Here are a few things that any organization deploying NLP models into production should be doing to ensure that those models continue to perform as expected.
Ensure consistency in data extraction pipelines and monitor for data drift.
Interact with feature scores and observing the impact on model outputs.
Identify the most important features impacting predictions of your NLP models.
Compare the similarity of new input documents to the documents used to train your NLP models
Detect biases in your NLP models by uncovering differences in accuracy and other performance metrics across different subgroups
Identify the specific words within a document that contributed the most to a given prediction
Thanks to Arthur, we know that our preventative care models are fair and that we can catch any potential issues before they impact our members.
– Heather Carroll Cox, Chief Data & Analytics Officer at Humana