The Model’s Shipped; What Could Possibly go Wrong

 By Medium - 2021-02-18

Description

In our last post we took a broad look at model observability and the role it serves in the machine learning workflow. In particular, we discussed the promise of model observability & model monitoring…

Summary

  • What Could Possibly go Wrong?
  • In this article we will be providing some more concrete examples of potential failure modes along with the most common symptoms that they exhibit in your production model’s performance.
  • Training vs Production Performance, Image by Author Training-production Skew In both the case of data drift and training-prod skew, the model is experiencing a distribution of model inputs in production that it did not see while training, which can lead to worse performance than when validating the model.
  • however, when you get back to your desk the next day after deploying the updated model you see a huge regression in Alexa’s accuracy in selecting the action that the user requested.

 

Topics

  1. NLP (0.15)
  2. Backend (0.09)
  3. Machine_Learning (0.07)

Similar Articles

Time-Series Forecasting with Google BigQuery ML

By Medium - 2021-02-16

If you have worked with any kind of forecasting models, you will know how laborious it can be at times especially when trying to predict multiple variables. From identifying if a time-series is…