The Playbook to Monitor Your Model’s Performance in Production

 By Medium - 2021-03-08

Description

As Machine Learning infrastructure has matured, the need for model monitoring has surged. Unfortunately this growing demand has not led to a foolproof playbook that explains to teams how to measure…

Summary

  • Performance Monitoring of ML Models As Machine Learning infrastructure has matured, the need for model monitoring has surged.
  • Once a model metric is determined, tracking this metric on a daily or a weekly cadence allows you to make certain that performance has not degraded drastically from when it was trained or when it was initially promoted to production.
  • Delayed Ground Truth, Image by Author In this above diagram, while we do see that ground truth for the model is eventually determined, the model’s predictions over the last month have not received their corresponding outcomes.
  • Though these lagging performance metrics are not quite as good at signaling a sudden model performance regression in a real time application, they still provide meaningful feedback to ensure that the models performance is moving in the right direction over time.

 

Topics

  1. NLP (0.11)
  2. Management (0.11)
  3. Backend (0.08)

Similar Articles

The Model’s Shipped; What Could Possibly go Wrong

By Medium - 2021-02-18

In our last post we took a broad look at model observability and the role it serves in the machine learning workflow. In particular, we discussed the promise of model observability & model monitoring…

Time-Series Forecasting with Google BigQuery ML

By Medium - 2021-02-16

If you have worked with any kind of forecasting models, you will know how laborious it can be at times especially when trying to predict multiple variables. From identifying if a time-series is…