Monitoring Machine Learning Models in Production
( reference : https://christophergs.com/machine%20learning/2020/03/14/how-to-monitor-machine-learning-models/ )
Contents
- ML System Life Cycle
- What makes ML System Monitoring hard
- Why you need monitoring
1. ML System Life Cycle
monitoring of ML models :
- track & understand model performance “in production”
- perspective in both (1) data science & (2) operation
Continuous Delivery for Machine Learning (CD4ML) ( Martin Fowler )
Phase 1. Model Building:
- define the problem
- data preparation, feature engineering, (initial) modeling code
Phase 2. Model Evaluation and Experimentation:
- feature selection, hyperparameter tuning
- compare with different algorithm
- check metrics
Phase 3. Productionize Model:
-
preparation for deployment
-
result : production-grade code
( can be in a completely different programming language / framework )
Phase 4. Testing:
-
check if it works as we expected
( check if it matches with (step 2) )
Phase 5. Deployment:
- put into production
- use APIs to access the model.
Phase 6. Monitoring and Observability:
- keep monitoring if it is working as we expected
Phase 1 & 2 ( model building & evaluation )
\(\rightarrow\) research environment ( by data scientists )
Phase 3 & 4 & 5 & 6
\(\rightarrow\) engineering, DevOps
Monitoring Scenarios
Scenario # 1 : deployment of brand new model
Scenario # 2 : replacement of existing model
Scenario # 3 : making small tweaks in current model
- ex) change in certain variable ( unable to use )
- ex) found a super feature! ( will add that feature )
2. What makes ML System Monitoring hard
The model is a tiny fraction of an overall ML system!
3 components of ML system
- 1) Code ( + config )
- 2) Model
- 3) Data
[ CAUTION ]
ML system’s is governed “not just by code”, but also by “model behavior learned from data”.
+ data inputs change over time!
Complexity of Code & Data ( + config )
- Entaglement
- Data changed \(\rightarrow\) Model weights changes
- changing anything changes everything
- Configuration
- slight change in hyperparmeter/version/feature can cause big change!
Responsibility Challenge
Teams of ML Systems
Data Scientists & Engineers/Devops
- Data Scientists
- focus on statistical tests on model inputs and outputs
- Engineers/Devops
- monitoring, production, deployment…
\(\rightarrow\) need BOTH perspectives!
Traditionally …
- monitoring & observability : just part of DevOps
But nowadays (ML System) …..
- unlikely that your DevOps team will have the necessary expertise to correctly monitor ML models
THEREFORE, whole team needs to work together !! MLOps!!
3. Why you need monitoring
“By failing to prepare, you are preparing to fail” ( Benjamin Franklin )
look for xxxxx