Deepchecks App: Monitoring Models in Production
Deepchecks app usually refers to the web interface teams use to inspect validation results and monitor deployed models over time. It turns check outputs into dashboards, trends and alerts, so problems such as drift or performance decay become visible long before customers notice them.
The following sections describe deployment options, the self-hosted setup flow, which signals to watch, how to design useful alerts and how to investigate them when they fire.
What the App Does
Notebook reports are excellent during development, but production models need continuous attention. The app runs the same Deepchecks checks on incoming data on a schedule, stores the results and presents them as time series, which makes gradual changes easy to spot.
Alerts can be configured on check results, so the team learns about a problem when it starts. Dynamic dashboards then help investigators drill into the affected features, segments or time windows to understand what changed.
Deployment Options
Two main paths exist: a self-hosted open-source deployment and managed offerings run by Deepchecks. The open-source monitoring project is built so that each deployment supports monitoring of a single model, which is enough for evaluation and smaller use cases.
| Option | Who Runs It | Good Fit |
|---|---|---|
| Open-source self-hosted | Your team | Trials, single-model monitoring, full control |
| Managed or enterprise | Deepchecks | Many models, collaboration and security needs |
Self-Hosted Setup Overview
The open-source README describes a short installation flow. You create a directory for the installation files, open a terminal there and run an installer package that deploys the monitoring system, which takes a few minutes to complete.
pip install deepchecks-installer
deepchecks-installer install-monitoring
Once the installation finishes, the deployment URL opens the onboarding flow, with localhost as the documented default. Server resources, container requirements and production hardening steps are covered in the official monitoring documentation, which you should follow closely.
What to Monitor After Launch
Good monitoring focuses on signals that predict business impact. Labels often arrive late in production, so data-based signals such as drift act as early warnings while performance metrics confirm the impact once ground truth is available.
- Feature drift: distribution changes in important input features.
- Prediction drift: shifts in the distribution of model outputs.
- Data integrity: nulls, new categories or type changes in incoming data.
- Performance: accuracy or error metrics when labels arrive.
- Segment health: groups whose results degrade faster than average.
Setting Alerts That Actually Help
Too many alerts train people to ignore them. Start with a small number tied to features that matter most, set thresholds using historical variation, and route each alert to the person who can act on it rather than to a crowded shared channel.
Review alert quality monthly. Raise thresholds that fire on harmless seasonal patterns and lower them where real incidents slipped through, so the monitoring setup becomes more precise with every review.
Investigating an Alert
When an alert fires, the dashboards help narrow the cause. Compare the alerting window with a stable reference period, find which features changed most, then check whether the shift comes from a genuine change in users or from a broken upstream pipeline.
- Confirm the data pipeline delivered complete, correctly typed data.
- Look for new product launches, regions or campaigns.
- Check whether only one segment is affected.
- Decide between fixing data, retraining or accepting the change.
License and Edition Notes
The monitoring repository is open source under AGPL-3.0, except for one enterprise directory that is subject to a commercial license and is not used by default. Enabling premium features in a self-hosted instance requires a Deepchecks license, so verify which features you plan to use.
Choosing Good Reference Data
Drift is always measured against a reference, so the reference choice shapes every alert. Training data is a natural starting point, but a recent stable production window often reflects real traffic better, especially when training data was filtered or resampled.
Document which reference each model uses and when it was last updated. Revisit it after retraining, because an outdated reference makes healthy new data look like drift.
Fitting Monitoring into Team Workflow
Monitoring works best when ownership is clear. Assign each model an owner, agree on response times for alerts and keep a short runbook describing what to check first, so incidents are handled consistently regardless of who is on duty.
- Name an owner and a backup for every monitored model.
- Link alerts to the runbook and the relevant dashboard.
- Log the outcome of each investigation for future reference.
Signals and Their Purpose
Each monitored signal answers a different question, and knowing which question you are asking makes dashboards easier to read during an incident.
| Signal | Question It Answers | Needs Labels |
|---|---|---|
| Feature drift | Are inputs changing? | No |
| Prediction drift | Are outputs changing? | No |
| Data integrity | Is incoming data well-formed? | No |
| Performance | Is the model still accurate? | Yes |
Data Retention and Cost
Storing every production sample forever is rarely necessary. Decide how long raw data and aggregated results should be kept, balancing investigation needs against storage costs and any privacy rules that apply to your organization.
Moving from Testing to Monitoring
Teams that already run suites in CI have a head start, because the checks they trust before release can continue in production. Reusing familiar checks shortens setup time and keeps quality definitions consistent from development to live traffic.
Frequently Asked Questions
The open-source monitoring deployment is free under AGPL-3.0 for single-model use. Managed and enterprise features require a commercial arrangement.
The README lists pip install deepchecks-installer followed by deepchecks-installer install-monitoring, run inside a dedicated directory.
The open-source deployment supports one model per deployment. Monitoring many models is a focus of the managed offerings.
No. Drift and integrity checks work without labels, while performance metrics become available once ground truth arrives.
Conclusion
Deepchecks app monitoring extends familiar checks into production, turning them into trends, dashboards and alerts. Start with a small set of meaningful signals, tune alerts over time, and choose between self-hosting and managed options based on how many models you need to watch.
Clear ownership, sensible thresholds and well-chosen reference data matter just as much as the tool itself, so invest time in those decisions from the first deployment onward.