How to guides

To get started with monitoring your model on VIANOPS, follow the three-step approach outlined below:

Step 1: Register your model and ingest inference data.

1. Begin by registering your model on VIANOPS. You can do this by providing minimum metadata about your model, such as the model name, description, type, and owner. You don’t need to provide the actual model file (e.g., pkl file) to register it on VIANOPS. You can use the VIANOPS API to register your model.

2. Once your model is registered on VIANOPS, you can begin ingesting inference data for that model. You can ingest historical inference data initially, and set up a data pipeline to ingest new inference data continuously in the future.

3. Ground truth for your model may not be immediately available or may be deferred for days, weeks, or months. VIANOPS supports deferred ground truth and runs performance monitoring once the ground truth is ingested.

4. You can slice the data into segments to monitor the model at a lower granularity for specific interests and narrow down the root cause analysis.

For more information on how to register your model and ingest inference data, see:

• Onboard your model

Step 2: Monitor your model using policies.

1. VIANOPS uses policies to monitor the model. You can create as many policies as you need, each with different configurations to monitor different risks, including performance drift, feature drift, prediction drift, bias and fairness, and more. You can create policies using the VIANOPS UI or API.

2. Once policies are created and activated, they will run based on the schedule and generate results for the dashboards. If a policy’s metric exceeds a threshold, it triggers alerts (either critical or warning).

For more information on how to monitor your model using policies, see:

• Monitor a model

Step 3: Perform root cause analysis.

Once you receive an alert from VIANOPS, you can use its flexible tools to identify the root cause of the issue. You can use relevant policies to determine when, where, and how the drift happened. Alternatively, you can observe the drifted features at the same time windows to explain the performance drip or prediction drift. You can also use drill down to the value distribution of a given feature to locate data quality issues in segments.

For more information on how to perform root cause analysis using VIANOPS, see:

• Root Cause Analysis