Release notes

VIANOPS v2.1 release notes

New in VIANOPS v2.1:

  1. Root Cause Analysis (RCA)

    • Hotspot analysis is now supported for feature drift, prediction drift and performance drift. A hotspot is a specific slice of data within a dataset where its drift shows a significantly higher value compared to other slices of data. Hotspot analysis is a technique that identifies hotspots.
    • By default, all categorical features are used for hotspot analysis. You can also use subset of categorical features within a policy for hotspot analysis through APIs - Hotspot analysis.
    • When a model triggers critical or warning alert, you can find a link on the alert called “hotspot”. This link opens a page with hotspot data and graphs to help you perform root cause analysis. You can find more details at UI - Hotspot analysis.
    • This feature helps you identify anomalous behaviors in subsets of data even if the impact of those anomalies on the full dataset does not trigger an alert.

  2. Model comparison

    You can compare performance metrics of two models that are the same model type (classification or regression) and with the same input features and output. Model comparison is typically used to compare new version of model to current model in production or multiple deployments of same model in production. Additionally, you can compare models by segments.

  3. Project dashboard
    • VIANOPS now opens to a project dashboard which provides a quick view on risk, volume, and recency for all models associated with the project.
    • Use projects to group similar models addressing a business problem. Generally, projects are tied to some business problem like reducing customer churn or reducing customer escalation, and the models are developed to deliver the best results.
    • UI - Projects
  4. Project management
  5. Performance policies
    • Performance policies now have the same targets and baselines as the feature drift and prediction drift policies (daily, week-to-date, month-to-date, and quarter-to-date).
    • UI - Performance policies

  6. Feature importance

    • You can now upload feature importance of the actual model via API for global explainability and then use feature importance as an additional weight method for overall feature drift.
    • Global feature importance appears at the bottom of the model dashboard if it is a SHAP (SHapley Additive exPlanations) value.
    • Global explainability helps you understand which features in a dataset have the strongest influence on a specific model’s output and which features are less important.
    • APIs - Feature importance
    • UI - Feature importance
  7. Custom binning for prediction (model output)
    • User-defined bins for distance-based prediction drift policies on regression models
    • The policy creation wizard flow includes bin entry for ease of creation. For example, define bins [10,15,25,50,100] results in four bins (10,15], (15, 25], (25, 50], (50-100], values fall outside left and right edge we still include them in the lowest and highest bins.
    • APIs - Distance-based drift on prediction data
    • UI - Prediction drift
  8. Segment management
  9. Data profile monitoring as part of feature drift policy

Improvements and enhancements in VIANOPS 2.1:

  1. Policy management
    • Visualize additional information including alert count, policy type, and next run
    • Toggle between active and inactive policies
  2. Policy management with Duplicate operation
    • In addition to create and edit, a new action is added for each policy in the Policy List to duplicate the policy. This is useful in creating a similar policy without having to start from scratch.
    • UI - Create a policy

  3. Scale: support unlimited number of features of a model

  4. Quick validation of a new segment during creation
    • When creating a segment, if you have loaded data with inference mapping, the Segments wizard shows how many rows of data are associated with the segment according to the conditions you have specified.
    • Segments
  5. Added performance metric for regression model policies in the UI
    • Added R-squared (R2) metric to the performance drift policies for regression models.
    • Metrics

  6. Added performance metrics for binary classification model policies in the UI

    The following metrics are now available for binary classification models:

    • Area under the curve (AUC) — The area under the ROC curve, which quantifies the overall performance of a binary classifier compared to a random classifier.
    • Gini coefficient — A measure of inequality or impurity in a set of values, often used in decision trees.
    • Lift — A measure of the effectiveness of a predictive model calculated as the ratio between the results obtained with and without the predictive model.
    • Log loss — A performance metric for classification where the model prediction is a probability value between 0 and 1. Measures divergence of model output probability from actual value. Logloss = 0 is a perfect classifier.
    • Probability calibration curve — A plot that compares the predicted probabilities of a model to the actual outcome frequencies, used to understand if a model’s probabilities can be taken at face value.

    See metrics.

  7. Search in tables now works throughout the UI.

VIANOPS documentation is now publicly available at https://docs.vianops.ai with API reference documentation available at https://developer.vianops.ai/reference/introduction.

VIANOPS v2.0 release notes

2.0 is the release with special focus on ML model monitoring. With the new name of VIANOPS, the platform now uniquely brings together observability across layered, high-volume, complex dimensions, with root cause analysis for high-risk hotspots that jeopardize model behavior, and the ability to drive high-performance ML operations across any cloud and any data source.

What’s new in VIANOPS 2.0:

  1. A free trial version of VIANOPS is available starting with this release. It includes a sample model that shows key parts of the platform flow.

  2. New user experience with dashboards.

    • New, more dynamic user interface.
    • A model dashboard with model performance and prediction metric graphs, alerts summary and recent alerts list, and easy access to policies and segments.

  3. Performance monitoring policy.

    • Use performance policies to monitor day-over-day performance changes and trigger alerts when performance drops significantly.
    • Performance
    • Deferred Ground Truth ingestion.
  4. Drift metric selection for feature drift or prediction drift monitoring.
    • You can choose either Population Stability Index (PSI) or Jensen-Shannon (JS) divergence as the metric for feature or prediction drift.

  5. Segments as model level objects and monitoring policies for the segments.

    • You can define any slice of data as a segment and monitor the feature drift, prediction drift, or performance of this segment in the corresponding policy.
    • You can select multiple segments in a single policy to easily compare the drift or performance across segments.
    • Segmentation enables monitoring for specific use cases as well as more granular root cause analysis when a model is not performing as expected.
    • Segments

  6. Custom bins for prediction or any feature.

    • You can set custom bins for distance-based drift policies for either prediction or feature drift.
    • Prediction drift (see the baseline_bins table item)
    • Feature drift (see the baseline_bins table item)

  7. VIANOPS REST APIs and SDK client

    • VIANOPS REST API provides you with programmatic access to the platform for managing and monitoring your MLOps projects and models.

      API documentation

    • You can use the SDK client to leverage VIANOPS API for managing and monitoring model deployments. The client simplifies the steps for deploying models and then managing and monitoring to sure the deployed models are trusted and their predictions accurate.

      Using the Python client SDK

  8. VIANOPS documentation is now publicly available at https://docs.vianops.ai with API reference documentation available at https://developer.vianops.ai/reference/introduction.
TABLE OF CONTENTS