Root cause analysis (RCA)

When models experience feature, prediction, or performance drift, VIANOPS policies detect the drift conditions and generate alerts. You’ll want to understand why your model is experiencing drift. Is production data different from training data? Have conditions for running the model in production changed? Segmentation and hotspot analysis help to uncover the root causes for detected drift.

• Segments—Runs policy drift conditions against narrow data segments and then compare results to other segments as well as all the data. Implementing segmentation before policies are run aids root cause analysis (RCA) by reducing the field of data. For example, the quick start model (from the sample notebook) predicts rates for taxi trips throughout NYC. Configured policies for that model look for drift indicators across the full dataset over different time periods. Those policies also run on configured narrower segments of data, i.e., the Williamsburg-Manhattan trip and Brooklyn-Manhattan trip.

• Hotspot analysis—Individual features tracked by a policy can provide deeper insight into detected model drift. When configured for a policy— feature drift, prediction drift, or performance drift —hotspot analysis calculates PSI, traffic, and impact for selected features. By calculating the features with highest impact on detected drift, hotspot analysis can surface details critical to understanding and addressing root cause issues.

The ability to include a feature in hotspot analysis is configured as part of inference mapping properties. In general, features of categorical data type configured to support drift and segmentation are included automatically for hotspot analysis unless otherwise specified (in the inference mapping configuration).

To see results of previously run hotspot analyses use /v1/hotspot-analyses/search.