Latest docs

Create sample model with Python SDK

Now that you’ve used the quickstart with the sample taxi fare model, it’s time to explore the sample notebook, monitor_placeholder_api_generic.ipynb. The notebook uses the Python client SDK to create a placeholder model, segments, and policies similar to what you experienced using the sample taxi fare model in the UI. Once created and submitted for monitoring, you use VIANOPS to get insight into model performance and accuracy.

To monitor a model, you need to:

• Create a placeholder model deployment
• Load inference data and ground truth
• Create and add segments and policies
• Monitor the deployed model in VIANOPS

Create a placeholder model deployment

To monitor a model deployed outside of VIANOPS, you create and deploy an internal placeholder model to operate as a reference for that model.

Using the sample notebook available from VIANOPS, you can deploy a placeholder model for your model. Note that many of the tasks illustrated in the notebook can be accomplished using either the API (SDK or REST API) or UI. The sample notebook uses the VIANOPS Python client SDK to create and deploy a new placeholder model. See the VIANOPS developer documentation for more information on the Python client SDK.

Explanation of the sample model and notebook

In this tutorial, a sample regression model that predicts the tip amount for taxi trips in New York City runs outside of VIANOPS. To monitor this model in VIANOPS, the sample notebook monitor_placeholder_api_generic.ipynb creates a placeholder model named “Taxi fare model” and associates it with the project, “Taxi fare project”. Sample data for taxi trips is used to create the placeholder model and simulate inference data for past and future timeframes. The sample data is available in a CSV file located in the /inputs folder. Ground truth (the actual values of those predictions) is added so VIANOPS can calculate model performance (the accuracy of its predictions give the actual data). Once the data is in place, the notebook creates segments to narrow drift analysis to specific populations in the data.

When finished, the configured policies run at scheduled intervals and detect feature and performance drift across the specified data and within specific time windows. Incoming inferences are compared against baseline data to detect values outside defined thresholds and metric settings. Policies run against all specified data (varies by policy configured); in addition, policies configured with segments then run again against the data defined by the segments, resulting in multiple policy runs that provide additional results to help understand your model. When alerts indicate drift-related issues, hotspot analysis aids root causes analysis by pinpointing the potential reasons.

Results are visualized in the policy and data segment pages in the UI. See the Explore sample taxi fare model tutorial for more details.

Create a placeholder model deployment in a notebook

1. Sign in to the VIANOPS notebook service for your platform, https://edahub.(your_platform_domain), using your VIANOPS login. For example, if you access VIANOPS at https://monitoring.company.vianai.site/ then the domain is company.vianai.site and the Jupyter notebook service is https://edahub.company.vianai.site.

   The sample notebook is available under notebooks > samples > monitor_placeholder. For the purposes of this tutorial look for the notebook (monitor_placeholder_api_generic.ipynb), Python file (helper.py), /inputs folder containing the dataset for the notebook. The dataset provides sample NYC Yellow Cab taxi fare data collected over a three-month period for Brooklyn and Manhattan routes.

2. First, you need to copy the monitor_placeholder folder structure to your home directory and create an /inputs subfolder.

   • Navigate to your home directory:

     

   • Open a terminal window (File > New > Terminal).
   • Enter the following code into the terminal:
     cp -R /notebooks/samples/monitor_placeholder/ .

   A new monitor_placeholder folder appears in your home directory. If the command fails try adding a destination folder: cp -R /notebooks/samples/monitor_placeholder/ -T dest_folder.

3. From the new monitor_placeholder folder, open the notebook monitor_placeholder_api_generic.ipynb.

4. Under the section “Specify imports and set up variables”, run the first cell to add all imports needed for the notebook; the Vianops_client imports are specified in the helper.py file.

5. Run the second cell to log in using the cluster and authenticated account information. These values are retrieved from VIANOPS configuration and helper file. If it cannot read these values from the system variables, the notebook prompts you to enter them.

   When prompted, enter the values:

   • cluster—The cluster location consists of the URL domain after monitoring and up to but not including the forward slash. For example, if the VIANOPS URL is https://monitoring.name.company.site then the cluster location is name.company.site.
   • username and password—For an account authenticated for the target cluster.

   The login() function defined in helper.py file reads the defined values and returns a token for accessing the database connection (passed to a script).

6. Click the third cell, review the values set for this placeholder model deployment, and press Run.

   • When running the sample notebook as-is, do not modify these values because they are configured specifically to support this notebook workflow.

   • If you want to extend the sample notebook with advanced monitoring, you likely need to modify one or more values. See the tutorial Extend sample model with Python SDK for help extending the sample model notebook.

   Notebook variable	Value
   allcolumns	Required. All columns in the feature set. By default, these are the columns in the sample notebook local data file provided in the /inputs folder. If you are using another feature set (different local file), make sure the columns listed here match the columns in your feature set. To specify different columns, see the “Change the feature set for the model” section of the tutorial Extend sample model with Python SDK for more details.
   continuouscolumns	Optional. Columns in the feature set containing continuous data. (Typically, these are numerical columns with integer or float data types.) Use empty brackets [] if there are no continuous columns. Make sure the columns listed here match the columns in your feature set. (By default, these are the continuouscolumns in the sample notebook local data file provided in the /inputs folder. To specify different continuous columns, see the “Change the feature set for the model” section of the tutorial Extend sample model with Python SDK for more details.)
   categoricalcolumns	Optional. Columns in the feature set containing categorical data. Use empty brackets [] if there are no categorical columns. Make sure the columns listed here match the columns in your feature set. (By default, these are the categoricalcolumns in the sample notebook local data file provided in the /inputs folder. To specify different categorical columns, see the section “Change the feature set for the model” of the tutorial Extend sample model with Python SDK for more details.)
   str_categorical_cols	Optional. Categorical columns in the feature set that should be set to string format. Use empty brackets [] if there are none. Make sure the columns listed here match the columns in your feature set. (By default, these are categoricalcolumns in the sample notebook local data file provided in the /inputs folder. To specify different columns, see the section “Change the feature set for the model” of the tutorialExtend sample model with Python SDK for more details.)
   feature_importance_payload	Optional. Contains three sections of information: Feature which identifies the columns in the feature set that contribute to feature importance, absolute_importance which provides the absolute importance values for that feature, and percent_contribution value identifying the percent importance/weight of the feature. If more information, see feature importance API documentation.
   targetcolumn	Required. Name of the feature set column that the model is predicting. By default, this is the targetcolumn for the sample notebook local data file provided in the /inputs folder. To specify a different column as the target, see the tutorial Extend sample model with Python SDK for help extending the sample notebook with advanced monitoring. Make sure this column matches a column in your feature set.
   offset_col	Required. Feature set column containing the offset value (datetime format), used for generating time/date stamps identifying when predictions were made. Offsets the given datetime to today -n number of days. Make sure this column matches a column in your feature set. (By default, this is the offset_col for the sample notebook local data file provided in the /inputs folder. To specify a different column for the offset value, see the “Change the feature set for the model” section of the tutorial Extend sample model with Python SDK for more details.)
   identifier_col	Required. Column containing the identifier feature that uniquely identifies predictions. (When the model sends predictions, VIANOPS uses the identifier column to map the model’s ground truth data.)
   directory	Optional. Set to inputs for the sample notebook. Local directory location containing data files for the notebook. These may be JSON, CSV, Parquet, or ZIP files (where each ZIP file contains only one compressed file of supported format). For the sample notebook. this must be a subdirectory in the notebook folder. If you change the directory location for reading local files, make sure to modify this setting for the new directory. To use different data for your placeholder model, see the “Change the feature set for the model” section of the tutorial, Extend sample model with Python SDK.
   upload_method	Required. Method for sending predictions received from model from the placeholder deployment to VIANOPS. Supported values: •json – (default if not defined) to create raw JSON inputs and outputs to POST to inference tracking API. •file – to cache uploaded files based on batch size and send cache key on POST to inference tracking API.
   batch_size	Required. Number of batches for splitting the prediction data (for either value of upload_method setting). If None, VIANOPS reads the full prediction file.
   n_days_data	Optional. Number of days’ worth of data present in the notebook. This value is specific to the taxi data and used in the helper Python file.
   days_future	Required. Used to offset the available data in the future. For example, if n_days_data is set to 92 and days_future is set to 5, the data is offset to provide 5 days of future data and 87 days of historical data, starting from today.
   days_drift	Required. Used to upload data from the latest date and the days leading up to that date. For example, if days_drift is set to 7 and days_future is set to 5 then data drifts over the last 2 days and then the next 5 days. (i.e., the offset is from today’s date plus days_future).
   policy_run_days	Optional. Number of previous days for running the policy. For example, if set to 10 and the current date is July 28th, then the policy runs from July 19th to July 28th (today).
   backend_connection	Required. Database location for the inference store and connection information. The notebook retrieves this automatically; do not modify.
   project_name	Required. Name for the placeholder project; set to Taxi fare project for the sample notebook. Model artifacts are associated with the project, including the experiment type, model, policies, and segments.
   experiment_type	Required. The sample notebook creates a regression model. Supported values: regression or classification. (Model data and target column must support the specified experiment type.)
   model_name	Required. Name for the placeholder model. When creating multiple models (i.e., running the notebook multiple times), the notebook appends a count and your username to the model names for all subsequent models.
   model_version	Required. Version of the placeholder model to deploy.
   model_stage	Required. Must be primary.
   model_class	Optional. ML model library, e.g., sklearn.
   segment1_name	Required. Identifies a segment that contains only data for the lower Manhattan taxi route.
   segment2_name	Required. Identifies a segment that contains only data for Williamsburg to Manhattan taxi route.
   segment3_name	Required. Identifies a distance segment (filtering on the est_trip_distance feature) to use with hotspot analysis as part of policy3_name.
   base_policy_name	Required. Identifies a policy that checks for feature drift each month using inferences from all data from the prior month (set as the baseline).
   mp_policy_name	Required. Identifies a policy that runs the MAE metric to calculate model accuracy and detect performance drift. Runs each day using inferences from the previous day as the baseline.

    

7. Run the fourth cell to create the unique identifying values this model: model and deployment name, model version, and model stage (using the values you specified as inputs and variables).

   With the variables set up, now you can create the project and model.

8. Under the section “Load training data”, run the cell. The taxi_training_data.csv file loads and will be available later as a baseline when you create policies.

9. Under the section “Create a project in modelstore” run the cell to create the project using upsert(projects_param) (SDK reference).

10. Under the section “Create a model in modelstore” run the cell to create the model using create(models_param)(SDK reference). The model is created with the following parameters: model class, model name, all columns, experiment type, training set, and the feature_importance_payload , which adds feature importance data for the model.

These methods create a new placeholder model deployment in VIANOPS, using the values configured in the notebook. When finished, you can find the new project and model in the UI; navigate using the project and model dropdown lists. See the Explore sample taxi fare model tutorial for more details.

Load inference data and ground truth

Next you load the inference mapping schema, and then inference data, predictions, and ground truth for the model. After that you can set up segments and policies (with hotspot analysis). When running, the policies analyze the new inference data and metrics to detect and identify feature and performance drift. In addition, the created feature drift policies use specified feature importance to apply correct weights to inference values.

As opposed to standard models, placeholder models are not trained for specific datasets; instead, they operate solely as references for deployed models. To support drift and performance monitoring for the model, VIANOPS creates an inference mapping identifying the schema for the model’s inferences. Once the inference mapping is in place, the model can send inferences/predictions to VIANOPS for policy monitoring and metrics analysis.

1. Run the cell “Load inference mapping schema” which creates the inference mapping for the model using create_inference_mapping(inference_mapping) (SDK reference) (and the helper.py script inference_mapping_generator()).

2. Run the cell “Send inference tracking to the backend”. This specifies the directory location of the inferences, how to read them (i.e., batch size, upload method, etc.), and how to create future and drift data. The helper.py method populate_data() reads inferences from the defined CSV file in the /inputs folder and uploads them to platform cache. Then, it reads the data from cache based on the defined batch_size.

3. Run the cell “Fill ground truth data for model performance”. This sets up values for ground truth then submits it to the backend database (using the helper.py script fill_ground_truths()).

4. Run the cell “Run a preprocess job for model performance”. submit(V1PreprocessingJoinInput) (SDK reference) joins the model’s inference mapping and ground truth values for the specified start and end time values, performance tasks, and policies to calculate model accuracy and detect performance drift.

At this point, you can find the model, columns with feature importance, and traffic in the VIANOPS UI. See the Explore sample taxi fare model tutorial for more details.

Now you can create segments and policies to monitor model performance for the deployed model.

Create and add segments and policies

The cells under the “Segments and Policies” section create segments (or subsets of data), and policies that, when deployed, enable VIANOPS to analyze inferences and metrics to detect issues with drift and model performance. Add segments to run policies on narrow populations of interest. When configured with a segment, a policy runs on all specified features (e.g., defined as part of feature_weights) and then runs separately for each defined segment.

The names for all segments and two policies are defined earlier in the notebook (in the section, “Specify imports and set up variables”). Additionally, the notebook uses a bulk create method to create three more feature drift policies.

Using create(V1PolicyModelList) (SDK reference) and create(V1SegmentBaseModelList (SDK reference), these cells create the following policies and segments for the defined variables:

• Month-to-month drift policy (base_policy_name variable) looks for feature drift in the inferences for the current month (target window) by comparing to them to the inferences from the prior month (baseline window). The policy weights the inferences using the values provided for feature importance (within the feature_importance_payload).
• Brooklyn-Manhattan segment (segment1_name variable) includes only data for pickup and dropoff locations in Brooklyn and Manhattan. The segment includes only data when the value of feature PULocation is either “Downtown Brooklyn/MetroTech, DUMBO/Vinegar Hill, or Brooklyn Heights, and the value of feature DOLocation is either Little Italy/NoLiTa or Lower East Side. This segment is created as inactive.
• Williamsburg-Manhattan segment (segment2_name variable) includes only data for pickup and dropoff locations in Williamsburg and Manhattan. The segment includes only data when the value of feature PuLocation is either Williamsburg (South Side) or South Williamsburg, and the value of feature DOLocation is either Little Italy/NoLiTa or Lower East Side. This segment is created as inactive.
• Distance segment (segment3_name variable) includes data only when the value of feature est_trip_distance is 2 (miles) or greater. This segment is created as inactive.
• MAE performance policy (mp_policy_name variable) looks for performance drift. Using the Mean Absolute Error (MAE) metric, it compares prediction and ground truth values for the current day (target window) by comparing to them to the data from the prior day (baseline window). If drift measure exceeds 20% a warning level flag is set; if it exceeds 50% a critical level flag is set. In addition to looking for performance drift violations across all data for the defined window, the policy also runs separately for the segments Brooklyn-Manhattan segment and Williamsburg-Manhattan segment.

Then, the cell “Create a day-to-day and week-to-week policy with segments” uses bulk_create() (SDK reference) to create two more policies:

• Day-to-day drift looks for feature drift in the inferences for the current day (target window) by comparing to them to the inferences from the prior day (baseline window).
• Week-to-week drift looks for feature drift in the inferences for the current week (target window) by comparing to them to the inferences from the prior week (baseline window).

The cell “Create a day-to-day policy for hotspot analysis” uses bulk_create() (SDK reference) to create a new policy that runs hotspot analysis on the data defined by the Distance segment.

Run each of the cells in this section, one after the other. When done, the new policies and segments are visible in the VIANOPS UI, in the Segments and Policies pages for the model. See the Explore sample taxi fare model tutorial for more details.

Now, run the cell “Run a preprocess job for policies”. Using submit(V1PreprocessingInput) (SDK reference) decreases policy processing time at runtime by reading the model inference mapping.

Run policies and model performance

At this point, you are ready to run the defined policies. You can either run the policies from the UI (from the policy page, Run Policy action) or from the notebook as explained here.

In the section “Trigger multiple runs for the policies”, run the two cells. run_policies() (SDK reference) creates the necessary data and then starts each policy.

Then, run the cell in the “Submit a model performance job” section to create the model performance payload and run the model performance job. submit(V1ModelPerformanceBody) (SDK reference) runs regression performance metrics (such as, MAE, MAPE, MSE, NMSE, NRMSE, NMAE, NMAPE, R2, RMSE, etc.). You can view and analyze the results of performance metrics in the Model Dashboard to track the effectiveness of the deployed model. See the Quickstart with sample taxi fare model for more details.

When you are ready to modify the notebooks to create segments and policies for your deployment, see the “Create a new segment for a policy” and “Create a new policy” sections of the tutorial, Extend sample model with Python SDK.

Monitor the deployed model in VIANOPS

To monitor your model, navigate to the deployment in the Model Dashboard and view the Latest Performance and Prediction Metrics charts. From the Policies page, review policy results and alerts as explained in the Explore sample taxi fare model tutorial.

Retrain and redeploy?

If you determine the model is drifting (and predictive accuracy is declining), you can retrain to get better performance and then redeploy the model to send new predictions.