APIs
You can manage and monitor your model deployments with the VIANOPS APIs. The APIs simplify the steps for deploying models to ensure the deployed models are trusted and their predictions accurate.
VIANOPS provides the following types of APIs for you to connect to your environment:
-
REST APIs — The VIANOPS REST APIs provide you CRUD operations to deploy, manage, and monitor your models.
-
Python client SDK — The Python client SDK is a wrapper around the VIANOPS REST API calls, making it easy to interact programmatically with the platform consistently and successfully.
REST API Documentation
The documentation for the REST APIs is available from your VIANOPS deployment by changing the subdomain to webservices and entering docs to the path of your deployment URL. For example, if your deployment base URL is https://monitoring.company.vianai.site/ then the REST API documentation is available at https://webservices.company.vianai.site/docs.
Python client SDK
The Python client SDK API reference explains the supported operations and provides JSON schema for Pydantic models.
Examples of use
The sample placeholder model notebook provides an end-to-end demonstration of using the Python API client to configure and deploy a placeholder model, and then monitor its performance in production. Look for the notebook, monitor_placeholder_api_generic.ipynb in the platform’s Jupyter notebook service, https://edahub.(your_VIANOPS_domain). This guide explains how to use the SDK by following the sample notebook.)
Requirements
- You must have authenticated access to VIANOPS v2.2 or later, either the licensed platform or free trial version.
- System requirements
- Python client installation
- Python >= Python 3.8
- pip >= 23.0
- Access to VIANOPS >= 2.2
Get started
The client is included in the VIANOPS installation and therefore available to access and use with edahub service notebooks. If you want to integrate external workflows with the SDK client, you need to install the client separately using pip install.
Install the client
pip install vianops_client
Once installed, import the vianops-client package and its subpackages/classes into your notebooks and python scripts. (See the list of all supported subpackages and classes.) For example, the following imports are defined for the sample, notebook monitor_placeholder_api_generic.ipynb.
# SDK imports
from vianops_client.models.commons import V1Filters
# different import within vianops_client
from vianops_client import CacheV1Api
from vianops_client import ExternalConnectionV1Api
from vianops_client import (
UserPreferenceV1Api,
V1UserPreferenceGetValueModelRequest,
V1UserPreferenceBody,
)
from vianops_client import V1PlaceholderDeployment, V1TrainingParams, V1ModelDetails
from vianops_client import (
DataloadingV1Api,
V1DataloadingJob,
)
from vianops_client import (
AuthV1Api,
ProjectsV1Api,
V1DeleteTypes,
V1Project,
V1ProjectList,
V1ProjectSearch,
V1ProjectUpdates,
)
from vianops_client import (
ModelsV1Api,
V1DeleteTypes,
V1ProjectFilters,
V1ModelsModel,
V1ModelsModelList,
V1ModelSearch,
V1ModelUpdates,
ModelDeploymentV1Api,
)
from vianops_client import (
V1ModelTagsModel,
)
from vianops_client import (
InferenceMappingV1Api,
V1InferenceMappingJob,
V1InferenceMappingSchema,
V1InferenceMappingColumnSchema,
V1InferenceMappingModel,
)
from vianops_client import (
InferenceTrackingV1Api,
V1InferenceTrackingModel,
V1AddGroundTruthModel,
)
from vianops_client import (
PreprocessingV1Api,
V1PreprocessingInput,
V1PreprocessingJoinInput,
V1PreprocessingJoinProcessWindow,
V1PreprocessingProcessWindow,
)
from vianops_client import (
V1PolicyRequestModel,
V1PolicyRequestModelList,
PoliciesV1Api,
V1PolicySearchRequest,
V1PolicyUpdateRequestModel,
V1PolicyUpdateModelList,
)
from vianops_client import (
SegmentsV1Api,
V1SegmentBaseModel,
V1SegmentBaseModelList,
V1SegmentModel,
)
from vianops_client import DriftdetectionV1Api, V1DriftDetectionJobModel
from vianops_client import ModelPerformanceV1Api, V1ModelPerformanceBody
Create clients from the imports
Using the imports, the sample notebook creates the clients as follows (and as shown in code snippets):
auth_api = AuthV1Api()
external_connections_api = ExternalConnectionV1Api()
user_preference_api = UserPreferenceV1Api()
deploy_api = ModelDeploymentV1Api()
project_api = ProjectsV1Api()
model_api = ModelsV1Api()
cache_api = CacheV1Api()
dataload_api = DataloadingV1Api()
inference_mapping_api = InferenceMappingV1Api()
inference_tracking_api = InferenceTrackingV1Api()
preprocessing_api = PreprocessingV1Api()
policies_api = PoliciesV1Api()
segments_api = SegmentsV1Api()
driftdetection_api = DriftdetectionV1Api()
modelperformance_api = ModelPerformanceV1Api()
Log in from the client
When ready to interact with the platform, the client needs to log in with an authenticated account. This endpoint passes parameters for username and password. When successful, the platform responds with an access token to include in subsequent API calls.
token = login(username=username, password=password, cluster=cluster)
Create and monitor a placeholder model
Start by setting up constants and variables to identify the model, feature set, connection, policies, segments, and so forth. Look at the sample notebook monitor_placeholder_api_generic.ipynb for examples of the data to set up.
Note: Column names in feature sets can contain only lowercase letters (a-z) or numbers (0-9), or the characters underscore (\_) or period (.).
Create project and model in modelstore
Create the project and then (placeholder) model in VIANOPS modelstore. If the model should be associated with an existing project, use search(data: V1ProjectSearch to find that project.
To create a new project, Vianai recommends using (upsert(data: V1ProjectList). This searches for existing projects of the same name before creating the new project, which ensures new projects have unique names. (The create(data: V1ProjectList) method is supported as an alternative but it does not enforce unique project names.) One project can have multiple associated models, and grouping multiple related/similar models as part of a single project is a recommended best practice.
Example from the notebook for using Upsert():
project_res = project_api.upsert(projects_param)
project_params = V1Project(
name="project123",
description="Description of this project",
status="active",
)
When successful, create() returns V1ProjectList.
"example": {
"name": "project123",
"description": "project description",
"status": "active",
"parent_project_uuid": "parent_project123_uuid",
"created_ts": 1675292368,
"modified_ts": 1675292368,
"created_by": "user1",
"modified_by": "user1",
"uuid": "0"
}
Use create(data: V1ModelsModelList) to create the model in the model store backend database, using the project uuid for that project. Specify metadata values using model tags, such as model_class, experiment_type, and train_set.
The notebook sets the model metadata as variables and then creates the model in model store using the tags where name, value, and status (active or inactive) are required for each model tag.
Note that several additional model tags are reserved for other functionality. For example, the feature_importance model tag enables you to add feature importance values to the model. (See how feature importance is added via defined feature_importance_payload variable for the notebook.) You can add feature importance to your models from offline files (csv, parquet, or json) or directly from API payloads.
Example from the notebook:
model_res = model_api.create(models_param)
model_params = V1ModelsModel(
name="model_abc",
version="1",
stage="primary",
status="active",
project_uuid="project123_uuid",
tags=model_tags_payload,
)
When successful, create() returns V1ModelsModelList.
"example": {
"version": "1",
"stage": "primary",
"status": "active",
"project_uuid": "project123_uuid",
"created_ts": 1675292368,
"modified_ts": 1675292368,
"created_by": "user1",
"modified_by": "user1",
"uuid": "model123_uuid",
"name": "model_abc",
"tags": []
}
Create inference mapping for your data
Use create_inference_mapping(data: V1InferenceMappingModel) to create an inference mapping for your model’s data. This specifies how raw inferences and predictions from the model are stored in the platform. Inference mapping must be created before the model sends new inference data. The Pydantic model V1InferenceMappingModel includes two other models: df_schema and columns (an argument to df_schema). df_schema arguments include datetime, target, and identifier columns, as well as columns[] which identifies properties for each column in the dataset, such as which features are enabled for drift detection, segmentation, and hotspot analysis. (Note that the platform sets the values for the hotspot and rca parameters based on feature configurations in inference mapping.)
columns_params = V1InferenceMappingColumnSchema [
{
"feature_id": 1,
"name": "feature1",
"dtype": "object",
"sql_type": "string",
"feature_type": "categorical",
"segmentation": true,
"drift": true,
"hotspot": true,
"rca": [
"targetfeature6",
"feature2"
],
"round": null
},
{
"feature_id": 2,
"name": "feature2",
"dtype": "object",
"sql_type": "text",
"feature_type": "categorical",
"segmentation": true,
"drift": true,
"hotspot": true,
"rca": [
"targetfeature6",
"feature1"
],
"round": null
},
{
"feature_id": 3,
"name": "feature3",
"dtype": "float64",
"sql_type": "Float32",
"feature_type": "continuous",
"segmentation": true,
"drift": true,
"hotspot": false,
"rca": [
"targetfeature6",
"feature1",
"feature2"
],
"round": null
},
{
"feature_id": 4,
"name": "feature4",
"dtype": "float64",
"sql_type": "Float32",
"feature_type": "continuous",
"segmentation": true,
"drift": true,
"hotspot": false,
"rca": [
"targetfeature6",
"feature1",
"feature2"
],
"round": null
},
{
"feature_id": 5,
"name": "datetimefeature5",
"dtype": "datetime",
"sql_type": "",
"feature_type": "unknown",
"segmentation": false,
"drift": false,
"hotspot": false,
"rca": [],
"round": null
},
{
"feature_id": 6,
"name": "targetfeature6",
"dtype": "object",
"sql_type": "text",
"feature_type": "categorical",
"segmentation": true,
"drift": true,
"hotspot": true,
"rca": [
"feature2",
"feature1",
],
"round": null
},
{
"feature_id": 7,
"name": "id_feature7",
"dtype": "object",
"sql_type": "text",
"feature_type": "unknown",
"segmentation": true,
"drift": true,
"hotspot": false,
"rca": [
"targetfeature6",
"feature1",
"feature2"
],
"round": null
},
]
df_schema = V1InferenceMappingSchema [
{
target_col: "targetfeature6",
identifier_cols: ["id_feature7"],
columns: columns_params,
datetime_col: "datetimefeature5",
predict_proba_col: None
}
]
Example from the notebook:
inference_mapping_res = inference_mapping_api.create_inference_mapping(
inference_mapping
)
inference_mapping = V1InferenceMappingJob(
deployment="deployment_xyx",
model_name="model_abc",
model_version="1",
model_stage="primary",
connection=clickhouse://default:@clickhouse/default", # replace with actual connection string
inference_table=inference_mapping_input["inference_table"],
df_schema= df_schema,
)
When successful, create_inference_mapping() returns V1JobModel.
"example": {
"index": 2,
"deployment": "deployment_xyz",
"model_name": "model_xyz",
"model_version": "1",
"model_stage": "primary",
"connection": "clickhouse://default:@clickhouse/default",
"df_schema": {},
"identifier_cols_schema": null,
"inference_table": "16849601825555_dit",
"ground_truth_table": "16849601825555_gt",
"joined_table": "16849601825555_joined",
"key_table": "16849601825555_key",
"create_ddl": null,
"postprocessors": null
}
Submit inference tracking job for your inference data
After inference mapping is created, the client can send model inferences to the backend via InferenceTrackingV1Api. VIANOPS supports file-based upload using cache_upload to first upload the data to cache (which is useful for large datasets); otherwise, you need to specify a list of dicts for inputs and list of outputs to fill the inputs and output fields in inference tracking call. (If doing a cache download as part of inference tracking, file_type specifies how to deserialize the file.)
Use create(data: V1InferenceTrackingModel) to track inferences from your model. create() takes an instance of V1InferenceTrackingModel which specifies the actual inference tracking data. Other arguments include the format for the data (raw or file), the file type for deserializing the data, if applicable (json, csv, or parquet), and the cache key for the cached inference data, if applicable.
V1InferenceTrackingModel specifies inference data for the model, including:
{
"uuid": "123-abc-456",
"deployment": "modeldeployment123",
"model_name": "modeldeployment123",
"model_version": "1",
"model_stage": "primary",
"hostname": "edahub",
"starttime": 1679184000000.0,
"endtime": 1679269500000.0,
"num_inferences": 1964,
"inputs": [],
"outputs": [],
"ground_truth": null,
"xformed_inputs": null,
"modified_ts": null,
"modified_indexes": null,
"inference_mapping_index": 3,
"inference_mapping": null
},
From the notebook
The monitor_placeholder_api_generic notebook uses a helper function to populate and submit inference tracking. populate_data() parses values of variables (from the notebook) to read data from specified notebook location (e.g., inputs directory). ` track_inference and post_track_inference` are used to specify the inference tracking payload and updates for the inference tracking payload, respectively.
Example from the notebook:
trackingResponse = inference_tracking_api.create(
inference_tracking_params,
format="file",
cache_key=cache_key,
file_type=file_type,
)
When successful, create() returns V1InferenceTrackingModel as a dict.
For example:
{
"uuid": "000121d1-e790-46f9-b603-c168ezzzzzzz",
"deployment": "deployment_xyz",
"model_name": "model_abc",
"model_stage": "primary",
"model_version": "1",
"hostname": "model_host",
"starttime": "1654703247907.512",
"endtime": "1654703247921.791",
"num_inferences": 1906,
"inputs": [],
"outputs": [],
"ground_truth": [
1
],
"xformed_inputs": [
1
],
"modified_ts": "1654703247907.512",
"modified_indexes": [
10000,
20000,
30000
],
"inference_mapping_index": 1,
"inference_mapping": [
{}
]
}
Submit ground truth for the inferences
Fill ground truth data for model inferences to enable VIANOPS to track and detect performance issues with models, and evaluate general model performance.
Use inference_tracking_ground_truth(data: V1AddGroundTruthMode) to submit ground truth data for the deployment. Ground truth can be submitted as part of inference tracking or any time after by using the method inference_tracking_ground_truth().
The Pydantic model V1AddGroundTruthModel specifies the deployment information, actual ground truth values(s), and format and file type or cache key identifying where to get the ground truth values.
"deployment": "deployment_xyz",
"model_name": "model_abc",
"model_stage": "primary",
"model_version": "1",
"ground_truths": [
{
"ground_truth": "13.8",
"inf_uuid": "65_209_2022_03_01 00:00:00_0"
}
],
"format": "file",
"file_type": "csv",
"cache_key": "temp.123456"
Example from the notebook (helper.py)
trackingGtResponse = inference_tracking_api.inference_tracking_ground_truth(
inference_tracking_gt_params)
When successful, inference_tracking_ground_truth() returns the message "Successfully added ground_truth."
Submit preprocess job for model performance
Before creating and running policies and segments, the client needs to join inference mapping and ground truth values to ensure the best performance when running those operations. To do this, use the Pydantic model PreprocessingV1Api to submit a dict of the inference mapping model. Set method as join;start_time and end_time values define the process window using the start and end times when ground truth data was submitted to VIANOPS.
payload = {
"inference_mapping": V1InferenceMappingModel(**inference_mapping_res),
"method": "join",
"process_window": V1PreprocessingJoinProcessWindow(
**{"start_time": gt_start_time, "end_time": gt_end_time}
),
}
Additional preprocess job, for policies
As part of the workflow for creating a placeholder model, the client submits another preprocessing job before running defined policies. As with the model performance preprocessing job this one also helps ensure policies are run with the best performance. The preprocess job for policies does not include the method join and instead uses start_date and end_date values to define the process window. These dates identify when predictions are made; since the notebook includes historical data as well, the start date is defined as today - x days and end date is set at today.
The payload for the policies preprocess job from the notebook example:
payload = {
"inference_mapping": V1InferenceMappingModel(**inference_mapping_res),
"process_window": V1PreprocessingProcessWindow(
**{"start_date": from_date, "end_date": to_date}
),
}
The notebook uses the following call to submit preprocessing jobs:
preprocessing_res = preprocessing_api.submit(payload)
When successful returns V1JobModel with information about the job. The result parameter identifies the input for the job and if it succeeded. The actual result value may vary depending on the type of preprocessing job.
For example:
{
"job_id": "preprocessing-queue-123456",
"job_type": "preprocessing",
"job_description": null,
"user_id": "user1",
"run_id": "None",
"status": "done",
"statusdetails": "",
"result": {
"success": true,
"error": " ",
"data": " ",
"input": "{}",
"error_type": " ",
},
"exception": " ",
"created": "2023-05-01T10:55:05.727056",
"started": "2023-05-01 10:55:10.141123",
"finished": "2023-05-01 10:55:10.452860",
"timestamp": 1682938510455.97,
"configmap":"config-preprocessing-queue-211558"
}
Create a segment
If you want to run policies on narrow populations of data (in addition to running on all data defined for the policy), create and add segments for those policies. Using segments, the client can narrow the scope and uncover patterns that may be occurring only in portions of the full data population for the model. One policy can include multiple segments, and one segment can be used (i.e., added to) multiple policies, as long as the configuration for the policy and segment match (e.g., same feature set).
If needed, look for more information about using segments in the API reference documentation.
You create a new segment using the V1SegmentBaseModel Pydantic model. Specify the segment as a combination of features, values, operators, and conjunctions. This configuration creates a filter to define a slice of the full population. You can include a combination of value and operators ( =, >, <, <=, >=, BETWEEN, TOP, BOTTOM, DISTINCT) and conjunctions (AND, OR, null (default value is null)). Additionally, specify the model id and a name for the segment as part of the payload.
payload = {
"name": "segment_x",
"description": "Segment description",
"filters": [
{
"feature_name": "feature-xyz",
"value": [
"10",
"5"
],
"operator": "=",
"conjunction": null,
},
{
"feature_name": "PULocation",
"operator": "=",
"value": ["Williamsburg (South Side)", "South Williamsburg"],
"conjunction": "and",
},
{
"feature_name": "DOLocation",
"operator": "=",
"value": ["Little Italy/NoLiTa", "Lower East Side"],
"conjunction": None,
},
],
"status": "inactive",
"model_uuid": "123-abc-456",
}
Example from the notebook:
segment2_params = segments_api.create(segment2_list)
When successful, create() returns V1SegmentModelList.
{
"model_uuid": "123-abc-456",
"name": "segment_1",
"description": "Segment description",
"filters": [
{
"feature_name": "feature-xyz",
"value": [
"10",
"5"
],
"operator": "=",
"conjunction": null,
"grouped_filters": null
}
],
"id": 100,
"status": "active",
"created_ts": 1682538384448.842,
"modified_ts": 1682538384448.842,
"created_by": "user1",
"modified_by": "user2",
"policies": null
},
Create a policy
To monitor deployed models and look for signs of potential issues, the client can create feature, prediction, and performance drift policies. The Pydantic model V1PolicyRequestModel identifies the payload for a policy, which varies depending on the type of policy. (See the API documentation for more information about policies.) You can configure segments and hotspot analysis for any of the drift policies to aid tracking drift and understanding model conditions.
To create just one policy, call create(data: V1PolicyRequestModelList). Define the type of policy (performance or drift), settings specific to that type, policy name and deployment information, any segments to run for the policy, hotspot analysis configuration, the data to use as the baseline and the target data for comparison, the initial status (active or inactive), etc.
Alternatively, to create multiple multiple feature drift policies at one time, use PoliciesV1Api.bulk_create(). By default, this method creates four feature drift policies (based on the default window_type parameter): day-to-day, week-to-week, month-to-month, quarter-to-quarter. Required values include model and project name, version, and stage. When run, bulk_create() apply equal distribution to specified features (if any); otherwise it retrieves values from the inference mapping defined for the model, again applying equal weighting. Any optional values specified for segment, window method, name prefix, window type, hotspot analysis, and so on are applied to the new policies.
Unless specified otherwise, the bulk-created policies run on all model features (of equal weight), hotspot analysis is enabled for all (categorical) columns set for drift and segmentation (based on inference mapping settings), and there are no defined segments. The baseline data for each policy is “prior” (this is the default window_method) and drift is measured with PSI.
The following example payload creates a single feature drift policy that includes one segment and runs hotspot analysis for two features.
"deployment": "deployment_xyz",
"model_name": "deployment_xyz",
"model_version": "1",
"model_stage": "primary",
"name": "Policy Name for distance-based feature drift",
"description": "Policy description",
"type": "drift",
"policy": {
"type": "feature-drift",
"drift_type": "distance",
"window_parameters": {
"target": {
"window_type": "day"
},
"baseline": {
"window_method": "prior",
"window_type": "day"
}
},
"select_features_type": "custom",
"feature_weightage": "equal",
"feature_weights": {
"feature_a": 25,
"feature_b": 25,
"feature_c": 25,
"feature_d": 25
},
"drift_measure": "PSI",
"warning_level": 1,
"critical_level": 2,
"schedule": "30 6 * * *",
"deployment_name": "deployment_xyz",
"method": "preprocess",
"hotspot_analysis": {
"method": "flat",
"value": 30,
"features": ["feature_a", "feature_b"],
},
},
"status": "active",
"segments": [
{
"model_uuid": "string",
"name": "segment_1",
"description": "Segment description",
"filters": [
{
"feature_name": "feature_a",
"value": [
"10",
"5"
],
"operator": ">=",
"conjunction": None,
}
],
"id": 1,
"status": "active",
"created_ts": 1679496214358.969,
"modified_ts": 1679496214358.969,
"created_by": "user1",
"modified_by": "user2"
}
]
},
Example from the notebook:
policy1_res = policies_api.create(policy1_data)
When successful, create() returns V1PolicyModelList information object.
"example": [
{
"uuid": "123_abcd_456",
"deployment": "deployment_xyz",
"model_name": "model_abc",
"model_version": "1",
"model_stage": "primary",
"name": "Policy with segments",
"description": "Policy description",
"type": "drift",
"policy": {
"type": "feature-drift",
"drift_type": "distance",
"window_parameters": {
"target": {
"window_type": "day"
},
"baseline": {
"window_method": "prior",
"window_type": "day"
}
},
"select_features_type": "custom",
"feature_weightage": "equal",
"feature_weights": {
"feature_a": 25,
"feature_b": 25,
"feature_c": 25,
"feature_d": 25
},
"drift_measure": "PSI",
"warning_level": 1,
"critical_level": 2,
"schedule": "0 0 ? * *",
"method": "preprocess"
},
"status": "active",
"created_ts": 1675292368,
"modified_ts": 1675292368,
"created_by": "user1",
"modified_by": "user2"
}
Run a preprocess job for policies
(See the section, “Submit preprocess job for model performance”.)
Example payload from the notebook:
payload = {
"inference_mapping": V1InferenceMappingModel(**inference_mapping_res),
"process_window": V1PreprocessingProcessWindow(
**{"start_date": from_date, "end_date": to_date}
),
}
Run the policies
Use driftdetectionV1Api submit for submitting policies to be run by VIANOPS.
How the sample notebook runs policies
The sample notebook is configured with multiple policies set to run for multiple days. To accomplish this, the notebook creates the offset in the create_offsets() helper script, sends process_date in the policy payload (which is returned from requesting access all of the policy payloads), and uses a loop to run the policies:
policy_search_params = V1PolicySearchRequest(**payload)
search_response = policies_api.search(policy_search_params)
See create_offsets in the helper file for details. For the sample notebook, the actual code for running the policies is defined in helper.py.
Example from the notebook:
def run_policy(policy):
payload = policy
driftdetection_res = driftdetection_api.submit(policy)
return driftdetection_res
Submit a model performance job
To access model performance metrics for comparison and analysis, use the model performance submit() method to run a model performance job for your model deployment. In the payload provide the model and deployment information and set the method aspreprocess.
For example, the performance policy configured for the sample notebook requires performance metrics to perform comparison. The notebook runs a model performance job for the deployed sample model as follows:
modelperformance_res = modelperformance_api.submit(modelperformance_payload)
After running a model performance job, metrics for the model are available via the API or in the UI, within performance trends.