Distance-based drift on feature data (APIs)
For policies configured to measure distance-based drift on model inputs or features, this object specifies how the policy operates, the data it analyzes (i.e., comparing current or past target data compared to current or past baseline data), the schedule it runs on, and thresholds for alert conditions.
"policy": {
"window_parameters": {
"target": {
"window_type": "day",
"process_date": null,
"offset_type": null,
"offset": null,
"start_of_week": null,
"quarter_start": null,
"year_start": null
},
"baseline": {
"window_method": "prior",
"window_type": "day",
"last_amount": null,
"process_date": null,
"offset_type": null,
"offset": null,
"start_of_week": null,
"quarter_start": null,
"year_start": null
}
},
"warning_level": ".1",
"critical_level": ".15",
"schedule": "0 0 0 ? * *",
"deployment_name": "deploymentxyz",
"method": "preprocess",
"hotspot_analysis": {
"method": "flat",
"value": 50,
"features": [
"PULocation",
"DOLocation"
]
},
"type": "feature-drift",
"drift_type": "distance",
"select_features_type": "custom",
"feature_weightage": "equal",
"feature_weights": {
"feature_a": 25,
"feature_b": 25,
"feature_c": 25,
"feature_d": 25
},
"drift_measure": "PSI",
"baseline_bins": null
}
Fields are case-insensitive for supported values.
{
"data": {
"h-0": "Property",
"h-1": "Type",
"h-2": "Description",
"0-0": "window_parameters",
"0-1": "array",
"0-2": "Settings for the target data (the policy runs on) and baseline data (the policy uses as comparison to target data to determine drift). See table, below.",
"1-0": "warning_level",
"1-1": "float",
"1-2": "Threshold value for flagging warning events. For example, if set to .1, then a warning alert is signaled when the drift measure exceeds .1.",
"2-0": "critical_level",
"2-1": "float",
"2-2": "Threshold value for flagging critical events. For example, if set to .15 then a critical alert is signaled when the drift measure exceeds .15.",
"3-0": "schedule",
"3-1": "cron string",
"3-2": "Schedule for calculating drift. Defined as a cron expression with a maximum of 5 fields. If needed, see an [example cron expression generator](https://crontab.cronhub.io/). For example, to calculate drift every day at 8am: \n`\"schedule\": \"0 8 \\* \\* \\*\"\\`.",
"4-0": "deployment_name",
"4-1": "string",
"4-2": "User-defined name of the deployment.",
"5-0": "method",
"5-1": "string",
"5-2": "Method for saving policy data. This value should be set this value to `preprocess` (the default value).",
"6-0": "hotspot_analysis",
"6-1": "array",
"6-2": "Settings for running [hotspot analysis](api-policies#hotspot-analysis): method for calculating the impact (`flat` or `percent`), integer value related to that method, and features on which to run hotspot analysis.",
"7-0": "type",
"7-1": "string",
"7-2": "Method of measuring drift; `feature-drift`.",
"8-0": "drift_type",
"8-1": "string",
"8-2": "Type of drift to measure; `distance`.",
"9-0": "select_features_type",
"9-1": "string",
"9-2": "Optional; Specifies the set of features to include in the drift policy. Supported values: `custom` (manually specified), `all` (all features). Defaults to `all`.",
"10-0": "feature_weightage",
"10-1": "string",
"10-2": "Weight applied to drift policy features. Supported values: \n`manual` (manually specified), `equal` (equal weightage percentages, adding up to 100). Defaults to `equal`.",
"11-0": "feature_weights",
"11-1": "array",
"11-2": "List of names for selected features and their weights, based on the selection method (`select_features_type`) and weighting method (`feature_weightage`).",
"12-0": "drift_measure",
"12-1": "string",
"12-2": "Statistical metric to use to measure drift, either Population Stability Index or Jensen Shannon Distance. Supported values: `PSI` (default) or `distance_JS`.",
"13-0": "baseline_bins",
"13-1": "string",
"13-2": "Enables custom binning. Specifies a list of numbers that define start and end bin edges for each custom bin. Can be used for some or all float or integer features in `feature_weights`. For each feature explicitly listed in `baseline_bins`, the platform applies that bin to the baseline and target on every run of this policy. For features not listed in `baseline_bins`, the platform applies the default bin method. For example, \n \n`\"baseline_bins\": { \n \"feature_a\": [-10, -5, 0, 3, 8, 27, 546], \n \"feature_c\": [-23.1, 12.55, 54.56, 123.456] \n } ` \n \nspecifies custom bins for two features. For example, the six bins for feature_a are: `-10 up to -5`, `-5 up to 0`, `0 up to 3`, `3 up to 8` `8 up to 27`, and `546 to the end of the data`. (These are bin edges (start and end of bins) so the number of bins is one less than the number of elements in each list/array.)"
},
"cols": 3,
"rows": 14,
"align": [
"left",
"left",
"left"
]
}
window_parameters object
Configures settings for target and baseline windows to specify the inference data to run the policy on and data to use as comparison. Additional (optional) parameters are available for configuration depending on policy settings; see table (below) for additional parameters.
"window_parameters": {
"target": {
"window_type": "day",
"process_date": null,
"offset_type": null,
"offset": null,
"start_of_week": null,
"quarter_start": null,
"year_start": null
},
"baseline": {
"window_method": "prior",
"window_type": "day",
"last_amount": null,
"process_date": null,
"offset_type": null,
"offset": null,
"start_of_week": null,
"quarter_start": null,
"year_start": null
}
}
{
"data": {
"h-0": "Property",
"h-1": "Type",
"h-2": "Description",
"0-0": "**target** ",
"0-1": "array",
"0-2": "Window of collected model inferences that the policy analyzes and compares to baseline data to detect drift or performance issues.",
"1-0": "window_type",
"1-1": "string",
"1-2": "Size of the target window. Supported values: `day`, `week`, `month`, `quarter`, `year`.",
"2-0": "process_date",
"2-1": "string",
"2-2": "Optional. Specifies an historical end date for the target window, in conjunction with `window_type`. Format as yyyy-mm-dd. \n \nFor example, if set to `2022-03-10` and `window_type` is set to `week`, the policy analyzes inferences received for the week leading up to 2022-03-10. \nIf set, the Baseline window is also reset accordingly. If not set, the default value is the date of the day the job is being run. \n \n**Note: **The policy uses the `process_date` instead of any configured offset offset. If you need a precise historical end date, you may want to use `process_date` rather than `offset`.",
"3-0": "offset_type",
"3-1": "string",
"3-2": "Optional. Specifies an historical timeframe for the target window data (in conjunction with the value of `offset`). Supported values: `day`, `week`, `month`, `quarter`. \n \nFor example, if set to `week` and `offset` is set to 3, the target window contains data up to 3 weeks ago (where amount of data is defined by `window_type`). If set, the baseline window is also reset accordingly. \n \n**Note: **If `process_date` is defined, the policy ignores any offset configuration.",
"4-0": "offset",
"4-1": "string",
"4-2": "Optional. Number for `offset_type`. For example, `2` (days, weeks, months, etc. in the past), depending on value of `offset_type`. Set `offset` to `1` to specify today, this week, this month, etc. \n \n**Note: **If `process_date` is defined, the policy ignores any offset configuration.",
"5-0": "start_of_week",
"5-1": "string",
"5-2": "Optional. Applies if `window_type` is `week`. \n \nSpecifies the start day for the target window other than default (Monday). Supported values: days of the week, `Monday` through `Sunday`.",
"6-0": "quarter_start",
"6-1": "string",
"6-2": "Optional. Applies if `window_type` is `quarter`. \n \nSpecifies the start day for the target window other than default first day of current quarter (i.e., January 1st, April 1st, July 1st, or October 1st). Format as yyyy-mm-dd.",
"7-0": "year_start",
"7-1": "string",
"7-2": "Optional. Applies if `window_type` is `year`. \n \nSpecifies the start day for the target window other than default (January 1st of the current year). Format as yyyy-mm-dd.",
"8-0": "**baseline**",
"8-1": "array",
"8-2": "Window of model data that the policy uses for comparison to the target data.",
"9-0": "window_method",
"9-1": "string",
"9-2": "Defines the baseline window of data to compare to the target window. \n \n**IMPORTANT**: Make sure to specify a baseline window that makes sense for the specified target window and that is EARLIER than the target window. Supported value depends on `window_type` setting. \n \nSupported values: `prior` (i.e., prior day, week, or month), `last` (i.e., last same weekdays, last number of weeks, last of months), `same_prior_year` (i.e., same quarter prior year), `training` (to use training data for baseline).",
"10-0": "window_type",
"10-1": "string",
"10-2": "Optional. Size of the baseline window, or specifies to use training data as baseline. Supported values: `day`, `week`, `month`, `quarter`, `year`, `training`. If not set, inherits the value of `window_type` set for the target window. \n \nRecommended to use the same value set for target window.",
"11-0": "last_amount",
"11-1": "integer",
"11-2": "Optional. Specifies the number of previous time periods to use as a dataset. Used when `window_method` is set to `last` (indicating last weekday, week, or month). \n \nFor example, if `window_type` is set to `day` and `last_amount` is set to 4, then the baseline windows contains all data for the last 4 days.",
"12-0": "process_date",
"12-1": "string",
"12-2": "Optional. Specifies an historical end date for the baseline window, in conjunction with `window_type`. Format as yyyy-mm-dd. \n \nFor example, if set to `2022-03-10` and `window_type` is set to `week`, the policy analyzes inferences received for the week leading up to 2022-03-10. \nIf set, the Target window is also reset accordingly. If not set, the default value is the date of the day the job is being run. \n \n**Note: **The policy uses the `process_date` instead of any configured offset offset. If you need a precise historical end date, you may want to use `process_date` rather than `offset`.",
"13-0": "offset_type",
"13-1": "string",
"13-2": "Optional. Specifies an historical timeframe for the baseline window data (in conjunction with the value of `offset`). Supported values: `day`, `week`, `month`, `quarter`. \n \nFor example, if set to `week` and `offset` is set to 3, the baseline window contains data up to 3 weeks ago (where amount of data is defined by `window_type`). If set, the target window is also reset accordingly. \n \n**Note: **If `process_date` is defined, the policy ignores any offset configuration.",
"14-0": "offset",
"14-1": "string",
"14-2": "Optional. Number for `offset_type`. For example, `2` (days, weeks, months, etc. in the past), depending on value of `offset_type`. Set `offset` to `1` to specify today, this week, this month, etc. \n \n**Note: **If `process_date` is defined, the policy ignores any offset configuration.",
"15-0": "start_of_week",
"15-1": "string",
"15-2": "Optional. Applies if `window_type` is `week`. \n \nSpecifies the start day for the baseline window other than default (Monday). Supported values: days of the week, `Monday` through `Sunday`.",
"16-0": "quarter_start",
"16-1": "string",
"16-2": "Optional. Applies if `window_type` is `quarter`. \n \nSpecifies the start day for the baseline window other than default first day of current quarter (i.e., January 1st, April 1st, July 1st, or October 1st). Format as yyyy-mm-dd.",
"17-0": "year_start",
"17-1": "string",
"17-2": "Optional. Applies if `window_type` is `year`. \n \nSpecifies the start day for the baseline window other than default (January 1st of the current year). Format as yyyy-mm-dd."
},
"cols": 3,
"rows": 18,
"align": [
"left",
"left",
"left"
]
}
TABLE OF CONTENTS