CLI (v2) schedule YAML schema for model monitoring

YAML syntax

Key	Type	Description
`$schema`	string	The YAML schema.
`name`	string	Required. Name of the schedule.
`version`	string	Version of the schedule. If omitted, Azure Machine Learning will autogenerate a version.
`description`	string	Description of the schedule.
`tags`	object	Dictionary of tags for the schedule.
`trigger`	object	Required. The trigger configuration to define rule when to trigger job. One of `RecurrenceTrigger` or `CronTrigger` is required.
`create_monitor`	object	Required. The definition of the monitor that will be triggered by a schedule. `MonitorDefinition` is required.

Trigger configuration

Recurrence trigger

Key	Type	Description	Allowed values
`type`	string	Required. Specifies the schedule type.	recurrence
`frequency`	string	Required. Specifies the unit of time that describes how often the schedule fires.	`minute`, `hour`, `day`, `week`, `month`
`interval`	integer	Required. Specifies the interval at which the schedule fires.
`start_time`	string	Describes the start date and time with timezone. If `start_time` is omitted, the first job will run instantly and the future jobs will be triggered based on the schedule, saying `start_time` will be equal to the job created time. If the start time is in the past, the first job will run at the next calculated run time.
`end_time`	string	Describes the end date and time with timezone. If `end_time` is omitted, the schedule will continue to run until it's explicitly disabled.
`timezone`	string	Specifies the time zone of the recurrence. If omitted, by default is UTC.	See appendix for timezone values
`pattern`	object	Specifies the pattern of the recurrence. If pattern is omitted, the job(s) will be triggered according to the logic of start_time, frequency and interval.

Recurrence schedule

Recurrence schedule defines the recurrence pattern, containing hours, minutes, and weekdays.

When frequency is day, pattern can specify hours and minutes.
When frequency is week and month, pattern can specify hours, minutes and weekdays.

Key	Type	Allowed values
`hours`	integer or array of integer	`0-23`
`minutes`	integer or array of integer	`0-59`
`week_days`	string or array of string	`monday`, `tuesday`, `wednesday`, `thursday`, `friday`, `saturday`, `sunday`

CronTrigger

Key	Type	Description	Allowed values
`type`	string	Required. Specifies the schedule type.	cron
`expression`	string	Required. Specifies the cron expression to define how to trigger jobs. expression uses standard crontab expression to express a recurring schedule. A single expression is composed of five space-delimited fields:`MINUTES HOURS DAYS MONTHS DAYS-OF-WEEK`
`start_time`	string	Describes the start date and time with timezone. If start_time is omitted, the first job will run instantly and the future jobs will be triggered based on the schedule, saying start_time will be equal to the job created time. If the start time is in the past, the first job will run at the next calculated run time.
`end_time`	string	Describes the end date and time with timezone. If end_time is omitted, the schedule will continue to run until it's explicitly disabled.
`timezone`	string	Specifies the time zone of the recurrence. If omitted, by default is UTC.	See appendix for timezone values

Monitor definition

Key	Type	Description	Allowed values	Default value
`compute`	Object	Required. Description of compute resources for Spark pool to run monitoring job.
`compute.instance_type`	String	Required. The compute instance type to be used for Spark pool.	'standard_e4s_v3', 'standard_e8s_v3', 'standard_e16s_v3', 'standard_e32s_v3', 'standard_e64s_v3'	n/a
`compute.runtime_version`	String	Optional. Defines Spark runtime version.	`3.1`, `3.2`	`3.2`
`monitoring_target`	Object	Azure Machine Learning asset(s) associated with model monitoring.
`monitoring_target.endpoint_deployment_id`	String	Optional. The associated Azure Machine Learning endpoint/deployment ID in format of `azureml:myEnpointName:myDeploymentName`. This field is required if your endpoint/deployment has enabled model data collection to be used for model monitoring.
`monitoring_target.model_id`	String	Optional. The associated model ID for model monitoring.
`monitoring_signals`	Object	Dictionary of monitoring signals to be included. The key is a name for monitoring signal within the context of monitor and the value is an object containing a monitoring signal specification. Optional for basic model monitoring that uses recent past production data as comparison baseline and has 3 monitoring signals: data drift, prediction drift, and data quality.
`alert_notification`	Object	Description of alert notification recipients.
`alert_notification.emails`	Object	List of email addresses to receive alert notification.

Monitoring signals

Data drift

As the data used to train the model evolves in production, the distribution of the data can shift, resulting in a mismatch between the training data and the real-world data that the model is being used to predict. Data drift is a phenomenon that occurs in machine learning when the statistical properties of the input data used to train the model change over time.

Key	Type	Description	Allowed values	Default value
`type`	String	Required. Type of monitoring signal. Prebuilt monitoring signal processing component is automatically loaded according to the `type` specified here.	`data_drift`	`data_drift`
`target_dataset`	Object	Optional. Description of production data to be analyzed for monitoring signal.
`target_dataset.dataset`	Object	Optional. Description of production data to be analyzed for monitoring signal.
`target_dataset.dataset.input_dataset`	Object	Optional. Description of input data source, see job input data specification.
`target_dataset.dataset.dataset_context`	String	The context of data, it refers model production data and could be model inputs or model outputs	`model_inputs`
`target_dataset.dataset.pre_processing_component`	String	Component ID in the format of `azureml:myPreprocessing@latest` for a registered component. This is required if `target_dataset.dataset.input_dataset.type` is `uri_folder`, see preprocessing component specification.
`target_dataset.data_window_size`	Integer	Optional. Data window size in days. This is the production data window to be computed for data drift.	By default the data window size is the last monitoring period.
`baseline_dataset`	Object	Optional. Recent past production data is used as comparison baseline data if this isn't specified. Recommendation is to use training data as comparison baseline.
`baseline_dataset.input_dataset`	Object	Description of input data source, see job input data specification.
`baseline_dataset.dataset_context`	String	The context of data, it refers to the context that dataset was used before	`model_inputs`, `training`, `test`, `validation`
`baseline_dataset.pre_processing_component`	String	Component ID in the format of `azureml:myPreprocessing@latest` for a registered component. This is required if `baseline_dataset.input_dataset.type` is `uri_folder`, see preprocessing component specification.
`features`	Object	Optional. Target features to be monitored for data drift. Some models might have hundreds or thousands of features, it's always recommended to specify interested features for monitoring.	One of following values: list of feature names, `features.top_n_feature_importance`, or `all_features`	Default `features.top_n_feature_importance = 10` if `baseline_dataset.dataset_context` is `training`, otherwise, default is `all_features`
`data_segment`	Object	Optional. Description of specific data segment to be monitored for data drift.
`data_segment.feature_name`	String	The name of feature used to filter for data segment.
`data_segment.feature_values`	Array	list of feature values used to filter for data segment
`alert_notification`	Boolean	Turn on/off alert notification for the monitoring signal. `True` or `False`
`metric_thresholds`	Object	List of metrics and thresholds properties for the monitoring signal. When threshold is exceeded and `alert_notification` is on, user will receive alert notification.		By default, the object contains `numerical` metric `population_stability_index` with threshold of `0.02` and `categorical` metric `normalized_wasserstein_distance` with threshold of `0.02`
`metric_thresholds.applicable_feature_type`	String	Feature type that the metric will be applied to.	`numerical` or `categorical`
`metric_thresholds.metric_name`	String	The metric name for the specified feature type.	Allowed `numerical` metric names: `jensen_shannon_distance`, `population_stability_index`, `two_sample_kolmogorov_test`. Allowed `categorical` metric names: `normalized_wasserstein_distance`, `chi_squared_test`
`metric_thresholds.threshold`	Number	The threshold for the specified metric.

Prediction drift

Prediction drift tracks changes in the distribution of a model's prediction outputs by comparing it to validation or test labeled data or recent past production data.

Key	Type	Description	Allowed values	Default value
`type`	String	Required. Type of monitoring signal. Prebuilt monitoring signal processing component is automatically loaded according to the `type` specified here	`prediction_drift`	`prediction_drift`
`target_dataset`	Object	Optional. Description of production data to be analyzed for monitoring signal.
`target_dataset.dataset`	Object	Optional. Description of production data to be analyzed for monitoring signal.
`target_dataset.dataset.input_dataset`	Object	Optional. Description of input data source, see job input data specification.
`target_dataset.dataset.dataset_context`	String	The context of data, it refers model production data and could be model inputs or model outputs	`model_outputs`
`target_dataset.dataset.pre_processing_component`	String	Component ID in the format of `azureml:myPreprocessing@latest` for a registered component. This is required if `target_dataset.dataset.input_dataset.type` is `uri_folder`, see preprocessing component specification.
`target_dataset.data_window_size`	Integer	Optional. Data window size in days. This is the production data window to be computed for prediction drift.	By default the data window size is the last monitoring period.
`baseline_dataset`	Object	Optional. Recent past production data is used as comparison baseline data if this isn't specified. Recommendation is to use training data as comparison baseline.
`baseline_dataset.input_dataset`	Object	Description of input data source, see job input data specification.
`baseline_dataset.dataset_context`	String	The context of data, it refers to the context that dataset come from.	`model_inputs`, `model_outputs`, `test`, `validation`
`baseline_dataset.target_column_name`	String	The name of target column.
`baseline_dataset.pre_processing_component`	String	Component ID in the format of `azureml:myPreprocessing@latest` for a registered component. This is required if `baseline_dataset.input_dataset.type` is `uri_folder`, see preprocessing component specification.
`alert_notification`	Boolean	Turn on/off alert notification for the monitoring signal. `True` or `False`
`metric_thresholds`	Object	List of metrics and thresholds properties for the monitoring signal. When threshold is exceeded and `alert_notification` is on, user will receive alert notification.		By default, the object contains `numerical` metric `population_stability_index` with threshold of `0.02` and `categorical` metric `normalized_wasserstein_distance` with threshold of `0.02`
`metric_thresholds.applicable_feature_type`	String	Feature type that the metric will be applied to.	`numerical` or `categorical`
`metric_thresholds.metric_name`	String	The metric name for the specified feature type.	Allowed `numerical` metric names: `jensen_shannon_distance`, `population_stability_index`, `two_sample_kolmogorov_test`. Allowed `categorical` metric names: `normalized_wasserstein_distance`, `chi_squared_test`
`metric_thresholds.threshold`	Number	The threshold for the specified metric.

Data quality

Data quality signal tracks data quality issues in production by comparing to training data or recent past production data.

Key	Type	Description	Allowed values	Default value
`type`	String	Required. Type of monitoring signal. Prebuilt monitoring signal processing component is automatically loaded according to the `type` specified here	`data_quality`	`data_quality`
`target_dataset`	Object	Optional. Description of production data to be analyzed for monitoring signal.
`target_dataset.dataset`	Object	Optional. Description of production data to be analyzed for monitoring signal.
`target_dataset.dataset.input_dataset`	Object	Optional. Description of input data source, see job input data specification.
`target_dataset.dataset.dataset_context`	String	The context of data, it refers model production data and could be model inputs or model outputs	`model_inputs`, `model_outputs`
`target_dataset.dataset.pre_processing_component`	String	Component ID in the format of `azureml:myPreprocessing@latest` for a registered component. This is required if `target_dataset.dataset.input_dataset.type` is `uri_folder`, see preprocessing component specification.
`target_dataset.data_window_size`	Integer	Optional. Data window size in days. This is the production data window to be computed for data quality issues.	By default the data window size is the last monitoring period.
`baseline_dataset`	Object	Optional. Recent past production data is used as comparison baseline data if this isn't specified. Recommendation is to use training data as comparison baseline.
`baseline_dataset.input_dataset`	Object	Description of input data source, see job input data specification.
`baseline_dataset.dataset_context`	String	The context of data, it refers to the context that dataset was used before	`model_inputs`, `model_outputs`, `training`, `test`, `validation`
`baseline_dataset.pre_processing_component`	String	Component ID in the format of `azureml:myPreprocessing@latest` for a registered component. This is required if `baseline_dataset.input_dataset.type` is `uri_folder`, see preprocessing component specification.
`features`	Object	Optional. Target features to be monitored for data quality. Some models might have hundreds or thousands of features. It's always recommended to specify interested features for monitoring.	One of following values: list of feature names, `features.top_n_feature_importance`, or `all_features`	Default to `features.top_n_feature_importance = 10` if `baseline_dataset.dataset_context` is `training`, otherwise default is `all_features`
`alert_notification`	Boolean	Turn on/off alert notification for the monitoring signal. `True` or `False`
`metric_thresholds`	Object	List of metrics and thresholds properties for the monitoring signal. When threshold is exceeded and `alert_notification` is on, user will receive alert notification.		By default, the object contains following `numerical` and `categorical` metrics: `null_value_rate`, `data_type_error_rate`, and `out_of_bounds_rate`
`metric_thresholds.applicable_feature_type`	String	Feature type that the metric will be applied to.	`numerical` or `categorical`
`metric_thresholds.metric_name`	String	The metric name for the specified feature type.	Allowed `numerical` and `categorical` metric names are: `null_value_rate`, `data_type_error_rate`, `out_of_bound_rate`
`metric_thresholds.threshold`	Number	The threshold for the specified metric.

Feature attribution drift

The feature attribution of a model may change over time due to changes in the distribution of data, changes in the relationships between features, or changes in the underlying problem being solved. Feature attribution drift is a phenomenon that occurs in machine learning models when the importance or contribution of features to the prediction output changes over time.

Key	Type	Description	Allowed values	Default value
`type`	String	Required. Type of monitoring signal. Prebuilt monitoring signal processing component is automatically loaded according to the `type` specified here	`feature_attribution_drift`	`feature_attribution_drift`
`target_dataset`	Object	Optional. Description of production data to be analyzed for monitoring signal.
`target_dataset.dataset`	Object	Optional. Description of production data to be analyzed for monitoring signal.
`target_dataset.dataset.input_dataset`	Object	Optional. Description of input data source, see job input data specification.
`target_dataset.dataset.dataset_context`	String	The context of data. It refers to production model inputs data.	`model_inputs`
`target_dataset.dataset.pre_processing_component`	String	Component ID in the format of `azureml:myPreprocessing@latest` for a registered component. This is required if `target_dataset.dataset.input_dataset.type` is `uri_folder`, see preprocessing component specification.
`target_dataset.lookback_period_days`	Integer	Lookback window to include extra data in current monitoring run, this is useful if you want model monitoring to run more frequently but the production data within monitoring period isn't enough or skewed.
`baseline_dataset`	Object	Required. It must be `training` data.
`baseline_dataset.input_dataset`	Object	Description of input data source, see job input data specification.
`baseline_dataset.dataset_context`	String	The context of data, it refers to the context that dataset was used before.	`training`
`baseline_dataset.pre_processing_component`	String	Component ID in the format of `azureml:myPreprocessing@latest` for a registered component. This is required if `baseline_dataset.input_dataset.type` is `uri_folder`, see preprocessing component specification.
`alert_notification`	Boolean	Turn on/off alert notification for the monitoring signal. `True` or `False`
`metric_thresholds`	Object	List of metrics and thresholds properties for the monitoring signal. When threshold is exceeded and `alert_notification` is on, user will receive alert notification.		By default, the object contains `normalized_discounted_cumulative_gain` metric with threshold of `0.02`
`metric_thresholds.applicable_feature_type`	String	Feature type that the metric will be applied to.	`all_feature_types`	`all feature_types`
`metric_thresholds.metric_name`	String	The metric name for the specified feature type.	`normalized_discounted_cumulative_gain`	`normalized_discounted_cumulative_gain`
`metric_thresholds.threshold`	Number	The threshold for the specified metric.		`0.02`

Remarks

The az ml schedule command can be used for managing Azure Machine Learning models.

Examples

Examples are available in the examples GitHub repository. A couple are as follows:

YAML: Schedule with recurrence pattern

APPLIES TO: Azure CLI ml extension v2 (current)

$schema: https://azuremlschemas.azureedge.net/latest/schedule.schema.jsonname: simple_recurrence_job_scheduledisplay_name: Simple recurrence job scheduledescription: a simple hourly recurrence job scheduletrigger: type: recurrence frequency: day #can be minute, hour, day, week, month interval: 1 #every day schedule: hours: [4,5,10,11,12] minutes: [0,30] start_time: "2022-07-10T10:00:00" # optional - default will be schedule creation time time_zone: "Pacific Standard Time" # optional - default will be UTCcreate_job: ./simple-pipeline-job.yml# create_job: azureml:simple-pipeline-job

YAML: Schedule with cron expression