"Deploy and monitor a model in Azure Machine Learning" deploy-prod fails at "Deploy model to managed online endpoint"

[mattflaschen]

Following Deploy and monitor a model in Azure Machine Learning, "/deploy-prod" fails.

After commenting "/deploy-prod", it is supposed to create an endpoint and deploy to it.

The GitHub action fails at the "Deploy model to managed online endpoint" step with the following error:

Run python src/deploy_to_online_endpoint.py \
Traceback (most recent call last):
Connecting to Azure Machine Learning workspace...
  File "/opt/hostedtoolcache/Python/3.10.21/x64/lib/python3.10/site-packages/azure/core/polling/base_polling.py", line 950, in run
Ensuring online endpoint 'diabetes-endpoint-73dcd7c9' exists...
    self._poll()
  File "/opt/hostedtoolcache/Python/3.10.21/x64/lib/python3.10/site-packages/azure/core/polling/base_polling.py", line 982, in _poll
    raise OperationFailed("Operation failed or canceled")
azure.core.polling.base_polling.OperationFailed: Operation failed or canceled

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/mslearn-mlops/mslearn-mlops/src/deploy_to_online_endpoint.py", line 119, in <module>
    main()
  File "/home/runner/work/mslearn-mlops/mslearn-mlops/src/deploy_to_online_endpoint.py", line 100, in main
    endpoint = ensure_endpoint(ml_client, args.endpoint_name)
  File "/home/runner/work/mslearn-mlops/mslearn-mlops/src/deploy_to_online_endpoint.py", line 57, in ensure_endpoint
    return ml_client.online_endpoints.begin_create_or_update(endpoint).result()
  File "/opt/hostedtoolcache/Python/3.10.21/x64/lib/python3.10/site-packages/azure/core/polling/_poller.py", line 323, in result
    self.wait(timeout)
  File "/opt/hostedtoolcache/Python/3.10.21/x64/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.21/x64/lib/python3.10/site-packages/azure/core/polling/_poller.py", line 342, in wait
    raise self._exception  # type: ignore
  File "/opt/hostedtoolcache/Python/3.10.21/x64/lib/python3.10/site-packages/azure/core/polling/_poller.py", line 247, in _start
    self._polling_method.run()
  File "/opt/hostedtoolcache/Python/3.10.21/x64/lib/python3.10/site-packages/azure/core/polling/base_polling.py", line 965, in run
    raise HttpResponseError(response=self._pipeline_response.http_response, error=err) from err
azure.core.exceptions.HttpResponseError: (SubscriptionNotRegistered) Resource provider [N/A] isn't registered with Subscription [N/A]. Please see troubleshooting guide, available here: https://aka.ms/register-resource-provider
Code: SubscriptionNotRegistered
Message: Resource provider [N/A] isn't registered with Subscription [N/A]. Please see troubleshooting guide, available here: https://aka.ms/register-resource-provider
Error: Process completed with exit code 1.

[mattflaschen]

This is an easy fix in the setup.sh script (just the az provider register line below):

Since it doesn't say which resource provider is missing ("N/A"), I added Microsoft.PolicyInsights (credit to https://learn.microsoft.com/en-us/answers/questions/5601678/azure-ml-endpoint-creation-fails-persistent-subscr and https://learn.microsoft.com/en-us/answers/questions/1983847/error-while-creating-a-managed-online-endpoint-in ) in Cloud Shell.

I was going to add others if that wasn't sufficient. For me, it worked with just Microsoft.PolicyInsights (though it took the overall deploy-prod action a full 26m 44s, the vast majority "Creating or updating deployment 'blue'..."; I was surprised when it eventually completed).

az provider register --namespace Microsoft.PolicyInsights

After adding that provider, I:

1. Waited until it was "Registered" ("Checking provider status" below)
2. Deleted the endpoint (after the original error, it stays in a failed state) and waited for it to fully delete
3. Retried the GitHub action (/deploy-prod):

Checking provider status:

az provider show -n Microsoft.PolicyInsights --query "registrationState"