Unable to upgrade AKS cluster above 1.25 when using ArcGIS Enterprise on Kubernetes

Description

This article describes a scenario where a customer is unable to upgrade past AKS version 1.25 when using ArcGIS Enterprise on Kubernetes due to internal validation failures.

Note:
ArcGIS Enterprise 11.1 on Kubernetes does not support Kubernetes versions above 1.25.x at the time of release and will require an upcoming patch to be installed to support versions 1.26.x and 1.27.x. If the cluster is upgraded before the organization is patched, HPA and backup/restore functionality will be adversely impacted.

Cause

The Azure Kubernetes service performs validation to determine whether deprecated APIs are in use prior to upgrading to a new control plane version. When customers attempt to upgrade the cluster to 1.26.x or higher, the following error is observed in the Azure logs:

Failed to save Kubernetes service '<clusterName>'.
Error: Control Plane upgrade is blocked due to recent usage of a Kubernetes API deprecated in the specified version.
Please refer to https://kubernetes.io/docs/reference/using-api/deprecation-guide to migrate the usage.
To bypass this error, set forceUpgrade in upgradeSettings.overrideSettings.
Bypassing this error without migrating usage will result in the deprecated Kubernetes API calls failing.
See details in https://aka.ms/aks/UpgradeAndDeprecatedAPIs.
Usage details: 2 errors occurred:
 	* usage has been detected on API autoscaling.horizontalpodautoscalers.v2beta2.list, and was recently seen at: <dateTimeStamp>, which will be removed in 1.26 	
 	* usage has been detected on API autoscaling.horizontalpodautoscalers.v2beta2.watch, and was recently seen at: <dateTimeStamp>, which will be removed in 1.26

These errors are produced due to the Kubernetes events monitoring framework that is incorporated within ArcGIS Enterprise on Kubernetes, but can also be due to other applications deployed within the cluster. Once an administrator is confident that all non-Esri applications have been migrated to use the newer autoscaling.horizontalpodautoscalers.v2 APIs, they can move forward with the steps below to bypass the validation.

Solution or Workaround

Steps to disable the validation engine so the control plane upgrade can proceed are as follows:

1. Use the Azure CLI (with the aks-preview extension installed) to enable the force-upgrade property on the cluster, then proceed with the upgrade via the Azure portal:
   1. Make sure both the Azure CLI (instructions) and aks-preview extension (instructions) are installed.
   2. Run the following command (update the datetime to a period in the future when the upgrade can be completed, see the Azure documentation for more information):

az aks update --name <clusterName> --resource-group <resourceGroup> --enable-force-upgrade --upgrade-override-until 2023-11-10T13:00:00Z