Querying a map service fails with a server log error

Description

Querying a map service fails with the following error in the server log:

Error: 
Wait timeout exceeded.

Cause

The issue is caused by insufficient machine process core and instances to run a query on large map services. By default, ArcGIS Server generates more ArcSOC.exe processes in the operating system to handle queries on large map services. For example, publishing 50 dynamic map services can generate over 100 ArcSOC.exe processes running on the machine. However, ArcGIS Server can be set to only generate a maximum of two instances per map service, which limits the server to process only up to 100. The lack of instances causes the error to occur and the query process to fail.

Solution or Workaround

The general rule of thumb for the Maximum Number of Instances parameter is n + 1, where n is the number of cores on the Server machine. If the maximum number of instances is set to more than the suggested parameter, users may encounter performance issues as ArcGIS Server consumes extra resources. This may lead to a service crash. Upgrading the machine's core processor according to desired capabilities is recommended to allow a higher number of instances. The maximum number of instances can be modified through ArcGIS Server Manager using either an administrator or publisher account.

To determine the optimal number of instances required in a service configuration, monitor the server over time. For example, how the applications use the instances. The number of available instances must be adjusted if requests are timing out or wait times for clients are too long. A server is ready to handle high volumes of requests even though it has just been turned on. It may launch over 90 threads with each thread capable of handling simultaneous requests. Each thread that loads to memory has full knowledge of ArcGIS Server security, properties, settings, and services, among others. The number of threads and memory increase when a system is put under load, to ensure the system operates at high performance even when it receives more load. Customization of the maximum number of instances and timeout must be considered thoroughly, to ensure it does not affect the machine and work performance. The server is already tuned for optimal performance, therefore, change must only take place when necessary.

The knowledge base article, Problem: The number of ArcSOC instances causes ArcGIS for Server performance issues provides instructions on how to modify the maximum number of instances per machine.