Problem: ArcGIS Server cannot start more service instances even though there is still plenty of RAM left on the machine
ArcGIS Server cannot start up more than 60 service instances on Windows XP or 120 service instances on Windows Server 2003.
ArcGIS Server service instances in high isolation consume limited Windows OS resources. For Windows 2003 Server, these resources are consumed when about 120 ArcSOC.exe processes have been launched. For Windows XP, this number is closer to 60. Windows 7 and Windows Server 2008 R2 do not have this problem.
Solution or Workaround
There are three solutions to this problem. The first involves configuring services to use the less performant, low isolation mode. The second solution is architectural and recommends scaling the ArcGIS Server system out rather than up. The third solution requires editing registry keys to increase the size of the interactive desktop heap.
Low Isolation Solution:
Consider configuring the ArcGIS Server services to run in low isolation. This allows multiple service instances to run in a single ArcSOC.exe process. Because this issue is related to the total number of ArcSOC.exe processes, forcing more service instances into each ArcSOC.exe process increases the number of service instances that can be accommodated by the ArcGIS Server before the process limit is reached.
The only disadvantage to this solution is that low isolation services are generally slightly less performant than high isolation services.
Performance tests run by Esri indicate that, generally, the maximum number of simultaneously processing instances that can be supported on a CPU core varies by service type, but rarely exceeds 4. This means that even if the ArcGIS Server SOC machine is very powerful, it can only really support 4 times the CPU cores' service instances. For example, a machine with 16 cores and 32GB of RAM can support 4*16 = 64 simultaneously executing instances. This does not mean only 64 users, because users are not normally using a service instance simultaneously and repeatedly with no rest between requests.
This means that there is little reason to have so many service instances configured for even a very powerful machine. Instead, the ArcGIS Server administrator should configure the services to run with Minimum Instances equal to zero, Capacity on the machine set to a number less than 60 or 120 on Windows XP and Windows Server 2003, respectively, and allow the internal pool-shrinking algorithms to optimize instance availability. With minimum instances set to 0 and Capacity properly set, the ArcGIS Server can have many (hundreds) services efficiently running simultaneously without running into the Windows process limit described in this article.
Follow the 'Cause 2' instructions in the Microsoft Support article KB184802 to reduce the size of the third Shared Section value (the limit is 128K). A link to the Microsoft Support KB article 184802 is in the Related Information below.
In the best case, this solution will only quadruple the number of processes to ~ 480. The reason this is risky is that it may reduce the functionality of other processes running on the server. If only ArcGIS Server SOC processes are running on the machine, this solution can be acceptable, but since these changes touch all processes on the machine, there may be unexpected consequences when a process tries to use a heap that is too small to hold the menus,windows,strings and hooks that it requires.