On Windows 2008 Server, Vista, or 7, lsass.exe grows in CPU usage and memory utilization under heavy load

Description

When ArcGIS Server .NET Web applications and Internet services are under heavy load (more than 25 concurrent requests per second), the Local Security Authority Subsystem Service (lsass.exe) system process can become overtaxed. This can cause system performance degradation and, in extreme cases, such as a heavy load over periods of 12 hours or more, machine shutdown.

This issue may also be encountered when exposing a public ArcGIS server instance with secure services. When the Googlebot (crawl-66-249-71-66.googlebot.com) attempts to index the REST services directory, it can produce an excessive number of Web service requests. Stopping Google from indexing the site or applying the solution provided below resolves the problem.

Cause

Web applications and services that work with ArcGIS Server .NET must run as users in the AGSUSERS and/or AGSADMIN operating system group. By default, this is accomplished by configuring a Web service or application to impersonate with a specified identity, when handled by the aspnet worker process.

The components of ArcGIS Server that handle Internet service requests, such as http://myArcGISServer/arcgis/services and http://myArcGISServer/arcgis/rest are, themselves, Web services. By default, these components impersonate the ArcGIS Web services account.

Every time a Web service or application that uses impersonation handles a request, the underlying ASP.NET worker process must use the Local Security Authority Subsystem Service process (lsass.exe) to authenticate. Under normal load conditions, this authentication operation is insignificant.

When a Web service or application that is impersonating is under heavy load (more than 25 simultaneous connections per second) for extended periods of time, the per request authentication operations begin to severely affect the memory and processing footprint of the lsass.exe process.

Solution or Workaround

1. Create a new IIS Application Pool and set its identity to the ArcGIS Web services account.
   1. Open Internet Information Services (IIS) Manager and navigate through the tree structure to the Application Pool folder.
   2. Right-click the Application Pool folder and click Add Application Pool.
   3. Give the application pool a name, such as ArcGIS Server Services Application Pool, and click OK to continue with the default settings.
   4. Right-click the new application pool and select Advanced Settings.
   5. Select the Identity parameter's value and click the ellipsis button (...).
   6. Select the Custom account option, and click Set.
   7. Enter the name and password of the ArcGIS Web Services account that was specified during the ArcGIS Server post installation process. Click OK.
   8. Click OK in the Application Pool Identity dialog box.
   9. Click OK in the Advanced Settings dialog box.

   Note:
   With ArcGIS Server 10, this application pool already exists. Apply steps d-h to this application pool. 

2. Grant Modify permissions to the ArcGIS Web services account for the C:\Windows\Temp directory.
   1. Open Windows Explorer and navigate to the C:\Windows\Temp directory.
   2. Right-click C:\Windows\Temp and click Properties.
   3. In the Properties dialog box, click the Security tab.
   4. In the Security dialog box, click Add. The Select Users dialog box opens.
   5. In the Select Users dialog box, change 'From this location', if necessary, to the location of the ArcGIS Web services account, and type the account in the lower box (or browse to it with the Advanced button). Click OK to return to the Properties dialog box.
   6. In the Properties dialog box, make sure the ArcGIS Web services user is highlighted, and in the Allow column, check the Modify box.
   7. Click OK to save and close the Properties dialog box. Windows Explorer may also be closed.
3. Configure the ArcGIS SOAP Web services to not use impersonation.
   1. Use Visual Studio or a text editor to open the web.config file in the C:\Inetpub\wwwroot\ArcGIS\Services folder (the ArcGIS Services folder may have been installed at a different location).
   2. In the web.config file, change the value for the Impersonate key to false:

      <appSettings>
          <add key="ServiceInfoRefreshTimeInSeconds" value="10" />
          <add key="GCInterval" value="10" />
          <add key="Impersonate" value="false" />
      </appSettings>
      

      Note:
      If the Impersonate key does not exist, add it by inserting the  element and set the key attribute to ''Impersonate'' and the value attribute to ''false'' as shown above.

   3. Save the web.config file.
4. Configure the ArcGIS REST Web services to not use impersonation.
   1. Use Visual Studio or a text editor to open the rest.config file in the C:\Inetput\wwwroot\ArcGIS\REST folder (the ArcGIS REST folder may have been installed at a different location).
   2. In the rest.config file, change the value for the Impersonate key to false:

      <?xml version="1.0" encoding="utf-8"?>
      <Config xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance xmlns:xsd="http://www.w3.org/2001/XMLSchema">
      ...
      ...
      <Impersonate>false</Impersonate>
      </Config> 
      

      Note:
      If the Impersonate key does not exist, add it by inserting the  element and set the value to 'false' as shown above.

   3. Save the rest.config file.
5. Add the ArcGIS Web Services (SOAP and REST)to the newly created application pool.
   1. Open the Internet Information Services (IIS) Manager from Control Panel > Administrative Tools.
   2. Expand the local computer node, the Sites node, the Default Web Site node, and the ArcGIS node.

      Note:
      ArcGIS is the default instance name. If a different instance name was selected during the post installation, use that name.

   3. Under the ArcGIS node, right-click Services and select Manage Application > Advanced Settings.
   4. In the Advanced Settings, select the 'Application pool' value and click the ellipsis button (...). Select the application pool created in step 1 of this article.
   5. Click OK in the Select Application Pool dialog box, and click OK in the Advanced Settings dialog box.
   6. Repeat steps c through e above for the REST services using 'REST' instead of 'Services' in step c.
   7. Close the IIS Manager.

   Note:
   With ArcGIS Server 10, the REST and SOAP endpoints are already in this application pool. This step can be skipped. 

6. Restart the ArcGIS Server Object Manager (SOM) service.
   1. Open the Services console from Control Panel > Administrative Tools.
   2. Right-click the ArcGIS Server Object Manager service and click 'Restart'.
   3. Close the Services window.