Knowledge Base - Technical Articles


Technical Article   Problem:  On Windows 2008 Server, Vista, or 7, lsass.exe grows in CPU usage and memory utilization under heavy load

Article ID: 37566
Bug Id: N/A
Software:  ArcGIS Server 9.3.1, 10
Platforms:  Windows Vista, 2008Server, Win 7

Description

When ArcGIS Server .NET Web applications and Internet services are under heavy load (more than 25 concurrent requests per second), the Local Security Authority Subsystem Service (lsass.exe) system process can become overtaxed. This can cause system performance degradation and, in extreme cases, such as a heavy load over periods of 12 hours or more, machine shutdown.

This issue may also be encountered when exposing a public ArcGIS server instance with secure services. When the Googlebot (crawl-66-249-71-66.googlebot.com) attempts to index the REST services directory, it can produce an excessive number of Web service requests. Stopping Google from indexing the site or applying the solution provided below resolves the problem.

Cause

Web applications and services that work with ArcGIS Server .NET must run as users in the AGSUSERS and/or AGSADMIN operating system group. By default, this is accomplished by configuring a Web service or application to impersonate with a specified identity, when handled by the aspnet worker process.

The components of ArcGIS Server that handle Internet service requests, such as http://myArcGISServer/arcgis/services and http://myArcGISServer/arcgis/rest are, themselves, Web services. By default, these components impersonate the ArcGIS Web services account.

Every time a Web service or application that uses impersonation handles a request, the underlying ASP.NET worker process must use the Local Security Authority Subsystem Service process (lsass.exe) to authenticate. Under normal load conditions, this authentication operation is insignificant.

When a Web service or application that is impersonating is under heavy load (more than 25 simultaneous connections per second) for extended periods of time, the per request authentication operations begin to severely affect the memory and processing footprint of the lsass.exe process.

Solution or Workaround

The burden on the lsass.exe process can be alleviated by altering the configuration of the aspnet worker process and the Web services or applications that are under heavy load.

The steps below outline how to configure the ArcGIS Web Services (SOAP and REST) to run in a separate IIS application pool with the identity of the ArcGIS Web services user, and how to disable per request impersonation.

The following instructions assume that the ArcGIS Web services account is called ArcGISWebServices (the default specified in the ArcGIS Server post installation utility). Modify this account name as appropriate for the system being used.

  1. Create a new IIS Application Pool and set its identity to the ArcGIS Web services account. -show me-

    a) Open Internet Information Services (IIS) Manager and navigate through the tree structure to the Application Pool folder.

    b) Right-click the Application Pool folder and click Add Application Pool.

    c) Give the application pool a name, such as ArcGIS Server Services Application Pool, and click OK to continue with the default settings.

    d) Right-click the new application pool and select Advanced Settings.

    e) Select the Identity parameter's value and click the ellipsis button (...).

    f) Select the Custom account option, and click Set.

    g) Enter the name and password of the ArcGIS Web Services account that was specified during the ArcGIS Server post installation process. Click OK.

    h) Click OK in the Application Pool Identity dialog box.

    i) Click OK in the Advanced Settings dialog box.


     With ArcGIS Server 10, this application pool already exists. Apply steps d-h to this application pool.

  2. Grant Modify permissions to the ArcGIS Web services account for the C:\Windows\Temp directory. -show me-

    a) Open Windows Explorer and navigate to the C:\Windows\Temp directory.

    b) Right-click C:\Windows\Temp and click Properties.

    c) In the Properties dialog box, click the Security tab.

    d) In the Security dialog box, click Add. The Select Users dialog box opens.

    e) In the Select Users dialog box, change 'From this location', if necessary, to the location of the ArcGIS Web services account, and type the account in the lower box (or browse to it with the Advanced button). Click OK to return to the Properties dialog box.

    f) In the Properties dialog box, make sure the ArcGIS Web services user is highlighted, and in the Allow column, check the Modify box.

    g) Click OK to save and close the Properties dialog box. Windows Explorer may also be closed.
  3. Configure the ArcGIS SOAP Web services to not use impersonation. -show me-

    a) Use Visual Studio or a text editor to open the web.config file in the C:\Inetpub\wwwroot\ArcGIS\Services folder (the ArcGIS Services folder may have been installed at a different location).

    b) In the web.config file, change the value for the Impersonate key to false:

    <appSettings>
    
    <add key="ServiceInfoRefreshTimeInSeconds" value="10" />
    <add key="GCInterval" value="10" />
    <add key="Impersonate" value="false" />
    </appSettings>


     If the Impersonate key does not exist, add it by inserting the <add> element and set the key attribute to ''Impersonate'' and the value attribute to ''false'' as shown above.

    c) Save the web.config file.
  4. Configure the ArcGIS REST Web services to not use impersonation. -show me-

    a) Use Visual Studio or a text editor to open the rest.config file in the C:\Inetput\wwwroot\ArcGIS\REST folder (the ArcGIS REST folder may have been installed at a different location).

    b) In the rest.config file, change the value for the Impersonate key to false:

    <?xml version="1.0" encoding="utf-8"?>
    
    <Config xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    ...
    ...
    <Impersonate>false</Impersonate>
    </Config>


     If the Impersonate key does not exist, add it by inserting the <Impersonate> element and set the value to 'false' as shown above.

    c) Save the rest.config file.
  5. Add the ArcGIS Web Services (SOAP and REST)to the newly created application pool. -show me-

    a) Open the Internet Information Services (IIS) Manager from Control Panel > Administrative Tools.

    b) Expand the local computer node, the Sites node, the Default Web Site node, and the ArcGIS node.

     ArcGIS is the default instance name. If a different instance name was selected during the post installation, use that name.

    c) Under the ArcGIS node, right-click Services and select Manage Application > Advanced Settings.

    d) In the Advanced Settings, select the 'Application pool' value and click the ellipsis button (...). Select the application pool created in step 1 of this article.

    e) Click OK in the Select Application Pool dialog box, and click OK in the Advanced Settings dialog box.

    f) Repeat steps c through e above for the REST services using 'REST' instead of 'Services' in step c.

    g) Close the IIS Manager.


     With ArcGIS Server 10, the REST and SOAP endpoints are already in this application pool. This step can be skipped.

  6. Restart the ArcGIS Server Object Manager (SOM) service. -show me-

    a) Open the Services console from Control Panel > Administrative Tools.

    b) Right-click the ArcGIS Server Object Manager service and click 'Restart'.

    c) Close the Services window.

Created: 2/8/2010
Last Modified: 3/2/2011

Article Rating: (1)
If you would like to post a comment, please login

Comments

By Kuvshinnikov - 02/14/2012 2:21 AM

The article is incorrect or the solution didn’t work.

Windows Server 2008 64bit, ArcGIS Server 10 SP3. Rest Services return error in restlog: Log Entry : 14.02.2012 12:06:05 ERROR:Couldn't get map service info http://server/ArcGIS/rest/services/service/MapServer :The type initializer for 'ESRI.ArcGIS.TokenService.Utility' threw an exception. And I cann't login in http://server/ArcGIS/rest/admin. If I add user "ArcGISWebservices" in local group "Administrators", then this solution work.

By Anonymous - 10/19/2010 2:21 PM

The article needs to be updated.

If this is being run on a 64bit server then you need to enable your new application pool to run 32bit applications This is missing from the information

Rating: