Knowledge Base - Technical Articles


Technical Article   Problem:  On Windows 2003 Server, the Local Security Authority Subsystem Service (lsass.exe) grows in CPU usage and memory utilization under heavy load

Article ID: 32620
Bug Id: N/A
Software:  ArcGIS Server 9.2, 9.3, 9.3.1, 10
Platforms:  Windows 2003Server

Description

When ArcGIS Server .NET Web applications and Internet services are under heavy load (more than 25 concurrent requests per second), the Local Security Authority Subsystem Service (lsass.exe) system process can become overtaxed. This behavior can cause system performance degradation and even machine shutdown in extreme cases, such as a heavy load over periods of 12 hours or more.

This issue may also be encountered when exposing a public ArcGIS server instance with secure services. When the Google bot (crawl-66-249-71-66.googlebot.com) attempts to index the REST services directory, it can produce an excessive number of Web service requests. Stopping the Google bot from indexing the site or applying the solution provided below resolves the problem.

Cause

Web applications and services that work with ArcGIS Server .NET must run as users in the AGSUSERS and/or AGSADMIN operating system group. By default, this is accomplished by configuring a Web service or application to impersonate a specified identity when handled by the aspnet worker process.

The components of ArcGIS Server that handle Internet service requests, such as http://myArcGISServer/arcgis/services and http://myArcGISServer/arcgis/rest are themselves Web services. By default, these components impersonate the ArcGIS Web services account.

Every time a Web service or application that uses impersonation handles a request, the underlying ASP.NET worker process must use the Local Security Authority Subsystem Service process (lsass.exe) to authenticate. Under normal load conditions, the effect of this authentication operation is insignificant.

When a Web service or application that it is impersonating is under heavy load (more than 25 simultaneous connections per second) for extended periods of time, the per request authentication operations begin to severely affect the memory and processing footprint of the lsass.exe process.

Solution or Workaround

The burden on the lsass.exe process can be alleviated by altering the configuration of the aspnet worker process and the Web services or applications that are under heavy load.

The steps below outline how to configure the ArcGIS Web Services (SOAP and REST) to run in a separate IIS application pool with the identity of the ArcGIS Web services user, and how to disable per request impersonation.

The following instructions assume that the ArcGIS Web services account is called ArcGISWebServices (the default specified in the ArcGIS Server post installation utility). Modify this account name as appropriate for the system being used.

  1. Create a new IIS Application Pool and set its identity to the ArcGIS Web services account. -show me-

    a) Open Internet Information Services (IIS) Manager and navigate through the tree structure to the Application Pool folder.

    b) Right-click the Application Pool folder and click New > Application Pool.

    c) Give the application pool an ID, such as ArcGIS Server Services Application Pool, and select the option to continue with the default settings.

    d) Right-click the new application pool and click Properties.

    e) Click the Identity tab and select Configurable.

    f) Enter the name and password of the ArcGIS Web services account that was specified during the ArcGIS Server post installation process. Click OK.

    g) Re-enter the password to confirm and click OK.

     With ArcGIS Server 10, this application pool already exists. Apply steps d through g to this application pool.

  2. Add the ArcGIS Web services account to the IIS_WPG local operating system group. -show me-

    a) From the Windows Control Panel > Administrative Tools, open the Computer Management console.

    b) In the navigation pane, under the System Tools group, expand the Local Users and Groups node. Click Groups.

    c) In the details pane, right-click the IIS_WPG group and select Properties.

    d) In the IIS_WPG Properties dialog box, click the Add button.

    e) In the Select Users, Computers, or Groups dialog box, change the entry under 'From this location', if necessary, to the location that contains the user account for the ArcGIS Web services (ArcGISWebServices). If the account is on the local computer, the location should be the local machine name.

    f) Type the account name into the box under ‘Enter the object names to select’, click Check Names, and select the user.

    Alternatively, click the ‘Advanced’ button, type account name or description and click ‘Find Now’. Select the user from search results, and click OK to return to the Select Users dialog box.

    g) Click OK in the two dialog boxes to save the settings. Close the Computer Management window.
  3. Grant the ArcGIS Web services account permissions to the IIS metabase.

    If the Microsoft .NET Framework SDK is installed on the machine follow these instructions: -show me-

    a) Open a .NET command prompt with Start > (All) Programs > Microsoft .NET Framework SDK v2.0 > SDK Command Prompt.

    b) Type the following command at the prompt, substituting the ArcGIS Web services account name as appropriate:

    aspnet_regiis -ga ArcGISWebServices

    c) Close the .NET command prompt by typing 'exit' and pressing the Enter key.

    If the Microsoft .NET Framework SDK is not installed, these instructions apply: -show me-

    a) Open a command prompt window by clicking Start > Run, typing 'cmd' in the Run dialog box, and pressing Enter.

    b) Type the following commands at the command prompt, substituting the ArcGIS Web services account name, as appropriate:

    cd C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727


    aspnet_regiis -ga ArcGISWebServices

    c) Close the command prompt window by typing 'exit' and pressing the Enter key.

  4. Grant Modify permissions to the ArcGIS Web services account for the C:\Windows\Temp directory. -show me-

    a) Open Windows Explorer and navigate to the C:\Windows\Temp directory.

    b) Right-click C:\Windows\Temp and click Properties.

    c) In the Properties dialog box, click the Security tab.

    d) In the Security dialog box, click Add. The Select Users dialog box opens.

    e) In the Select Users dialog box, change 'From this location', if necessary, to the location of the ArcGIS Web services account, and type the account in the lower box (or browse to it with the Advanced button). Click OK to return to the Properties dialog box.

    f) In the Properties dialog box, make sure the ArcGIS Web services user is highlighted, and in the Allow column, check the Modify box.

    g) Click OK to save and close the Properties dialog box. Windows Explorer may also be closed.
  5. Configure the ArcGIS SOAP Web services to not use impersonation. -show me-

    a) Use Visual Studio or a text editor to open the web.config file in the C:\Inetpub\wwwroot\ArcGIS\Services folder (the ArcGIS Services folder may have been installed at a different location).

    b) In the web.config file, change the value for the Impersonate key to false:

    <appSettings>
    
    <add key="ServiceInfoRefreshTimeInSeconds" value="10" />
    <add key="GCInterval" value="10" />
    <add key="Impersonate" value="false" />
    </appSettings>


     If the Impersonate key does not exist, add it by inserting the <add> element and set the key attribute to ''Impersonate'' and the value attribute to ''false'' as shown above.

    c) Save the web.config file.
  6. Configure the ArcGIS REST Web services to not use impersonation. -show me-

    a) Use Visual Studio or a text editor to open the rest.config file in the C:\Inetput\wwwroot\ArcGIS\REST folder (the ArcGIS REST folder may have been installed at a different location).

    b) In the rest.config file, change the value for the Impersonate key to false:

    <?xml version="1.0" encoding="utf-8"?>
    
    <Config xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    ...
    ...
    <Impersonate>false</Impersonate>
    </Config>


     If the Impersonate key does not exist, add it by inserting the <Impersonate> element and set the value to 'false' as shown above.

    c) Save the rest.config file.
  7. Add the ArcGIS Web Services (SOAP and REST) to the newly created application pool. -show me-

    a) Open the Internet Information Services (IIS) Manager from Control Panel > Administrative Tools.

    b) Expand the local computer node, the Web sites node, the Default Web site node, and the ArcGIS node.

    c) In the ArcGIS node, right-click Services and click Properties. The Properties dialog box opens for Services.

    d) On the Virtual Directory tab, select the 'Application pool' drop-down list and select the application pool created in step 1 of this article.

    e) Click OK to save and close the Properties dialog box.

    f) Repeat steps c through e for the REST services using 'REST' instead of 'Services' in step c.

    g) Close the IIS Manager.

     With ArcGIS Server 10, the REST and SOAP endpoints are already in this application pool. This step can be skipped.

  8. Restart the ArcGIS Server Object Manager (SOM) service. -show me-

    a) Open the Services console from Control Panel > Administrative Tools.

    b) Right-click the ArcGIS Server Object Manager service and click 'Restart'.

    c) Close the Services window.

Created: 2/5/2007
Last Modified: 2/25/2011

Article Rating: (6)
If you would like to post a comment, please login

Comments

By wpgis5673 - 06/19/2012 9:54 AM

The article is incorrect or the solution didn’t work.

Removing impersonation causes IE to repeatedly request credentials if one uses AD for AGS security. There must be more to this solution that has not been explained.

By dur3511 - 07/04/2011 9:25 AM

The article needs to be updated.

Looks working with 10 no issues on cached services (they all appear fine after the procedure applied), but requires IIS restart as stated by comment dated 01/26/2011

Rating:

By Anonymous - 01/26/2011 9:12 AM

I would like to see a new article that discusses the topic outlined below.

Making changes to IIS often requires the IIS service to be restarted. I think an IIS restart should be included in these instructions.

Rating:

By Anonymous - 08/26/2010 11:17 AM

I have suggested related resources/links that can enhance this article. See below.

Yes, but not in ArcGIS 10. See: http://resources.arcgis.com/content/kbase?fa=articleShow&d=37566

Rating:

By Anonymous - 08/26/2010 5:28 AM

I have suggested related resources/links that can enhance this article. See below.

Is this also an issue with Windows 2008 Server?

By Anonymous - 01/08/2010 6:03 PM

I have suggested related resources/links that can enhance this article. See below.

Applying this article to token secured ArcGIS Server .NET will break token security. See: NIM052716 Please update the article to include the workaround for NIM052716: Assuming the identity user is MACHINENAME\ArcGISWeb and the database where the user accounts are stored in named "aspnetdb" 1. The MACHINENAME\ArcGISWeb user needs to be added to the logins in the SQLServer with "aspnetdb" set as the default database. 2. And in the user mapping for this user, allow aspnet_membership_FullAccess and aspnet_Roles_FullAccess.

By Anonymous - 09/10/2008 10:58 AM

The article contains a typographical or grammatical error.

Step 6 (c) says to save the web.config file. In step 6 (a) and (b) you are modifying the rest.config file. Step 6 (c) should say to save the rest.config file not web.config file.

Rating:

By Anonymous - 02/13/2008 8:27 AM

I followed the article’s instructions, but experienced another problem. I’ve provided details below.

Good morning. I have followed the article’s instructions of http://support.esri.com/index.cfm?fa=knowledgebase.techarticles.articleShow&d=32620 but I´ve a new problem. In this case we have a ArcGIS Explorer application an we need to connect with the ArcGIS Server with the article’s configuration above, it’s impossible, we obtain an connection error. Then we re-assigned the ArcGIS web Services application to the default application pool of IIS (the connection is possible. Could you help us? Thanks

Rating:

By Anonymous - 01/17/2008 8:58 PM

Other - See details below.

I followed these instructions to the letter. Now all my cached services will not display anything - as if they now have no idea where the arcgiscache directory is. I have many GB of caches that took literally hundreds of hours to create, so I'm more than a little upset that my services now seem disconnected from them because I followed your KB article. I'd appreciate a remedy ASAP.

Rating: