Problem: On Windows 2003 Server, the Local Security Authority Subsystem Service (lsass.exe) grows in CPU usage and memory utilization under heavy load
When ArcGIS Server .NET Web applications and Internet services are under heavy load (more than 25 concurrent requests per second), the Local Security Authority Subsystem Service (lsass.exe) system process can become overtaxed. This behavior can cause system performance degradation and even machine shutdown in extreme cases, such as a heavy load over periods of 12 hours or more.
This issue may also be encountered when exposing a public ArcGIS server instance with secure services. When the Google bot (crawl-66-249-71-66.googlebot.com) attempts to index the REST services directory, it can produce an excessive number of Web service requests. Stopping the Google bot from indexing the site or applying the solution provided below resolves the problem.
Web applications and services that work with ArcGIS Server .NET must run as users in the AGSUSERS and/or AGSADMIN operating system group. By default, this is accomplished by configuring a Web service or application to impersonate a specified identity when handled by the aspnet worker process.
The components of ArcGIS Server that handle Internet service requests, such as http://myArcGISServer/arcgis/services and http://myArcGISServer/arcgis/rest are themselves Web services. By default, these components impersonate the ArcGIS Web services account.
Every time a Web service or application that uses impersonation handles a request, the underlying ASP.NET worker process must use the Local Security Authority Subsystem Service process (lsass.exe) to authenticate. Under normal load conditions, the effect of this authentication operation is insignificant.
When a Web service or application that it is impersonating is under heavy load (more than 25 simultaneous connections per second) for extended periods of time, the per request authentication operations begin to severely affect the memory and processing footprint of the lsass.exe process.
Solution or Workaround
The burden on the lsass.exe process can be alleviated by altering the configuration of the aspnet worker process and the Web services or applications that are under heavy load.
The steps below outline how to configure the ArcGIS Web Services (SOAP and REST) to run in a separate IIS application pool with the identity of the ArcGIS Web services user, and how to disable per request impersonation.
The following instructions assume that the ArcGIS Web services account is called ArcGISWebServices (the default specified in the ArcGIS Server post installation utility). Modify this account name as appropriate for the system being used.
- Create a new IIS Application Pool and set its identity to the ArcGIS Web services account.
- Open Internet Information Services (IIS) Manager and navigate through the tree structure to the Application Pool folder.
- Right-click the Application Pool folder and click New > Application Pool.
- Give the application pool an ID, such as ArcGIS Server Services Application Pool, and select the option to continue with the default settings.
- Right-click the new application pool and click Properties.
- Click the Identity tab and select Configurable.
- Enter the name and password of the ArcGIS Web services account that was specified during the ArcGIS Server post installation process. Click OK.
- Re-enter the password to confirm and click OK.
Note: With ArcGIS Server 10, this application pool already exists. Apply steps d through g to this application pool.
- Add the ArcGIS Web services account to the IIS_WPG local operating system group.
- From the Windows Control Panel > Administrative Tools, open the Computer Management console.
- In the navigation pane, under the System Tools group, expand the Local Users and Groups node. Click Groups.
- In the details pane, right-click the IIS_WPG group and select Properties.
- In the IIS_WPG Properties dialog box, click the Add button.
- In the Select Users, Computers, or Groups dialog box, change the entry under 'From this location', if necessary, to the location that contains the user account for the ArcGIS Web services (ArcGISWebServices). If the account is on the local computer, the location should be the local machine name.
- Type the account name into the box under ‘Enter the object names to select’, click Check Names, and select the user.
- Click OK in the two dialog boxes to save the settings. Close the Computer Management window.
- Grant the ArcGIS Web services account permissions to the IIS metabase.
- Open a .NET command prompt with Start > (All) Programs > Microsoft .NET Framework SDK v2.0 > SDK Command Prompt.
- Type the following command at the prompt, substituting the ArcGIS Web services account name as appropriate:
aspnet_regiis -ga ArcGISWebServices
- Close the .NET command prompt by typing 'exit' and pressing the Enter key.
- Open a command prompt window by clicking Start > Run, typing 'cmd' in the Run dialog box, and pressing Enter.
- Type the following commands at the command prompt, substituting the ArcGIS Web services account name, as appropriate:
aspnet_regiis -ga ArcGISWebServices
- Close the command prompt window by typing 'exit' and pressing the Enter key.
- Grant Modify permissions to the ArcGIS Web services account for the C:\Windows\Temp directory.
- Open Windows Explorer and navigate to the C:\Windows\Temp directory.
- Right-click C:\Windows\Temp and click Properties.
- In the Properties dialog box, click the Security tab.
- In the Security dialog box, click Add. The Select Users dialog box opens.
- In the Select Users dialog box, change 'From this location', if necessary, to the location of the ArcGIS Web services account, and type the account in the lower box (or browse to it with the Advanced button). Click OK to return to the Properties dialog box.
- In the Properties dialog box, make sure the ArcGIS Web services user is highlighted, and in the Allow column, check the Modify box.
- Click OK to save and close the Properties dialog box. Windows Explorer may also be closed.
- Configure the ArcGIS SOAP Web services to not use impersonation.
- Use Visual Studio or a text editor to open the web.config file in the C:\Inetpub\wwwroot\ArcGIS\Services folder (the ArcGIS Services folder may have been installed at a different location).
- In the web.config file, change the value for the Impersonate key to false:
<appSettings> <add key="ServiceInfoRefreshTimeInSeconds" value="10" /> <add key="GCInterval" value="10" /> <add key="Impersonate" value="false" /> </appSettings>
Note: If the Impersonate key does not exist, add it by inserting the <add> element and set the key attribute to 'Impersonate' and the value attribute to 'false' as shown above.
- Save the web.config file.
- Configure the ArcGIS REST Web services to not use impersonation.
- Use Visual Studio or a text editor to open the rest.config file in the C:\Inetput\wwwroot\ArcGIS\REST folder (the ArcGIS REST folder may have been installed at a different location).
- In the rest.config file, change the value for the Impersonate key to false:
<?xml version="1.0" encoding="utf-8"?> <Config xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance xmlns:xsd="http://www.w3.org/2001/XMLSchema"> ... ... <Impersonate>false</Impersonate> </Config>
Note: If the Impersonate key does not exist, add it by inserting the <Impersonate> element and set the value to 'false' as shown above.
- Save the rest.config file.
- Add the ArcGIS Web Services (SOAP and REST) to the newly created application pool.
- Open the Internet Information Services (IIS) Manager from Control Panel > Administrative Tools.
- Expand the local computer node, the Web sites node, the Default Web site node, and the ArcGIS node.
- In the ArcGIS node, right-click Services and click Properties. The Properties dialog box opens for Services.
- On the Virtual Directory tab, select the 'Application pool' drop-down list and select the application pool created in step 1 of this article.
- Click OK to save and close the Properties dialog box.
- Repeat steps c through e for the REST services using 'REST' instead of 'Services' in step c.
- Close the IIS Manager.
Note: With ArcGIS Server 10, the REST and SOAP endpoints are already in this application pool. This step can be skipped.
- Restart the ArcGIS Server Object Manager (SOM) service.
- Open the Services console from Control Panel > Administrative Tools.
- Right-click the ArcGIS Server Object Manager service and click 'Restart'.
- Close the Services window.