The ArcGIS Server logs cannot be queried

Description

When attempting to query logs in the ArcGIS Server Manager, the display says "Refreshing log message display..." but never opens or continues to load, as shown in the image below. The same issue occurs when accessing the Server Administrator interface.

Cause

To determine a possible cause, look through all the logs on the local drive to see if any records are listed there, or if the logs are even being generated or not. Typically, the logs are found where the ArcGIS Server is installed:

<install location>\arcgisserver\logs

To narrow down the causes of the problem, it is recommended to change the log settings such as the source, time, level, and machine (if it is a multi-machine site), and so on, see if there is a specific pattern in which we can query from a specific machine, source, or time. 

Some questions to consider:

• The operating system the clients are using
• Whether or not it is a federated/hosted server
• If it is a single or a multi-machine site?
• Any recent changes that occurred prior to the onset of the problem. Windows Update, new antivirus, or changes on the firewall.
• When this problem first appeared. Has it been occurring consistently, or does it happen on and off?
• Does it happen using all browsers?
• Does it happen using the port number, web adaptor with FQDN, and the DNS URLs if there are any?
• Does it occur on the machine that hosts the ArcGIS Server?

There are numerous causes for this problem, a few of which are listed below:

• The ArcGIS Server Windows account running the service does not have full access to the ArcGIS Server logs directory.
• The disk space on the drive where the ArcGIS Server content (including logs) is stored is insufficient.
• The CPU usage/memory% consumption is extremely high, resulting in poor performance.
• Antivirus/firewall/network issues
• The ArcGIS Data Store Windows account that runs the service, lacks full access to the ArcGIS Data Store logs directory.
• New patches and/or Windows Updates have been installed
• If the ArcGIS Server site has more than one machine, a miscommunication between the machines might cause this issue
• The arcgis-logsettings.json file is either missing or corrupt.

Solution or Workaround

A single machine site

Here are some potential solutions for the aforementioned causes:

Note:
It is recommended to have a snapshot before attempting any workflows as a precaution)

• The path where the server is reading the logs is incorrect or inaccessible
  • This can be checked from the logs’ settings

• The ArcGIS Server Windows account that runs the service lacks full access to the ArcGIS Server logs directory
  • Examine the permissions on all ArcGIS Server directories for the account that runs the service and ensure that these permissions are passed to the logs folder inside.
  • Run the configure ArcGIS Server account utility
  • Remove all permissions and read them on all directories
  • Change the account that runs the service to another account to ensure that the problem is not an account issue (especially if it is a local account, change it to a domain account). See: Change the ArcGIS Server account

• There is insufficient disk space on the drive where the ArcGIS Server content (including logs) is stored
  • Increase the desk space for at least 20 GB on the drive that has the logs
  • Delete some of the old logs if the size of the logs folder is large (in GB)

• The CPU usage/memory% is extremely high, causing a performance issue
  • Check if the system requirements of the ArcGIS Server are met
  • Check the number of ArcSOC processes  running on the ArcGIS Server machine (It shouldn't return more than 200, preferably 180, or else some of the services must be tuned down)
    • For Windows:

      • tasklist | find "ArcSOC.exe" /c 

    • For Linux:

      • ps -C arcsoc --no-headers | wc -l

  • Determine the source of the CPU/Memory consumption
  • Increase the resources of the machine

Note:
Check if the memory consumption increases as soon as you hit the query.

• An issue with the ArcGIS Data Store
  • This can be confirmed when we can query the logs from all sources except the data store source
  • The Windows account that runs the ArcGIS Server and ArcGIS Data Store services lack full access to the ArcGIS Data Store logs directory. Check permissions over all the directories of the ArcGIS Data Store for the account/s that runs the services (server and data store) and check that these permissions are passed to the logs folder inside.
  • Unregister the data store, check if you can query the logs after. If yes, then the issue is with the data store and you should troubleshoot it to be able to re-register it again.
  • The primary data store and the standby data store are out of sync,  follow the steps listed here Error: ArcGIS Data Store has detected a replication failure

• Antivirus/firewall/Network issues
  • Temporarily disable the antivirus on that machine completely. Then restart the ArcGIS Server, clear the browser cache, and see if the behavior persists. if it did not, exclude the ArcGIS Server logs from the antivirus
  • Check the ports by running this command
    • For Windows:

      • netstat -ao | findstr "LISTENING" | findstr "4000 4001 4002 4003 6080 6443 6006 1098 6099"

    • For Linux:

      • sudo netstat -tulpn | grep LISTEN

  • Check what is used in HTTP and HTTPS Protocol and TLSv1.2, TLSv1.1 SSL Protocols for the server admin interface security > config, based on each version Secure ArcGIS Server communication
  • Check the F12 console and Network tab, and Fiddler for any errors that could help identify any network errors 

• The arcgis-logsettings.json file is either missing or corrupt.
  • This can be found in this path C:\Program Files\ArcGIS\Server\framework\etc\arcgis-logsettings.json
  • replace it with one from a healthy environment

• The subkey has a false configuration: this must be done with a high degree of caution and the appropriate snapshot\backup must be taken before editing this 

Warning:   
The instructions below include making changes to essential parts of your operating system. It is recommended that you backup your 
operating system and files, including the registry, before proceeding. Consult a qualified computer systems professional, if necessary.

Esri cannot guarantee results from incorrect modifications while following these instructions. Therefore, use caution and proceed 
at your own risk.

1. Open the Registry Editor. Press Windows + R and type regedit.
2. Go to the Application eventlog key located in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\EventLog\Application\ArcGIS Server
3. Locate the project’s subkey and check if this subkey has the following configuration:

Name: EventMessageFile
Type: REG_EXPAND_SZ
Data: Drive:\Program Files\ArcGIS\Server\framework\etc\service\bin\ArcGISServer.exe

If the ArcGIS Server site has more than one machine

Check if it is one of the machine’s issues by changing from all machines to one machine at a time when querying the logs and see if you are able to query from one but not the other.

Additionally, test the following:

• A miscommunication between the machines might cause this issue
  • Ping both machines to each other
• Since the logs folder is saved locally on each machine, check the path and permission over all of the directories

Further steps to try

• Restart the server and the data store service
• Reboot the machine
• Repair ArcGIS Server
• Repair the data store if you have confirmed that the issue is with it alone
• Re-create the ArcGIS Server site with a new installation