Stopping transport because the required layer [DatastoreConnection][Layer Name] [LayerID] [Feature Server] is not accessible after sync'ing has completed

Error Message

The following error is returned by an output connector in GeoEvent Manager (https://server.domain.com:6143/geoevent/manager/index.html), if the feature service is stopped on the server side (by ArcGIS Server):

Error: Stopping transport because the required layer [DatastoreConnection][Layer Name] [LayerID] [Feature Server] is not accessible after sync'ing has completed.

GeoEvent’s output lost the URL it requires to make requests on a feature service’s REST endpoint, the output should remain in its ‘Started’ state and not transition to an ‘Error’ state.

This behavior is expected and seems to be a limitation of any service‑oriented architecture (SOA) if an external service unexpectedly goes offline.
Depending on the type of inbound or outbound connector, it is possible that the HTTP client used by the connector may receive an HTTP/500 “general server/service” error when making a request on an external web service. An input in this case stops querying for or receiving data. An output connector stops making requests on the external service to disseminate data from processed event records.

The key here is to recognize that an Enterprise portal or ArcGIS Server hosted feature service does not benefit from special treatment or handling because it is inside the Esri ecosystem. GeoEvent Server treats any web service being polled or to which it sends requests as an external web service.

For the period of time the outbound connector remains disconnected, in an error state, any real-time data received will not be disseminated through the outbound connector. Event records received will continue to be “lost” as they are routed to the outbound connector which is no longer connected and running.

The solution for the error includes some basic troubleshooting steps along with best practices to prevent connectors from entering an error state and avoiding data loss.

Cause

The Update a Feature output connector transitions into an error state when the REST endpoint of the output feature service is unavailable. The feature service could be unavailable due to : 

• A planned outage such as server patching
• Safekeeping and maintenance of the referenced data source (Enterprise Geodatabase)
• Unplanned outages such as service corruption or when the ArcGIS Server site is offline
• If the output feature service is somehow stopped (The error can be replicated by simply stopping the feature service from the respective Server Manager/Admin.) 
• The feature service is being recycled
• Loss in communication between the GeoEvent server and ArcGIS Server hosting the feature service

This would be expected of a service which is suddenly unavailable, because the service had been stopped (or the ArcGIS Server taken offline) and would produce an HTTP/500 error if a running GeoEvent connector (input or output) made a request on the unavailable service.

If a GeoEvent Server connector (inbound or outbound) were to make a request on a feature service which was currently being recycled, an HTTP/500 server response would also be expected. The service is not available.

Solution or Workaround

To troubleshoot the error message, confirm the point of failure based on the above potential causes. 

• Is the ArcGIS Server as a whole functional? 
• Check if the REST endpoint of the service is available. 
• Confirm the status of the service, either Started or Stopped from Server Manager or Admin.
• Are there related ArcSOCs for the feature service?
• Validate the referenced data source.
• Access the REST endpoint from the GeoEvent machine. Confirm if it is accessible. 
• Examine ArcGIS Server logs to troubleshoot further. 

The output connector can be recovered or brought back online by: 

• Restarting the specific output connector from the GeoEvent Manager endpoint.
• Stopping and starting the output connector. 
• Force syncing the GeoEvent Datastore connection from GeoEvent Manager > Site > Data Stores. Sync the connection to which the service is published. 
• Waiting for the next service discovery to be triggered. 

Note:
In most cases waiting for the next service discovery should be avoided. If the service discovery coincided with a service being temporarily unavailable and GeoEvent were unable to restore its input(s) and/or output(s), it would be necessary to wait until the next service discovery (several hours later), at which time the feature service(s) are available.

It is easy to get impatient in this case, and click to request a new service discovery after several minutes, when it may take 20, 30, 45 or more minutes to complete a service discovery for a couple of hundred map/feature services being hosted by a hosting server whose infrastructure may have multiple ArcGIS Server instances.

For more information on tracking a service discovery, follow the Geonet blog on Monitoring a registered web server's service discovery.

Recommended best practices

• System administrators who are aware that servers are going to be rebooted or who know that specific web services are going to be temporarily unavailable share a responsibility to suspend or stop real-time processing, which leverages those web services. Otherwise, they may be reasonably expected to have to stop and restart individual connectors – or in the worst case, restart their GeoEvent Server to re-establish service connectivity. Stop the input, output connectors and GeoEvent service during patching/maintenance window. 
• Increase the Discovery Rate on their registered server connection(s) from the default – which is 60 minutes – to something more like 1440 minutes. It can also be set it much greater value.
• Limit the number of features services that are discoverable each time GeoEvent Server runs its service discovery.
• Also consider not federating the ArcGIS Server beneath which GeoEvent Server is running, and configuring that ArcGIS Server with its own managed relational geodatabase, so that feature services whose feature record sets are truly being queried and/or updated in real-time are not comingled with the Enterprise portal’s hosted feature layers. Unless there is a spatiotemporal big data store in the system architecture, it may be feasible to not federate GeoEvent Server’s ArcGIS Server, and not even registering a connection to the Enterprise portal’s hosted server.

How do we prevent the loss of data if their services unexpectedly become unavailable?

The way the connectors are implemented, it may not be possible. 

• GeoEvent Server’s output isn’t actively or even passively connected to a feature service. The output can only send a series of discrete requests to the feature service and discover – through failure – that feature service is unavailable at the current time.
• The output may try again in a few hundred milliseconds, but if it continues to make repeated calls on an unavailable service every second, or every 100 milliseconds indefinitely, it impairs GeoEvent Server’s ability to perform other event processing tasks and work with other inbound/outbound web services which are available. The only choice is to allow the output to suspend operations until it is stopped and restarted.