Error: Failed to connect to the specified server. Failure to access the DBMS server
When attempting to make an ArcSDE Direct Connection to a SQL Server database, in which SQL Server was installed on a non-standard port, (the default SQL Server port is 1433) the following error occurs.
"Error: Failed to connect to the specified server. Failure to access the DBMS server."
In most cases, if a non-standard SQL Server port is specified, and the SQL Server Browser service is run, the service communicates with the client informing the client what port number is being used for the instance in which a connection is being requested. This eliminates the client from having to know the port number when connecting to an instance of SQL Server.
Solution or Workaround
In a more secure environment, typically the SQL Server Browser service is not running. By not running the SQL Server Browser service, the client needs to identify the correct non-standard port in the connection string in order to connect to SQL Server. Therefore, the port can be programmatically specified in the connection string, which would be done by placing ',<port number>' immediately following the server name.
Here is how the syntax looks when attempting to establish a direct connection from ArcCatalog.
Here is how the syntax looks when attempting to create an ArcSDE service in which the SQL Server instance is installed on a non-standard port and the SQL Server Browser Service is not enabled.
For information regarding an additional option to create a SQL Server alias for use by the client machine, please review the following article from Microsoft.
How to: Create a Server Alias for Use by a Client (SQL Server Configuration Manager)