Unable to reproduce the problem. Registering PostgreSQL 9.5.4 to ArcGIS Server 10.5.1 on RHEL 7.5 was successful. PostgreSQL client libraries was bundled in the ArcGIS Server setup, no client configuration was needed. Make sure there was no other PostgreSQL client libraries in the $PATH that may have caused conflict. Also tested registering PostgreSQL database to ArcGIS Server 10.6.1 on CentOS 7.5.1804, and it was successful.