Answer
The Database Connection dialog box does not support environment variables. Instead, set environment variables at the operating system (OS) level or in the user's shell.
It is also possible to use the Create ArcSDE Connection File geoprocessing tool and the direct connection syntax used by older ArcGIS releases, which can include environment variables. However, the recommended method is to set an OS or shell variable.