This is expected behavior. The HTTP referrer option is causing the token to not be accepted by ArcGIS for Server when provided directly. This is working as designed.
The ArcGIS Server REST API documents that setting as follows;
HTTP Referer: When this (default) option is selected, the issued token can only be used in requests referred by the specified URL. This is the URL of the page from which the request is made to the ArcGIS resource. Use this approach when building an application with the ArcGIS API for JavaScript or other REST-based applications, in which individual clients requests maps and data directly from the ArcGIS for Server web service.
解决办法
When using Request IP, the generated token used to open the secure map service works properly.