ArcGIS Enterprise 11.1 does not recognize the Active Directory group when adding new members if the Active Directory group store name has a space in the name.
Groups with one or more spaces in the name are fully supported. When attempting to add members based on groups, the format of the group name must correspond with the group store type. When the group store type is Windows, the group name format must be 'domain/group name'. When the group store type is Lightweight Directory Access Protocol (LDAP), the group name format is just 'group name'. Searching for the group and selecting the group prior to clicking 'Add' is recommended.