Surveys can be downloaded by all members of the organization using the Survey123 desktop app despite the survey being restricted to a group and its members.
This issue is not reproducible with version 3.0 or above. A signed-in user who is not in the group the survey is shared with can not download or see the survey in the Connect or Field app. Note that if the user is signed-in as an administrator of the organisation, all surveys will be available to them regardless of if they are in the group the survey has been specifically share with.