When using IFeatureWorkspace :: CreateFeatureDataset out-of-process, you get an automation error when a new spatial reference is passed as the second parameter. You would encounter this if, for example, you create a standalone application that passes a new spatial reference via this method to a workspace object instantiated by ArcCatalog.
This is caused by a known issue in marshaling of the spatial reference. See the ArcObjects developer help for more on marshaling and out-of-process servers.
Get help from ArcGIS experts
Download the Esri Support App