A. Instead of building these custom symbols as EMF pictures from Illustrator, purchase font creation software and make a TrueType font with these symbols, which can then be used in ArcMap as character markers. The Feature Outline Mask tool works just fine on symbols based on character markers.B. Build up custom symbols using the ArcMap symbol editor and save them to a style. Again, the Feature Outline Mask tool works just fine on the symbols based on character markers.C. Build up custom cartographic representation symbols in the cartographic representation Marker Editor. The Feature Outline Mask tool also works just fine on symbols manually created in this editor.D. Export the symbols out of Illustrator as BMPs and use the BMPs as the picture marker instead of the EMF in the workflow.