Answer
This problem is addressed in ArcGIS 8.1 Service Pack 1. See the Related Information section to download this pack.
Rolling back a transaction using either the Defined Layers command 'Dataset Transaction rollback' or the ArcObjects ITransaction’s method 'AbortTransaction' will fail and return no error.