Unable to save a large diagram layout in SDE.
This is a problem with the 'maxblobsize' and/or the 'blobmem' parameters in the SDE configuration.
To figure out the setting, the best thing to do is create the 'normal’ type of diagram and apply an algorithm, such as the Hierarchical - Smart Tree. Save and close the diagram. Re-open and verify that the layout was saved.
If it did not save, then the parameters need to be increased. There is no method for figuring out the size without testing it due to many factors, such as the number of labels and subtypes, that influence the final storage size.
sdeconfig -o alter -u sde -p sde -N -q -v maxblobsize=4000000