When using the Georeferencing Rotate tool, and the cursor is placed over the X in the center of the image, the cursor changes from a circle with an arrow to a box with four directional triangles. This signifies that the rotation point can be shifted.
In the Layout mode, the cursor does not change; thus, the rotation point cannot be adjusted. In order to get the tool to enter the shift mode in Layout, you must first select the raster by activating the Select Elements tool (solid black pointer) in the Tools toolbar and double-clicking inside the extent of the image. The image extent will draw with a diagonally hatched border. The georeferencing Rotate tool will now activate the shift cursor when placed over the X at the center of the raster.