FAQ: Why is the XY Tolerance Calculated Differently Depending on Data Type?

Answer

The XY tolerance for geodatabase feature classes and shapefiles are calculated differently.

Note:
Storage for shapefiles and geodatabase feature classes differ based on the precision of the data. Please check the following documentation for a detailed explanation: Download: ESRI Shapefile Technical Description.

The XY tolerance and XY resolution for a geodatabase feature class can only be changed when initially creating the feature dataset.

Extent of geodatabase feature classes
In a geodatabase feature class, the XY tolerance is calculated as a function of the projection definition assigned to the feature class. If the data has a Projected Coordinate System (PCS) and the unit is set to meters for example, then the default XY tolerance is .001 meters, or 1 millimeter.

The projection definition, along with the XY tolerance, is used to calculate the pecision for data storage for the geodatabase feature class. It is also used to calculate the extent for which data can be created in the feature class.

The screenshot below shows the Extent of a geodatabase feature class in a PCS. Note the tiny size of South Korea, and compare with the red line that bounds the allowed size of the feature class.
[O-Image] Example of the extent for a feature class in a PCS from a geodatabase

[O-Image] Example of the extent for a feature class in a PCS from a geodatabase

Extent of a Shapefile
Shapefiles do not have XY tolerance, XY resolution, or a default extent assigned. The XY tolerance for a shapefile is calculated as 1/10,000 of the height or width of the shapefile, whichever is larger. The shapefile extent is limited to the geometry of the existing features. The following screenshot shows the extent of the same South Korea data in a shapefile.
[O-Image] Shapefile extent

Below is the XY tolerance calculation for a shapefile
Height = Top - Bottom = Y
Width = Left - Right = X
XY tolerance = The larger value is selected and divided by 10000. That is the XY tolerance for the shapefile in the units of the coordinate system.

This screenshot shows the actual coordinate extent for the shapefile.
[O-Image] Shapefile Coordinates

We use those extent values to calculate the XY tolerance for the shapefile.
Height = Top 670285 – Bottom 66807 = 603478 meters
Width = Left 117163 – Right 433255 = -316092 meters

For this example, the height of 603478 meters is selected because it is larger. The XY tolerance of the shapefile is 603478 / 10000, or 60.78 meters.

Comparison results
Given this comparison, the geodatabase feature class can store data with a much larger extent, and greater precision, than is available with the shapefile data model.

Article ID: 000012626

Software: