FAQ: How Does ArcGIS Desktop Calculate Coordinates for Data Storage in the Geodatabase?

Answer

Coordinate values for data stored in the high precision geodatabase do not match values arrived at through arithmetic calculations.

Coordinates in the high-precision geodatabase are stored as floating point numbers. Floating point numbers are approximations that cannot precisely mimic real numbers, and floating-point operations cannot exactly mimic true arithmetic operations.

The Institute of Electrical and Electronics Engineers (IEEE) adopted a standard referred to as IEEE 754 in 1985. This standard is a method of storing floating-point numbers in a compact way that is easy to manipulate. This standard is used by Intel coprocessors and most PC-based programs that implement floating-point math.

IEEE 754 specifies that numbers be stored in binary format to reduce storage requirements and allow the built-in binary arithmetic instructions that are available on all microprocessors to process the data in a relatively rapid fashion.

However, some numbers that are simple, nonrepeating decimal numbers are converted into repeating binary numbers that cannot be stored with perfect accuracy.

For example, the number 1/10 can be represented in a decimal number system with a simple decimal: 0.1

However, the same number in binary format becomes the repeating binary decimal: .0001100011000111000111...

Since this number cannot be represented in a finite amount of space, this number is rounded down by approximately -2.78E-17 when it is stored.

If several arithmetic operations are performed on floating point numbers to obtain a given result, these rounding errors may be cumulative.

Portions of the IEEE documentation on standard 754-1985 for Binary Floating-Point Arithmetic can be accessed in the Related Information section below. Other technical information related to this topic is also available from the links below.

Article ID:000010968

Software: