Frequently asked question

How does ArcGIS Desktop calculate coordinates for data storage in the geodatabase?

Last Published: April 25, 2020


Coordinate values for data stored in the high precision geodatabase do not match values arrived at through arithmetic calculations.

Coordinates in the high-precision geodatabase are stored as floating point numbers. Floating point numbers are approximations that cannot precisely mimic real numbers, and floating-point operations cannot exactly mimic true arithmetic operations.

The Institute of Electrical and Electronics Engineers (IEEE) adopted a standard referred to as IEEE 754 in 1985. This standard is a method of storing floating-point numbers in a compact way that is easy to manipulate. This standard is used by Intel coprocessors and most PC-based programs that implement floating-point math.

IEEE 754 specifies that numbers be stored in binary format to reduce storage requirements and allow the built-in binary arithmetic instructions that are available on all microprocessors to process the data in a relatively rapid fashion.

However, some numbers that are simple, nonrepeating decimal numbers are converted into repeating binary numbers that cannot be stored with perfect accuracy.

For example, the number 1/10 can be represented in a decimal number system with a simple decimal: 0.1

However, the same number in binary format becomes the repeating binary decimal: .0001100011000111000111...

Since this number cannot be represented in a finite amount of space, this number is rounded down by approximately -2.78E-17 when it is stored.

If several arithmetic operations are performed on floating point numbers to obtain a given result, these rounding errors may be cumulative.

Portions of the IEEE documentation on standard 754-1985 for Binary Floating-Point Arithmetic can be accessed in the Related Information section below. Other technical information related to this topic is also available from the links below.

Article ID:000010968

  • ArcMap

Get help from ArcGIS experts

Contact technical support

Download the Esri Support App

Go to download options

Related Information

Discover more on this topic