Frequently asked question
What is the precision of the grid format?
Answer
Coordinate:----------
Grids store coordinate information as double precision (or "8 byte") floating point values. Double precision uses two consecutive 32-bit words and provides the following range of values:
2.2250738585072014e-308 to 1.7976931348623158e+308
(with 15 decimal digits of precision)
The precision offered here is sufficient to map any point on the Earth to better than a millimeter of accuracy.
Value:
-----
The Grid format supports single precision (or "4 byte") value ranges, both as 32-bit signed integer and 32-bit signed floating point values.
The ranges of values this supports are:
Floating point:
1.175494351e-38 to 3.402823466e+38
(with 6 decimal digits of precision)
Integer:
-2147483648 to +2147483647 (-2**31 to (2**31 - 1))
(Note: For integer grids, this applies only to the VALUE item. An integer grid may have other items added to its Attribute Table whose range of values depends on the item definition.)
Notes:
- the Grid environment respects the setting of the PROJECTCOMPARE command.
- Single precision number representation is according to IEEE standard of a 4-byte real number. Of the 32 bits allocated to storing the number, 1 bit represents sign, 23 bits represent the mantissa, and 8 bits represent the exponent. That means that the number of significant digits, regardless of the exponent, is limited to that which can be represented by 2 to the 23rd power, or 8388608. This provides for six digits of accuracy.
Article ID:000004232
Software:
- ArcMap 8 x
- Legacy Products
Get help from ArcGIS experts
Download the Esri Support App
Discover more on this topic
Search for related information
Find training related to this topic
Explore ideas and give feedback