This formula:
.55*100
results in this result: 55.00000000000001!
Some input values give surprising results, other do not:
.5x100 results in 5
.52x100 results in 52
.57x100 results in 56.99999999999999
.59x100 results in 59.
and so forth.
I can fix this problem with a round statement, but I would like understand why this is happening. Does anyone have an explanation?
Panorama uses double precision floating point arithmetic.
It uses 53 significant binary digits. This is equivalent to log10(2^53)=15.95 significant decimal digits. Panorama’s default output pattern uses 16 significant decimal digits. 15.95 is a lot closer to 16 than it is to 15, but it doesn’t quite get there. It would appear that the closest the binary can get to 0.57 is a little closer to 0.5699999999999999 than it is to 0.5700000000000000. The nearest it can get to 0.55 is closer to 0.5500000000000001 than it is to 0.5500000000000000.
To elaborate a bit, this wasn’t really a “choice” on ProVUE’s point, IEEE 754 is the arithmetic scheme built into macOS, so virtually all Mac software uses it. In fact, IEEE 754 is built into the CPU chip, so it’s used by almost all software on Mac, Window, and UNIX.
Floating point can only deal with an approximation of real numbers. In this case, the value 0.57 cannot be specified exactly as a IEEE 754 floating point number, so the multiplication answer is not quite exact either. This is simply the nature of computer floating point calculations, so usually you want to round the display. There is even a blog that is entirely devoted to the topic of edge cases in floating point calculation:
And what a wonderful blog it is! Tom, if you have 10 minutes to spare, go there and read the ‘Please Calculate This Circle’s Circumference’ topic. Ignore the C++ esoterica and focus on the main question. It’s reminiscent of the Panorama X/Excel problem that Kurt and I are still struggling with.