Setting a variable equal to a number containing one or more digits after the decimal point will set the data type of that variable to Float.
Presumably, this mean that any subsequent calculation involving that variable will be carried out to the maximum achievable precision. Does this mean I don’t need to use the float( function in those calculations?
For instance, if I have
f = 1/298.2572221
and f is subsequently used in a calculation, will I get a different result if I had used
f = float(1)/298.2572221
Does it matter that, in the first expression, the first number encountered in the calculation is not floating point?
The divide operator is always a floating point operation. So even if both operands are integers, the result will be floating point.
1/2 ☞ 0.5
Even if the result could be integer, it will be floating point. So the result of
4/2 ☞ 2.00000
is a floating point value.
If you want to do integer division, you have to use the integer divide operator.
4\2 ☞ 2
You used division as an example, but I suspect you were looking for a more general answer. However, there isn’t one. For example, addition, subtraction and multiplication will return an integer if both operands are integers, but floating point if either operand is floating point. Different operators and functions have different rules. In general, these rules are documented in the help pages for each specific operator or function. It’s a lot simpler than Panorama 6, since there are no fixed point numbers to worry about.
I just noticed that the * operator is not included in the help file! The documentation was written, but somehow it didn’t get into the master file. Same for subtraction. Whoops!
So does this mean that the float( function will be rarely needed?
My primary reason for asking is that comparing identical code (which is stuffed full of float(functions) for high-precision geodetic calculations, Pan6 gives the correct result (ie, it exactly matches an online calculation posted by a government agency) and the PanX result is slightly different. I haven’t yet checked it on another iMac which is my next step.
The float function should rarely be needed. Any numeric operation that has a floating point number as one of its operands will be a floating point operation and return a floating point result. The trig functions all return floating point numbers. Floating point numbers have the equivalent of somewhere between 15 and 16 significant decimal figures, and you should only need 11 to calculate the circumference of the equator to the nearest millimeter.
If you aren’t getting that kind of precision in Panorama X, when you were getting it in Panorama 6, it probably means that numbers are being converted to text at some point. When Panorama 6 did that, it kept 15 significant figures by default. Panorama X keeps 6.
Spot on Dave! It never ceases to amaze me how you get to the nub of these things so quickly. In my conversion of calculated decimal values to sexagesimal format, I was using the str function. Replacing that with the pattern( function gives me the precise results that I want.
I need to calculate latitude and longitude to 0.00001 seconds of arc and azimuths to 0.01 seconds of arc so six significant figures just doesn’t cut it.
I find the limitation of the str( function pretty scary and I’ve posted a separate comment on that.