Can someone please explain this behaviour and suggest a way around it?
In the command window on VFP 9.
Test 1
a = 7003602346555440
? a
Displays correct value.
Test 2
a = 7003602346555438
? a
Still fine.
Test 3
a = 7003602346555439
? a
Displays incorrect value of 7003602346555440
Test 4
? a=7003602346555439
Returns .T. as you'd expect.
Test 5
? VAL(7003602346555439)
Displays incorrect value of 7003602346555440
Clearly something odd going on with converting the numeric into the textual representation for display, but can anyone suggest a way to avoid this and ensure I always get the correct text version of the numeric?
Source from this article
SUMMARY
Visual FoxPro is documented as having 16 digits of precision. This is an
approximation: the actual maximum exactly representable number is
9007199254740992 (2^53).
MORE INFORMATION
Floating point numbers are stored in 8-byte or 64-bit representations. There are
12 bits of overhead, leaving 52 bits to store the number. There is one more
implied bit that gives you 2^53 as the maximum. The maximum number that can be
stored by Visual FoxPro is 2^1023. The highest power of two that is printed out
exactly using the ? command with the default setting of SET DECIMALS TO 2 is
2^43.
The following code demonstrates this:
SET DECIMALS TO 2
? 2^43 && All digits displayed
? 2^44 && Scientific notation
SET DECIMALS TO 5
? 2^53 && Maximum exact number
? 2^53 - 1 && Correct result
? 2^53 + 1 && Incorrect result: rounded in floating point
? 2^1023 && Cannot display: *'s will be printed
? 2^1022 && Can display
Even though the documentation says that val() will round up after 16 digits, it often rounds up at 16 and above. The example you are showing uses 16 digits and that is causing val() to round up.
Related
In Hive I was performing some tests with operations and got a behaviour that I could not understand.
While doing that operation, Hive is returning below results with 17 decimals after the comma:
select 500/3260;
> 0.15337423312883436
But when I am triyng to cast it with decimal format, the result is:
select cast(500 as decimal(38,18)) / cast(3260 as decimal(38,18));
> 0.153374
I would like to have 18 decimals, but only 6 are displayed.
Could you please explain me why it is giving this result?
Thank you in advance for your help.
I think it's due to decimal division rules detailed here. The link is for SQL server, but apparently the behaviour shown here is the same.
The scale will be set to 6 if it's greater than 6 and if the integral part is greater than 32. In this case, both integral part and scale would be reduced and resulting type is decimal(38,6). Result might be rounded to 6 decimal places or the overflow error will be thrown if the integral part can't fit into 32 digits.
The SQL server implementation was referenced in this Hive document (page 3).
Concerning the following line(from here):
"%.8x" % 7929856 #=> "00790000"
I don't understand what is done with 7929856 to get the value "00790000". I know that 0x790000 is 7929856 in hexadecimal, but I don't know where the two leading zeros came from. Is this simply a method of converting the number to hexadecimal? Can someone explain what is happening there?
The "%.8x" is %x with a minimum precision specified. In this case, 8 digits. So if you gave it:
"%.8x" % 1
> '00000001'
The result will always have at least 8 digits, maybe more.
I'm not terribly familiar with Ruby, but my guess (from similar syntax in C) is that the "8" in "%.8x" means to display 8 digits.
I'm using VBScript in ASP on IIS and I cannot seem to get the annoying error to go anyway so this makes debugging so much harder not to mention it does not tell me the exact error except the same error message so I can only assume what's wrong in my code.
My question is: Using the logical operators AND,XOR,NOT,OR in VBScript , is there a limit on the range to what the operands can be? I have implemented a bit shift right function and used the mod operator and I didn't notice until now that my function was causing the error.
My right shift function
function rshift(number,n)
'Shifts a number's bits n bits to the right
for i=1 to n
if number mod 2 = 0 then
number = number / 2
else
number = (number - 1) / 2
end if
next
rshift = number
end function
'Fails
rshift(1125899906842624,2)
I think for values larger than 2^32 ( or 31) - 1 that the operators do not work. Tried googling for the range of the operands but couldn't find anything helpful. I saw someone posted a topic about logical operators not working on large values but I can't seem to find that anymore.
Can someone verify this ?
Edit: Found a topic which gives more information on using the mod operator on signed 32 bit integers http://blogs.msdn.com/b/ericlippert/archive/2004/12/01/integer-arithmetic-in-vbscript-part-one.aspx
In VBScript there are several subtypes for integers including integer and long. VBScript will attempt to determine what type of value you are using and use the appropriate subtype.
Integer can store a value between -32,768 to 32,767 and long can store a value between -2,147,483,648 to 2,147,483,647. That value that you are using is greater than this and will result in an overflow.
You can use the VarType function to see what type your number is interpreted as. You may use Double to represent larger values.
This answer looks interesting. Maybe you can use it as part of your function.
In TI-Basic, there's a Fix function to limit the number of displayed decimal places. For example, Fix 2 would display only 2 decimal digits. However, when I try to convert a number to Degree-Minute-Second notation, I sometimes get more than the number of "fixed" decimal digits. For example,
1.12345678901
Float
Disp Ans►DMS
Fix 2
Disp Ans►DMS
Float
Disp Ans
Fix 2
Disp Ans
displays
1°7'24.444"
1°7'24.444"
1.123456789
1.12
The normal decimals act as expected. However, I would expect the second line to display 1°7'24.44. Is this possible? Or would I have to somehow convert it to a string and prune afterwards? (Keep in mind that I want to shorten the decimal because of the display constraints; I want to display text next to it without overlap).
extra info: TI-84+ Silver Ed'n, OS version 2.55 w/MathPrint
►DMS will display 0 to 3 digits after the decimal point, solely depending on the length of the decimal. The Fix command, set programmatically or through MODE does not affect this.
Storing a number formatted in DMS in a variable will undo the DMS formatting, and it cannot be stored in a string.
My suggestion would be isolating the degrees, minutes, and seconds in separate variables and working with them from there. In this way, they would also all be affected by the Fix command.
In Mma, for example, I want to calculate
1.0492843824838929890231*0.2323432432432432^3
But it does not show the full precision. I tried N or various other functions but none seemed to work. How to achieve this? Many thanks.
When you specify numbers using decimal point, it takes them to have MachinePrecision, roughly 16 digits, hence the results typically have less than 16 meaningful digits. You can do infinite precision by using rational/algebraic numbers. If you want finite precision that's better than default, specify your numbers like this
123.23`100
This makes Mathematica interpret the number as having 100 digits of precision. So you can do
ans=1.0492843824838929890231`100*0.2323432432432432`100^3
Check precision of the final answer using Precision
Precision[ans]
Check tutorial/ArbitraryPrecisionNumbers for more details
You may do:
r[x_]:=Rationalize[x,0];
n = r#1.0492843824838929890231 (r#0.2323432432432432)^3
Out:
228598965838025665886943284771018147212124/17369643723462006556253010609136949809542531
And now, for example
N[n,100]
0.01316083216659453615093767083090600540780118249299143245357391544869\
928014026433963352910151464006549
Sometimes you just want to see more of the machine precision result. These are a few methods.
(1) Put the cursor at the end of the output line, and press Enter (not on the numeric keypad) to copy the output to a new input line, showing all digits.
(2) Use InputForm as in InputForm[1.0/7]
(3) Change the setting of PrintPrecision using the Options Inspector.