Let's say I have any array Trial() As Integer
I have two following variables defined as Integer:
Dim Left As Integer
Dim Right As Integer
Now I am increasing the array index of trial
ReDim Preserve Trial(Left+Right)
Now If My total (Left+Right) exceeds Integer limit the above will give error.
And if redeclared Left as Long then it will work fine.
Actually I want to understand the internal calculation for (Left+Right).
Does it allocate the space for total depending on the datatype of "Left"?
Or it may also depends on datatype of "Right"?
It depends on both. The compiler will examine both variables and determine from both what data type it needs. For example. If you were to add (or multiple or divide) an Integer and a Long then the result will give you a long.
The calculation Left + Right is done presuming the result was an integer too, that's where the overflow occurs.
If you go CLng(Left) + CLng(Right) first, it is done the same way, only that the result will be a long, and thus no overflow occurs.
Related
I inherited an application that took 2281534395476 and incorrectly casted it to an integer of 906761300. I'm looking to essentially un-cast it back to its proper and correct (long) data type.
What is a reproducible algorithm for doing so that could be run over a list? I'm looking for pseudocode or something of the sort, although the end result will be written in Ruby.
One simply cannot get the value back when already casted from long to int. When you are casting from long to int, you are losing some bits. So, you just can't get back those bits and can't get back your lost long value by using a generic algorithm.
In a system , where long and int uses same number of bits, the conversion is trivial; it simply copies the value. In this case, you may get the value back.
On a system where long is wider than int, if the long value won't fit in an int, then the result of the conversion is implementation-defined. What typically happens is that the high-order bits are discarded. Your system seems to be this one. So, you can't get back the long value.
I'm using VBScript in ASP on IIS and I cannot seem to get the annoying error to go anyway so this makes debugging so much harder not to mention it does not tell me the exact error except the same error message so I can only assume what's wrong in my code.
My question is: Using the logical operators AND,XOR,NOT,OR in VBScript , is there a limit on the range to what the operands can be? I have implemented a bit shift right function and used the mod operator and I didn't notice until now that my function was causing the error.
My right shift function
function rshift(number,n)
'Shifts a number's bits n bits to the right
for i=1 to n
if number mod 2 = 0 then
number = number / 2
else
number = (number - 1) / 2
end if
next
rshift = number
end function
'Fails
rshift(1125899906842624,2)
I think for values larger than 2^32 ( or 31) - 1 that the operators do not work. Tried googling for the range of the operands but couldn't find anything helpful. I saw someone posted a topic about logical operators not working on large values but I can't seem to find that anymore.
Can someone verify this ?
Edit: Found a topic which gives more information on using the mod operator on signed 32 bit integers http://blogs.msdn.com/b/ericlippert/archive/2004/12/01/integer-arithmetic-in-vbscript-part-one.aspx
In VBScript there are several subtypes for integers including integer and long. VBScript will attempt to determine what type of value you are using and use the appropriate subtype.
Integer can store a value between -32,768 to 32,767 and long can store a value between -2,147,483,648 to 2,147,483,647. That value that you are using is greater than this and will result in an overflow.
You can use the VarType function to see what type your number is interpreted as. You may use Double to represent larger values.
This answer looks interesting. Maybe you can use it as part of your function.
In VB6, what is the likely cause of an Overflow error? I am using strings when it occurs.
Is it RAM, or hard-drive space? Or is it something internal to VB6?
I am going to take a stab in the dark here, some code would help as Hogan said. Typically overflow error occur with string when VB6 things it is dealing with integer or longs in math formulas and the results are too large for a integer or long to hold.
Depending on the nature of your formula you may get away from the problem by forcing the system to use the numbers as floating point by adding a '.0" at the end. Otherwise use the various Cxxx function to explicitly convert the numbers to a type that has a greater range.
The one thing you consider is that floating point are less precise than whole number integers so make sure you don't lose needed precision when you do the conversion.
Another 'stab', since no code is provided ...
The following will produce an overflow error:
Dim x as integer
x = len(longstring) 'longstring over 32,768 characters in length
Would cause an overflow error.
Dim x as long
x = len(longstring) 'longstring over 32,768 characters in length
Works fine.
Another example of an overflow from Microsoft Support here:
EDIT
A more subtle situation that can catch you off guard:
You attempt to use a number in a calculation, and that number is coerced into an integer, but the result is larger than an integer.
Dim x As Long
x = 2000 * 365 ' Error: Overflow
To work around this situation, type the number, like this:
Dim x As Long
x = CLng(2000) * 365
I have the following code :
Dim L as Integer
Dim R as Integer
Dim a as Integer
a=((L+R)/2)
Now (L+R) exceeds limit of Integer.
In order to handle this case:
I have following three options:
Define L (or R) as Long
Write a= ((CLng(L)+R)/2)
Declare new variable as Long :
Like this
Dim S as Long
S=S+L+R
I am confused which one is the best to implement?
Change all the variables to Long.
The code will be more robust.
The code will execute faster.
The additional 2 bytes of memory per variable is totally insignificant, unless you have many millions of these integer variables in use simultaneously.
You've already posted several questions here about integer overflow errors. With all respect, I really advise you to just change all your Integer variables to Long and get on with your coding.
I'd pick #2. I think (not sure) that this uses a little less memory than #1 because there's only one Long -value in the equation where as changing L or R to Long would require space for 2 Long values.
I'm thinking #2 and #3 might end up looking the same (or pretty damn close) after compile and I personally think that in this case an extra variable wouldn't make it more readable. The difference of course is that in #2 the result of the L+R might not need to be saved anywhere, but only moved between registers for the calculation.
I'm thinking alot here, but I'm posting this partly because I hope that if I'm wrong, someone would correct me. Anyway, with the reasoning above, I'd go with #2. Edit: at least I'm quite certain that if one of the options uses less memory than the others, it's #2, but they might all be the same in that regard.
As I tried to debug, I found that : just as I type in
Dim value As Double
value = 0.90000
then hit enter, and it automatically converts to 0.9
Shouldn't it keep the precision in double in visual basic?
For my calculation, I absolutely need to show the precision
If precision is required then the Currency data type is what you want to use.
There are at least two representations of your value in play. One is the value you see on the screen -- a string -- and one is the internal representation -- a binary value. In dealing with fractional values, the two are often not equivalent and where they aren't, it's because they can't be.
If you stick with doubles, VB will maintain 53 bits of mantissa throughout your calculations, no matter how they might appear when printed. If you transition through the string domain, say by saving to a file or DB and later retrieving, it often has to leave some of that precision behind. It's inevitable, because the interface between the two domains is not perfect. Some values that can be exactly represented as strings (or Decimals, that is, powers of ten) can't be exactly represented as fractional powers of 2.
This has nothing to do with VB, it's the nature of floating point. The best you can do is control where the rounding occurs. For this purpose your friend is the Format function, which controls how a value appears in string form.
? Format$(0.9, "0.00000") will show you an example.
You are getting what you see on the screen confused with what bits are being set in the Double to make that number.
VB is simply being "helpful", and simply knocking off excess zeros. But for all intents and purposes,
0.9
is identical to
0.90000
If you don't believe me, try doing this comparison:
Debug.Print CDbl("0.9") = CDbl("0.90000")
As has already been said, displayed precision can be shown using the Format$() function, e.g.
Debug.Print Format$(0.9, "0.00000")
No, it shouldn't keep the precision. Binary floating point values don't retain this information... and it would be somewhat odd to do so, given that you're expressing the value in one base even though it's being represented in another.
I don't know whether VB6 has a decimal floating point type, but that's probably what you want - or a fixed point decimal type, perhaps. Certainly in .NET, System.Decimal has retained extra 0s from .NET 1.1 onwards. If this doesn't help you, you could think about remembering two integers - e.g. "90000" and "100000" in this case, so that the value you're representing is one integer divided by another, with the associated level of precision.
EDIT: I thought that Currency may be what you want, but according to this article, that's fixed at 4 decimal places, and you're trying to retain 5. You could potentially just multiply by 10, if you always want 5 decimal places - but it's an awkward thing to remember to do everywhere... and you'd have to work out how to format it appropriately. It would also always be 4 decimal places, I suspect, even if you'd specified fewer - so if you want "0.300" to be different to "0.3000" then Currency may not be appropriate. I'm entirely basing this on articles online though...
You can also enter the value as 0.9# instead. This helps avoid implicit coercion within an expression that may truncate the precision you expect. In most cases the compiler won't require this hint though because floating point literals default to Double (indeed, the IDE typically deletes the # symbol unless the value was an integer, e.g. 9#).
Contrast the results of these:
MsgBox TypeName(0.9)
MsgBox TypeName(0.9!)
MsgBox TypeName(0.9#)