Single strange behavior #48466
-
Try this in C# var x = 123456781.0F;
Console.WriteLine(x); // 123456784
var x = 123456785.0F;
Console.WriteLine(x); // 123456784 Why are Dim x As Single = 123456779.0F
|
Beta Was this translation helpful? Give feedback.
Replies: 4 comments
-
Double also does such exotic things. For example: |
Beta Was this translation helpful? Give feedback.
-
https://en.wikipedia.org/wiki/Single-precision_floating-point_format
These are not integers with 7 or fewer digits. Your integers have 9 digits and are not a power of 2. So they cannot be converted exactly. As such, you get a close value as per the specific "string to FP" algorithm used. |
Beta Was this translation helpful? Give feedback.
-
I recommend reading https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html to better understand this space and the subtleties therein.
The reason this happens is because those values cannot be represented exactly by either 32bit or 64bit floating point numbers. You can use System.Decimal here. However, that too has fixed amount of space, and there are values it cannot represent. When deciding which data type to use for numeric values, it's necessary to understand their precision and range and how that will affect your domain. |
Beta Was this translation helpful? Give feedback.
-
@VBAndCs You may want to consider posting your questions to gitter.im/csharplang or dicord.gg/csharp as they're better avenues for questions/answers. :) |
Beta Was this translation helpful? Give feedback.
I recommend reading https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html to better understand this space and the subtleties therein.
The reason this happens is because those values cannot be represented exactly by either 32bit or 64bit floating point numbers. You can use System.Decimal here. However, that too has fixed amount of space, and there are values it cannot represent. When deciding which data type to use for numeric values, it's necessary to understand their precision and range and how that will affect your domain.