Skip to content

Single strange behavior #48466

Answered by CyrusNajmabadi
VBAndCs asked this question in Q&A
Discussion options

You must be logged in to vote

I recommend reading https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html to better understand this space and the subtleties therein.

This doesn't seem a round operation to me, and there is no obvious reason to do any rounding here.

The reason this happens is because those values cannot be represented exactly by either 32bit or 64bit floating point numbers. You can use System.Decimal here. However, that too has fixed amount of space, and there are values it cannot represent. When deciding which data type to use for numeric values, it's necessary to understand their precision and range and how that will affect your domain.

Replies: 4 comments

Comment options

You must be logged in to vote
0 replies
Comment options

You must be logged in to vote
0 replies
Comment options

You must be logged in to vote
0 replies
Answer selected by jmarolf
Comment options

You must be logged in to vote
0 replies
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet
2 participants
Converted from issue

This discussion was converted from issue #48466 on October 10, 2020 09:12.