quasi-1BRC in Nim clarifying relevance of challenge #95
Replies: 1 comment 1 reply
-
|
The problem is solvable in like 1-2 seconds. It's silly to say "Almost all of our tools are 10-1000x too slow at text parsing, therefor we shouldn't care about parsing." Real computers spend a lot of time on parsing, formatting, and codecs. Real people spend a lot of time waiting for those computers to do those things. It's worth doing them right eventually. |
Beta Was this translation helpful? Give feedback.
-
|
It's also silly to say "real people spend a lot of time waiting" yet also seem to criticize a recommendation for formats orders of magnitude more efficient for calculation when ASCII can be avoided (which in my experience is very often, but sure - not "always", and sure often not worth it for "small" data). One could view this thread as merely me pointing out how avoidable text may have been in this situation. We agree 100% that it's worth doing them right eventually (which is why I think observing this here not silly). What I am saying is that part of that is often that you should "not just parse but transform". Or "save the parse". Or stay in binary. This is not radical advice. Saving an answer to not repeat work / avoiding unnecessary round-trips are early performance lessons for most. I actually explicitly acknowledged "nice to have a fast transformation" and have certainly spent many hours of my own life optimizing such transforms (in both directions even). I did say you should ("maybe!") not care about the combination strategy this challenge suggests. Challenges like this which combine parsing & simple calculation suggest to the naive that parsing is "merely one important part" of doing said calculation or that it's a good idea to "calculate alongside the parse -- look ma, one pass!" when the parsing is orders of magnitude more work - the whole calculation, really. If there is any chance of follow-on calculations, "convert don't just parse" or "stay in binary" seems good advice to mention alongside the challenge. I expect many will still have bad takeaways regardless of me saying anything at all simply from the opening sentence "can be pushed for aggregating one billion rows from a text file". Sorry if you disagree. (and FWIW, you seem like you personally know all this and much more anyway, and I don't meant to suggest otherwise but you inspired a defensive mode.) In some ways, the entire project of "DBs" for 70 years has been about better prepared formats. There is sadly a culture of trapping data behind complex &| proprietary formats &| COBOL-esque query languages. It's been getting better recently but with a tendency to jump immediately to distributed systems worm cans most need not suffer if they manage their "small" data better. I never said anything about codecs, but agree AV codecs are often a good example of efficient binary formats. To end on a more positive/constructive note, if one cares about Computer Slowness/electricity in general, it would be better to have challenges to transform into a re-usable efficient formats / have faster DB loaders, challenges to have DBs more easily configurable to realize SIMD/DIMM saturation, etc. { or for some DBs to even have that ability } or even more general JSON parser/formatters (probably too much to hope for bold warnings to "avoid using unless truly necessary"). People are less likely to be confused by the point of such exercises. Here it seems to be "How far can we push text inappropriately?" while it would be better to ask/answer "How easy can we make efficiency?". But I've written far too much on this and risk being misunderstood on a volume-basis of my own inefficient text. ;-) |
Beta Was this translation helpful? Give feedback.
-
This breaks most written (& probably unwritten) rules of this challenge, but otherwise replicates the central calculation so to emphasize in a very hands-on way what the rules make this challenge actually about - ASCII parsing, not numeric reductions in DB systems as was mentioned on Lobsters -- as well as to show maybe why you shouldn't care. The "real" answer is to avoid ASCII as much as possible, just loading a binary format/DB once. Then the speed of code like the actual 1BRC doesn't matter so much (well, it can be nice to have a fast transformation to binary, but that should be run only once {or zero times if you can originate in binary as I also show in
mkSplit.nimbelow}). I try to make binary (almost) as simple as CSV over in nio.Here is a HOW2 I wrote up and the 3 Nim files (one of which depends upon my
cligenwhich you can clone and put in yournim c --path), oh and theruused in the HOW2 is from bu, but you can replace that with some timing utility of your own preference:and then
brc.nim:and
mkSplit.nim:and finally the main event
doSplit.nim:Beta Was this translation helpful? Give feedback.
All reactions