1BRC in .NET

[buybackoff]

.NET 8 version without any dependencies on 6 cores (i5-12500) with ~30GB unused RAM (DDR4-3200 CL22) finishes in ~5.3 4.25 seconds.

This approximately scales to 8.8 seconds on the evaluation server (by comparing local run of the top reported result to the reported time).

No assumptions neither on name nor on temperature format/length/etc. Supports \r\n lines endings, since it runs on Windows and the Java generator from this repo uses the OS default new line separator.

https://github.com/buybackoff/1brc

[garfbradaz]

Amazing!

[pietrzak-chapsvision]

Impressive !

Is the call to GetHashCode optimized by the JIT ?

[adamrodger]

You may get some more performance by implementing IComparable on Utf8Span so the OrderBy doesn't need to create strings before comparing.

You may also trim a small amount of performance by using a StringBuilder when printing the results instead of doing Console.WriteLine and interpolated strings in a loop.

There's also AOT compilation, preallocating collection sizes (e.g. the dictionary) if that's not cheating, and tweaking GC settings (such as basically disabling it or using Server GC so it runs much less often).

[buybackoff]

Order and print take together < 30 msec, the code is zero alloc on the main path. Yes I do ToString twice by accident (I could materialize span to string using Select() before ordering and print the string) - but it does not matter at all. Only the main loop in ProcessChunk matters.

[adamrodger]

Could you get anything out of AOT so it doesn't have to JIT anything? Or explicit SIMD usage

[buybackoff]

The code is so small and straightforward, it should be JITted very fast. As for SIMD, why should I bother if the Span API IndexOf/Any does it for me x-plat x-arch!?
The only scenario when manual SIMD may be interesting is to apply simdjson/simdcsv idea of finding boundaries first and then processing stuff inside the boundaries.

There are multiple known ways to cut probably 0.30+ seconds depending on how much "cheating" is allowed. I will wait for the top results and see how much additional knowledge they apply. At the moment my implementation is general-purpose x-plat x-arch and pretty fast. I would prefer not to cut corners but try to improve the code itself.

[bricedenice59]

Woww amazing! I didn't know anything about named memory mapped files.
Unfortunately not available on unix platform as I see : Unhandled exception. System.PlatformNotSupportedException: Named maps are not supported.

[buybackoff]

Oh, I need to fix that. Should work on Linux. Thanks!

[bricedenice59]

That would be lovely; thanks in advance :)

[buybackoff]

Should work now

[buybackoff]

Updated with comparable performance numbers. Now I have a dedicated machine for that.

The performance is measured on an idle 6C/12T Alder Lake CPU fixed at 2.5 GHz (no turbo), 32GB DDR4 3200, Debian 12 in LXC. At least 5 runs, showing the best result.

• The top two Java results (reported as < 8.0): ~4.1 sec
• My code: 4.59 sec
• Current No 3 Java (9.625): 4.75 sec

Original numbers were measured inside the app. However, in the latest change I found such a trivial omission that improved the perf by more than 1/8, so that is compensating and all initial numbers are more or less valid still.

[yunusefendi52]

@buybackoff I notice that you use StringComparer.InvariantCulture which has different result than the java version like "Zürich" should be first and then "Ürümqi", then "İzmir". Have you tried using StringComparer.Ordinal?

[buybackoff]

Specs are not mentioning how to sort exactly. For performance it does not matter at all, there are just too few unique names.

[oleggolovkov]

• What if you pass Summary by ref to Merge, would this improve timing?
• Also I think this struct is 16 bytes and for min/max int is not needed, short should be enough.
• Also I think .Select(x => (Name: x.Key.ToString(), x.Value)) for final result may not be needed, instead Utf8Span could implement IComparable<Utf8Span> and do comparison on Span
• Console.Write($"{pair.Name} = {pair.Value}"); I think allocates an unnecessary string, could instead rewrite it as Console.Write(pair.Name); Console.Write(" = "); Console.Write(pair.Value) however not sure what is the performance of actual Console api, maybe the better strategy would be to just form 1 big resulting string and invoke a single Console.Write?

[buybackoff]

@oleggolovkov Could you, please, open in issue with this in the repo? Thanks!