stream spits garbage when used with high rate uart 921600

[LZ7AA]

Good day gentlemen,
I'm having a blocking issue with the latest ESPHome and latest version of the stream-server. All was working good and stable with my initial setup I did an year ago (spring of 2022). I upgraded the other day my HomeAssistant (2023.3.1) with latest ESPHome 2023.2.4 and the issue popped-up. Hence, I am attributing it to the version of ESPHome framework and the stream-server component.
My use case is: ADSB receiver decoding airplane positions and providing data as uart stream (one direction Tx only) at 921600 bps. The UART is hardwired to GPIO1 and 3 on ESP8266 (nodemcuv2) and oxan/esphome-stream-server serving the data as TCP stream. Then I have a client app READSB connecting to ESP over WiFi and taking the data for further processing. HomeAssistant has nothing to do with the stream-server, I just use ESPhome framework as a handy tool for programming the ESP.
The issue:
I observe a lot of garbage in the stream, which does not come from the ADSB receiver uart, but definitely comes from the esp8266 memory. The garbage is predominantly full of zeros but it has fragments of SSID list of nearby WiFi networks (heard by the ESP).
I suspect something's wrong with the buffer handling. The issue is directly related to the amount of data coming from the UART. When there is no data or small amount of data - everything is OK (clean TCP stream), but when the UART data is huge the garbage is very pronounced leading to impossibility to further processing. The other observation is that when I increase the UART rx_buffer_size from the default 256 to 512 or 1024 the TCP stream is almost full of garbage and when I decrease it to 64 it gets almost clean, but lots of incoming data gets dropped (so not usable either).
The other observation is: I reboot the ESP, connect to the TCP stream and the stream is clean. Then I disconnect and reconnect and the stream is immediately garbled.
Oxan, Joshua, Bojan, can you help me investigate the issue and diagnose it, please?
Here is my config:
ADSB.txt
And here is a capture of the TCP stream.
capture.txt
Lines 1-6 are normal ADSB messages.
Lines 7-18 are garbage.
Lines 19-22 are normal ADSB messages.

I am available to provide further information or give you access to my system for live investigation.
Thank you in advance,
Kiril

[LZ7AA]

I was able to conduct an experiment and collect some more information. I was able to dig-in and dust-off an old Raspberry with HA core-2022.7.0 and ESPHome 2022.6.2. I compiled my ADSB project (same yaml code) there and installed it.
I observed the same bad output as before, so this rules out HA and ESPHome framework versions. There are times when the stream-server runs without problems for more than 10 minutes, but it starts bad output within a few minutes most of the times. I am able to brake it with certainty by disconnecting and re-connecting the TCP client.

I strongly believe the issue is with the latest stream-server version. Issue #34 seems related and proves the issue with the steam-serer is present even with much lower baud rate of 115k. So I think the hardware is not overloaded.

@oxan can you please help?
Can you provide the previous version of your component? The one circa jun/july 2022. I am sure it worked correctly.

[LZ7AA]

To summarize my findings so far.
Async server is capable of handling 300-400 kbps (may be more, but I have no more data at the moment to confirm higher rates) with 100 packets per second. UART buffer = 512
Socket based (new) version was tested with 8k=8k buffers and produces stuttering output with "slow" rate of 50kbps and 10 packets per second. The "fast" rate went up to 170kpbs and 20 packets per second. The "slow" stream is predominant with only occasional switch (on the fly, changing by it's own) to "fast" stream.
With 4096k buffer the fast and slow periods are of equal weight/time. With 1024k buffer the fast period is predominant. Here I noticed a dependency on the client attached. When I test with RealTerm attached I see switch from fast to slow and back. When I test with the real client tracking ADSB messages from airplanes it is always slow. I have never saw it processing more than 50kbps. Interesting to mention here the ADSB client (readsb) reports processing of 50 messages per second with average message size 30 - 50 ASCII characters. Now calculating 50 messages at 50kbps results in 1k per message, which is not true. Messages are much smaller. Perhaps messages are garbled and discarded by the ADSB processing software. Unfortunately they (their content) are not shown nor logged.
Same ADSB software connected to the async-server works flawlessly.
Here is reference to the ADSB processing software. You may make more sense of the message / TCP stream handling than me, as I am no developer.
https://github.com/wiedehopf/readsb

[LZ7AA]

I checked the TCP receiver buffer is 256kb by default.
TCP buffer size 64Kb * (2^n) (default: n=2, 256Kb)
And I sniffed the packets from the async server. Here are their sizes (including headers):
182, 118, 139, 182, 208, 182, 164, 182, 299, 182, 164, 182, 97, , 182, 151, 110, 70, 347, ...
Longest packet I have seen tonight is 944 Bytes.