Follow up on Crate "Union all" bug

[c0c0n3]

Is your feature request related to a problem? Please describe.

PR #300 implemented a workaround to crate/crate#9854, have a look over there.

Describe the solution you'd like

Follow up with the Crate guys. If the issue gets fixed, revert back PR #300's workaround.

Describe alternatives you've considered

An alternative would be not to order the result set. Basically we're just returning entity IDs, perhaps time oder isn't actually that important. If that's the case, we can keep things as they are, i.e. no extra work. (Well maybe just get rid of the TODO in the code since it could be confusing if somebody bumps into it a year from now :-)

Additional context
PR #300, crate/crate#9854

[amotl]

Dear @c0c0n3 and @chicco785,

thanks for tracking this. I just wanted to quickly take the chance to get back to you and leave a remark that crate/crate#9854 has been fixed since CrateDB 4.2.0.

Somehow related to #157, you might want to consider upgrading to CrateDB 4.3.1, which is the current release. Also, I would like to point out that CrateDB 4.3.2 is around the corner.

With kind regards,
Andreas.

[chicco785]

thx @amotl we will surely look into that!

[c0c0n3]

Dear @amotl,

Thank you very much for notifying us of the fix. At the moment we're using Crate 4.1.8 in our production environments, but
as @chicco785 pointed out we will definitely put some thought into upgrading to 4.3.2. On this note, it seems Crate has quite fast-paced development cycles, so I was wondering if you recommend upgrading frequently (because e.g. a 3 month development cycle usually brings in significant new features and enhancements) or longer term upgrades would be better---e.g. for stability. The reason why I am asking is that at the moment we upgrade our environments about once a year...

[amotl]

Dear @c0c0n3 and @chicco785,

thanks for asking. I would definitively recommend to upgrade more regularly. We are always fixing bugs and bringing in improvements on different levels. I will be happy to work closer together with you on this matter and beyond.

I haven't looked into the test harness implementation of ngsi-timeseries-api thoroughly, but if essential things are covered there, trying out version bumps on the infrastructure should be no hassle, right?

In any case, I very much appreciate the work you are doing here and look forward to learn more details about QuantumLeap. I've already identified some aspects on the issue tracker where I might attach and followup to, specifically regarding eventual bottlenecks you might be observing.

Keep up the spirit and with kind regards,
Andreas.

[c0c0n3]

Hi @amotl, hope you won't mind me keeping the conversation informal going forward, it feels like we made a new friend :-)

I would definitively recommend to upgrade more regularly

Noted, thanks!!

trying out version bumps on the infrastructure should be no hassle, right?

Well, that's the theory. Practice can be a bumpy road sometimes. To be fair, Crate has fairly good, well-documented upgrade paths and most of the time upgrading is a smooth ride. But from time to time we got ourselves into trouble with prod clusters---e.g. last month we got stuck on some nasty (Lucene) index corruption issues when upgrading from v3 to v4 which costed us about a week of dev since we had to recover about 20M records scattered about a dozen tables.

I very much appreciate the work you are doing here

Likewise, we really value Crate as a product and all the hard, quality work you guys are putting into it.

look forward to learn more details about QuantumLeap.

For a 5-min visual tour, you could try this Wiki page. The official docs go into more detail and there's also a task-oriented, hands-on tutorial over here if you like the learning-by-doing approach. Surely, there's no substitute for reading the source code, but that's not for the faint-hearted---we've been wanting to untangle the spaghetti for a while, but have been to hard-pressed to get actual features out of the door, you probably know how the story goes...

identified some aspects on the issue tracker where I might attach and followup to, specifically regarding eventual bottlenecks you might be observing.

That's great. We'd really appreciate any suggestions you might have, but even more so if could tell us what we're doing wrong. (No worries, we won't take it badly :-) Truth be told, there's still alot we've got to learn about Crate before we'll be able to tune performance like a pro :-)

[amotl]

Dear @c0c0n3,

thanks for your kind words and sorry to hear about the hassle you had with your production cluster. So, you made the transition to CrateDB 4.x successfully now? If so, the road should be paved for further upgrades but I see that you might be hesitant on this as you are aligning the development head with the actual infrastructure you are running on production, right?

Thanks also for referencing those documentation items. You are having some nice drawings and prose over there, kudos! Parts of that remember me about the work we have been doing over at [1] and [2], also touching [3] and [4] and still available for exploration on [5] these days. However, everything has been done without holding on to standards like NGSI, so I well recognize the work you are doing here by implementing this.

We'd really appreciate any suggestions you might have, but even more so if could tell us what we're doing wrong.

I will try to find some time to address some aspects I have been observing here. Please also bear with me that I am not involved in the very details and might get something wrong when attaching to different things. Saying that, I will be happy to learn and look forward to your feedback eventually adjusting my false assumptions.

No worries, we won't take it badly ;].

Dito ;].

With kind regards,
Andreas.

[1] https://github.com/daq-tools/kotori
[2] https://github.com/panodata/grafana-map-panel
[3] https://github.com/panodata/luftdatenpumpe
[4] https://github.com/earthobservations/wetterdienst
[5] https://weather.hiveeyes.org/

[c0c0n3]

Hello @amotl !

made the transition to CrateDB 4.x successfully now?

yep, we're on version 4.1.8 at the moment.

aligning the development head with the actual infrastructure you are running on production, right?

Pretty much. We strive to implement features that could be useful to others too as much as we can, but at the end of the day our eensy-weensy budget rules out pretty much anything that isn't readily exploitable by our clients.

some nice drawings and prose over there

Thanks! These days sketching apps on tablets got so good that even a monkey like me can whip together decent looking diagrams :‑O

work we have been doing over at [1] and [2]

I've had a quick look at Kotori, I really like the idea, good stuff! Also we've been using the Grafana map panel ourselves, so now we know who to thank :-)

everything has been done without holding on to standards like NGSI, so I well recognize the work you are doing here by implementing this.

In principle it should be possible to make Kotori speak NGSI too since it looks like you have a plugin architecture in place?
I really feel there should be a community-driven, high-quality implementation of an NGSI library that others could easily use to build NGSI services or connect to them. For example, it'd be nice to have a proper AST representation, a streaming parser, an NGSI-to-tabular interpreter, etc. If we had that, both QL and Kotori could readily benefit from that, IMO.

I will try to find some time to address some aspects I have been observing here.

We welcome any kind of feedback :-)

[amotl]

Hi @c0c0n3 and @chicco785,

sorry for the late response.

We strive to implement features that could be useful to others too as much as we can, but at the end of the day our eensy-weensy budget rules out pretty much anything that isn't readily exploitable by our clients.

I see.

I've had a quick look at Kotori, I really like the idea, good stuff! Also we've been using the Grafana map panel ourselves, so now we know who to thank :-)

Thanks! Also, I still have to answer @chicco785 at grafana-toolbox/panodata-map-panel#87. Apologies.

I really feel there should be a community-driven, high-quality implementation of an NGSI library that others could easily use to build NGSI services or connect to them.

Indeed, that would be nice.

Now, while I appreciate the fruitful discussion here, I want to apologize that I kind of started to hijack the original issue which was just about the "Union all bug". So, let's get back to the topic of CrateDB here and maybe continue this discussion within the "Discussions" section?

On the other hand, as we already have some unrelated noise here, can I humbly ask you to unlock the "Issues" section on the crate-ce repository [1]? I would like to share some news which probably would fit best into that repository.

With kind regards,
Andreas.

[1] https://github.com/orchestracities/crate-ce

[c0c0n3]

hi @amotl !

So, let's get back to the topic of CrateDB here and maybe continue this discussion within the "Discussions" section?

Excellent idea. We enabled GitHub Discussions for this repo a while back but then I sort of plain forgot we have it on. Sorry I should;ve advertised that, but going forward, yes, please use Discussions as you see fit.

unlock the "Issues" section on the crate-ce repository [1]?

It looks like you should be able to open issues on that repo too, let us know if you're still having trouble with that. Thanks!

[github-actions]

Stale issue message