V3 Treesitter Engine

[vhyrro]

Would've been cooler if it were called the V8 engine but it's supposed to be the 3rd version of the parser so I can't just do that :p

It's finally here! The parser is so close to completion that I can already start migrating all of Neorg to the new engine. It's fast, it's much more reliable and it's 7.7x smaller than the previous iteration(s). And if it ever breaks, then it's simple to debug and actually consistent in when it errors in my experience. V1 would work for 90% of cases and then break super randomly in irreproducible environments.

Thanks to this we can also use treesitter as a source of truth for pandoc parsers or highlighting engines (looking at you, github) 👀

EDIT: It also now compiles natively on MacOS! Wohoo!!!

TODOs

There's a lot to do for this PR alone, and there's still some preliminary stuff. Here's the checklist:

• Implement inline maths and inline macros parsing and make conflicts work as intended within TS.
• Implement free-form attached modifiers in the parser. These are pretty hard, but they're about halfway there.
• Test the parser in MacOS environments to see if it compiles natively without hacks.
• Fix up some odd edge cases
• Put it all in Neorg!

[c3n21]

Hi! Just saw your PR and was wondering when this will be merged because I'm implementing a LaTeX concealer inside Neorg and with the new Neorg parser it seems that it should be simpler to query injected LaTeX nodes, am I right?

Because right now I need to create a new parser for each inline_math and fetch all the nodes so that I can conceal them

[boltlessengineer]

Hi! Just saw your PR and was wondering when this will be merged because I'm implementing a LaTeX concealer inside Neorg and with the new Neorg parser it seems that it should be simpler to query injected LaTeX nodes, am I right?
Because right now I need to create a new parser for each inline_math and fetch all the nodes so that I can conceal them

To merge this PR, we have to complete the v3 parser first, and to do that, we should make a new spec because v1 spec currently has several issues to discuss.

[benlubas]

I'm implementing a LaTeX concealer inside Neorg

I just want to make sure that you're aware of core.renderer.latex which conceals inline math with a rendered version of the latex.

Assuming that you know about that and you want to conceal like \epsilon into the actual symbol (and similar conceals). I'll say that we already inject the latex parser into math blocks, and inline math blocks are already query-able with the current parser. So I can't imagine that your implementation would be affected too heavily by the new parser. After all, language injections are defined as queries, not by the parser.

[c3n21]

I'm implementing a LaTeX concealer inside Neorg

I just want to make sure that you're aware of core.renderer.latex which conceals inline math with a rendered version of the latex.

Assuming that you know about that and you want to conceal like \epsilon into the actual symbol (and similar conceals). I'll say that we already inject the latex parser into math blocks, and inline math blocks are already query-able with the current parser. So I can't imagine that your implementation would be affected too heavily by the new parser. After all, language injections are defined as queries, not by the parser.

Thanks for your comment, I'm already aware of that.

As a newbie of the matter I dig through the whole codebase.

But to achieve what I need to do (which is exactly what you said, concealing each command with the actual symbol) I also need to parse the Latex inside the inline_math node, am I right?

Only then I can effectively substitute each command node's text with the symbol

[max397574]

But to achieve what I need to do (which is exactly what you said, concealing each command with the actual symbol) I also need to parse the Latex inside the inline_math node, am I right?

yes I think. But for that it doesn't matter which neorg parser is used I guess
This should already be doable with the current implementation and won't be easier with v3

[flexagoon]

As someone suggested in the Discord, since the V2 parser never really existed and was just a failed experiment, wouldn't it make sense to call this the V2 parser to make it match the version of the specification?