Parser Plugins

[SomeRanDev]

Understandable if this does not get added, but wanted to try and make it happen.

This pull request adds the ability for Haxe compiler plugins to modify the behavior of the parser. This is achieved by:

• Adding hooks to the parser at certain points to "override" the result of some functions.
• Adding the --load-plugin argument to load plugins immediately before the parsing phase.

In addition, a new example plugin has been added. It acts as a small tutorial by for creating parser plugins by implementing the pipe forward operator |> in Haxe. It can be found in plugins/parser_hooks.

---

Available Hooks

These represent lists stored at Parser.parser_hooks

• on_expr : Parser.token_stream -> (Ast.expr option)
  Hooks at the beginning of the expression parsing function.
• on_expr_next : (Parser.token_stream * Ast.expr) -> (Ast.expr option)
  Hooks at the beginning of "expr_next".
• on_expr_expected : Parser.token_stream -> (Ast.expr option)
  Hooks after an expression attempted to be parsed but failed. Can return an expression to prevent "Expected expression" error,
• on_type_decl : (Parser.token_stream * Parser.type_decl_completion_mode) -> (Ast.type_decl option)
  Hooks at the start of the type declaration parsing function.
• on_class_field : (Parser.token_stream * bool) -> (Ast.class_field option)
  Hooks at the start of the class field parsing function.

Motivation

In general, this adds a lot to the existing plugin system, giving it the power it deserves.

Firstly, this change offers a way for full-project expression scanning and replacing with little performance drawbacks. Libraries that require global @:build macros to support features could be optimized through native implementation with a plugin. Furthermore, at least from this version forward, users with personal projects on older versions of Haxe could backport certain new features without having to deal with other internal incompatibilities. (Or unreleased features can be ported, like how I really want to use ?. but 4.3 hasn't released yet ;p)

As demonstrated with the example plugin, this also allows Haxe programmers to make modular changes to their Haxe syntax (within the limitations of available Lexer tokens). While maybe not the ultimate solution, it provides at least one solution to DSLs in Haxe. A plugin could validate a DSL's structure, using error reporting functions at appropriate positions if necessary. With smart retention of positional data, the generated expression can also report back to the right spots for typing issues. Internal functions for parsing an expression can also be used within the DSL parse. All while retaining editor support.

Drawbacks(?)

Of course, code is read more than it is written, so allowing anyone to just modify the Haxe syntax for their project could be seen as detrimental. However, I believe writing a compiler plugin creates a barrier of entry that would prevent it from becoming prevalent. It wouldn't be much different from someone just creating their own fork of Haxe for their project, which can already be done.

Not to mention, plugins must be recompiled and maintained for new versions of Haxe, so that creates an additional barrier that prevents misuse. Obsolescent or unmaintained plugins inherently would not work.

Finally, --load-plugin provides an explicit, clean method of checking for parser plugin usage. Due to the fact it requires a local path to the project, parser plugins cannot be automatically installed in libraries through extraParams.hxml. So any distributed plugins likely will require instructions for installing, thus creating a clear association with the plugin and the new syntax.

---

All in all, I believe parser plugins are a nice way to resolve some niche issues without any impact on existing Haxe syntax, so I hope it will be considered! Thank you for reading and feedback is appreciated.

[Simn]

This is quite interesting from a technical standpoint, but yeah, I don't think we want this in the official release. Mostly due to the first drawback you mention, there's a difference between forking a codebase completely and utilizing an official tool feature. And I don't like when the latter causes syntax fragmentation...

As for your implementation, one thing to consider is the compilation server. If I'm reading this correctly, subsequent compilations would fill up these hooks lists because that aren't reset. In fact, I think you can even reproduce this with a --load-plugin --next compilation, where the second run should be affected by the --load-plugin of the first one.

[SomeRanDev]

Ahhh, I knew I had to be missing something. XD

Anyway, understandable! Thank you for the feedback. I'll try and think of other methods cause definitely would love to see some conclusion to dsl in haxe. But guess it's not that easy. Xd