[Library] A native Luals parser

[CppCXY]

I have completed most of the work, but there may still be many bugs in the details that I will gradually fix in future use. I will implement a new language server based on this. If you are interested, you can continue to follow my progress.

the library: https://crates.io/crates/emmylua_parser

a simple example:

let code = r#"
    local a = 1
    local b = 2
    local function foo(x)
        return x + 1
    end
    function f()
    end
    local c = foo(a + b)
    if c > 2 then
        print(c)
    elseif c == 2 then
    else
        print("c is not greater than 2")
    end
    for i = 1, 10 do
        print(i)
    end
    for i, v in ipairs({1, 2, 3}) do
        print(i, v)
    end
    while c < 10 do
        c = c + 1
    end
    repeat
        c = c - 1
    until c == 0
    ::ll::
    goto ll
    local a = c["string"]
"#;
let tree = LuaParser::parse(code, ParserConfig::default());
let chunk = tree.get_chunk_node();
for node in chunk.descendants::<LuaAst>() {
    match node {
        LuaAst::LuaChunk(lua_chunk) => {
            assert!(lua_chunk.get_block().is_some());
        }
        LuaAst::LuaBlock(lua_block) => {
            assert!(lua_block.get_stats().next().is_some());
        }
        LuaAst::LuaAssignStat(lua_assign_stat) => {
            let (var_list, expr_list) = lua_assign_stat.get_var_and_expr_list();
            assert!(var_list.len() > 0);
            assert!(expr_list.len() > 0);
        }
        LuaAst::LuaLocalStat(lua_local_stat) => {
            let mut name_list = lua_local_stat.get_local_name_list();
            let local_name = name_list.next().unwrap();
            assert!(local_name.get_name_token().is_some());
            let mut expr_list = lua_local_stat.get_value_exprs();
            assert!(expr_list.next().is_some());
        }
        LuaAst::LuaCallExprStat(lua_call_expr_stat) => {
            assert!(lua_call_expr_stat.get_call_expr().is_some());
        }
        LuaAst::LuaLabelStat(lua_label_stat) => {
            assert!(lua_label_stat.get_label_name_token().is_some());
            assert_eq!(
                lua_label_stat
                    .get_label_name_token()
                    .unwrap()
                    .get_name_text(),
                "ll"
            );
        }
        LuaAst::LuaGotoStat(lua_goto_stat) => {
            assert!(lua_goto_stat.get_label_name_token().is_some());
            assert_eq!(
                lua_goto_stat
                    .get_label_name_token()
                    .unwrap()
                    .get_name_text(),
                "ll"
            );
        }
        LuaAst::LuaDoStat(lua_do_stat) => {
            assert!(lua_do_stat.get_block().is_some());
        }
        LuaAst::LuaWhileStat(lua_while_stat) => {
            assert!(lua_while_stat.get_condition_expr().is_some());
            assert!(lua_while_stat.get_block().is_some());
        }
        LuaAst::LuaRepeatStat(lua_repeat_stat) => {
            assert!(lua_repeat_stat.get_block().is_some());
            assert!(lua_repeat_stat.get_condition_expr().is_some());
        }
        LuaAst::LuaIfStat(lua_if_stat) => {
            assert!(lua_if_stat.get_condition_expr().is_some());
            assert!(lua_if_stat.get_block().is_some());
            assert!(lua_if_stat.get_else_if_clause_list().next().is_some());
            assert!(lua_if_stat.get_else_clause().is_some());
        }
        LuaAst::LuaForStat(lua_for_stat) => {
            assert!(lua_for_stat.get_var_name().is_some());
            assert!(lua_for_stat.get_block().is_some());
            assert_eq!(lua_for_stat.get_iter_expr().count(), 2);
        }
        LuaAst::LuaForRangeStat(lua_for_range_stat) => {
            assert_eq!(lua_for_range_stat.get_var_name_list().count(), 2);
            assert!(lua_for_range_stat.get_block().is_some());
            assert!(lua_for_range_stat.get_expr_list().next().is_some());
        }
        LuaAst::LuaFuncStat(lua_func_stat) => {
            assert!(lua_func_stat.get_func_name().is_some());
            assert!(lua_func_stat.get_closure().is_some());
        }
        LuaAst::LuaLocalFuncStat(lua_local_func_stat) => {
            assert!(lua_local_func_stat.get_local_name().is_some());
            assert!(lua_local_func_stat.get_closure().is_some());
        }
        LuaAst::LuaReturnStat(lua_return_stat) => {
            assert!(lua_return_stat.get_expr_list().next().is_some());
        }
        LuaAst::LuaNameExpr(lua_name_expr) => {
            assert!(lua_name_expr.get_name_token().is_some());
        }
        LuaAst::LuaIndexExpr(lua_index_expr) => {
            assert!(lua_index_expr.get_prefix_expr().is_some());
            assert!(lua_index_expr.get_indexed_expr().is_some());
        }
        LuaAst::LuaTableExpr(lua_table_expr) => {
            assert!(lua_table_expr.is_array());
        }
        _ => {}
    }
}

[Frityet]

is there a performance comparison with the existing parser?

[CppCXY]

is there a performance comparison with the existing parser?

The libraries and design concepts used come from rust-analyzer, and it can be said to be a Lua port of rust-analyzer's parser. Although it has not been extensively tested, I can confirm that it is not too slow.

Using the Rowan library for tree construction, it can share many identical nodes, so the memory usage will not be significant

[Mc-GrowlR]

I'm more concerned about memory usage

[CppCXY]

I'm more concerned about memory usage

Don't worry, I have extensive experience. Memory usage will definitely not be an issue once it's release.

[Mc-GrowlR]

I'm more concerned about memory usage我更关心内存使用情况

Don't worry, I have extensive experience. Memory usage will definitely not be an issue once it's release.别担心，我有丰富的经验。一旦发布，内存使用绝对不会成为问题。

yes，I'd like to run it in my termux environment as my tablet has a lot of memory and I hope it performs very well.
This will be extremely friendly for environments with small memory

[sewbacca]

Is this an attempt to make LuaLS faster or to be a complete rewrite? I am unsure if replacing just the parser is a well thought through approach, as LuaLS uses AFAIK LPeg to parse Lua syntax , which is already native and fast. For a proper speed improvement, I'd recommend profiling the LuaLS to see which parts actually need optimizing. Did someone profile the language server?

[CppCXY]

Is this an attempt to make LuaLS faster or to be a complete rewrite? I am unsure if replacing just the parser is a well thought through approach, as LuaLS uses AFAIK LPeg to parse Lua syntax , which is already native and fast. For a proper speed improvement, I'd recommend profiling the LuaLS to see which parts actually need optimizing. Did someone profile the language server?

It's not just about parsing, storage efficiency is more important. I am not just trying to make LuaLS faster; I am rewriting a language server. The important work is already done, and now it's just about gradually adding features. I can share some data.

Currently, it processes very quickly and uses very little memory, approximately only 8% of LuaLS's memory usage. I will make an announcement when the language server is complete

[Mc-GrowlR]

What？ Compared to Lua, the memory usage is only 8%?Looking at my luals taking up more than 1G of memory, and now stuttering vscode. I'm looking forward to your work more and more.

…

---- Replied Message ---- | From | ***@***.***> | | Date | 12/15/2024 19:59 | | To | ***@***.***> | | Cc | ***@***.***>***@***.***> | | Subject | Re: [LuaLS/lua-language-server] [Library] A native Luals parser (Issue #2910) | Is this an attempt to make LuaLS faster or to be a complete rewrite? I am unsure if replacing just the parser is a well thought through approach, as LuaLS uses AFAIK LPeg to parse Lua syntax , which is already native and fast. For a proper speed improvement, I'd recommend profiling the LuaLS to see which parts actually need optimizing. Did someone profile the language server? It's not just about parsing, storage efficiency is more important. I am not just trying to make LuaLS faster; I am rewriting a language server. The important work is already done, and now it's just about gradually adding features. I can share some data. Currently, it processes very quickly and uses very little memory, approximately only 8% of LuaLS's memory usage. I will make an announcement when the language server is complete — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>