Neovim badly interacts with server when position_encoding is set to "utf-8"

[fit-sizhe]

Describe the bug
When typing Chinese in neovim, syntax highlighting is off, and unclosed delimiter errors are everywhere

Package/Software version:
tinymist version: v0.12.0, installed through mason.nvim
neovim version:

NVIM v0.10.2
Build type: RelWithDebInfo
LuaJIT 2.1.1713484068

[fit-sizhe]

my lsp settings for tinymist is mentioned in #761.

[oakrotka]

I'm also experiencing this with polish characters. My lsp settings:

lspconfig.tinymist.setup {
    on_attach = on_attach,
    capabilities = capabilities,
    offset_encoding = "utf-8",
    settings = {
      semantic_tokens = "disable",
    }
}

[random-username-here]

Got a similar error when using cyrrillic letters in Neovim.

The error can be reproduced like this:

1. Open a file
2. Type **
3. Move cursor to the middle, write аа (cyrillic letter A two times)
4. Get that "unclosed delimiter" error

When looking through the logs, I see this:

[2024-11-09T15:38:57Z INFO  sync_lsp] notifing textDocument/didChange succeeded in 152.43µs
[2024-11-09T15:38:57Z INFO  tinymist::actor::editor] received status request(primary) Compiling
error: unclosed delimiter
  ┌─ /home/i-s-d/homework/test.typ:1:0
  │
1 │ *а*а
  │ ^
[2024-11-09T15:38:57Z INFO  tinymist::actor::editor] received diagnostics from primary:9: diag(Some(1))
2024-11-09T15:38:57Z INFO  tinymist::actor::editor] received status request(primary) CompileError
[2024-11-09T15:38:57Z INFO  tinymist::task::cache] CacheEvictTask: evict cache in 8.48µs

Which probably means there is something wrong with incremental document changes.

If I type аааа instead of аа, it also has that error, but it also panics:

[ERROR][2024-11-09 18:38:57] .../vim/lsp/rpc.lua:770    "rpc"   "/usr/bin/tinymist"     "stderr"
  [2024-11-09T15:38:57Z INFO  sync_lsp] notifying textDocument/didChange - at Instant { tv_sec: 27190, tv_nsec: 960419617 }
  thread 'main' panicked at crates/tinymist/src/server.rs:962:26:
  invalid range
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[ERROR][2024-11-09 18:39:04] .../vim/lsp/rpc.lua:770    "rpc"   "/usr/bin/tinymist"     "stderr"
  [2024-11-09T15:39:04Z INFO  sync_lsp] handling shutdown - (7) at Instant { tv_sec: 27197, tv_nsec: 433124982 }
  [2024-11-09T15:39:04Z INFO  sync_lsp] handled  shutdown - (7) in 51.50µs

(I formatted the LSP log a little bit, because it joined multiple server logs into LSP client log)

My current LSP config:

lspconfig.tinymist.setup {
  capabilities = capabilities,
  offset_encoding = "utf-8",
  setup = {
    cmd = { 'tinymist' }
  },
  settings = {
    semantic_tokens = "disable",
  }
}

I tried setting RUST_BACKTRACE=1 in the cmd, but seems like it is ignored somehow.

Removing offset_encoding = "utf-8" fixes this, but results in errors from semantic token handler in Neovim, (which can be disabled with autocmd on LspAttach and setting client.server_capabilities.semanticTokensProvider = nil).

So, my guess is what those Chinese characters take two utf16 codepoints, and the deal with them is the same with Cyrillic/Polish characters, which take two utf8 codepoints. Maybe try setting offset_encoding to utf32 (or similar property, if you use something other than lspconfig). It worked for me.

But in that case out-of-range array access errors from /usr/share/nvim/runtime/lua/vim/lsp/semantic_tokens.lua will happen, which are not directly related to this issue. To test the changes, you can disable semantic tokens like this:

vim.api.nvim_create_autocmd("LspAttach", {
  callback = function(args)
    local client = vim.lsp.get_client_by_id(args.data.client_id)
    client.server_capabilities.semanticTokensProvider = nil
  end,
});

tinymist --version:

tinymist
Build Timestamp:     2024-10-19T17:10:46.837035115Z
Build Git Describe:  v0.12.0
Commit SHA:          4437b3bffb5f01f06bd64e4e424480ba72a88619
Commit Date:         None
Commit Branch:       None
Cargo Target Triple: x86_64-unknown-linux-gnu
Typst Version:       0.12.0
Typst Source:        git+https://github.com/Myriad-Dreamin/typst.git?tag=tinymist-v0.12.0#58426a90a7ef721738a01be09793d33e55eb75a9

[Eric-Song-Nop]

A temporary solution might be to upgrade or downgrade your Neovim, I personally think supporting only utf-16 from Tinymist is better.

[fit-sizhe]

@Eric-Song-Nop I don't think upgrading will help much, as my neovim version is the latest.

[Eric-Song-Nop]

The requirement for utf-8 offset_encoding should be fixed in neovim/neovim#30747 , after that pr I hope that not using utf-8 offset_encoding should be enough.

[fit-sizhe]

@Eric-Song-Nop Nope, removing "offset_encoding" causes the issue #761.

[Myriad-Dreamin]

I wouldn't like to make an assertion but there should be some bug(s) on neovim and a way without having to change tinymist to make semantic highlighting perfect on neovim, because we haven't received any confused semantic highlighted errors on VS Code for months.

I also add another recent finding about it, hinting there might be inconsistent uses of position encoding on some distros of neovim, which causes desync state between LSP server (tinymist) and clients (neovim). This happens even if tinymist and neovim have negotiated a position encoding: folke/trouble.nvim#606

More information is needed as I'm not a proficient neovim user.