[BUG] watch fails to watch dependency files in large go packages

[asfaltboy]

NeoVim Version

NVIM v0.10.3
Build type: RelWithDebInfo
LuaJIT 2.1.1731601260
Run "nvim -V1 -v" for more info

Describe the bug

On larger go projects, the watch functionality is very slow and borderline unusable, as a trigger for re-running tests in larger packages. See more details in context section below.

To Reproduce

Unfortunately, I struggle to get 'watch' feature working with dependency files using a minimal config (it only watches for changes in the actual test file), though the dependency files are watched when I use the LazyVim distro config (here). I think it has something to do with my LSP client config, would appreciate a minimal config example for supporting watch (it doesn't seem to work out of the box without LSP client/server configuration).

Here's my minimal config:

vim.opt.runtimepath:remove(vim.fn.expand("~/.config/nvim"))
vim.opt.packpath:remove(vim.fn.expand("~/.local/share/nvim/site"))

local lazypath = "/tmp/lazy/lazy.nvim"

if not (vim.uv or vim.loop).fs_stat(lazypath) then
  vim.fn.system({
    "git",
    "clone",
    "--filter=blob:none",
    "https://github.com/folke/lazy.nvim.git",
    "--branch=stable", -- latest stable release
    lazypath,
  })
end

vim.opt.rtp:prepend(lazypath)

require("lazy").setup({
  "nvim-neotest/neotest",
  dependencies = {
    "nvim-neotest/nvim-nio",
    "nvim-lua/plenary.nvim",
    "antoinemadec/FixCursorHold.nvim",
    "nvim-treesitter/nvim-treesitter",
    "neovim/nvim-lspconfig",
    -- Install adapters here
    "fredrikaverpil/neotest-golang",
  },
  config = function()
    -- Install any required parsers
    require("nvim-treesitter.configs").setup({
      ensure_installed = { "go" },
    })

    require("neotest").setup({
      -- Add adapters to the list
      adapters = {
        require("neotest-golang"),
      },
      log_level = "debug",
    })

    require("lspconfig").gopls.setup({
      staticcheck = true,
      directoryFilters = { "-.git" },
    })
  end,
})

I was able to reproduce this issue with the full config, within the traefik repo, with one package very slow (api) and another package not so slow (udp).

Steps to reproduce the behavior:

1. Clone the traefik/traefik repo, e.g with gh cli: gh repo clone traefik/traefik
2. Run 'Neotest summary' and wait for it to complete (took 17s for a larger file)
3. Pick a large package test (handler_entrypoint_test.go), and attempt to watch the file or a test case.
4. See no watch is active (or with minimal config, watch only active for the test file itself, and not deps)
5. See around ~400 calls to symbol definitions, taking over 1,420 seconds.
6. Watch does not seem to trigger test upon changing the implementation file (with a large package).

Repeat the test with a smaller file (conn_test.go) to see it working mostly fine.

Expected behavior

Watch should run quick, even for larger projects, and result in working listener within a few seconds at most.

Logs

I've attached a few examples, reproducing with minimal config, and a full config for two files: pkg/udp/conn_test.go and pkg/api/handler_entrypoint_test.go (the implementation is in the file of the same name sans-test suffix).

• entrypoint_test_big.log
• entrypoint_test_min.log
• conn_test_big.log
• conn_test_min.log

Look for Built dependencies in X ms in the files to find these results.

Additional context

Looking at the logs, the function slowness seems related to the _build_dependencies function running many times for every linked file.

I noticed this issue first when working on a work/private Golang project, so couldn't share details until now. But, I was able to reproduce on OSS project - traefik. It's possibly limited to Golang, though I haven't tried reproducing it on a larger project in another language.

Note: I've been trying to debug the issue with watch for a while, but I'm a bit blocked by lack of understanding of how this works. I would appreciate a bit more guidance on the inner-workings / architecture of this feature, so that I can add more debug statements, or re-produce on a smaller project, and better understand where is the bottleneck is.

[Su3h7aM]

Regarding the issue of the watcher not detecting changes in dependencies, it seems to be a regression.
I was racking my brains over this until I decided to take a look at the commit history and noticed recent changes in the watcher part (#479). For me, in version 5.6.1, the Watcher works with dependencies.

I tried to reproduce the problem with large packages, and for me, in both versions (5.8.0 and 5.6.1), I had the same result as you.