Bundling Dependencies

[bennypowers]

TL;DR:

dev server is buildless, so we load every module in your app individually, which lets us rely on built-in browser goodies like caching, but if you have a complicated SPA with lots of node_modules, that means the waterfall can get both very tall and very deep (tall meaning your code imports lots of dependencies, deep meaning those dependencies each kick off long chains of importing-other-modules-which-also-import-other-modules-which-). Now, even though the browser caches all those so it takes 0ms to load each module, it still takes some time for the browser to traverse the waterfall as a whole. One solution we are mulling over is whether or not to sneakily bundle your node_modules behind the scenes, what to make the waterfall shallower

@davidclark87 From Adobe mentioned recently on the Polymer slack an interest in automatically bundling dependencies in order to reduce waterfall times when using @web/dev-server. This reignited a previous discussion of the feature among @LarsDenBakker @daKmoR, @Westbrook and myself. @justinfagnani also expressed interest in such a tool. I'll try to summarize the points raised here in this post, and we can continue the discussion here:

Problem

In large apps, the request waterfall can get exceedingly deep for dependencies, and this severely impacts page reload performance even when those dependencies are cached by the browser. In some cases it can get to the point where the benefits of buildless development vs bundled development become questionable.

Straw Solution

Bundle dependencies in a separate thread to reduce the waterfall depth.

Preferably, this would also still enable users to edit node_modules/**/*.* in-place (i.e. a patch-package workflow and see their changes reflected on the reloaded page.

We'd also prefer to use (and/or contribute to) a lower-level 'dependency bundler' that could be shared among the different toolchains in the broader JS and web ecosystem, e.g. https://www.npmjs.com/package/esinstall which powers snowpack.

0. exclude node_modules from the watcher
1. setting a watcher on node_modules to run a build on deps (perhaps taking the app's module graph into account)
2. proxy those artifacts to node_modules in wds
3. set a watcher on the artifacts to reload the page

Alternative Solutions

HMR might address the problem of development cycle time (i.e. the time it takes to observe a code change after hitting "save"). HMR is not appropriate to every project though and also isn't bulletproof, the occasional page reload is still advised.

[simi]

@bennypowers would it be possible to share some example repo facing the problem this proposal is trying to solve?

[bennypowers]

I can't on account of NDA, and I suspect anyone facing this issue would say the same thing, as this problem will tend to affect larger codebases for commercial dashboard SPA and such.

But I can show you this devtools network panel screenshot, and note the timings

[stefanholzapfel]

@bennypowers isn't vite already addressing this? https://vitejs.dev/guide/dep-pre-bundling.html#the-why

[bennypowers]

Vite is a tool based on the concepts first expressed in es-dev-server, and it's focus is primarily on legacy frameworks. We're aiming for something that will be useful to a wider community. Moreover, this whole thread is about implementing a plugin, not an all-in-one solution

[stefanholzapfel]

Afaik they are using esbuild for the prebundling (https://twitter.com/youyuxi/status/1352359127897550851), but you mean you want a plugin that is independent from any 3rd party tools and work in any "bundler chain"?

[bennypowers]

That could be one approach. The purpose of the thread is to collect comments like yours on his best to accomplish the goal of faster reloads