Consider dfns curation process

[tidoust]

Via w3c/reffy#1110

We don't have a curation mechanism in place right now for dfns because the intent was to propagate updates as soon as new crawl results are available to cross-references databases, and CSS/IDL data curation requires manual fixing on a regular basis. To prevent stalls, we'd need a way to have dfns curation continue even when CSS/IDL curation fails. That could perhaps be achieved simply by running the curation separately in yet another branch.

Dfns curation would also be useful to detect and prevent situations where crawl drops a large number of definitions by mistake, e.g. because the crawl succeeds and yet the crawled spec is temporarily broken to the point where no dfns can be extracted (something similar happened in Shepherd/Bikeshed recently for instance).

This would also allow us to create patches if that seems needed. We managed to survive without that for dfns until now though.

[darobin]

If this issue one that would lead to the release of a @webref/dfns? If not I can file separately.

I'm using the definitions for a project but they're not released separately so I have to use github:w3c/webref as a dependency. Unfortunately, that fails when installed as a dependency (the prepare script runs before the dependencies are installed, which shouldn't happen but I don't understand what's happening).

Anyway, it would be neat to be able to reference prose definitions!

[tidoust]

This issue is certainly a prerequisite to the possible release of a @webref/dfns package. We did not necessarily have that in mind until now though ;)

For other @webref/xxx packages, we review diffs manually and release new versions of the packages once per week on average (unless nothing changed, of course). In other words, data in the packages may be one or two weeks late. Would that be enough for your use case?

I'm asking because authoring tools typically want fresh data for their cross-reference database and would probably continue to use the repository contents directly even if we release a @webref/dfns package.

Unfortunately, that fails when installed as a dependency (the prepare script runs before the dependencies are installed, which shouldn't happen but I don't understand what's happening).

Ah, I don't understand what's happening either, but I'll have a look. Maybe there's a simple fix (or maybe we don't need to have that script run as a prepare script).

[darobin]

Yes, I would be fine with the data being a week or two late. Most of what we rely on is pretty foundational stuff that's not changing all the time. I tried to use the online data but there was too much latency and that caused intermittent failures.

Ah, I don't understand what's happening either, but I'll have a look. Maybe there's a simple fix (or maybe we don't need to have that script run as a prepare script).

To give some more details: if I npm install --save github:w3c/webref as a dependency, it'll just work. But then if I install the npm package of the tool I'm making elsewhere (or globally) it fails. For some reason, in that case the prepare script runs before the devDependencies have been installed and so the script fails because it can't find rimraf. I tried a bunch of things like switching to dependencies but it changes nothing. The npm docs say that that's not the normal behaviour. I wonder if it's a weird thing where for some reason the node called by the prepare script isn't the right one but I got tired of hunting that down as I need to ship something so I forked this repo and made a version without prepare 😬. Let me know if I can help hunt this down.

[tidoust]

I wonder if it's a weird thing where for some reason the node called by the prepare script isn't the right one but I got tired of hunting that down as I need to ship something so I forked this repo and made a version without prepare 😬

I don't know what happens, but I don't think the prepare script in webref should run automatically on installs in any case, for reasons mentioned in #914. So I dropped it as well (or rather renamed it to "curate").

A @webref/dfns package would still be better for you in the sense that having webref as a dependency means downloading ~80MB instead of "just" ~25MB of definitions, but at least this should not fail.