Process for keeping BCD in sync with test (collector) results

[foolip]

What would you like to see added to BCD?

OWD has a project to Fix errors in BCD for API, CSS and JS going back to at least 2020 , which means making BCD match the results of https://github.com/GooborgStudios/mdn-bcd-collector.

When that is done, there will be a need to keep the data in sync over time.

The most basic form this could take is to regularly apply all changes that the update-bcd script would make to BCD, and treat that as a burndown list. For each change, either review it and merge into BCD, or fix the test if the results are wrong.

I also have an a more elaborate setup in mind that I think would spread the load between BCD contributors/reviewers better. In short, it would be automatic PR creation, and low-overhead ways of flagging certain changes incorrect/unwanted. In more detail:

• The collector automatically gets test results for all post-2020 browser releases every release (@queengooborg already has this set up)
• For every new round of results, the collector automatically opens PRs with changes. It probably makes sense to group changes by which browser+release is affected. That script has to also be able to look at already open PRs and avoid including the same changes.
• When changes are wrong, there's a low-overhead way of adding them to an ignorelist, perhaps by adding a comment on a specific issue.

It would still be a bunch of work to work through that ignorelist and fix tests, etc.

How impactful do you think this enhancement will be?

The impact is twofold.

First, we'd ensure that BCD stays accurate to a high degree. This is of course useful at the granular level, but importantly it makes it much more tractable to do work like web-platform-dx/web-features#40, where multiple BCD entries are combined. The reason is that a 1% error rate of individual entries compounds to a 9.6% error rate when combining 10 entries, making the resulting data just not reliable enough.

Second, ~85% of API, CSS and JS entries are currently covered by the collector, and that number can be pushed up. If we have changes for 85%+ of the data data being submitted automatically, including new entries, then the number of PRs that have to be researched and created by humans should be significantly reduced.

Do you have anything more you want to share?

No response

[foolip]

@Elchi3 here's the issue I promised to file.

@ddbeck FYI

[queengooborg]

Since we have openwebdocs/project#168 to track this as an Open Web Docs project, should we close this issue?

[Elchi3]

Agreed