Add mdn-bcd-collector update script

[mzgoddard]

Summary

Add update script from https://github.com/GooborgStudios/mdn-bcd-collector. Add needed utilities (logger and ua-parser) and npm dependencies.

Test results and supporting details

I ran the update script locally with https://github.com/GooborgStudios/mdn-bcd-collector and https://github.com/GooborgStudios/mdn-bcd-results/tree/3af0b0ad1e7fd53098f266c80387e359288e21ad.

Running once completes without throwing an error.
Running a second time throws an error.

Error

~/browser-compat-data/node_modules/compare-versions/lib/esm/index.js:120
        throw new Error(`Invalid argument not valid semver ('${version}' received)`);
              ^
Error: Invalid argument not valid semver ('13.1> 14.1' received)
    at validateAndParse (~/browser-compat-data/node_modules/compare-versions/lib/esm/index.js:120:15)
    at compareVersions (~/browser-compat-data/node_modules/compare-versions/lib/esm/index.js:10:16)
    at compareVersions (~/browser-compat-data/node_modules/compare-versions/lib/esm/index.js:64:17)
    at update (file://~/browser-compat-data/scripts/update.ts:576:11)
    at main (file://~/browser-compat-data/scripts/update.ts:686:22)
    at async file://~/browser-compat-data/scripts/update.ts:746:3

I made a number of changes to the script:

• Add constants from mdn-bcd-collector/lib/constants.ts to update script
• Add relevant types from mdn-bcd-collector/types/types.ts to update script
• Replace Array<string, string, string, TestResultValue> in Overrides with [string, string, string, TestResultValue]
• Add type definition ManualOverride = [string, string, string, TestResultValue]
• Type Array.isArray in overrides.filter as (item: unknown) => item is ManualOverride
• Add MDN_COLLECTOR_DIR for loading overrides file
• Changed overrides file path to path.join(MDN_COLLECTOR_DIR, 'custom/overrides.json'),

Dependencies added to package:

• fs-extra
• klaw
• ua-parser-js
• winston
• @google-cloud/logging-winston

Related issues

#19868
#18514

[mzgoddard]

After speaking with @foolip I've made this as a first step in reducing tech debt in the update script by moving it into browser-compat-data. I tried to keep the number of changes low to the script itself low.

There will be a lot of steps to do after merging this PR that we can list in #19868 or its own issue.

[foolip]

I think this is only used for fs.readJson(). Can you get rid of it and just use plain fs and JSON.parse() like other existing code in BCD?

[mzgoddard]

As part of this I'm going to make reading the json files serial like what is done in index.ts. update at the moment reads the json in parallel. Since it is as of this time 1455 json files, some systems may run in EMFILE or ENFILE errors opening that many files at one time.

[foolip]

Makes sense, I guess @queengooborg and I never ran into that.

[foolip]

@mzgoddard the fs-extra dependency is still here. Can it be removed now?

[github-actions]

This pull request has merge conflicts that must be resolved before it can be merged.

[mzgoddard]

I resolved conflicts and tested this script vs the one in GooborgStudios/mdn-bcd-collector.

There were no differences between the changes both scripts made.

Here is a gist with the diffs of each: https://gist.github.com/mzgoddard/b1ff3687630db9c6cf05579fb8413674

[foolip]

Brilliant, thank you @mzgoddard! I'll go ahead and merge this before there are more conflicts.

[foolip]

Oh no, the Node LTS CI job is failing. @mzgoddard can you check if it's caused by these changes?

[mzgoddard]

@foolip I checked out the CI job. Its an error from swc during generate-types.

@swc/core threw an error when attempting to validate swc compiler options.

I don't think its related to the change in the PR. I think its a flaky error. If you re-run the job I think it will pass. (I don't have the permission needed to re-run.)

[foolip]

@mzgoddard I re-ran the job but it's still failing. Unless it's also failing in other recent PRs I guess it was caused by the changes here?

[github-actions]

This pull request has merge conflicts that must be resolved before it can be merged.

[queengooborg]

To help reduce potential for merge conflicts, we may want to split this down into even smaller chunks. For example, the UA parser I think could be its own PR (and I have some change requests regarding it specifically that would help turn it into a useful utility).

[mzgoddard]

I found the cause of the issue in CI testing. I made a mistake on one of the package-lock.json merges and it was persisting an older @swc/core dependency. I've checked out package-lock.json from main and reinstalled the new dependencies to resolve this.

[mzgoddard]

To help reduce potential for merge conflicts, we may want to split this down into even smaller chunks. For example, the UA parser I think could be its own PR (and I have some change requests regarding it specifically that would help turn it into a useful utility).

I'll make a PR with just UA parser and its test file. (That may not reduce merge conflicts by much since UA parser has its own dependency, ua-parser-js and the merge conflicts are happening in package.json and package-lock.json.)

[github-actions]

This pull request has merge conflicts that must be resolved before it can be merged.

[foolip]

Thank you @mzgoddard for sorting out the CI issues and sending #20210 separately. I'll go ahead and merge this now.