Switch from testing Edge 17 to 18

[foolip]

Edge 44.17763.1.0 / EdgeHTML 18.17763 is available on Sauce Labs.

This is the latest stable version of Edge, and testing it will help validate
the setup for Edge on Azure Pipelines:
web-platform-tests/wpt#14836

[foolip]

@jugglinmike I'm not sure if there's anything that can be tested about this, but I presume that it'll require restarting the Buildbot master?

[jugglinmike]

This looks good to me, but I'd like to wait on merging for a few days. We have an open issue about an Edge-specific failure. While my latest change seems to have fixed it, it's not conclusive. I'd like to see a few more successes before calling the problem resolved. With that, I'll be more comfortable making further changes to the configuration.

It looks like Azure is configured to segment WPT into 20 "chunks," while this project uses 100. For the purposes of comparison, differences in test suite segmentation adds noise to the data. Although my offer to change the Safari configuration stands, I'm less inclined to do the same for Edge. Even a segment as small as 1% of WPT requires a significant amount of time in Edge via Sauce Labs (as long as three hours). With larger "chunks," intermittent failures will be much more costly, and we may not be able to recover.

[foolip]

We could use more chunks in Azure Pipelines instead perhaps.

[foolip]

@jugglinmike, if I do 100 chunks in Azure Pipelines, is this the setup I should replicate?

results-collection/src/master/wpt_run_step.py

Lines 34 to 46 in 43658dd

	def makeWptRunCommand(properties):
	browser_id = None
	browser_name = properties.getProperty('browser_name')
	command = [
	'run-and-verify.py',
	'--max-attempts', properties.getProperty('max_attempts'),
	'--log-wptreport', properties.getProperty('log_wptreport'),
	'--log-raw', properties.getProperty('log_raw'),
	'--',
	'--log-mach', '-',
	'--this-chunk', properties.getProperty('this_chunk'),
	'--total-chunks', properties.getProperty('total_chunks')
	]

That is, no special chunking?

[foolip]

https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=5841 is an attempted run with 100 jobs and the default chunk type. I expect it'll fail in some mysterious way, but we'll see.

[jugglinmike]

@foolip see also the arguments we set for Sauce-Labs-mediated browsers:

results-collection/src/master/wpt_run_step.py

Lines 71 to 81 in 43658dd

	command.extend([
	'--sauce-platform', sauce_platform_id,
	'--sauce-user', 'wpt-%s' % workername,
	'--sauce-key', key,
	'--sauce-tunnel-id', properties.getProperty('workername'),
	'--sauce-connect-binary', 'sc',
	'--sauce-connect-arg=--logfile=/var/log/sauce-connect/sc.log',
	'--sauce-init-timeout', '45',
	'--no-restart-on-unexpected',
	'--run-by-dir', '3'
	])

Most of those are irrelevant for Azure Pipelines, but that patch is missing --run-by-dir. I honestly don't know how that interacts with the "chunking" heuristic (it's a setting we inherited from the original automation script), so I'd just as soon remove it from this project.

Meanwhile, that intermittent bug is still biting us, so I'm going to continue research using Edge 17.

[jgraham]

--run-by-dir operates after chunking, so it doesn't make much sense if you're chunking by hash of full path. But it is intended to work well with the default directory-hash based chunking.

[foolip]

Trying again with --run-by-dir 3 in https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=5920

[jugglinmike]

This necessitated restarting the Buildbot master, so I've done that.