Run CI Daily

[DaAwesomeP]

This pull causes the CI to run everyday at 12am UTC. This should hopefully help catch issues before they end up failing in unrelated pull requests. This should definitely help catch the frequent new codespell matches.

[DaAwesomeP]

There seems to be a race condition with python/ola/ClientWrapperTest.sh. It failed on bullseye last run and then this run in sid.

Fitting that this is discovered in this pull request of all pull requests.

[DaAwesomeP]

The error:

FAIL: python/ola/ClientWrapperTest.sh
=====================================

...F/usr/lib/python3.11/unittest/case.py:622: ResourceWarning: unclosed <socket.socket fd=4, family=1, type=1, proto=0>
  with outcome.testPartExecutor(self):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
..
======================================================================
FAIL: testEventLoop (__main__.ClientWrapperTest.testEventLoop)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/ola/ola/./python/ola/ClientWrapperTest.py", line 209, in testEventLoop
    self.assertAlmostEqual(d_diff, datetime.timedelta(milliseconds=15),
AssertionError: datetime.timedelta(microseconds=17990) != datetime.timedelta(microseconds=15000) within datetime.timedelta(microseconds=750) delta (datetime.timedelta(microseconds=2990) difference)

----------------------------------------------------------------------
Ran 6 tests in 0.021s

FAILED (failures=1)
FAIL python/ola/ClientWrapperTest.sh (exit status: 1)

[peternewman]

Yeah this has unfortunately been kicking around for a while, it's supposed to respond within a 750 uS window and its taken 2990.

microseconds=750) delta (datetime.timedelta(microseconds=2990

ola/python/ola/ClientWrapperTest.py

Lines 191 to 210 in f82bbc5

	# Check when the callbacks were called. Allow 500 microseconds of drift.
	# Called immediately
	a_diff = results.a_called - self.start
	self.assertAlmostEqual(a_diff, datetime.timedelta(milliseconds=0),
	delta=datetime.timedelta(microseconds=750))
	# Called in 5 milliseconds
	b_diff = results.b_called - self.start
	self.assertAlmostEqual(b_diff, datetime.timedelta(milliseconds=5),
	delta=datetime.timedelta(microseconds=750))
	# Called in 10 milliseconds
	c_diff = results.c_called - self.start
	self.assertAlmostEqual(c_diff, datetime.timedelta(milliseconds=10),
	delta=datetime.timedelta(microseconds=750))
	# Called in 15 milliseconds
	d_diff = results.d_called - self.start
	self.assertAlmostEqual(d_diff, datetime.timedelta(milliseconds=15),
	delta=datetime.timedelta(microseconds=750))

Although now writing this I'm slightly suspicious that's its micro and I wonder/assumed it was supposed to be milli (without rechecking the code). Having checked micro makes sense, given we're asking for tens of milliseconds for the window. It does inherently need to be fairly tight unfortunately, due to the nature of the check it's running.

But either way its definitely unfortunately worse on our various VMs/images.

[peternewman]

I can see the general sense, I wonder if we should stagger things a bit though, I guess it rather depends on the frequency of PRs coming in, but is daily an appropriate frequency for doing all the full builds? I wonder if they should be less often given only external package changes can break them.

I'm more onboard with doing the spelling CI regularly given as you say that will often get tripped up by external factors.

The other thing is unless these raise an issue or something, will anyone actually spot the CI issues?

[DaAwesomeP]

I can see the general sense, I wonder if we should stagger things a bit though, I guess it rather depends on the frequency of PRs coming in, but is daily an appropriate frequency for doing all the full builds? I wonder if they should be less often given only external package changes can break them.

My thinking is that daily/nightly allows for us to find those external breaking factors before they make it into an unrelated PR. We could switch to weekly if you would prefer. We could run lint daily and builds weekly.

I'm more onboard with doing the spelling CI regularly given as you say that will often get tripped up by external factors.

I think running all of it on some frequency is nice because you find changes that you really aren't expecting (i.e. maybe Debian changes how Python libraries are installed or Autoconf behavior changes in a new version). Importantly, it's much nicer to find these changes one at a time and not many at once after a longer period of inactivity on the repo.

The other thing is unless these raise an issue or something, will anyone actually spot the CI issues?

IIRC repo watchers will get a notification that a run failed.