Shutdown never finishes when background tasks are cancelled but need some time to finish

[aldem]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

I have an app that has background task, and this background task needs some time to finish once shutdown (initiated by SIGINT) is requested. Unfortunately, this seldom works as expected - often it never finishes and gets stuck in shutdown_tasks().

The are couple problems that I recognize but could not find a solution yet. The part of the code that is handling shutdown:

sanic/sanic/app.py

Lines 1910 to 1918 in acb29c9

	if timeout is None:
	timeout = self.config.GRACEFUL_SHUTDOWN_TIMEOUT
	while len(self._task_registry) and timeout:
	with suppress(RuntimeError):
	running_loop = get_running_loop()
	running_loop.run_until_complete(asyncio.sleep(increment))
	self.purge_tasks()
	timeout -= increment

• by default increment = 0.1, eventually it becomes negative due to imprecise nature of floats, thus loop will never finish as long as there are running tasks. Actually there is no "safe" value for an increment as long as either timeout or increment are not integers.
• there is always running (background) task as asyncio loop is killed at this point and the while loop is in eternal cycle (and get_running_loop() produces an exception), so it eats 100% CPU as well.

Since background_task() is perfectly terminates by SIGINT when run directly using asyncio.run(background_task()) (after cancellation), I believe the problem is in Sanic.

The following snippet could be used to reproduce the problem (I ran it on Debian 12, Python 3.11):

Code snippet

import asyncio
import signal

from sanic import Sanic

app = Sanic("sanic-bug")
app.config.GRACEFUL_SHUTDOWN_TIMEOUT = 5

bg_task: asyncio.Task | None = None

async def background_task():
    print("Background task running")

    try:
        print("Sleep one")
        await asyncio.sleep(30)
    except asyncio.CancelledError:
        # This get executed when Ctrl-C is hit as task is cancelled
        print("Sleep one cancelled")

    try:
        print(f"Sleep two {asyncio.get_running_loop()}")
        # This never finishes as long as timeout is > 0
        # and Sanic.shutdown_tasks() is in eternal loop,
        await asyncio.sleep(5)
    except asyncio.CancelledError:
        print("Sleep two cancelled")
        asyncio.current_task().uncancel()
    except BaseException as e:
        print(f"Oops: {e!r}")
    print("Background task finished")
    return

@app.before_server_start
async def prepare_start(*args):
    print("Starting background task")
    _ = app.add_task(background_task(), name="background-task", register=True)
    print("Started background task")


if __name__ == "__main__":
    app.run(host="0.0.0.0", single_process=True)

The output:

Starting background task
Started background task
Background task running
Sleep one
[2024-01-19 13:48:59 +0100] [857208] [INFO] Starting worker [857208]
^CSleep one cancelled
Sleep two <uvloop.Loop running=True closed=False debug=False>
[2024-01-19 13:49:02 +0100] [857208] [INFO] Stopping worker [857208]

At this point it hangs forever. This also affect multi-process mode.

Expected Behavior

I would expect that asyncio loop is not touched until and unless all background tasks are finished or when GRACEFUL_SHUTDOWN_TIMEOUT expires, in the latter case it could be forcibly killed.

How do you run Sanic?

As a script (app.run or Sanic.serve)

Operating System

Linux

Sanic Version

23.12.1

Additional context

No response