From fab1c67cf0c3e3452e794c1be355f20e25d42b8b Mon Sep 17 00:00:00 2001 From: "Nathaniel J. Smith" Date: Sun, 22 Jul 2018 23:40:56 -0700 Subject: [PATCH 01/12] First draft of manual --- docs/source/index.rst | 42 ++++ docs/source/quickstart.rst | 389 +++++++++++++++++++++++++++++++++++++ docs/source/reference.rst | 176 +++++++++++++++++ 3 files changed, 607 insertions(+) create mode 100644 docs/source/quickstart.rst create mode 100644 docs/source/reference.rst diff --git a/docs/source/index.rst b/docs/source/index.rst index 118c45b..1b7af1a 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -8,9 +8,51 @@ pytest-trio: Pytest plugin for trio =================================== +This is a pytest plugin to help you test projects that use `Trio +`__, a friendly library for concurrency +and async I/O in Python. Features include: + +* Async tests without the boilerplate: just write ``async def + test_whatever(): ...``. + +* Useful fixtures included: use ``autojump_clock`` for easy testing of + code with timeouts. + +* Write your own async fixtures: start up a Trio-based server inside a + fixture, and then connect to it from your test. + +* Use `Hypothesis `__? This plugin + integrates with Hypothesis, so your async tests can use + property-based testing. + + +Vital statistics +================ + +* Install: ``pip install pytest-trio`` + +* Documentation: https://pytest-trio.readthedocs.io + +* Issue tracker, source code: https://github.com/python-trio/pytest-trio + +* License: MIT or Apache 2, your choice + +* Contributor guide: https://trio.readthedocs.io/en/latest/contributing.html + +* Code of conduct: Contributors are requested to follow our `code of + conduct + `__ in + all project spaces. + .. toctree:: :maxdepth: 2 + quickstart.rst + reference.rst + +.. toctree:: + :maxdepth: 1 + history.rst ==================== diff --git a/docs/source/quickstart.rst b/docs/source/quickstart.rst new file mode 100644 index 0000000..fb90500 --- /dev/null +++ b/docs/source/quickstart.rst @@ -0,0 +1,389 @@ +Quickstart +========== + +Enabling Trio mode and running your first async tests +----------------------------------------------------- + +.. note:: If you used `cookiecutter-trio + `__ to set up + your project, then pytest-trio and Trio mode are already + configured! You can write ``async def test_whatever(): ...`` and it + should just work. Feel free to skip to the next section. + +Let's make a temporary directory to work in, and write two trivial +tests: one that we expect should pass, and one that we expect should +fail:: + + # test_example.py + import trio + + async def test_sleep(): + start_time = trio.current_time() + await trio.sleep(1) + end_time = trio.current_time() + assert end_time - start_time >= 1 + + async def test_should_fail(): + assert False + +If we run this under pytest normally, then we get a strange result: + +.. code-block:: none + + $ pytest test_example.py + + ======================== test session starts ========================= + platform linux -- Python 3.6.5, pytest-3.6.3, py-1.5.4, pluggy-0.6.0 + rootdir: /tmp, inifile: + collected 2 items + + test_example.py .. [100%] + + ========================== warnings summary ========================== + test_example.py::test_sleep + .../_pytest/python.py:196: RuntimeWarning: coroutine 'test_sleep' was never awaited + testfunction(**testargs) + + test_example.py::test_should_fail + .../_pytest/python.py:196: RuntimeWarning: coroutine 'test_should_fail' was never awaited + testfunction(**testargs) + + -- Docs: http://doc.pytest.org/en/latest/warnings.html + ================ 2 passed, 2 warnings in 0.02 seconds ================ + +So ``test_sleep`` passed, which is what we expected... but +``test_should_fail`` also passes, which is strange. And it says that +the whole test run completed in 0.02 seconds, which is weird, because +``test_sleep`` should have taken at least second to run. And then +there are these strange warnings at the bottom... what's going on +here? + +The problem is that our tests are async, and pytest doesn't know what +to do with it. So it basically skips running them entirely, and then +reports them as passed. This is not very helpful! If you see warnings +like this, or if your tests seem to pass but your coverage reports +claim that they weren't run at all, then this might be the problem. + +Here's the fix: + +1. Install pytest-trio: ``pip install pytest-trio`` + +2. In your project root, create a file called ``pytest.ini`` with + contents: + + .. code-block:: none + + [pytest] + trio_mode = true + +And we're done! Let's try running pytest again: + +.. code-block:: none + + $ pip install pytest-trio + + $ cat <pytest.ini + [pytest] + trio_mode = true + EOF + + $ pytest test_example.py + ======================== test session starts ========================= + platform linux -- Python 3.6.5, pytest-3.6.3, py-1.5.4, pluggy-0.6.0 + rootdir: /tmp, inifile: pytest.ini + plugins: trio-0.4.2 + collected 2 items + + test_example.py .F [100%] + + ============================== FAILURES ============================== + __________________________ test_should_fail __________________________ + + async def test_should_fail(): + > assert False + E assert False + + test_example.py:7: AssertionError + ================= 1 failed, 1 passed in 1.05 seconds ================= + +Notice that now it says ``plugins: trio``, which means that +pytest-trio is installed, and the results make sense: the good test +passed, the bad test failed, no warnings, and it took just over 1 +second, like we'd expect. + + +Trio's magic autojump clock +--------------------------- + +Tests involving time are often slow and flaky. But we can +fix that. Just add the ``autojump_clock`` fixture to your test, and +it will run in a mode where Trio's clock is virtualized and +deterministic. Essentially, the clock doesn't move, except that whenever all +tasks are blocked waiting, it jumps forward until the next time when +something will happen:: + + # Notice the 'autojump_clock' argument: that's all it takes! + async def test_sleep_efficiently_and_reliably(autojump_clock): + start_time = trio.current_time() + await trio.sleep(1) + end_time = trio.current_time() + assert start_time - end_time == 1 + +In the version of this test we saw before that used real time, at the +end we had to use a ``>=`` comparison, in order to account for +scheduler jitter and so forth. If there were a bug that caused +:func:`trio.sleep` to take 10 seconds, our test wouldn't have noticed. +But now we're using virtual time, so the call to ``await +trio.sleep(1)`` takes *exactly* 1 virtual second, and the ``==`` test +will pass every time. Before, we had to wait around for the test to +complete; now, it completes essentially instantaneously. (Try it!) +And, while here our example is super simple, it's integration with +Trio's core scheduling logic allows this to work for arbitrarily +complex programs (as long as they aren't interacting with the outside +world). + + +Async fixtures +-------------- + +Now that we have ``trio_mode`` activated, we can also write async +fixtures:: + + import pytest + import trio + + @pytest.fixture + async def fixture_that_sleeps_for_some_reason(): + await trio.sleep(1) + return "slept" + + async def test_example(fixture_that_sleeps_for_some_reason): + assert fixture_that_sleeps_for_some_reason == "slept" + +If you need to run teardown code, you can use ``yield``, just like a +regular pytest fixture:: + + @pytest.fixture + async def connection_to_httpbin(): + # Setup code + stream = await trio.open_tcp_stream("httpbin", 80) + + # The value of this fixture + yield stream + + # Teardown code, executed after the test is done + await stream.close() + + +.. _server-fixture-example: + +Running a background server from a fixture +------------------------------------------ + +Here's some code to implement an echo server. It's supposed to take in +arbitrary data, and then send it back out again:: + + async def echo_server_handler(stream): + while True: + data = await stream.receive_some(1000) + if not data: + break + await stream.send_all(data) + + # Usage: await trio.serve_tcp(echo_server_handler, ...) + +Since this is such complicated and sophisticated code, we want to +write lots of tests to make sure it's working correctly. To test it, +we need to start one task to run the echo server, and then connect to +it from a second task and send it test data. Here's a first attempt:: + + # Let's cross our fingers and hope no-one else is using this port... + PORT = 14923 + + # Don't copy this -- we can do better + async def test_attempt_1(): + async with trio.open_nursery() as nursery: + # Start server running in the background + nursery.start_soon( + partial(trio.serve_tcp, echo_server_handler, port=PORT) + ) + + # Connect to the server. + client = await trio.open_tcp_stream("127.0.0.1", PORT) + # Send some test data, and check that it gets echoed back + async with client: + for test_byte in [b"a", b"c", b"c"]: + await client.send_all(test_byte) + assert await client.receive_some(1) == test_byte + +This will mostly work, but it has a few problems. The first one is +that there's a race condition: We spawn a background task to call +``serve_tcp``, and then immediately try to connect to that server. +Sometimes this will work fine. But it takes a little while for the +server to start up and be ready to accept connections – so other +times, randomly, our connection attempt will happen too quickly, and +error out. After all – ``nursery.start_soon`` only promises that the +task will be started *soon*, not that it's actually happened. + +To solve this race condition, we should switch to using ``await +nursery.start(...)``. You should `read its docs for full details +`__, +but basically the idea is that this starts up a background task, and +waits for it to finish getting started before returning. This requires +some cooperation from the function you're calling: it has to tell +``nursery.start`` when it's finished setting up. Fortunately, +:func:`trio.serve_tcp` is already set up to cooperate with +``nursery.start``, so we can write:: + + # Let's cross our fingers and hope no-one else is using this port... + PORT = 14923 + + # Don't copy this -- we can do better + async def test_attempt_2(): + async with trio.open_nursery() as nursery: + # Start server running in the background + # AND wait for it to finish starting up before continuing + await nursery.start( + partial(trio.serve_tcp, echo_server_handler, port=PORT) + ) + + # Connect to the server + client = await trio.open_tcp_stream("127.0.0.1", PORT) + async with client: + for test_byte in [b"a", b"c", b"c"]: + await client.send_all(test_byte) + assert await client.receive_some(1) == test_byte + +That solves our race condition. Next issue: hardcoding the port number +like this is a bad idea, because port numbers are a machine-wide +resource, so if we're unlucky some other program might already be +using it. What we really want to do is to tell :func:`~trio.serve_tcp` +to pick a random port that no-one else is using. It turns out that +this is easy: if you request port 0, then the operating system instead +picks an unused one for you automatically. Problem solved! + +But wait... if the operating system is picking the port for us, how do +we know figure out which one it picked, so we can connect? + +Well, there's no way to predict the port ahead of time. But after +:func:`~trio.serve_tcp` has opened a port, it can check and see what +it got. So we need some way to pass this data back out of +:func:`~trio.serve_tcp`. Fortunately, ``nursery.start`` handles this +too: it lets the task pass out a piece of data after it's started. And +it just so happens that what :func:`~trio.serve_tcp` passes out is a +list of :class:`~trio.SocketListener` objects. And there's a handy +function called :func:`trio.testing.open_stream_to_socket_listener` +that can take a :class:`~trio.SocketListener` and make a connection to +it. + +Putting it all together:: + + from trio.testing import open_stream_to_socket_listener + + # Don't copy this -- we can do better + async def test_attempt_3(): + async with trio.open_nursery() as nursery: + # Start server running in the background + # AND wait for it to finish starting up before continuing + # AND find out where it's actually listening + listeners = await nursery.start( + partial(trio.serve_tcp, echo_server_handler, port=0) + ) + + # Connect to the server. + # There might be multiple listeners (example: IPv4 and + # IPv6), but we don't care which one we connect to, so we + # just use the first. + client = await open_stream_to_socket_listener(listeners[0]) + async with client: + for test_byte in [b"a", b"c", b"c"]: + await client.send_all(test_byte) + assert await client.receive_some(1) == test_byte + +Okay, this is getting closer... but if we try to run it, we'll find +that it just hangs instead of completing. What's going on? + +The problem is that after we finish doing our tests, we need to shut +down the server – otherwise it will keep waiting for new connections +forever, and the test will never finish. We can fix this by cancelling +the nursery once we're done with it:: + + # Don't copy this -- it finally works, but we can still do better! + async def test_attempt_4(): + async with trio.open_nursery() as nursery: + # Start server running in the background + # AND wait for it to finish starting up before continuing + # AND find out where it's actually listening + listeners = await nursery.start( + partial(trio.serve_tcp, echo_server_handler, port=0) + ) + + # Connect to the server. + # There might be multiple listeners (example: IPv4 and + # IPv6), but we don't care which one we connect to, so we + # just use the first. + client = await open_stream_to_socket_listener(listeners[0]) + async with client: + for test_byte in [b"a", b"c", b"c"]: + await client.send_all(test_byte) + assert await client.receive_some(1) == test_byte + + # Shut down the server now that we're done testing it + nursery.cancel_scope.cancel() + +Okay, finally this test works correctly. But that's a lot of +boilerplate. Remember, we need to write lots of tests for this server, +and we don't want to have to copy-paste all that stuff into every +test. Let's factor it out into a fixture. + +Probably our first attempt will look something like:: + + # DON'T DO THIS, IT DOESN'T WORK + @pytest.fixture + async def echo_server_connection(): + async with trio.open_nursery() as nursery: + await nursery.start(...) + yield ... + nursery.cancel_scope.cancel() + +Unfortunately, this doesn't work. **You cannot make a fixture that +opens a nursery, and then yields from inside the nursery block.** +Sorry. + +Instead, pytest-trio provides a built-in fixture called +``test_nursery``. This is a nursery that pytest-trio creates +internally, that lasts for as long as the test is running, and then +pytest-trio cancels it. Which is exactly what we need – in fact it's +even simpler than our first try, because now we don't need to worry +about cancelling the nursery ourselves. + +So here's our complete, final version:: + + # Final version -- copy this! + from functools import partial + import pytest + import trio + from trio.testing import open_stream_to_socket_listener + + async def echo_server_handler(stream): + while True: + data = await stream.receive_some(1000) + if not data: + break + await stream.send_all(data) + + @pytest.fixture + async def echo_server_connection(test_nursery): + listeners = await test_nursery.start( + partial(trio.serve_tcp, echo_server_handler, port=0) + ) + client = await open_stream_to_socket_listener(listeners[0]) + async with client: + yield client + + async def test_final(echo_server_connection): + for test_byte in [b"a", b"c", b"c"]: + await echo_server_connection.send_all(test_byte) + assert await echo_server_connection.receive_some(1) == test_byte + +No race conditions, no hangs, simple, clean, and reusable. diff --git a/docs/source/reference.rst b/docs/source/reference.rst new file mode 100644 index 0000000..c23a1d3 --- /dev/null +++ b/docs/source/reference.rst @@ -0,0 +1,176 @@ +Reference +========= + +Trio mode +--------- + +Most users will want to enable "Trio mode". Without Trio mode: + +* Async tests have to be decorated with ``@pytest.mark.trio`` +* Async fixtures have to be decorated with + ``@pytest_trio.trio_fixture`` (instead of ``@pytest.fixture``). + +When Trio mode is enabled, two things happen: + +* Async tests automatically have the ``trio`` mark added, so you don't + have to do it yourself. +* Async fixtures using ``@pytest.fixture`` automatically get converted + to Trio fixtures. + +There are two ways to enable Trio mode. + +The first option is to **use a pytest configuration file**. The exact +rules for how pytest finds configuration files are `a bit complicated +`__, but you want to +end up with something like: + +.. code-block:: ini + + # pytest.ini + [pytest] + trio_mode = true + +The second option is **use a conftest.py file**. Inside your tests +directory, create a file called ``conftest.py``, with the following +contents:: + + # conftest.py + from pytest_trio.enable_trio_mode import * + +This does exactly the same thing as setting ``trio_mode = true`` in +``pytest.ini``, except for two things: + +* Some people like to ship their tests as part of their library, so + they (or their users) can test the final installed software by + running ``pytest --pyargs PACKAGENAME``. In this mode, + ``pytest.ini`` files don't work, but ``conftest.py`` files do. + +* Enabling Trio mode in ``pytest.ini`` always enables it globally for + your entire testsuite. Enabling it in ``conftest.py`` only enables + it for test files that are in the same directory as the + ``conftest.py``, or its subdirectories. + +If you have software that uses multiple async libraries, then you can +use ``conftest.py`` to enable Trio mode for just the part of your +testsuite that uses Trio; or, if you need even finer-grained control, +you can leave Trio mode disabled and use ``@pytest.mark.trio`` +explicitly on all your Trio tests. + + +Trio fixtures +------------- + +Normally, pytest runs fixture code before starting the test, and +teardown code afterwards. For technical reasons, we can't wrap this +whole process in :func:`trio.run` – only the test itself. As a +workaround, pytest-trio introduces the concept of a "Trio fixture", +which acts like a normal fixture for most purposes, but actually does +the setup and teardown inside the test's call to :func:`trio.run`. + +The following fixtures are treated as Trio fixtures: + +* Any function decorated with ``@pytest_trio.trio_fixture``. +* If Trio mode is enabled, then any *async* function decorated with + ``@pytest.fixture`` automatically becomes a Trio fixture. +* Any fixture which depends on a Trio fixture. + +The most notable difference between regular fixtures and Trio fixtures +is that regular fixtures can't use Trio APIs, but Trio fixtures can. +Most of the time you don't need to worry about this, because you +normally only call Trio APIs from async functions, and when Trio mode +is enabled, all async fixtures are automatically Trio fixtures. +However, if for some reason you do want to use Trio APIs from a +synchronous fixture, then you'll have to use +``@pytest_trio.trio_fixture``:: + + # This fixture is not very useful + # But it is an example where @pytest.fixture doesn't work + @pytest_trio.trio_fixture + def trio_time(): + return trio.current_time() + +Only Trio tests can use Trio fixtures. If you have a regular +(synchronous) test that tries to use a Trio fixture, then that's an +error. + +And finally, regular fixtures can be `scoped to the test, class, +module, or session +`__, +but Trio fixtures **must be test scoped**. Class, module, and session +scope are not supported. + + +Built-in fixtures +----------------- + +These fixtures are automatically available to any code using +pytest-trio. + +.. data:: autojump_clock + + A :class:`trio.testing.MockClock`, configured with ``rate=0, + autojump_threshold=0``. + +.. data:: mock_clock + + A :class:`trio.testing.MockClock`, with its default configuration + (``rate=0, autojump_threshold=inf``). + +What makes these particularly useful is that whenever pytest-trio runs +a test, it checks the fixtures to see if one of them is a +:class:`trio.abc.Clock` object. If so, it passes that object to +:func:`trio.run`. So if your test requests one of these fixtures, it +automatically uses that clock. + +If you implement your own :class:`~trio.abc.Clock`, and implement a +fixture that returns it, then it will work the same way. + +Of course, like any pytest fixture, you also get the actual object +available. For example, you can call +:meth:`~trio.testing.MockClock.jump`:: + + async def test_time_travel(mock_clock): + assert trio.current_time() == 0 + mock_clock.jump(10) + assert trio.current_time() == 10 + +.. data:: test_nursery + + A nursery created and managed by pytest-trio itself. When + pytest-trio runs a test, it performs these steps in this order: + + 1. Open the ``test_nursery`` + 2. Set up all Trio fixtures. + 3. Run the test. + 4. Tear down all Trio fixtures. + 5. Cancel the ``test_nursery``. + + See :ref:`server-fixture-example` for an example of how this can be + used. + + +Integration with the Hypothesis library +--------------------------------------- + +There isn't too much to say here, since the obvious thing just works:: + + from hypothesis import given + import hypothesis.strategies as st + + @given(st.binary()) + async def test_trio_and_hypothesis(data): + ... + +Under the hood, this requires some coordination between Hypothesis and +pytest-trio. Hypothesis runs your test multiple times with different +examples of random data. For each example, pytest-trio calls +:func:`trio.run` again (so you get a fresh clean Trio environment), +sets up any Trio fixtures, runs the actual test, and then tears down +any Trio fixtures. Most of the time this shouldn't matter (and `is +probably what you want anyway +`__), but in +some unusual cases it could surprise you. And note that this only +applies to Trio fixtures – if a Trio test uses a mix of Trio fixtures +and regular fixtures, then the regular fixtures will still only +`instantiated once and re-used for all examples +`__. From 0217e26a537a80ca671c7763498e7b55d666e239 Mon Sep 17 00:00:00 2001 From: "Nathaniel J. Smith" Date: Mon, 23 Jul 2018 22:48:38 -0700 Subject: [PATCH 02/12] Tweak fixture semantics MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The fixture setup code doesn't have convenient access to config file settings, and also keying the automagic-async-fixture-handling off of the trio-mode setting would be a compatibility break. Not the end of the world, but kind of annoying. Instead, let's continue to key the automagic-async-fixture-handling off of the requesting test being a Trio test. It's just as convenient, and while technically could be a little more error-prone – specifically, if you accidentally use an async fixture with a non-async test, then we won't be able to catch that as an error – I doubt it matters much in practice. And if people turn out to make that mistake a lot, we can always add some check for it later. --- docs/source/reference.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/reference.rst b/docs/source/reference.rst index c23a1d3..ba0d5c9 100644 --- a/docs/source/reference.rst +++ b/docs/source/reference.rst @@ -70,8 +70,8 @@ the setup and teardown inside the test's call to :func:`trio.run`. The following fixtures are treated as Trio fixtures: * Any function decorated with ``@pytest_trio.trio_fixture``. -* If Trio mode is enabled, then any *async* function decorated with - ``@pytest.fixture`` automatically becomes a Trio fixture. +* Any async function decorated with ``@pytest.fixture``, *if* + requested by a Trio test. * Any fixture which depends on a Trio fixture. The most notable difference between regular fixtures and Trio fixtures From e5289d015bd5f6b8f615db174dd748bac0af2779 Mon Sep 17 00:00:00 2001 From: "Nathaniel J. Smith" Date: Tue, 24 Jul 2018 20:29:25 -0700 Subject: [PATCH 03/12] Edits after review --- docs/source/index.rst | 10 ++--- docs/source/quickstart.rst | 78 ++++++++++++++++++++------------------ docs/source/reference.rst | 36 ++++++++++++------ 3 files changed, 70 insertions(+), 54 deletions(-) diff --git a/docs/source/index.rst b/docs/source/index.rst index 1b7af1a..4682c05 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -18,12 +18,12 @@ and async I/O in Python. Features include: * Useful fixtures included: use ``autojump_clock`` for easy testing of code with timeouts. -* Write your own async fixtures: start up a Trio-based server inside a - fixture, and then connect to it from your test. +* Write your own async fixtures: set up an async database connection + or start a server inside a fixture, and then use it in your tests. -* Use `Hypothesis `__? This plugin - integrates with Hypothesis, so your async tests can use - property-based testing. +* Integration with the fabulous `Hypothesis + `__ library, so your async tests can use + property-based testing: just use ``@given`` like you're used to. Vital statistics diff --git a/docs/source/quickstart.rst b/docs/source/quickstart.rst index fb90500..4fa313d 100644 --- a/docs/source/quickstart.rst +++ b/docs/source/quickstart.rst @@ -146,33 +146,31 @@ world). Async fixtures -------------- -Now that we have ``trio_mode`` activated, we can also write async -fixtures:: - - import pytest - import trio +We can write async fixtures:: @pytest.fixture - async def fixture_that_sleeps_for_some_reason(): - await trio.sleep(1) - return "slept" + async def db_connection(): + return await some_async_db_library.connect(...) - async def test_example(fixture_that_sleeps_for_some_reason): - assert fixture_that_sleeps_for_some_reason == "slept" + async def test_example(db_connection): + await db_connection.execute("SELECT * FROM ...") If you need to run teardown code, you can use ``yield``, just like a regular pytest fixture:: + # DB connection that wraps each test in a transaction and rolls it + # back afterwards @pytest.fixture - async def connection_to_httpbin(): + async def rolback_db_connection(): # Setup code - stream = await trio.open_tcp_stream("httpbin", 80) + connection = await some_async_db_library.connect(...) + await connection.execute("START TRANSACTION") # The value of this fixture - yield stream + yield connection # Teardown code, executed after the test is done - await stream.close() + await connection.execute("ROLLBACK") .. _server-fixture-example: @@ -194,8 +192,9 @@ arbitrary data, and then send it back out again:: Since this is such complicated and sophisticated code, we want to write lots of tests to make sure it's working correctly. To test it, -we need to start one task to run the echo server, and then connect to -it from a second task and send it test data. Here's a first attempt:: +we want to start a background task running the echo server itself, and +then we'll connect to it, send it test data, and see how it responds. +Here's a first attempt:: # Let's cross our fingers and hope no-one else is using this port... PORT = 14923 @@ -226,14 +225,15 @@ error out. After all – ``nursery.start_soon`` only promises that the task will be started *soon*, not that it's actually happened. To solve this race condition, we should switch to using ``await -nursery.start(...)``. You should `read its docs for full details +nursery.start(...)``. You can `read its docs for full details `__, -but basically the idea is that this starts up a background task, and -waits for it to finish getting started before returning. This requires -some cooperation from the function you're calling: it has to tell -``nursery.start`` when it's finished setting up. Fortunately, -:func:`trio.serve_tcp` is already set up to cooperate with -``nursery.start``, so we can write:: +but basically the idea is that both ``nursery.start_soon(...)`` and +``await nursery.start(...)`` create background tasks, but only +``start`` waits for the new task to finish getting itself set up. This +requires some cooperation from the background task: it has to notify +``nursery.start`` when it's ready. Fortunately, :func:`trio.serve_tcp` +already knows how to cooperated with ``nursery.start``, so we can +write:: # Let's cross our fingers and hope no-one else is using this port... PORT = 14923 @@ -259,11 +259,11 @@ like this is a bad idea, because port numbers are a machine-wide resource, so if we're unlucky some other program might already be using it. What we really want to do is to tell :func:`~trio.serve_tcp` to pick a random port that no-one else is using. It turns out that -this is easy: if you request port 0, then the operating system instead -picks an unused one for you automatically. Problem solved! +this is easy: if you request port 0, then the operating system will +pick an unused one for you automatically. Problem solved! But wait... if the operating system is picking the port for us, how do -we know figure out which one it picked, so we can connect? +we know figure out which one it picked, so we can connect to it later? Well, there's no way to predict the port ahead of time. But after :func:`~trio.serve_tcp` has opened a port, it can check and see what @@ -303,10 +303,10 @@ Putting it all together:: Okay, this is getting closer... but if we try to run it, we'll find that it just hangs instead of completing. What's going on? -The problem is that after we finish doing our tests, we need to shut -down the server – otherwise it will keep waiting for new connections -forever, and the test will never finish. We can fix this by cancelling -the nursery once we're done with it:: +The problem is that we need to shut down the server – otherwise it +will keep waiting for new connections forever, and the test will never +finish. We can fix this by cancelling everything in the nursery once +our test is finished:: # Don't copy this -- it finally works, but we can still do better! async def test_attempt_4(): @@ -347,15 +347,16 @@ Probably our first attempt will look something like:: nursery.cancel_scope.cancel() Unfortunately, this doesn't work. **You cannot make a fixture that -opens a nursery, and then yields from inside the nursery block.** -Sorry. +opens a nursery or cancel scope, and then yields from inside the +nursery or cancel scope block.** Sorry 🙁. We're `working on it +`__. Instead, pytest-trio provides a built-in fixture called -``test_nursery``. This is a nursery that pytest-trio creates -internally, that lasts for as long as the test is running, and then -pytest-trio cancels it. Which is exactly what we need – in fact it's -even simpler than our first try, because now we don't need to worry -about cancelling the nursery ourselves. +``test_nursery``. This is a nursery that's created before each test, +and then automatically cancelled after the test finishes. Which is +exactly what we need – in fact it's even simpler than our first try, +because now we don't need to worry about cancelling the nursery +ourselves. So here's our complete, final version:: @@ -365,6 +366,7 @@ So here's our complete, final version:: import trio from trio.testing import open_stream_to_socket_listener + # The code being tested: async def echo_server_handler(stream): while True: data = await stream.receive_some(1000) @@ -372,6 +374,7 @@ So here's our complete, final version:: break await stream.send_all(data) + # The fixture: @pytest.fixture async def echo_server_connection(test_nursery): listeners = await test_nursery.start( @@ -381,6 +384,7 @@ So here's our complete, final version:: async with client: yield client + # A test using the fixture: async def test_final(echo_server_connection): for test_byte in [b"a", b"c", b"c"]: await echo_server_connection.send_all(test_byte) diff --git a/docs/source/reference.rst b/docs/source/reference.rst index ba0d5c9..ed64f3d 100644 --- a/docs/source/reference.rst +++ b/docs/source/reference.rst @@ -6,16 +6,21 @@ Trio mode Most users will want to enable "Trio mode". Without Trio mode: -* Async tests have to be decorated with ``@pytest.mark.trio`` -* Async fixtures have to be decorated with - ``@pytest_trio.trio_fixture`` (instead of ``@pytest.fixture``). +* Pytest-trio only handles tests that have been decorated with + ``@pytest.mark.trio`` +* Pytest-trio only handles fixtures if they're async *and* used by a + test that's decorated with ``@pytest.mark.trio``, or if they're + decorated with ``@pytest_trio.trio_fixture`` (instead of + ``@pytest.fixture``). -When Trio mode is enabled, two things happen: +When Trio mode is enabled, two extra things happen: * Async tests automatically have the ``trio`` mark added, so you don't have to do it yourself. * Async fixtures using ``@pytest.fixture`` automatically get converted - to Trio fixtures. + to Trio fixtures. (The main effect of this is that it helps you + catch mistakes where a you use an async fixture with a non-async + test.) There are two ways to enable Trio mode. @@ -166,11 +171,18 @@ pytest-trio. Hypothesis runs your test multiple times with different examples of random data. For each example, pytest-trio calls :func:`trio.run` again (so you get a fresh clean Trio environment), sets up any Trio fixtures, runs the actual test, and then tears down -any Trio fixtures. Most of the time this shouldn't matter (and `is -probably what you want anyway +any Trio fixtures. Notice that this is a bit different than regular +pytest fixtures, which are `instantiated once and then re-used for all +`__. Most of the time +this shouldn't matter (and `is probably what you want anyway `__), but in -some unusual cases it could surprise you. And note that this only -applies to Trio fixtures – if a Trio test uses a mix of Trio fixtures -and regular fixtures, then the regular fixtures will still only -`instantiated once and re-used for all examples -`__. +some unusual cases it could surprise you. And this only applies to +Trio fixtures – if a Trio test uses a mix of regular fixtures and Trio +fixtures, then the regular fixtures will be reused, while the Trio +fixtures will be repeatedly reinstantiated. + +Also, pytest-trio only handles ``@given``\-based tests. If you want to +write `stateful tests +`__ for +Trio-based libraries, then check out `hypothesis-trio +`__. From 2165d1061c42ef930b094a14c484a87e0fcdcf52 Mon Sep 17 00:00:00 2001 From: "Nathaniel J. Smith" Date: Tue, 24 Jul 2018 20:59:54 -0700 Subject: [PATCH 04/12] Catch up the reference docs on what counts as a trio fixture --- docs/source/reference.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/source/reference.rst b/docs/source/reference.rst index ed64f3d..1021093 100644 --- a/docs/source/reference.rst +++ b/docs/source/reference.rst @@ -76,7 +76,8 @@ The following fixtures are treated as Trio fixtures: * Any function decorated with ``@pytest_trio.trio_fixture``. * Any async function decorated with ``@pytest.fixture``, *if* - requested by a Trio test. + Trio mode is enabled *or* this fixture is being requested by a Trio + test. * Any fixture which depends on a Trio fixture. The most notable difference between regular fixtures and Trio fixtures From 8660bb22dc3826d8ffb96af1aabdac665aeeb2e4 Mon Sep 17 00:00:00 2001 From: "Nathaniel J. Smith" Date: Tue, 24 Jul 2018 21:06:31 -0700 Subject: [PATCH 05/12] Speling --- docs/source/quickstart.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/quickstart.rst b/docs/source/quickstart.rst index 4fa313d..04b3cc1 100644 --- a/docs/source/quickstart.rst +++ b/docs/source/quickstart.rst @@ -161,7 +161,7 @@ regular pytest fixture:: # DB connection that wraps each test in a transaction and rolls it # back afterwards @pytest.fixture - async def rolback_db_connection(): + async def rollback_db_connection(): # Setup code connection = await some_async_db_library.connect(...) await connection.execute("START TRANSACTION") From e52357e1e8e4d0bd2b9fd3daafe3250c725af7c3 Mon Sep 17 00:00:00 2001 From: "Nathaniel J. Smith" Date: Fri, 17 Aug 2018 02:59:51 -0700 Subject: [PATCH 06/12] Don't call it Pytest_Trio that just looks silly --- docs/source/history.rst | 14 +++++++------- pyproject.toml | 1 + 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/source/history.rst b/docs/source/history.rst index 38c7930..a27ad57 100644 --- a/docs/source/history.rst +++ b/docs/source/history.rst @@ -5,7 +5,7 @@ Release history .. towncrier release notes start -Pytest_Trio 0.4.2 (2018-06-29) +pytest-trio 0.4.2 (2018-06-29) ------------------------------ Features @@ -16,20 +16,20 @@ Features using Trio. (`#42 `__) -Pytest_Trio 0.4.1 (2018-04-14) +pytest-trio 0.4.1 (2018-04-14) ------------------------------ No significant changes. -Pytest_Trio 0.4.0 (2018-04-14) +pytest-trio 0.4.0 (2018-04-14) ------------------------------ - Fix compatibility with trio 0.4.0 (`#25 `__) -Pytest_Trio 0.3.0 (2018-01-03) +pytest-trio 0.3.0 (2018-01-03) ------------------------------ Features @@ -39,7 +39,7 @@ Features `__) -Pytest_Trio 0.2.0 (2017-12-15) +pytest-trio 0.2.0 (2017-12-15) ------------------------------ - Heavy improvements, add async yield fixture, fix bugs, add tests etc. (`#17 @@ -53,14 +53,14 @@ Deprecations and Removals `__) -Pytest_Trio 0.1.1 (2017-12-08) +pytest-trio 0.1.1 (2017-12-08) ------------------------------ Disable intersphinx for trio (cause crash in CI for the moment due to 404 in readthedoc). -Pytest_Trio 0.1.0 (2017-12-08) +pytest-trio 0.1.0 (2017-12-08) ------------------------------ Initial release. diff --git a/pyproject.toml b/pyproject.toml index 77448c7..407fb8b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -2,5 +2,6 @@ package = "pytest_trio" filename = "docs/source/history.rst" directory = "newsfragments" +title_format = "pytest-trio {version} ({project_date})" underlines = ["-", "~", "^"] issue_format = "`#{issue} `__" From 735b68e9536182d5c4f8dcca5fc0c3e25ada06d2 Mon Sep 17 00:00:00 2001 From: "Nathaniel J. Smith" Date: Fri, 17 Aug 2018 03:55:51 -0700 Subject: [PATCH 07/12] Update docs for the latest changes in gh-50 Also wrote some draft release notes --- docs/source/history.rst | 20 +++++ docs/source/index.rst | 10 ++- docs/source/quickstart.rst | 148 +++++++++++++++++++------------------ docs/source/reference.rst | 74 ++++++++++++++++--- 4 files changed, 170 insertions(+), 82 deletions(-) diff --git a/docs/source/history.rst b/docs/source/history.rst index a27ad57..2f8fce7 100644 --- a/docs/source/history.rst +++ b/docs/source/history.rst @@ -5,6 +5,26 @@ Release history .. towncrier release notes start +pytest-trio 0.5.0 (????-??-??) +------------------------------ + +This is a major release, including a rewrite of large portions of the +internals. We believe it should be backwards compatible with existing +projects, but major new features include: + +* "trio mode": no more writing ``@pytest.mark.trio`` everywhere! +* it's now safe to use nurseries inside fixtures (`#55 + `__) +* new ``@trio_fixture`` decorator to explicitly mark a fixture as a + trio fixture +* a number of easy-to-make mistakes are now caught and raise + informative errors +* the :data:`nursery` fixture is now 87% more magical + +For more details, see the manual. Oh right, speaking of which: we +finally have a manual! You should read it. + + pytest-trio 0.4.2 (2018-06-29) ------------------------------ diff --git a/docs/source/index.rst b/docs/source/index.rst index 4682c05..9af200c 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -15,16 +15,22 @@ and async I/O in Python. Features include: * Async tests without the boilerplate: just write ``async def test_whatever(): ...``. -* Useful fixtures included: use ``autojump_clock`` for easy testing of - code with timeouts. +* Useful fixtures included: use :data:`autojump_clock` for easy + testing of code with timeouts, or :data:`nursery` to easily set up + background tasks. * Write your own async fixtures: set up an async database connection or start a server inside a fixture, and then use it in your tests. + If you have multiple async fixtures, pytest-trio will even do + setup/teardown concurrently whenever possible. * Integration with the fabulous `Hypothesis `__ library, so your async tests can use property-based testing: just use ``@given`` like you're used to. +* Supports testing projects that use Trio exclusively, and also + projects that support multiple async libraries. + Vital statistics ================ diff --git a/docs/source/quickstart.rst b/docs/source/quickstart.rst index 04b3cc1..1c0682d 100644 --- a/docs/source/quickstart.rst +++ b/docs/source/quickstart.rst @@ -172,6 +172,13 @@ regular pytest fixture:: # Teardown code, executed after the test is done await connection.execute("ROLLBACK") +If you need to support Python 3.5, which doesn't allow ``yield`` +inside an ``async def`` function, then you can define async fixtures +using the `async_generator +`__ +library – just make sure to put the ``@pytest.fixture`` *above* the +``@async_generator``. + .. _server-fixture-example: @@ -191,10 +198,11 @@ arbitrary data, and then send it back out again:: # Usage: await trio.serve_tcp(echo_server_handler, ...) Since this is such complicated and sophisticated code, we want to -write lots of tests to make sure it's working correctly. To test it, -we want to start a background task running the echo server itself, and -then we'll connect to it, send it test data, and see how it responds. -Here's a first attempt:: +write lots of tests to make sure it's working correctly. Our basic +strategy for each test will be to start the echo server running in a +background task, and then in the test body we'll connect to our +server, send it some test data, and see how it responds. Here's a +first attempt:: # Let's cross our fingers and hope no-one else is using this port... PORT = 14923 @@ -208,12 +216,12 @@ Here's a first attempt:: ) # Connect to the server. - client = await trio.open_tcp_stream("127.0.0.1", PORT) + echo_client = await trio.open_tcp_stream("127.0.0.1", PORT) # Send some test data, and check that it gets echoed back - async with client: - for test_byte in [b"a", b"c", b"c"]: - await client.send_all(test_byte) - assert await client.receive_some(1) == test_byte + async with echo_client: + for test_byte in [b"a", b"b", b"c"]: + await echo_client.send_all(test_byte) + assert await echo_client.receive_some(1) == test_byte This will mostly work, but it has a few problems. The first one is that there's a race condition: We spawn a background task to call @@ -232,7 +240,7 @@ but basically the idea is that both ``nursery.start_soon(...)`` and ``start`` waits for the new task to finish getting itself set up. This requires some cooperation from the background task: it has to notify ``nursery.start`` when it's ready. Fortunately, :func:`trio.serve_tcp` -already knows how to cooperated with ``nursery.start``, so we can +already knows how to cooperate with ``nursery.start``, so we can write:: # Let's cross our fingers and hope no-one else is using this port... @@ -248,11 +256,11 @@ write:: ) # Connect to the server - client = await trio.open_tcp_stream("127.0.0.1", PORT) - async with client: - for test_byte in [b"a", b"c", b"c"]: - await client.send_all(test_byte) - assert await client.receive_some(1) == test_byte + echo_client = await trio.open_tcp_stream("127.0.0.1", PORT) + async with echo_client: + for test_byte in [b"a", b"b", b"c"]: + await echo_client.send_all(test_byte) + assert await echo_client.receive_some(1) == test_byte That solves our race condition. Next issue: hardcoding the port number like this is a bad idea, because port numbers are a machine-wide @@ -294,11 +302,11 @@ Putting it all together:: # There might be multiple listeners (example: IPv4 and # IPv6), but we don't care which one we connect to, so we # just use the first. - client = await open_stream_to_socket_listener(listeners[0]) - async with client: - for test_byte in [b"a", b"c", b"c"]: - await client.send_all(test_byte) - assert await client.receive_some(1) == test_byte + echo_client = await open_stream_to_socket_listener(listeners[0]) + async with echo_client: + for test_byte in [b"a", b"b", b"c"]: + await echo_client.send_all(test_byte) + assert await echo_client.receive_some(1) == test_byte Okay, this is getting closer... but if we try to run it, we'll find that it just hangs instead of completing. What's going on? @@ -311,54 +319,54 @@ our test is finished:: # Don't copy this -- it finally works, but we can still do better! async def test_attempt_4(): async with trio.open_nursery() as nursery: - # Start server running in the background - # AND wait for it to finish starting up before continuing - # AND find out where it's actually listening - listeners = await nursery.start( - partial(trio.serve_tcp, echo_server_handler, port=0) - ) - - # Connect to the server. - # There might be multiple listeners (example: IPv4 and - # IPv6), but we don't care which one we connect to, so we - # just use the first. - client = await open_stream_to_socket_listener(listeners[0]) - async with client: - for test_byte in [b"a", b"c", b"c"]: - await client.send_all(test_byte) - assert await client.receive_some(1) == test_byte - - # Shut down the server now that we're done testing it - nursery.cancel_scope.cancel() + try: + listeners = await nursery.start( + partial(trio.serve_tcp, echo_server_handler, port=0) + ) + echo_client = await open_stream_to_socket_listener(listeners[0]) + async with echo_client: + for test_byte in [b"a", b"b", b"c"]: + await echo_client.send_all(test_byte) + assert await echo_client.receive_some(1) == test_byte + finally: + # Shut down the server now that we're done testing it + nursery.cancel_scope.cancel() Okay, finally this test works correctly. But that's a lot of -boilerplate. Remember, we need to write lots of tests for this server, -and we don't want to have to copy-paste all that stuff into every -test. Let's factor it out into a fixture. +boilerplate. Can we slim it down? We can! -Probably our first attempt will look something like:: +First, pytest-trio provides a magical fixture that takes care of the +nursery management boilerplate: just request the :data:`nursery` fixture, +and you get a nursery object that's already set up, and will be +automatically cancelled when you're done:: - # DON'T DO THIS, IT DOESN'T WORK - @pytest.fixture - async def echo_server_connection(): - async with trio.open_nursery() as nursery: - await nursery.start(...) - yield ... - nursery.cancel_scope.cancel() + # Don't copy this -- it finally works, but we can *still* do better! + async def test_attempt_5(nursery): + listeners = await nursery.start( + partial(trio.serve_tcp, echo_server_handler, port=0) + ) + echo_echo_client = await open_stream_to_socket_listener(listeners[0]) + async with echo_client: + for test_byte in [b"a", b"b", b"c"]: + await echo_echo_client.send_all(test_byte) + assert await echo_echo_client.receive_some(1) == test_byte -Unfortunately, this doesn't work. **You cannot make a fixture that -opens a nursery or cancel scope, and then yields from inside the -nursery or cancel scope block.** Sorry 🙁. We're `working on it -`__. +And finally, remember, we need to write lots of tests, and we don't +want to have to copy-paste the server setup code every time. Let's +factor it out into a fixture:: -Instead, pytest-trio provides a built-in fixture called -``test_nursery``. This is a nursery that's created before each test, -and then automatically cancelled after the test finishes. Which is -exactly what we need – in fact it's even simpler than our first try, -because now we don't need to worry about cancelling the nursery -ourselves. + @pytest.fixture + async def echo_server_connection(nursery): + listeners = await nursery.start( + partial(trio.serve_tcp, echo_server_handler, port=0) + ) + echo_client = await open_stream_to_socket_listener(listeners[0]) + async with echo_client: + yield echo_client -So here's our complete, final version:: +And now in tests, all we have to do is request the ``echo_client`` +fixture, and we get a background server and a client stream connected +to it. So here's our complete, final version:: # Final version -- copy this! from functools import partial @@ -376,18 +384,18 @@ So here's our complete, final version:: # The fixture: @pytest.fixture - async def echo_server_connection(test_nursery): - listeners = await test_nursery.start( + async def echo_client(nursery): + listeners = await nursery.start( partial(trio.serve_tcp, echo_server_handler, port=0) ) - client = await open_stream_to_socket_listener(listeners[0]) - async with client: - yield client + echo_client = await open_stream_to_socket_listener(listeners[0]) + async with echo_client: + yield echo_client # A test using the fixture: - async def test_final(echo_server_connection): - for test_byte in [b"a", b"c", b"c"]: - await echo_server_connection.send_all(test_byte) - assert await echo_server_connection.receive_some(1) == test_byte + async def test_final(echo_client): + for test_byte in [b"a", b"b", b"c"]: + await echo_client.send_all(test_byte) + assert await echo_client.receive_some(1) == test_byte No race conditions, no hangs, simple, clean, and reusable. diff --git a/docs/source/reference.rst b/docs/source/reference.rst index 1021093..8ddd954 100644 --- a/docs/source/reference.rst +++ b/docs/source/reference.rst @@ -106,6 +106,42 @@ but Trio fixtures **must be test scoped**. Class, module, and session scope are not supported. +Concurrent setup/teardown +------------------------- + +If your test uses multiple fixtures, then for speed, pytest-trio will +try to run their setup and teardown code concurrently whenever this is +possible while respecting the fixture dependencies. + +Here's an example, where a test depends on ``fix_b`` and ``fix_c``, +and these both depend on ``fix_a``:: + + @trio_fixture + def fix_a(): + ... + + @trio_fixture + def fix_b(fix_a): + ... + + @trio_fixture + def fix_c(fix_a): + ... + + @pytest.mark.trio + async def test_example(fix_b, fix_c): + ... + +When running ``test_example``, pytest-trio will perform the following +sequence of actions: + +1. Set up ``fix_a`` +2. Set up ``fix_b`` and ``fix_c``, concurrently. +3. Run the test. +4. Tear down ``fix_b`` and ``fix_c``, concurrently. +5. Tear down ``fix_a``. + + Built-in fixtures ----------------- @@ -140,16 +176,34 @@ available. For example, you can call mock_clock.jump(10) assert trio.current_time() == 10 -.. data:: test_nursery - - A nursery created and managed by pytest-trio itself. When - pytest-trio runs a test, it performs these steps in this order: - - 1. Open the ``test_nursery`` - 2. Set up all Trio fixtures. - 3. Run the test. - 4. Tear down all Trio fixtures. - 5. Cancel the ``test_nursery``. +.. data:: nursery + + A nursery created and managed by pytest-trio itself, which + surrounds the test/fixture that requested it, and is automatically + cancelled after the test/fixture completes. Basically, these are + equivalent:: + + # Boring way + async def test_with_background_task(): + async with trio.open_nursery() as nursery: + try: + ... + finally: + nursery.cancel_scope.cancel() + + # Fancy way + async def test_with_background_task(nursery): + ... + + For a fixture, the cancellation always happens after the fixture + completes its teardown phase. (Or if it doesn't have a teardown + phase, then the cancellation happens after the teardown phase + *would* have happened.) + + This fixture is even more magical than most pytest fixtures, + because if it gets requested several times within the same test, + then it creates multiple nurseries, one for each fixture/test that + requested it. See :ref:`server-fixture-example` for an example of how this can be used. From 14570bef2b75572f4cab3eead4b3a085a16c5ce4 Mon Sep 17 00:00:00 2001 From: "Nathaniel J. Smith" Date: Fri, 17 Aug 2018 04:28:49 -0700 Subject: [PATCH 08/12] rearrange background server quickstart to flow more smoothly --- docs/source/quickstart.rst | 199 ++++++++++++++++++++----------------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/docs/source/quickstart.rst b/docs/source/quickstart.rst index 1c0682d..0c6f918 100644 --- a/docs/source/quickstart.rst +++ b/docs/source/quickstart.rst @@ -197,12 +197,12 @@ arbitrary data, and then send it back out again:: # Usage: await trio.serve_tcp(echo_server_handler, ...) -Since this is such complicated and sophisticated code, we want to -write lots of tests to make sure it's working correctly. Our basic -strategy for each test will be to start the echo server running in a -background task, and then in the test body we'll connect to our -server, send it some test data, and see how it responds. Here's a -first attempt:: +Now we need to test it, to make sure it's working correctly. In fact, +since this is such complicated and sophisticated code, we're going to +write lots of tests for it. And they'll all follow the same basic +pattern: we'll start the echo server running in a background task, +then connect to it, send it some test data, and see how it responds. +Here's a first attempt:: # Let's cross our fingers and hope no-one else is using this port... PORT = 14923 @@ -223,17 +223,76 @@ first attempt:: await echo_client.send_all(test_byte) assert await echo_client.receive_some(1) == test_byte -This will mostly work, but it has a few problems. The first one is -that there's a race condition: We spawn a background task to call -``serve_tcp``, and then immediately try to connect to that server. -Sometimes this will work fine. But it takes a little while for the -server to start up and be ready to accept connections – so other +This will mostly work, but it has a few problems. The most obvious one +is that when we run it, even if everything works perfectly, it will +hang at the end of the test – we never shut down the server, so the +nursery block will wait forever for it to exit. + +To avoid this, we should cancel the nursery at the end of the test: + +.. code-block:: python3 + :emphasize-lines: 7,20,21 + + # Let's cross our fingers and hope no-one else is using this port... + PORT = 14923 + + # Don't copy this -- we can do better + async def test_attempt_2(): + async with trio.open_nursery() as nursery: + try: + # Start server running in the background + nursery.start_soon( + partial(trio.serve_tcp, echo_server_handler, port=PORT) + ) + + # Connect to the server. + echo_client = await trio.open_tcp_stream("127.0.0.1", PORT) + # Send some test data, and check that it gets echoed back + async with echo_client: + for test_byte in [b"a", b"b", b"c"]: + await echo_client.send_all(test_byte) + assert await echo_client.receive_some(1) == test_byte + finally: + nursery.cancel_scope.cancel() + +In fact, this pattern is *so* common, that pytest-trio provides a +handy :data:`nursery` fixture to let you skip the boilerplate. Just +add ``nursery`` to your test function arguments, and pytest-trio will +open a nursery, pass it in to your function, and then cancel it for +you afterwards: + +.. code-block:: python3 + :emphasize-lines: 5 + + # Let's cross our fingers and hope no-one else is using this port... + PORT = 14923 + + # Don't copy this -- we can do better + async def test_attempt_3(nursery): + # Start server running in the background + nursery.start_soon( + partial(trio.serve_tcp, echo_server_handler, port=PORT) + ) + + # Connect to the server. + echo_client = await trio.open_tcp_stream("127.0.0.1", PORT) + # Send some test data, and check that it gets echoed back + async with echo_client: + for test_byte in [b"a", b"b", b"c"]: + await echo_client.send_all(test_byte) + assert await echo_client.receive_some(1) == test_byte + +Next problem: we have a race condition. We spawn a background task to +call ``serve_tcp``, and then immediately try to connect to that +server. Sometimes this will work fine. But it takes a little while for +the server to start up and be ready to accept connections – so other times, randomly, our connection attempt will happen too quickly, and error out. After all – ``nursery.start_soon`` only promises that the -task will be started *soon*, not that it's actually happened. +task will be started *soon*, not that it's actually happened. So this +test will be flaky, and flaky tests are the worst. -To solve this race condition, we should switch to using ``await -nursery.start(...)``. You can `read its docs for full details +Fortunately, Trio makes this easy to solve, by switching to using +``await nursery.start(...)``. You can `read its docs for full details `__, but basically the idea is that both ``nursery.start_soon(...)`` and ``await nursery.start(...)`` create background tasks, but only @@ -241,26 +300,28 @@ but basically the idea is that both ``nursery.start_soon(...)`` and requires some cooperation from the background task: it has to notify ``nursery.start`` when it's ready. Fortunately, :func:`trio.serve_tcp` already knows how to cooperate with ``nursery.start``, so we can -write:: +write: + +.. code-block:: python3 + :emphasize-lines: 6-10 # Let's cross our fingers and hope no-one else is using this port... PORT = 14923 # Don't copy this -- we can do better - async def test_attempt_2(): - async with trio.open_nursery() as nursery: - # Start server running in the background - # AND wait for it to finish starting up before continuing - await nursery.start( - partial(trio.serve_tcp, echo_server_handler, port=PORT) - ) + async def test_attempt_4(nursery): + # Start server running in the background + # AND wait for it to finish starting up before continuing + await nursery.start( + partial(trio.serve_tcp, echo_server_handler, port=PORT) + ) - # Connect to the server - echo_client = await trio.open_tcp_stream("127.0.0.1", PORT) - async with echo_client: - for test_byte in [b"a", b"b", b"c"]: - await echo_client.send_all(test_byte) - assert await echo_client.receive_some(1) == test_byte + # Connect to the server + echo_client = await trio.open_tcp_stream("127.0.0.1", PORT) + async with echo_client: + for test_byte in [b"a", b"b", b"c"]: + await echo_client.send_all(test_byte) + assert await echo_client.receive_some(1) == test_byte That solves our race condition. Next issue: hardcoding the port number like this is a bad idea, because port numbers are a machine-wide @@ -284,79 +345,39 @@ function called :func:`trio.testing.open_stream_to_socket_listener` that can take a :class:`~trio.SocketListener` and make a connection to it. -Putting it all together:: - - from trio.testing import open_stream_to_socket_listener - - # Don't copy this -- we can do better - async def test_attempt_3(): - async with trio.open_nursery() as nursery: - # Start server running in the background - # AND wait for it to finish starting up before continuing - # AND find out where it's actually listening - listeners = await nursery.start( - partial(trio.serve_tcp, echo_server_handler, port=0) - ) - - # Connect to the server. - # There might be multiple listeners (example: IPv4 and - # IPv6), but we don't care which one we connect to, so we - # just use the first. - echo_client = await open_stream_to_socket_listener(listeners[0]) - async with echo_client: - for test_byte in [b"a", b"b", b"c"]: - await echo_client.send_all(test_byte) - assert await echo_client.receive_some(1) == test_byte +Putting it all together: -Okay, this is getting closer... but if we try to run it, we'll find -that it just hangs instead of completing. What's going on? +.. code-block:: python3 + :emphasize-lines: 1,8,13-16 -The problem is that we need to shut down the server – otherwise it -will keep waiting for new connections forever, and the test will never -finish. We can fix this by cancelling everything in the nursery once -our test is finished:: + from trio.testing import open_stream_to_socket_listener # Don't copy this -- it finally works, but we can still do better! - async def test_attempt_4(): - async with trio.open_nursery() as nursery: - try: - listeners = await nursery.start( - partial(trio.serve_tcp, echo_server_handler, port=0) - ) - echo_client = await open_stream_to_socket_listener(listeners[0]) - async with echo_client: - for test_byte in [b"a", b"b", b"c"]: - await echo_client.send_all(test_byte) - assert await echo_client.receive_some(1) == test_byte - finally: - # Shut down the server now that we're done testing it - nursery.cancel_scope.cancel() - -Okay, finally this test works correctly. But that's a lot of -boilerplate. Can we slim it down? We can! - -First, pytest-trio provides a magical fixture that takes care of the -nursery management boilerplate: just request the :data:`nursery` fixture, -and you get a nursery object that's already set up, and will be -automatically cancelled when you're done:: - - # Don't copy this -- it finally works, but we can *still* do better! async def test_attempt_5(nursery): + # Start server running in the background + # AND wait for it to finish starting up before continuing + # AND find out where it's actually listening listeners = await nursery.start( partial(trio.serve_tcp, echo_server_handler, port=0) ) - echo_echo_client = await open_stream_to_socket_listener(listeners[0]) + + # Connect to the server. + # There might be multiple listeners (example: IPv4 and + # IPv6), but we don't care which one we connect to, so we + # just use the first. + echo_client = await open_stream_to_socket_listener(listeners[0]) async with echo_client: for test_byte in [b"a", b"b", b"c"]: - await echo_echo_client.send_all(test_byte) - assert await echo_echo_client.receive_some(1) == test_byte + await echo_client.send_all(test_byte) + assert await echo_client.receive_some(1) == test_byte -And finally, remember, we need to write lots of tests, and we don't -want to have to copy-paste the server setup code every time. Let's -factor it out into a fixture:: +Now, this works – but there's still a lot of boilerplate. Remember, we +need to write lots of tests for this server, and we don't want to have +to copy-paste all that stuff into every test. Let's factor out the +setup into a fixture:: @pytest.fixture - async def echo_server_connection(nursery): + async def echo_client(nursery): listeners = await nursery.start( partial(trio.serve_tcp, echo_server_handler, port=0) ) @@ -398,4 +419,4 @@ to it. So here's our complete, final version:: await echo_client.send_all(test_byte) assert await echo_client.receive_some(1) == test_byte -No race conditions, no hangs, simple, clean, and reusable. +No hangs, no race conditions, simple, clean, and reusable. From 75befd350f354a34601a9f81e58435ff522340dd Mon Sep 17 00:00:00 2001 From: "Nathaniel J. Smith" Date: Wed, 22 Aug 2018 02:46:42 -0700 Subject: [PATCH 09/12] Add docs for ContextVar support --- docs/source/index.rst | 15 +++++++++++---- docs/source/reference.rst | 26 ++++++++++++++++++++++++++ 2 files changed, 37 insertions(+), 4 deletions(-) diff --git a/docs/source/index.rst b/docs/source/index.rst index 9af200c..fd29f82 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -21,15 +21,22 @@ and async I/O in Python. Features include: * Write your own async fixtures: set up an async database connection or start a server inside a fixture, and then use it in your tests. - If you have multiple async fixtures, pytest-trio will even do - setup/teardown concurrently whenever possible. + +* If you have multiple async fixtures, pytest-trio will even do + setup/teardown concurrently whenever possible. (Though honestly, + we're not sure whether this is a good idea or not and might remove + it in the future. If it makes your tests harder to debug, or + conversely provides you with big speedups, `please let us know + `__.) * Integration with the fabulous `Hypothesis `__ library, so your async tests can use property-based testing: just use ``@given`` like you're used to. -* Supports testing projects that use Trio exclusively, and also - projects that support multiple async libraries. +* Support for testing projects that use Trio exclusively and want to + use pytest-trio everywhere, and also for testing projects that + support multiple async libraries and only want to enable + pytest-trio's features for a subset of their test suite. Vital statistics diff --git a/docs/source/reference.rst b/docs/source/reference.rst index 8ddd954..ccdae6a 100644 --- a/docs/source/reference.rst +++ b/docs/source/reference.rst @@ -141,6 +141,32 @@ sequence of actions: 4. Tear down ``fix_b`` and ``fix_c``, concurrently. 5. Tear down ``fix_a``. +We're `seeking feedback +`__ on whether +this feature's benefits outweigh its negatives. + + +Handling of ContextVars +----------------------- + +The :mod:`contextvars` module provides a way to create task-local +variables called :class:`~contextvars.ContextVars`. Normally, in Trio, +each task gets its own :class:`~contextvars.Context`. But pytest-trio +overrides this, and for each test it uses a single +:class:`~contextvars.Context` which is shared by all fixtures and the +test function itself. + +The benefit of this is that you can set +:class:`~contextvars.ContextVars` inside a fixture, and your settings +will be visible in dependent fixtures and the test itself. + +The downside is that if two fixtures are run concurrently (see +previous section), and both mutate the same +:class:`~contextvars.ContextVars`, then there will be a race condition +and the the final value will be unpredictable. If you make one fixture +depend on the other, then this will force an ordering and make the +final value predictable again. + Built-in fixtures ----------------- From d63783c3870b465ea4d7a951385aac7b93b210df Mon Sep 17 00:00:00 2001 From: "Nathaniel J. Smith" Date: Wed, 22 Aug 2018 03:04:14 -0700 Subject: [PATCH 10/12] fix xrefs --- docs/source/reference.rst | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/source/reference.rst b/docs/source/reference.rst index ccdae6a..541b7e7 100644 --- a/docs/source/reference.rst +++ b/docs/source/reference.rst @@ -149,20 +149,22 @@ this feature's benefits outweigh its negatives. Handling of ContextVars ----------------------- -The :mod:`contextvars` module provides a way to create task-local -variables called :class:`~contextvars.ContextVars`. Normally, in Trio, -each task gets its own :class:`~contextvars.Context`. But pytest-trio +The :mod:`contextvars` module lets you create +:class:`~contextvars.ContextVar` objects to represent task-local +variables. Normally, in Trio, each task gets its own +:class:`~contextvars.Context`, so that :class:`~ContextVar` mutations +are only visible to the task that performs them. But pytest-trio overrides this, and for each test it uses a single :class:`~contextvars.Context` which is shared by all fixtures and the test function itself. The benefit of this is that you can set -:class:`~contextvars.ContextVars` inside a fixture, and your settings -will be visible in dependent fixtures and the test itself. +:class:`~contextvars.ContextVar` values inside a fixture, and your +settings will be visible in dependent fixtures and the test itself. The downside is that if two fixtures are run concurrently (see previous section), and both mutate the same -:class:`~contextvars.ContextVars`, then there will be a race condition +:class:`~contextvars.ContextVar`, then there will be a race condition and the the final value will be unpredictable. If you make one fixture depend on the other, then this will force an ordering and make the final value predictable again. From b89b34d993ecbc16c0ec8c3f5f61d0edd5e11bb8 Mon Sep 17 00:00:00 2001 From: "Nathaniel J. Smith" Date: Wed, 22 Aug 2018 03:06:26 -0700 Subject: [PATCH 11/12] this time for sure... --- docs/source/reference.rst | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/docs/source/reference.rst b/docs/source/reference.rst index 541b7e7..6a56539 100644 --- a/docs/source/reference.rst +++ b/docs/source/reference.rst @@ -152,15 +152,19 @@ Handling of ContextVars The :mod:`contextvars` module lets you create :class:`~contextvars.ContextVar` objects to represent task-local variables. Normally, in Trio, each task gets its own -:class:`~contextvars.Context`, so that :class:`~ContextVar` mutations -are only visible to the task that performs them. But pytest-trio -overrides this, and for each test it uses a single -:class:`~contextvars.Context` which is shared by all fixtures and the -test function itself. +:class:`~contextvars.Context`, so that changes to +:class:`~contextvars.ContextVar` objects are only visible inside the +task that performs them. But pytest-trio overrides this, and for each +test it uses a single :class:`~contextvars.Context` which is shared by +all fixtures and the test function itself. The benefit of this is that you can set :class:`~contextvars.ContextVar` values inside a fixture, and your settings will be visible in dependent fixtures and the test itself. +For example, `trio-asyncio `__ +uses a :class:`~contextvars.ContextVar` to hold the current asyncio +loop object, so this lets you open a loop inside a fixture and then +use it inside other fixtures or the test itself. The downside is that if two fixtures are run concurrently (see previous section), and both mutate the same From 8ed00927a3b8d3b2a3beb612d61597f9fc0dca22 Mon Sep 17 00:00:00 2001 From: "Nathaniel J. Smith" Date: Fri, 24 Aug 2018 05:06:27 -0700 Subject: [PATCH 12/12] Update README also --- README.rst | 44 ++++++++++++++++++++++++++++++++++++++------ 1 file changed, 38 insertions(+), 6 deletions(-) diff --git a/README.rst b/README.rst index fe1b031..e919304 100644 --- a/README.rst +++ b/README.rst @@ -1,17 +1,49 @@ pytest-trio =========== +.. image:: https://img.shields.io/badge/chat-join%20now-blue.svg + :target: https://gitter.im/python-trio/general + :alt: Join chatroom + +.. image:: https://img.shields.io/badge/docs-read%20now-blue.svg + :target: https://pytest-trio.readthedocs.io/en/latest/?badge=latest + :alt: Documentation Status + +.. image:: https://img.shields.io/pypi/v/pytest-trio.svg + :target: https://pypi.org/project/pytest-trio + :alt: Latest PyPi version + .. image:: https://travis-ci.org/python-trio/pytest-trio.svg?branch=master - :target: https://travis-ci.org/python-trio/pytest-trio + :target: https://travis-ci.org/python-trio/pytest-trio + :alt: Automated test status (Linux and MacOS) .. image:: https://ci.appveyor.com/api/projects/status/aq0pklx7hanx031x?svg=true - :target: https://ci.appveyor.com/project/touilleMan/pytest-trio + :target: https://ci.appveyor.com/project/touilleMan/pytest-trio + :alt: Automated test status (Windows) .. image:: https://codecov.io/gh/python-trio/pytest-trio/branch/master/graph/badge.svg - :target: https://codecov.io/gh/python-trio/pytest-trio + :target: https://codecov.io/gh/python-trio/pytest-trio + :alt: Test coverage + +This is a pytest plugin to help you test projects that use `Trio +`__, a friendly library for concurrency +and async I/O in Python. For an overview of features, see our `manual +`__, or jump straight to the +`quickstart guide +`__. + + +Vital statistics +---------------- + +**Documentation:** https://pytest-trio.readthedocs.io -Welcome to `pytest-trio `__! +**Bug tracker and source code:** +https://github.com/python-trio/pytest-trio -Pytest plugin for trio +**License:** MIT or Apache 2, your choice. -License: Your choice of MIT or Apache License 2.0 +**Code of conduct:** Contributors are requested to follow our `code of +conduct +`__ +in all project spaces.