5

I am trying to write tests for an asyncio-based Python app that uses Redis.

I reproduced the Python code from the official Redis as a document database quick start guide and turned it into tests:

import pytest

from redis.asyncio import Redis
from redis.commands.json.path import Path
from redis.commands.search import AsyncSearch
from redis.commands.search.field import NumericField, TagField, TextField
from redis.commands.search.indexDefinition import IndexDefinition, IndexType
from redis.commands.search.query import Query
from redis.exceptions import ResponseError

@pytest.fixture(scope="session")
async def redis():
    redis = Redis(host="localhost", port=6379, db=0, decode_responses=True)
    yield redis
    await redis.aclose()

bicycle_ = {
    "brand": "Velorim",
    "model": "Jigger",
    "price": 270,
    "description": (
        "Small and powerful, the Jigger is the best ride "
        "for the smallest of tikes! This is the tiniest "
        "kids’ pedal bike on the market available without"
        " a coaster brake, the Jigger is the vehicle of "
        "choice for the rare tenacious little rider "
        "raring to go."
    ),
    "condition": "new",
}

bicycles_ = [
    bicycle_,
    {
        "brand": "Bicyk",
        "model": "Hillcraft",
        "price": 1200,
        "description": (
            "Kids want to ride with as little weight as possible."
            " Especially on an incline! They may be at the age "
            'when a 27.5" wheel bike is just too clumsy coming '
            'off a 24" bike. The Hillcraft 26 is just the solution'
            " they need!"
        ),
        "condition": "used",
    },
    {
        "brand": "Nord",
        "model": "Chook air 5",
        "price": 815,
        "description": (
            "The Chook Air 5  gives kids aged six years and older "
            "a durable and uberlight mountain bike for their first"
            " experience on tracks and easy cruising through forests"
            " and fields. The lower  top tube makes it easy to mount"
            " and dismount in any situation, giving your kids greater"
            " safety on the trails."
        ),
        "condition": "used",
    },
]

schema = (
    TextField("$.brand", as_name="brand"),
    TextField("$.model", as_name="model"),
    TextField("$.description", as_name="description"),
    NumericField("$.price", as_name="price"),
    TagField("$.condition", as_name="condition"),
)

@pytest.fixture(scope="session")
async def create_bicycle_index(redis: Redis):
    index = redis.ft("idx:bicycle")
    try:
        await index.dropindex()
    except ResponseError as e:
        pass
    await index.create_index(
        schema,
        definition=IndexDefinition(prefix=["bicycle:"], index_type=IndexType.JSON),
    )

@pytest.fixture(scope="session")
async def bicycles(redis: Redis):
    for bid, bicycle in enumerate(bicycles_):
        await redis.json().set(f"bicycle:{bid}", Path.root_path(), bicycle)

@pytest.fixture()
async def bicycle_idx(create_bicycle_index, redis: Redis):
    index = redis.ft("idx:bicycle")
    return index

async def test_search_all_bicycles(bicycles, bicycle_idx: AsyncSearch, redis: Redis):
    res = await bicycle_idx.search(Query("*"))
    assert res.total == 3

async def test_search_jigger_bicycle(bicycles, bicycle_idx: AsyncSearch, redis: Redis):
    res = await bicycle_idx.search(Query("@model:Jigger"))
    assert res.total == 1

Unfortunately it throws the following errors:

E           RuntimeError: Event loop is closed

/usr/lib/python3.12/asyncio/base_events.py:539: RuntimeError
E           RuntimeError: Task <Task pending name='Task-5' coro=<test_search_all_bicycles() running at /home/duranda/devel/redis-pytest/test_redis.py:94> cb=[_run_until_complete_cb() at /usr/lib/python3.12/asyncio/base_events.py:180]> got Future <Future pending> attached to a different loop

/usr/lib/python3.12/asyncio/streams.py:542: RuntimeError

The full trace is available on Pastebin.

I tried to add the code below on top of my module to solve this error; without success.

@pytest.fixture(scope="session")
def event_loop():
    try:
        loop = asyncio.get_running_loop()
    except RuntimeError:
        loop = asyncio.new_event_loop()
    yield loop
    loop.close()

Note that you need to install pytest-asyncio and redis-py and to run a redis-stack server in order to test the above code, e.g.:

pip install pytest-asyncio redis
docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest

I also added the following to pytest.ini in order to not having to use @pytest.mark.asyncio. Note that ( I also tried marking all my function with @pytest.mark.asyncio which did not help.

[pytest]
asyncio_mode = auto
2
  • Please show the full tracebacks Commented Feb 3, 2024 at 23:27
  • I edited the question and added the full trace. Commented Feb 4, 2024 at 10:32

5 Answers 5

3

There are issues with this answer and I am still looking for a better solution.

I was able to fix this issue by passing the event_loop fixture to the redis fixture:

@pytest.fixture(scope="session")
def event_loop():
    try:
        loop = asyncio.get_running_loop()
    except RuntimeError:
        loop = asyncio.new_event_loop()
    yield loop
    loop.close()

@pytest.fixture(scope="session")
async def redis(event_loop):
    redis = Redis(host="localhost", port=6379, db=0, decode_responses=True)
    yield redis
    await redis.aclose()

However, this throws some warnings:

venv/lib/python3.12/site-packages/pytest_asyncio/plugin.py:229
  /home/duranda/devel/redis-geospatial/venv/lib/python3.12/site-packages/pytest_asyncio/plugin.py:229: PytestDeprecationWarning: redis is asynchronous and explicitly requests the "event_loop" fixture. Asynchronous fixtures and test functions should use "asyncio.get_running_loop()" instead.
    warnings.warn(

app/tests/test_app.py::test_search_all_bicycles
  /home/duranda/devel/redis-test/venv/lib/python3.12/site-packages/pytest_asyncio/plugin.py:751: DeprecationWarning: The event_loop fixture provided by pytest-asyncio has been redefined in
  /home/duranda/devel/redis-test/app/tests/test_app.py:13
  Replacing the event_loop fixture with a custom implementation is deprecated
  and will lead to errors in the future.
  If you want to request an asyncio event loop with a scope other than function
  scope, use the "scope" argument to the asyncio mark when marking the tests.
  If you want to return different types of event loops, use the event_loop_policy
  fixture.
Sign up to request clarification or add additional context in comments.

Comments

2

I had a similar problem but for parameterized tests that run an async function.

I used the pytest-asyncio library and decorated my test with @pytest.mark.asyncio(scope="session") and that worked for me. Note that if you don't set the scope, then I was getting the same loop closed error.

Comments

2

The correct answer is to close the Redis client when you finish the tests. For example:

from redis import asyncio as aioredis
client = aioredis.Redis(...)
# do something
client.close()

Comments

1

The issue arises from the difference in scopes between your fixtures and tests.

By default, pytest-asyncio creates a new event loop per function. If you wish to have asyncio session fixtures, your code must run with the same event loop as those fixtures, instead of recreating a new loop each function.

Per the official guide, you can add this code to your conftest.py file (creating one if it doesn't exist):

import pytest

from pytest_asyncio import is_async_test


def pytest_collection_modifyitems(items):
    pytest_asyncio_tests = (item for item in items if is_async_test(item))
    session_scope_marker = pytest.mark.asyncio(scope="session")
    for async_test in pytest_asyncio_tests:
        async_test.add_marker(session_scope_marker)

This ensures all tests run with the same event loop.

3 Comments

I tried this as well as marking all my tests with pytest.mark.asyncio(scope="session") but it did not solve the issue.
Best explanation so far, that I found on SO. Too bad I cannot use this solution, so I will live with the deprecation as long as it holds :-) Why? I'm using an alembic migration fixture and doing it scope="function" instead of "session" would be brutal on test-speed.
@ChristofKälin some time has passed since I've answered, but it looks like it changes everything to session scope instead of function.
0

In addition to existing answers which correctly state that the issue is the scope of the fixtures, setting a session scope for asyncio loop for both fixtures and individual tests can be done globally using the following pytest.ini configuration:

[pytest]
asyncio_mode = auto
asyncio_default_fixture_loop_scope = session
asyncio_default_test_loop_scope = session

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.