How many assertions per test?

Having multiple assert statements in a test is totally fine...

sometimes...

How multiple assert statements per test can bite you

There is a downside to consider: some test frameworks (like pytest) will stop the test on the first assertion that fails so you won't know the outcomes of any following assertions. This becomes more important the longer it takes for you to run your test suite.

When multiple assert statements is fine

Knowing that, multiple assertions are great when they are designed and ordered in such a way that if one of them fails, the following assertions will certainly fail.

For example, checking for a specific key in a dictionary then checking the value of that key.

If the key isn't in the dictionary, then the following assertion will certainly fail so we aren't missing information by not reaching it.

In the following snippet, the 2nd parameter is a good example of a case where this approach can work well.

import pytest

@pytest.mark.parametrize("json_data", [
    {"key": "good afternoon!"},
    {},
    {"key": "good morning!", "evil key!": "i shouldn't be here!"}
])
def test_multiple_assert_statements(json_data: dict):
    assert "key" in json_data
    assert json_data["key"] == "good afternoon!"

The "key" key is not in json_data so it will fail and the following assertion won't run.

That's not so bad considering what the following assertion does:

    assert json_data["key"] == "good afternoon!"

We already know "key" isn't in json_data. So we won't learn anything new if pytest could evaluate the following assertion because it'd be a KeyError.

Multiple assert statements can hide information

But let's say we find a bug where an "evil key!" makes its way into json_data. We need to be certain that "evil key!" is never included in json_data. So, recognizing a pattern in the unit test, we add a second following assertion.

Here's what the test looks like now with our new assertion:

@pytest.mark.parametrize("json_data", [
    {"key": "good afternoon!"},
    {},
    {"key": "good morning!", "evil key!": "i shouldn't be here!"}
])
def test_multiple_assert_statements(json_data: dict):
    assert "key" in json_data
    assert json_data["key"] == "good afternoon!"
    assert "evil key!" not in json_data

Let's think through what will happen with the 2nd parameter again, the empty dictionary: {}.

We've been over what it means for the first and second assertions: we can infer the result of the second assertion given the first one fails.

But could we learn something new if pytest could reach the third assertion?

    ...
    assert "evil key!" not in json_data

We definitely could.

For the {} case, the only thing we know is that "key" is not in json_data. We don't know if json_data had an "evil key!". And since pytest won't reach this assertion, we won't know until we fix the missing "key" bug.

This example is contrived. We're not actually using the parameters as input to any function or method. There are no "act" steps.

Here's a more realistic example:

import pytest
from my_application import my_application

@pytest.mark.parametrize("input_data", [
    "alpha",
    "beta",
    "gamma"
])
def test_multiple_assert_statements(input_data: str):
    json_data = my_application(input_data)
    assert "key" in json_data
    assert json_data["key"] == "good afternoon!"
    assert "evil key!" not in json_data

Each of our parameters will now be passed into my_application which will return the json_data we care so much about.

Let's say we run this test and the "beta" parameter fails here:

    ...
    json_data = my_application("beta")
  > assert "key" in json_data
    assert json_data["key"] == "good afternoon!"
    assert "evil key!" not in json_data

We can still infer that the 2nd assertion would certainly have failed as well. But we can't know for certain the outcome of the 3rd assertion.

To do that, we have to revisit the my_application function and fix the missing "key" bug before we can learn whether or not the "evil key!" bug regressed.

If your test suite takes a long time to run, this can be a painful situation.

Maybe it's okay if we rearrange them carefully?

To fix this, we might try moving the last assertion to run before the others like this:

import pytest
from my_application import my_application

@pytest.mark.parametrize("input_data", [
    "alpha",
    "beta",
    "gamma"
])
def test_multiple_assert_statements(input_data: str):
    json_data = my_application(input_data)
    assert "evil key!" not in json_data
    assert "key" in json_data
    assert json_data["key"] == "good afternoon!"

This way we'll always know if "evil_key!" was included when it shouldn't be.

But we're still going to be missing information if the first assertion ever fails. From the contrived example, consider what would happen if json_data was the third case:

    ...
    json_data = {"key": "goop", "evil key!": "i shouldn't be here!"}
    assert "evil key!" not in json_data
    assert "key" in json_data
    assert json_data["key"] == "good afternoon!"

We will the "evil key!" bug came back. But we'll miss a new bug with "key": it's been incorrectly set to "goop". Nobody wants "goop".

Alternative approaches to multiple assert statements

1. There are pytest plugins (e.g., pytest-assume) that provide what is sometimes called a "soft assert" which is basically an assertion that doesn't fail immediately. These "soft" assertions are accumulated during the course of the test and then at the end it asserts on all of them at once.
2. Build 2 identical data structures: one will represent the expected data and the other will represent the actual data. Hard-code the values you'd like to see for the expected data and fill in the values for the actual data using system under test's output. Then assert once, comparing them with ==.

Here's an example implementation of #2 using the previous example:

import pytest

@pytest.mark.parametrize("json_data", [
    {"key": "good afternoon!"},
    {},
    {"key": "good morning!", "evil key!": "i shouldn't be here!"}
])
def test_multiple_assertions(json_data: dict):
    actual = {
        "key is present": "key" in json_data,
        "key value is": json_data.get("key"),
        "evil key is missing": "evil key!" not in json_data
    }

    expected = {
        "key is present": True,
        "key value is": "good afternoon!",
        "evil key is missing": True
    }

    assert expected == actual

This way, we'll always know the outcome of each of our assertions regardless of any that failed.

Other benefits of this approach:

1. No additional dependencies
2. When your test fails, pytest will print a very readable test report

Here's an edited example of that to show only the assertion messages:

> pytest ./test_multiple_assertions.py::test_multiple_assertions
<snip>
=============================== FAILURES ================================
_________________ test_multiple_assertions[json_data1] __________________

json_data = {}

<snip>

>       assert expected == actual
E       AssertionError: assert {'evil key is...d afternoon!'} == {'evil key is...lue is': None}
E         Omitting 1 identical items, use -vv to show
E         Differing items:
E         {'key value is': 'good afternoon!'} != {'key value is': None}
E         {'key is present': True} != {'key is present': False}
E         Use -v to get the full diff

<snip>
_________________ test_multiple_assertions[json_data2] __________________

json_data = {'evil key!': "i shouldn't be here!", 'key': 'good morning!'}

<snip>

>       assert expected == actual
E       AssertionError: assert {'evil key is...d afternoon!'} == {'evil key is...ood morning!'}
E         Omitting 1 identical items, use -vv to show
E         Differing items:
E         {'key value is': 'good afternoon!'} != {'key value is': 'good morning!'}
E         {'evil key is missing': True} != {'evil key is missing': False}
E         Use -v to get the full diff

<snip>