How to allow a certain differences in a text assertion for a test case that has some expected changes in data using phpunit?

Question

I am setting up a golden master test case, yet the result may be off at certain points due to side-effects changing some strings at known places.

Getting rid of those side-effects would be best, yet for now, I am OK with those differences being filtered out and the test case succeeding.

Yet a simple:

$this->assertSame($expected, $actual);

fails now every time.

How can I allow expected differences?

This is an example:

        // THIS SHOULD PASS EVEN THOUGH TEXT IS NOT ENTIRLY THE SAME

        $actual =<<<TXT
This should be the same
Here might be an expected DIFF 123 That should be ignored
This is again the same
Also this
Yet another line containing an 23 expected diff
TXT;

        $expected =<<<TXT
This should be the same
Here might be an expected DIFF 177 That should be ignored
This is again the same
Also this
Yet another line containing an 42 expected diff
TXT;

Example of what should fail:

$actual =<<<TXT
This should be the same
Here might be an expected DIFF 123 That should be ignored
This is again the same
Also this
Yet another line containing an 23 expected diff
TXT;

        $expected =<<<TXT
This should be the same
Here might be an expected DIFF 177 That should be ignored
This is again the same
Also this SHOULD be the same but is not
Yet another line containing an 42 expected diff
TXT;

@AlexHowansky No, they might contain special chars, like :. Yet they should always have the same length, and appear at the same order. Each line contains part of a string that remains the same. — k0pernikus
– k0pernikus, Commented May 7 at 18:38
golden master allowing noise. hmm. not sure if that is a good idea. better store the mismatch next to the golden master and then review failures and promote the new golden masters (bless the tests). do that until you've found out that you don't need a golden master any longer (e.g. you've finally implemented an object that can represent the value and is comparable and under test.) just my 2 cents. — hakre
– hakre, Commented May 8 at 0:51

Alex Howansky · Accepted Answer · 2025-05-07 19:58:35Z

2

I think your proposed solution is unnecessarily complex. You should be able to achieve an equal or higher detection rate with just two lines of code.

Since you've stated that the expected diffs always have the same size, I'd start with a simple strlen() check:

$this->assertSame(strlen($expected), strlen($actual));

Then I'd use an amount-of-change check like levenshtein(). It will return the number of characters that need to be replaced/added/removed to get from A to B. So, if you have (for example) 2 points that might contain different text, and each contains a 6-character string, then you know the maximum number of characters difference is 12. Thus, if more than 12 characters differ between the two strings, you have an error.

$this->assertLessThanOrEqual(12, levenshtein($expected, $actual));

answered May 7 at 19:58

Alex Howansky

54k8 gold badges85 silver badges102 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

1stthomas · Accepted Answer · 2025-05-09 11:36:40Z

I went with this naive approach:

$expectedDiffCount = 2;

try {
    self::assertSame($expected_script, $body);
} catch (ExpectationFailedException $e) {
    if ($expectedDiffCount === 0) {
        throw $e;
    }

    $diff = $e->getComparisonFailure()->getDiff();
    $diff = explode("\n", $diff);

    $plus_count = 0;
    $minus_count = 0;

    foreach ($diff as $line) {
        if (str_starts_with($line, '++') || str_starts_with($line, '--')) {
            // false positive in diff output
            continue;
        }


        if (str_starts_with($line, '+')) {
            $plus_count++;
        }

        if (str_starts_with($line, '-')) {
            $minus_count++;
        }
    }

    if ($minus_count === $expectedDiffCount && $plus_count === $expectedDiffCount) {
        return;
    }

    throw $e;
}

This does not check for the actual content of the difference, so if randomly two other errors would appear and the expected ones won't appear, this could pass incorrectly, yet this is not very likely and good enough for my use case.

Collectives™ on Stack Overflow

How to allow a certain differences in a text assertion for a test case that has some expected changes in data using phpunit?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related