Fix infinite loop in JBIG2 decoder with >4 referred-to segments

[Gaurang-5]

Fixes #20439

Summary

Fixes an infinite loop issue when decoding JBIG2 images with more than 4 referred-to segments.

Details

The bug had two parts:

1. Incorrect condition check: The code was checking the entire referredFlags byte (referredFlags === 7) instead of checking the extracted referredToCount value (the top 3 bits: referredToCount === 7)

2. Incorrect byte count calculation: The calculation for retention flags bytes was (referredToCount + 7) >> 3 when it should be (referredToCount + 8) >> 3

According to the JBIG2 specification, retention flags require (referredToCount + 1) bits total: 1 bit for the segment itself plus 1 bit for each referred segment. The correct byte count is ceil((referredToCount + 1) / 8) which equals (referredToCount + 8) >> 3.

This issue was causing the parser to read incorrect positions, resulting in an infinite loop in the readSegments function.

Testing

• Tested with the provided PDF from the issue (hmm.pdf)
• All existing unit tests pass (1065 specs, 0 failures)
• Fix aligns with solutions in other PDF renderers (SerenityOS, PDFium)

[timvandermeij]

Please add a test case (see e.g. https://github.com/mozilla/pdf.js/pull/20270/files for how to do that) which serves as a regression test. After that we can trigger the tests here.

[nico]

Feel free to use the file attached to the issue :)

[calixteman]

@nico, it's super easy to add the test yourself:

• copy your file in test/pdfs/
• add an entry in test_manifest.json very similar to:

  pdf.js/test/test_manifest.json

  Lines 42 to 46 in ddf3a98

  	"id": "tracemonkey-eq",
  	"file": "pdfs/tracemonkey.pdf",
  	"md5": "9a192d8b1a7dc652a19835f6f08098bd",
  	"rounds": 1,
  	"type": "eq"

commit and push.

[Gaurang-5]

@timvandermeij
I've added a regression test for this issue. The test uses the PDF file from the bug report and verifies that it can be decoded without hanging. All tests pass locally.

The test file is located at:

• test/pdfs/issue20439.pdf
• Entry added to test/test_manifest.json

Please let me know if you need any changes!
Thanks

[timvandermeij]

Looks good to me like this, with the comment addressed, the commits squashed into one and passing tests. Thanks!

[calixteman]

/botio test

[moz-tools-bot]

From: Bot.io (Windows)

---

Received

Command cmd_test from @calixteman received. Current queue size: 0

Live output at: http://54.193.163.58:8877/73e7f20e09be864/output.txt

[moz-tools-bot]

From: Bot.io (Linux m4)

---

Received

Command cmd_test from @calixteman received. Current queue size: 0

Live output at: http://54.241.84.105:8877/5a1410188c64b28/output.txt

[moz-tools-bot]

From: Bot.io (Linux m4)

---

Failed

Full output at http://54.241.84.105:8877/5a1410188c64b28/output.txt

Total script time: 40.08 mins

• Unit tests: Passed
• Integration Tests: FAILED
• Regression tests: FAILED

  different ref/snapshot: 1

Image differences available at: http://54.241.84.105:8877/5a1410188c64b28/reftest-analyzer.html#web=eq.log

[moz-tools-bot]

From: Bot.io (Windows)

---

Failed

Full output at http://54.193.163.58:8877/73e7f20e09be864/output.txt

Total script time: 75.00 mins

• Unit tests: FAILED
• Integration Tests: FAILED
• Regression tests: FAILED

  different ref/snapshot: 1

Image differences available at: http://54.193.163.58:8877/73e7f20e09be864/reftest-analyzer.html#web=eq.log

[timvandermeij]

Could you squash the commits into one so we have a single commit for the change (see https://github.com/mozilla/pdf.js/wiki/Squashing-Commits if you're not familiar with how to do that)? This should be good to merge then.