Repository navigation
Conversation
…sage C extensions
There was a problem hiding this comment.
🟡 Changes recommended
The boundary tests use payload sizes that place both documents in each pair within the same buffer capacity, missing the intended exact-fit and one-byte-overflow cases.
1 open issue
What changed in this PR
Adds deterministic robustness tests for PyMongo’s BSON decoder and C OP_MSG builders, extending sanitizer coverage without changing production code. ThreadSanitizer work remains separate under PYTHON-6168.
Changes:
- Adds BSON round-trip, forged-length, truncation, and byte-mutation tests.
- Exercises C OP_MSG builders directly with wire-layout and buffer-growth checks.
- Includes message tests in the ASan/UBSan task.
| File | Description |
|---|---|
test/test_message.py |
Adds direct C OP_MSG builder tests, skipped when the extension is unavailable. |
test/test_bson.py |
Adds value-extreme, size-boundary, and hostile-input BSON tests. |
.evergreen/scripts/setup_tests.py |
Adds message tests to sanitizer coverage and updates explanatory comments. |
Review effort: Balanced
| max_message_size = 48 * 1024 * 1024 | ||
|
|
||
|
|
||
| class TestOpMsgBuilders(unittest.TestCase): |
There was a problem hiding this comment.
Should this also have a case that forces a batch split due to the number of documents exceeding max_write_batch_size?
| self.assertEqual(max_doc_size, 0) | ||
|
|
||
| def test_op_msg_multiple_buffer_doublings(self): | ||
| docs = [{"s": "x" * n} for n in (5000, 50, 5)] |
There was a problem hiding this comment.
Can this have a comment explaining how/why these sizes cause the buffer to double multiple times? It's not obvious from the code.
| 1, {"update": "coll"}, docs, True, _OPTS, _FakeBulkWriteContext() | ||
| ) | ||
| self.assertEqual(struct.unpack("<i", msg[:4])[0], len(msg)) | ||
| self.assertIsInstance(to_publish, list) |
There was a problem hiding this comment.
Should we also check the length of to_publish, similar to the test above?
| return doc | ||
|
|
||
|
|
||
| def _typed_documents() -> list[dict[str, Any]]: |
There was a problem hiding this comment.
| def _typed_documents() -> list[dict[str, Any]]: | |
| def _extreme_typed_documents() -> list[dict[str, Any]]: |
The name is a little too generic considering this is specifically about extreme values.
| class TestBSONEncodeDecodeBoundaries(unittest.TestCase): | ||
| """Encode every BSON type at value extremes and re-encode byte-stably. | ||
|
|
||
| Drives buffer growth transitions and every _downcast_and_check path |
There was a problem hiding this comment.
I'm not sure what this sentence means. Do the tests cause buffer growth transitions intentionally? What happens to every _downcast_and_check path exactly?
| def test_element_string_lengths(self): | ||
| base = bson.encode({"a": "x" * 20}) | ||
| # The string length field starts at offset 7. | ||
| for length in (-1, 0x7FFFFFFF): |
There was a problem hiding this comment.
0x7FFFFFFF should be a defined constant since it's used several times in this file.
| def test_mutations_decode_cleanly(self): | ||
| for doc in _typed_documents(): | ||
| data = bson.encode(doc, codec_options=_DATETIME_MS_CODEC) | ||
| for corrupted in self._mutated(data): | ||
| try: | ||
| bson.decode(corrupted, codec_options=_DATETIME_MS_CODEC) | ||
| except (BSONError, ValueError): | ||
| pass | ||
| for truncated in self._truncated(data): | ||
| try: | ||
| bson.decode(truncated, codec_options=_DATETIME_MS_CODEC) | ||
| except (BSONError, ValueError): | ||
| pass |
There was a problem hiding this comment.
| def test_mutations_decode_cleanly(self): | |
| for doc in _typed_documents(): | |
| data = bson.encode(doc, codec_options=_DATETIME_MS_CODEC) | |
| for corrupted in self._mutated(data): | |
| try: | |
| bson.decode(corrupted, codec_options=_DATETIME_MS_CODEC) | |
| except (BSONError, ValueError): | |
| pass | |
| for truncated in self._truncated(data): | |
| try: | |
| bson.decode(truncated, codec_options=_DATETIME_MS_CODEC) | |
| except (BSONError, ValueError): | |
| pass | |
| def test_mutations_decode_cleanly(self): | |
| def _assert_decodes_or_errors(data): | |
| for doc in data: | |
| try: | |
| bson.decode(doc, codec_options=_DATETIME_MS_CODEC) | |
| except (BSONError, ValueError): | |
| pass | |
| for doc in _typed_documents(): | |
| data = bson.encode(doc, codec_options=_DATETIME_MS_CODEC) | |
| _assert_encodes_or_errors(self._mutated(data) | |
| _assert_encodes_or_errors(self._truncated(data) |
Reduce duplication.
| def test_op_msg_no_identifier(self): | ||
| _, msg, total_size, max_doc_size = _cmessage._op_msg(0, {"ismaster": 1}, "", None, _OPTS) | ||
| self.assertIsInstance(msg, bytes) | ||
| self.assertEqual(struct.unpack("<i", msg[:4])[0], len(msg)) |
There was a problem hiding this comment.
struct.unpack("<i", msg[:4])[0] is repeated 6 times, should it be a helper method?
| self.assertIsInstance(msg, bytes) | ||
| self.assertEqual(struct.unpack("<i", msg[:4])[0], len(msg)) | ||
| # 21 bytes of header precede the command document. | ||
| self.assertEqual(total_size, len(msg) - 21) |
There was a problem hiding this comment.
The 21 bytes of header should be a defined constant.
| def test_round_trip_size_boundaries(self): | ||
| # Encoded sizes that exactly fill the C encoder's buffer at each | ||
| # doubling boundary (no realloc), and one byte more (forces a | ||
| # realloc). Encoding {"a": "x" * n} adds 13 bytes of BSON |
There was a problem hiding this comment.
| # realloc). Encoding {"a": "x" * n} adds 13 bytes of BSON | |
| # realloc). Encoding {"a": "x" * n} adds 13 bytes of BSON |
| for doc in _extreme_typed_documents(): | ||
| data = bson.encode(doc, codec_options=_DATETIME_MS_CODEC) | ||
| for corrupted in self._mutated(data): | ||
| self._assert_decodes_or_errors(corrupted) | ||
| for truncated in self._truncated(data): | ||
| self._assert_decodes_or_errors(truncated) |
There was a problem hiding this comment.
Each self._assert_decodes_or_errors call should be in its own subtest.
| # doubling boundary (no realloc), and one byte more (forces a | ||
| # realloc). Encoding {"a": "x" * n} adds 13 bytes of BSON | ||
| # overhead, so use n = capacity - 13 and capacity - 12. | ||
| for capacity in (256, 512, 1024, 2**16, 2**17): |
There was a problem hiding this comment.
Should we also have a case that tests against _MAX_INT32?
There was a problem hiding this comment.
Not, it would require too much memory, added a comment.
| """Decode must either succeed or raise; never crash.""" | ||
| try: | ||
| bson.decode(data, codec_options=_DATETIME_MS_CODEC) | ||
| except (BSONError, ValueError): |
There was a problem hiding this comment.
Are either of these equally valid? Or should we expect decode to always raise a BSONError?
There was a problem hiding this comment.
Tightened to only BSONError

PYTHON-6285
Changes in this PR
Adds deterministic hostile-input and boundary tests for the C BSON decoder and OP_MSG builders, with no production code changes. Ports the test-only portion of PYTHON-6168 (its TSan work stays with that ticket) and adds the OP_MSG tests to the ASan/UBSan sanitizer task.
Test Plan
pytest test/test_bson.py -k "Boundaries or Hostile or Mutation",pytest test/test_message.py -k OpMsg), 14 passed against the pure-Python decoder, clean skips when_cmessageis absent.test_bson.pyandtest_message.pysuites: 158 passed, 2 pre-existing skips;mypyand pre-commit hooks clean.Checklist
Checklist for Author
Checklist for Reviewer