Silent data loss: emptying an implicitly-created table (dotted key) makes it vanish from as_string() while unwrap() still contains it

[Master-Norna]

Summary

After removing keys from a table that was created implicitly (via a dotted key, with no [header] of its own), an empty intermediate table remains in the in-memory model (unwrap() / dict view) but is silently dropped from as_string() / dumps() output. The rendered document no longer round-trips: parse(doc.as_string()).unwrap() != doc.unwrap() — data is lost without any warning.

This is a silent data loss: no exception, no error, the output is perfectly valid TOML — it just no longer contains a table the user can still see in the in-memory model.

Minimal reproduction (copy-paste)

import tomlkit

doc = tomlkit.parse("[a]\nb.c = 1\nd = 2\n")
del doc["a"]["b"]["c"]          # empty the implicitly-created table `a.b`

s = doc.as_string()
print(repr(s))                  # '[a]\nd = 2\n'   <- no trace of `a.b`
print(doc.unwrap())             # {'a': {'b': {}, 'd': 2}}   <- `a.b` still here
print(tomlkit.parse(s).unwrap())# {'a': {'d': 2}}            <- `a.b` lost

Also works with pure top-level dotted keys, remove(), and clear():

doc = tomlkit.parse("a.b.c = 1\na.d = 2\n")
doc["a"]["b"].remove("c")
# as_string() -> 'a.d = 2\n'
# unwrap()    -> {'a': {'b': {}, 'd': 2}}
# reparse     -> {'a': {'d': 2}}

doc = tomlkit.parse("[a]\nb.c = 1\nd = 2\n")
doc["a"]["b"].clear()
# same mismatch

What is NOT affected (verified)

• Tables with an explicit [a.b] header: the empty header is preserved ([a.b]\n), round-trips fine.
• Deleting the whole table (del doc["a"]["b"]): consistent.
• Inline tables and AoT elements: consistent.

So the bug is specific to implicit tables — those created by dotted keys (b.c = 1, a.b.c = 1) or by out-of-order [a.b] sections where the parent got no header of its own.

Frequency

A random-mutation fuzz (parse -> 1-4 random delete/add/replace ops -> compare unwrap() vs parse(as_string()).unwrap()) hit this in 274 of 3000 runs. It is the dominant round-trip failure mode.

Root cause

When the parser creates an intermediate table from a dotted key, it sets Table._is_super_table = True (items.py:1948-1949). is_super_table() returns that frozen flag first, so the if not self: return False fallback (which would correctly make an empty table render its own header) is never reached after the table has been emptied.

Container._render_table (container.py:672-708) then treats the table as a super table: no [a.b] header is emitted, and with no children left to render, the table renders as the empty string — while Container.remove / _remove_at (container.py:476-514) never invalidate the frozen _is_super_table flag on the emptied table, and the table stays in the parent's _body and dict view (hence unwrap() still contains it).

Expected

Either the emptied implicit table keeps rendering (e.g. as [a.b], matching the explicit-header behavior), or it is removed from the in-memory model together with the rendered output — but unwrap() and parse(as_string()).unwrap() should never disagree.

Related

• Deleting a table creates invalid output #204 "Deleting a table creates invalid output" (closed) — different symptom (duplicate headers when deleting a whole multi-tier table); this one is about the contents of an implicit table being emptied.

Environment

tomlkit master @ 8c959b5 (0.15.1+), Python 3.12.

[dimbleby]

Duplicate #553

[Master-Norna]

Confirmed — this is the super-table half of #553, and PR #590 already implements the fix (render the header of an emptied dotted-key super table), with a regression test covering this exact repro. I won't open a competing PR.

Two notes for the thread:

1. tomlkit does not serialize some empty arrays and dictionaries #553's repro also contains an array-of-tables half ([[a]] emptied via pop()), which Render the header of emptied super tables created from dotted keys #590 explicitly does not address ("fixes only the super-table half of tomlkit does not serialize some empty arrays and dictionaries #553"). So tomlkit does not serialize some empty arrays and dictionaries #553 remains partially open even once Render the header of emptied super tables created from dotted keys #590 lands — the AoT half still needs a fix.
2. For frequency data: a random-mutation fuzz (parse → 1–4 random delete/add/replace ops → compare unwrap() vs parse(as_string()).unwrap()) hit the super-table half in 274 of 3000 runs — the dominant round-trip failure mode in that sample. The root-cause analysis in the issue body (_is_super_table frozen at parse, never invalidated on the delete path) matches what Render the header of emptied super tables created from dotted keys #590 changes.

[dimbleby]

You forgot to close this issue