Skip to content

Document table-level retention and refresh the README for the merged DAOs - #117

Merged
CritasWang merged 1 commit into
apache:masterfrom
PDGGK:feature/wk10-table-ttl
Aug 3, 2026
Merged

Document table-level retention and refresh the README for the merged DAOs#117
CritasWang merged 1 commit into
apache:masterfrom
PDGGK:feature/wk10-table-ttl

Conversation

@PDGGK

@PDGGK PDGGK commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Retention / TTL

Physical retention was never documented for this module, even though the shipped schema already declares telemetry with WITH (TTL=DEFAULT). This adds a Retention / TTL section to the README covering what an operator actually has to do:

  • IoTDB Table Mode expresses TTL as a table property in milliseconds, and the accepted spellings are narrow: a concrete retention is a bare, unquoted long (TTL=604800000); never-expire is the quoted string TTL='INF' (the form entity_attributes and telemetry_latest already ship with); TTL=DEFAULT inherits the database default. An unquoted TTL=INF is parsed as an identifier and rejected, and any other quoted value — a quoted number ('604800000') or a duration ('7d') — fails with ttl value must be 'INF' or a long literal. All four forms were checked against a 2.0.8 container.
  • A concrete retention is set either at create time (CREATE TABLE telemetry (...) WITH (TTL=604800000)) or at runtime (ALTER TABLE telemetry SET PROPERTIES TTL=604800000), and read back from SHOW TABLES or from information_schema.tables, where the column has to be quoted as "ttl(ms)".
  • telemetry_latest stays TTL='INF' deliberately: it holds one row per identity and is the authority for a latest value, so expiring it would drop the latest reading while history remained.

It also records a limitation that was implicit in the code and easy to misread: ThingsBoard's TimeseriesDao.save(..., long ttl) carries a per-data-point TTL, but IoTDB retention can only be expressed per table, so the two cannot be reconciled faithfully. The module uses that argument (with iotdb.defaultTtlMs) only for ThingsBoard's storage data-point accounting, never as an instruction to expire rows.

IoTDBTableTtlIT pins the operator paths against a real apache/iotdb:2.0.8-standalone container: the schema bootstrap resolving TTL=DEFAULT, the shipped TTL='INF' on entity_attributes reading back as INF, ALTER TABLE ... SET PROPERTIES TTL=<ms> and back to DEFAULT, and a CREATE TABLE ... WITH (TTL=<ms>) read back exactly. It asserts the TTL property mechanism only — not physical row eviction, which is asynchronous and compaction-driven and so is not deterministic inside a test.

README accuracy

Writing that section surfaced that the rest of the README still describes the module as it was several changes ago, so this brings it back in line with the code:

  • Scope, the configuration table and Status said time-bucketed aggregation was not implemented and that aggregation, latest telemetry and the attribute DAO were outside the current scope. All three are implemented, so those sections now describe the three DAOs and their independent selectors.
  • The iotdb.ts.experimental-raw-only row now says what the flag really does: the name predates aggregation support and is kept for compatibility, but it is the opt-in for the backend as a whole rather than a raw-only switch.
  • The latest-telemetry limitation bullet said key discovery returns an empty list. findAllKeysByEntityIds reads DISTINCT key from both telemetry and the telemetry_latest overlay; only the batch findLatestByEntityIds deferral remains, and that bullet is kept.
  • The overview said the integration tests cover the write path; they cover write, read, aggregation, latest telemetry, attributes and now retention.

Tests

Full module build is green on JDK 17: 190 unit tests and 54 container integration tests, including the two new TTL tests.

…DAOs

Add a Retention / TTL section describing how physical retention is set on
the IoTDB side: it is a table property in milliseconds, declared as
WITH (TTL=DEFAULT) in the shipped schema and changed with ALTER TABLE ...
SET PROPERTIES TTL=<ms>, readable back from information_schema.tables. It
also records the Phase-1 limitation that ThingsBoard's per-save ttl
argument cannot be a physical-retention directive, because IoTDB
retention is table-wide, so the module uses it only for storage
data-point accounting.

IoTDBTableTtlIT pins both the schema-default and the runtime-ALTER paths
against a real IoTDB 2.0.8 container. It asserts the TTL property
mechanism only, not physical row eviction, which is asynchronous and
compaction-driven and therefore not deterministic in a test.

While writing that section the surrounding README turned out to describe
the module as it was several changes ago, so this also brings it back in
line with the code: time-bucketed aggregation, latest telemetry and the
attribute DAO are implemented rather than out of scope, and key discovery
now reads DISTINCT keys from both telemetry and the telemetry_latest
overlay instead of returning an empty list. The batch findLatestByEntityIds
deferral is unchanged and stays documented.

Signed-off-by: Zihan Dai <99155080+PDGGK@users.noreply.github.com>
@CritasWang
CritasWang merged commit d6245f3 into apache:master Aug 3, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants