Skip to content

Shared timestamp-axis policy header (pj_plugins/sdk/timestamp_policy.hpp): one detector, one plausibility rule, one seconds→ns conversion #186

Description

@GNERSIS

Problem

Five plugins in pj-official-plugins independently decide which column is the time axis, with four different rules and no test that any two agree:

Plugin Location Rule
parser_arrow (PR #279) parser_arrow/src/table_shaper.cpp detectTimestampLeaf first TIMESTAMP leaf, then names timestamp_ns, recording_timestamp_ns, timestamp, time, ts — case-sensitive, scalar leaves of TIMESTAMP/int64/uint64/double only
toolbox_mosaico (PR #282) toolbox_mosaico/src/arrow_ipc_message.cpp detectTimestampLeaf same five names, no type filter on the name pass
data_load_parquet data_load_parquet/parquet_helpers.hpp findTimestampColumn first TIMESTAMP field, then timestamp, time, t, ts, time_stamp, datetime, date_time, _timestamp, _timecase-insensitive
data_load_lerobot data_load_lerobot/lerobot_plugin.cpp literal "timestamp", else frame_index / fps
data_load_ulog data_load_ulog/ulog_flatten.hpp literal "timestamp" (a ULog format rule, listed for completeness)

Consequences already observed:

  • The same Parquet file gets a different time axis through the parquet loader than through arrow-ipcparser_arrow (parquet finds t, datetime, Timestamp; parser_arrow finds timestamp_ns, recording_timestamp_ns; neither finds the other's).
  • Before #279's fix, a column named ts of type int8 silently became a topic's time axis — every integer width was accepted as a nanosecond epoch (uint8 can express 255 ns since 1970). Mosaico's copy in #282 still has no type filter, so it now names such a column explicitly and the parser accepts it with a warning.
  • The unit is inferred from storage type (integers = nanoseconds, float/double = seconds), so a float32 epoch axis cannot distinguish samples 128 s apart. Nothing in the contract lets a producer say timestamp_unit: s.
  • #282 also carries a verbatim copy of parser_arrow's floatingSecondsToNanoseconds with a comment asserting bit-for-bit equality; #279 has since replaced that function, so the copy is now the divergent one.

Every new Arrow-facing plugin copies one of these; the copies drift on the day they are made.

Precedent

SDK 0.24.0 added pj_plugins/sdk/parser_array_policy.hpp — "the cross-parser max-array-size + clamp/skip contract with its canonical and legacy JSON keys" — precisely so plugins stopped carrying private copies of a shared contract. parser_arrow already consumes it. This asks for the same move for the timestamp axis.

Proposal: pj_plugins/sdk/timestamp_policy.hpp

Header-only, no virtuals, no Arrow dependency in the contract (each plugin adapts its own schema — nanoarrow or libarrow — into candidates):

namespace PJ::sdk {

enum class TimeKind : uint8_t { kNativeTimestamp, kInt64, kUInt64, kUInt32, kNarrowInt, kFloat32, kFloat64, kOther };

struct TimestampCandidate {
  std::string_view name;   // flattened leaf path, '/'-separated, dots normalized
  TimeKind kind;
  bool is_list_element;    // never auto-selected
};

struct TimestampPolicy {
  std::span<const std::string_view> names;   // priority order
  bool case_insensitive;
};

inline constexpr std::array kCanonicalTimestampNames = { /* union of the lists in use, priority order */ };
inline constexpr TimestampPolicy kCanonicalPolicy{kCanonicalTimestampNames, /*case_insensitive=*/true};

/// Type pass (first native TIMESTAMP scalar), then name pass restricted to plausible kinds.
[[nodiscard]] std::optional<std::size_t> detectTimestampColumn(std::span<const TimestampCandidate>,
                                                               const TimestampPolicy& = kCanonicalPolicy);

/// For an explicitly configured column: ok, or a warning the plugin should surface.
[[nodiscard]] std::optional<std::string_view> explicitAxisWarning(TimeKind kind);

/// Seconds → int64 ns, integer-split (platform-independent; round half away from zero).
[[nodiscard]] std::optional<int64_t> secondsToNanoseconds(double seconds) noexcept;

}  // namespace PJ::sdk

Rules the shared policy owns:

  1. Type pass before name pass; native TIMESTAMP wins.
  2. Never auto-select a narrow integer, uint32 or float32 as an axis (cannot express an epoch nanosecond beyond seconds); never a list element.
  3. One agreed name list (union of parquet's and parser_arrow's), case-insensitive, with an exact-case-first tie rule.
  4. Explicit selection of an implausible type warns rather than silently producing epoch-adjacent timestamps.
  5. Canonical config keys: timestamp_column, and timestamp_unit: ns|us|ms|s so the unit stops being inferred from storage type.
  6. One seconds→nanoseconds conversion so plugins stop carrying copies of each other's.

Adopters

parser_arrow, toolbox_mosaico, data_load_parquet, data_load_lerobot. Not data_load_csv (it parses untyped text; column selection is the user's in its dialog) and not data_load_ulog (format-mandated field).

Blast radius

New header + SDK minor bump. No ABI change. Adopters opt in per plugin.

Context

Found while reviewing pj-official-plugins #279 (parser_arrow) and #282 (Mosaico transport). The two PRs land the fifth and a reworked second copy of this heuristic within a day of each other.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions