Problem
Five plugins in pj-official-plugins independently decide which column is the time axis, with four different rules and no test that any two agree:
| Plugin |
Location |
Rule |
parser_arrow (PR #279) |
parser_arrow/src/table_shaper.cpp detectTimestampLeaf |
first TIMESTAMP leaf, then names timestamp_ns, recording_timestamp_ns, timestamp, time, ts — case-sensitive, scalar leaves of TIMESTAMP/int64/uint64/double only |
toolbox_mosaico (PR #282) |
toolbox_mosaico/src/arrow_ipc_message.cpp detectTimestampLeaf |
same five names, no type filter on the name pass |
data_load_parquet |
data_load_parquet/parquet_helpers.hpp findTimestampColumn |
first TIMESTAMP field, then timestamp, time, t, ts, time_stamp, datetime, date_time, _timestamp, _time — case-insensitive |
data_load_lerobot |
data_load_lerobot/lerobot_plugin.cpp |
literal "timestamp", else frame_index / fps |
data_load_ulog |
data_load_ulog/ulog_flatten.hpp |
literal "timestamp" (a ULog format rule, listed for completeness) |
Consequences already observed:
- The same Parquet file gets a different time axis through the parquet loader than through
arrow-ipc → parser_arrow (parquet finds t, datetime, Timestamp; parser_arrow finds timestamp_ns, recording_timestamp_ns; neither finds the other's).
- Before #279's fix, a column named
ts of type int8 silently became a topic's time axis — every integer width was accepted as a nanosecond epoch (uint8 can express 255 ns since 1970). Mosaico's copy in #282 still has no type filter, so it now names such a column explicitly and the parser accepts it with a warning.
- The unit is inferred from storage type (integers = nanoseconds, float/double = seconds), so a
float32 epoch axis cannot distinguish samples 128 s apart. Nothing in the contract lets a producer say timestamp_unit: s.
- #282 also carries a verbatim copy of parser_arrow's
floatingSecondsToNanoseconds with a comment asserting bit-for-bit equality; #279 has since replaced that function, so the copy is now the divergent one.
Every new Arrow-facing plugin copies one of these; the copies drift on the day they are made.
Precedent
SDK 0.24.0 added pj_plugins/sdk/parser_array_policy.hpp — "the cross-parser max-array-size + clamp/skip contract with its canonical and legacy JSON keys" — precisely so plugins stopped carrying private copies of a shared contract. parser_arrow already consumes it. This asks for the same move for the timestamp axis.
Proposal: pj_plugins/sdk/timestamp_policy.hpp
Header-only, no virtuals, no Arrow dependency in the contract (each plugin adapts its own schema — nanoarrow or libarrow — into candidates):
namespace PJ::sdk {
enum class TimeKind : uint8_t { kNativeTimestamp, kInt64, kUInt64, kUInt32, kNarrowInt, kFloat32, kFloat64, kOther };
struct TimestampCandidate {
std::string_view name; // flattened leaf path, '/'-separated, dots normalized
TimeKind kind;
bool is_list_element; // never auto-selected
};
struct TimestampPolicy {
std::span<const std::string_view> names; // priority order
bool case_insensitive;
};
inline constexpr std::array kCanonicalTimestampNames = { /* union of the lists in use, priority order */ };
inline constexpr TimestampPolicy kCanonicalPolicy{kCanonicalTimestampNames, /*case_insensitive=*/true};
/// Type pass (first native TIMESTAMP scalar), then name pass restricted to plausible kinds.
[[nodiscard]] std::optional<std::size_t> detectTimestampColumn(std::span<const TimestampCandidate>,
const TimestampPolicy& = kCanonicalPolicy);
/// For an explicitly configured column: ok, or a warning the plugin should surface.
[[nodiscard]] std::optional<std::string_view> explicitAxisWarning(TimeKind kind);
/// Seconds → int64 ns, integer-split (platform-independent; round half away from zero).
[[nodiscard]] std::optional<int64_t> secondsToNanoseconds(double seconds) noexcept;
} // namespace PJ::sdk
Rules the shared policy owns:
- Type pass before name pass; native TIMESTAMP wins.
- Never auto-select a narrow integer, uint32 or float32 as an axis (cannot express an epoch nanosecond beyond seconds); never a list element.
- One agreed name list (union of parquet's and parser_arrow's), case-insensitive, with an exact-case-first tie rule.
- Explicit selection of an implausible type warns rather than silently producing epoch-adjacent timestamps.
- Canonical config keys:
timestamp_column, and timestamp_unit: ns|us|ms|s so the unit stops being inferred from storage type.
- One seconds→nanoseconds conversion so plugins stop carrying copies of each other's.
Adopters
parser_arrow, toolbox_mosaico, data_load_parquet, data_load_lerobot. Not data_load_csv (it parses untyped text; column selection is the user's in its dialog) and not data_load_ulog (format-mandated field).
Blast radius
New header + SDK minor bump. No ABI change. Adopters opt in per plugin.
Context
Found while reviewing pj-official-plugins #279 (parser_arrow) and #282 (Mosaico transport). The two PRs land the fifth and a reworked second copy of this heuristic within a day of each other.
Problem
Five plugins in pj-official-plugins independently decide which column is the time axis, with four different rules and no test that any two agree:
parser_arrow(PR #279)parser_arrow/src/table_shaper.cppdetectTimestampLeaftimestamp_ns, recording_timestamp_ns, timestamp, time, ts— case-sensitive, scalar leaves of TIMESTAMP/int64/uint64/double onlytoolbox_mosaico(PR #282)toolbox_mosaico/src/arrow_ipc_message.cppdetectTimestampLeafdata_load_parquetdata_load_parquet/parquet_helpers.hppfindTimestampColumntimestamp, time, t, ts, time_stamp, datetime, date_time, _timestamp, _time— case-insensitivedata_load_lerobotdata_load_lerobot/lerobot_plugin.cpp"timestamp", elseframe_index / fpsdata_load_ulogdata_load_ulog/ulog_flatten.hpp"timestamp"(a ULog format rule, listed for completeness)Consequences already observed:
arrow-ipc→parser_arrow(parquet findst,datetime,Timestamp; parser_arrow findstimestamp_ns,recording_timestamp_ns; neither finds the other's).tsof typeint8silently became a topic's time axis — every integer width was accepted as a nanosecond epoch (uint8can express 255 ns since 1970). Mosaico's copy in #282 still has no type filter, so it now names such a column explicitly and the parser accepts it with a warning.float32epoch axis cannot distinguish samples 128 s apart. Nothing in the contract lets a producer saytimestamp_unit: s.floatingSecondsToNanosecondswith a comment asserting bit-for-bit equality; #279 has since replaced that function, so the copy is now the divergent one.Every new Arrow-facing plugin copies one of these; the copies drift on the day they are made.
Precedent
SDK 0.24.0 added
pj_plugins/sdk/parser_array_policy.hpp— "the cross-parser max-array-size + clamp/skip contract with its canonical and legacy JSON keys" — precisely so plugins stopped carrying private copies of a shared contract.parser_arrowalready consumes it. This asks for the same move for the timestamp axis.Proposal:
pj_plugins/sdk/timestamp_policy.hppHeader-only, no virtuals, no Arrow dependency in the contract (each plugin adapts its own schema — nanoarrow or libarrow — into candidates):
Rules the shared policy owns:
timestamp_column, andtimestamp_unit: ns|us|ms|sso the unit stops being inferred from storage type.Adopters
parser_arrow,toolbox_mosaico,data_load_parquet,data_load_lerobot. Notdata_load_csv(it parses untyped text; column selection is the user's in its dialog) and notdata_load_ulog(format-mandated field).Blast radius
New header + SDK minor bump. No ABI change. Adopters opt in per plugin.
Context
Found while reviewing pj-official-plugins #279 (
parser_arrow) and #282 (Mosaico transport). The two PRs land the fifth and a reworked second copy of this heuristic within a day of each other.