fix(backtest): don't append symbol_suffix to symbols the broker carries bare, + two operator scripts

[Marinski]

Two independent changes. Happy to split them if you'd rather review separately — say the word.

---

1. symbol_suffix was appended to symbols the broker carries bare

_normalize_symbol appended symbol_suffix to every [Tester].Symbol that did not already end with it. Brokers rarely suffix their whole book: Eightcap Global carries 56 suffixed FX pairs (EURUSD.i) alongside 785 bare metals, indices and crypto (XAUUSD, BTCUSD, ASX200). Every non-FX backtest there asked the tester for a symbol that does not exist and came back empty.

The suffix is now skipped when the broker's symbol list has the bare name and lacks the suffixed one. Brokers that carry both forms — BlackBull lists AUDUSD and AUDUSDp — still get the suffix, so symbol_suffix: p keeps meaning "use the prime variant".

That check needs a symbol list, and there is no way to get one at INI-build time: a mode: backtest terminal never attaches the SDK, and Bases/<server>/symbols/*.dat is encrypted. So GET /symbols (unfiltered only) persists what it saw to <terminal>/mt5api-symbols.json, and the INI builder reads that back.

With no cache present the previous append-always behaviour is used unchanged, so a terminal that has never been primed cannot regress. Priming is one GET /<broker>/<account>/symbols per terminal.

Covered by tests/test_symbol_suffix_remap.py: bare-only, suffixed-only, both-forms, and the no-cache fallback.

2. Two operator scripts

Neither is imported by the API; both are run by hand.

prune-terminal-logs.sh — terminal64.exe and metatester64.exe write UTF-16 logs inside each terminal directory. MT5 has no retention setting and nothing prunes them: rotate-logs.sh only covers the API's own logs, and the log-rotator sidecar only ever saw data/shared/logs. They grow without bound.

measure-broker-offsets.py — config.yaml's per-terminal utc_offset is a static number subtracted from every broker timestamp, and nothing in the stack is DST-aware, so a value that is right in August is an hour wrong after the autumn rollover. This reports the measured offset per terminal.

It carries a prominent warning: hitting an SDK route on a mode: backtest terminal launches terminal64.exe, and POST /terminal/shutdown only detaches the SDK client — so the terminal is left holding MT5's single-instance lock, and the next backtest there returns an empty Bars=0 Ticks=0 Symbols=0 report. It must be followed by docker compose down && ./run.sh.

---

make test-unit: 433 passed, 2 skipped. make verify-binaries: OK.

[psyb0t]

I reproduced a stale-cache failure in the current head.

symbol_cache.save() records updated, and age_seconds() can calculate the age, but production never calls age_seconds(). load() returns cached symbols regardless of whether updated is missing, malformed, or years old. I wrote a cache containing XAUUSD, changed its timestamp to epoch, and load() still returned {XAUUSD} with an age of over 1.7 billion seconds.

That makes the stale cache authoritative in _normalize_symbol, so a broker symbol that changes from bare to suffixed, or vice versa, can remain normalized incorrectly until somebody happens to refresh /symbols. This contradicts the module's stated stale-cache fallback behavior.

Please give the cache a finite configured maximum age, have load() treat absent or invalid timestamps as stale, and add an integration-level normalization test proving a stale bare-symbol cache falls back to the suffix behavior.

I also ran scripts/prune-terminal-logs.sh in dry-run mode against old fixture logs. It selected Tester/logs/tester.log, but did not select the documented root terminal/logs/terminal.log. find_logs() only matches Tester/logs and Tester/Agent-*/logs. Please include the root terminal logs path and add coverage for root, Tester, and Agent logs.

[Marinski]

Both fixed in 14394f7.

1. The cache is no longer authoritative forever

You named the design flaw precisely: age_seconds() existed and production never called it. The age check now lives inside load() itself, so no caller can forget it — enforcing it anywhere else is how this bug happened.

load() treats as stale, and returns None for: a cache older than symbol_cache_max_age (default 7 days; SYMBOL_CACHE_MAX_AGE in the environment; clamped rather than raised, since config.py is imported by the whole API), a missing updated stamp, and a malformed one — non-int, bool, zero, negative, "yesterday" are all covered by a parametrized matrix. Stale degrades to the conservative append-always fallback that shipped before the cache existed, never to an old answer presented as a current one. The warning names the remedy (GET /symbols on that terminal).

The integration-level test you asked for runs through the real INI-builder path (_normalize_symbol), both directions in one test so the staleness is provably what flips it: a fresh cache knowing XAUUSD as bare suppresses the remap; the same cache aged to epoch stops suppressing it and XAUUSD.i comes back. Your exact reproduction — epoch timestamp, ~1.7 billion seconds old — is the unit variant beside it. All fail against the previous code.

2. The root terminal log is pruned now

find_logs matched only Tester/logs and Tester/Agent-*/logs, so the documented <terminal>/logs — the terminal's own log, the biggest of the three — was invisible to both the age and the size pass, and grew without bound while every pass reported clean. One */logs/*.log pattern now covers all three locations (find's -path crosses /), with MQL5/ excluded by name: its expert logs already miss the case-sensitive pattern on Linux, but these trees sit on Windows-backed shares where a case-insensitive mount would match them, and truncating an EA's own log mid-run is not this script's call to make.

New behavioral suite (tests/test_prune_terminal_logs.py) runs the real script against a fixture tree rather than asserting on its source text: all three documented locations pruned by age, MQL5/Logs untouched (terminal-level and agent-level), a recent root log surviving, the size cap truncating — not deleting — an oversized root log, and dry-run naming the root log without touching anything. That last one is your reproduction: the previous script fails it by omission. Dockerfile.test now copies the script so the offline suite can execute it.

Merge order

#15 → #16 → #18 → #10, this third. #16 previously carried a stale copy of prune-terminal-logs.sh swept in by its squash — evicted from #16 in its own review round, so this branch is the file's single home and there is no add/add collision. I will rebase this onto master promptly once #16 lands.

Also rebased onto current master (post-#17); the one conflict was the Dockerfile.test COPY line, resolved as the union — this branch adds prune-terminal-logs.sh to it so the offline suite can execute the real script.

Full suite green, lint clean.