Expected behavior
On a AccountType.Cash account, the proceeds of the automatic "Liquidate from delisting" fill should settle like any other equity sale: DelayedSettlementModel (T+1, 06:00 exchange time by default) moves the amount from UnsettledCashBook to CashBook once the settlement time has passed, so the cash becomes buying power again.
Actual behavior
The liquidation fill lands and the proceeds are added to UnsettledCashBook, but they are never settled for the rest of the backtest. Portfolio.UnsettledCash stays at the full proceeds, Portfolio.Cash and MarginRemaining never recover, while TotalPortfolioValue still counts the money, so the equity curve looks unaffected while the strategy silently cannot redeploy the capital.
Same setup with an ordinary MarketOrder sale of the same security a week earlier settles T+1 correctly, so this is specific to the delisting path.
Cause
AlgorithmManager.Run performs the hourly settlement scan by iterating algorithm.Securities.Values (Engine/AlgorithmManager.cs, foreach (var security in algorithm.Securities.Values) { ... security.SettlementModel.Scan(...) }).
SecurityManager.Values is documented as "Excludes non active or delisted securities" (Common/Securities/SecurityManager.cs). UniverseSelection.HandleDelisting (Engine/DataFeeds/UniverseSelection.cs) sets IsDelisted = true and calls _algorithm.Securities.Remove(data.Symbol) on the Delisted event, in the same time step in which BrokerageTransactionHandler.HandleDelistingNotification fills the synthetic liquidation order and enqueues the proceeds on that security's DelayedSettlementModel.
From that point on the scan loop never visits the delisted security, so DelayedSettlementModel.Scan is never called for it and its _unsettledCashAmounts queue is never drained. Calling Scan by hand from the algorithm drains it immediately (see the log below), which shows the settlement time and the queued amount are both correct and only the scan is unreachable. Securities.ContainsKey still returns true for the symbol (it reads the complete collection), which hides the removal from a casual check.
Margin accounts are unaffected: DefaultBrokerageModel.GetSettlementModel only assigns DelayedSettlementModel to equities when AccountType == Cash.
Reproduce
QC Cloud, LEAN v2.5.0.0.18130 (master). Backtest aff9fc27e21470a270605e2967c91ea1. FMR (First Mercury Financial) was acquired for cash and its map file ends 2011-02-09; any equity held through a delisting reproduces it.
from AlgorithmImports import *
class CashScanProbe(QCAlgorithm):
def initialize(self):
self.set_start_date(2011, 1, 20)
self.set_end_date(2011, 3, 15)
self.set_cash(100000)
self.set_brokerage_model(BrokerageName.INTERACTIVE_BROKERS_BROKERAGE, AccountType.CASH)
eq = self.add_equity("FMR", Resolution.DAILY,
data_normalization_mode=DataNormalizationMode.RAW)
self.fmr = eq.symbol
self.anchor = self.add_equity("SPY", Resolution.DAILY).symbol # keeps on_data firing after FMR is gone
self.bought = False
self.scanned = False
def on_data(self, slice):
bar = slice.bars[self.fmr] if slice.bars.contains_key(self.fmr) else None
syms = sorted(kvp.key.value for kvp in self.securities)
sm = self.securities[self.fmr].settlement_model
self.log(f"STATE {self.time:%Y-%m-%d %H:%M} secKeys={syms} "
f"modelUnsettled={sm.get_unsettled_cash().amount} "
f"portfolioUnsettled={self.portfolio.unsettled_cash:.2f} cash={self.portfolio.cash:.2f}")
for s, d in slice.delistings.items():
self.log(f"DELISTING {s.value} type={d.type} time={d.time:%Y-%m-%d %H:%M}")
if not self.bought and bar is not None:
self.market_order(self.fmr, 1000)
self.bought = True
# diagnostic only: explicit scan well after the T+1 settlement time has passed
if not self.scanned and self.time >= datetime(2011, 2, 14):
self.scanned = True
p = ScanSettlementModelParameters(self.portfolio, self.securities[self.fmr], self.utc_time)
self.log(f"MANUALSCAN before unsettled={self.portfolio.unsettled_cash:.2f}")
sm.scan(p)
self.log(f"MANUALSCAN after unsettled={self.portfolio.unsettled_cash:.2f} cash={self.portfolio.cash:.2f}")
def on_order_event(self, oe):
self.log(f"ORDEREVENT {oe.utc_time:%Y-%m-%d %H:%M:%S}utc status={oe.status} "
f"qty={oe.fill_quantity} px={oe.fill_price} tag={oe.message}")
Log (trimmed):
2011-02-09 00:00 DELISTING FMR type=WARNING time=2011-02-09 00:00
2011-02-09 16:00 STATE secKeys=['FMR', 'SPY'] modelUnsettled=0.0 portfolioUnsettled=0.00 cash=83545.00
2011-02-10 00:00 ORDEREVENT 2011-02-10 05:00:00utc status=FILLED qty=-1000.0 px=16.5 tag=Liquidate from delisting
2011-02-10 00:00 STATE secKeys=['SPY'] modelUnsettled=16500.0 portfolioUnsettled=16500.00 cash=83545.00
2011-02-10 00:00 DELISTING FMR type=DELISTED time=2011-02-10 00:00
2011-02-11 16:00 STATE secKeys=['SPY'] modelUnsettled=16500.0 portfolioUnsettled=16500.00 cash=83545.00
2011-02-14 16:00 STATE secKeys=['SPY'] modelUnsettled=16500.0 portfolioUnsettled=16500.00 cash=83545.00
2011-02-14 16:00 MANUALSCAN before unsettled=16500.00
2011-02-14 16:00 MANUALSCAN after unsettled=0.00 cash=100045.00
Without the manual scan (same algorithm, the MANUALSCAN block removed) the 16,500 is still unsettled at the end of the run on 2011-03-15.
Control (same setup, ordinary market_order(self.fmr, -1000) on 2011-02-01 instead of holding through the delisting): unsettled 16,450 at the 2011-02-02 fill, cash=99990.00 unsettled=0.00 on 2011-02-03. T+1 settlement works on the normal path.
Proposed change
Run the settlement scan over the securities that can still hold unsettled funds rather than only the active ones. The smallest change is to iterate algorithm.Securities.Total for the SettlementModel.Scan call in AlgorithmManager.Run (keeping Values for the margin-interest call if delisted securities should not accrue interest), so a delisted security keeps being scanned until its queue is empty.
An alternative is to move pending settlements off the per-security model on delisting (e.g. UniverseSelection.HandleDelisting or HandleDelistingNotification forcing a final scan at the settlement time), but the queued item's settlement time is in the future at that point, so a scan-on-removal alone does not settle it.
Open questions
- Should a delisted security also keep accruing margin interest in that loop, or only settle?
- Whether the same scan gap affects securities removed by universe selection while they still hold unsettled proceeds (a member removed right after a sale on a cash account). Not tested here.
Reported via Intercom conversation 215476138524644.
Expected behavior
On a
AccountType.Cashaccount, the proceeds of the automatic "Liquidate from delisting" fill should settle like any other equity sale:DelayedSettlementModel(T+1, 06:00 exchange time by default) moves the amount fromUnsettledCashBooktoCashBookonce the settlement time has passed, so the cash becomes buying power again.Actual behavior
The liquidation fill lands and the proceeds are added to
UnsettledCashBook, but they are never settled for the rest of the backtest.Portfolio.UnsettledCashstays at the full proceeds,Portfolio.CashandMarginRemainingnever recover, whileTotalPortfolioValuestill counts the money, so the equity curve looks unaffected while the strategy silently cannot redeploy the capital.Same setup with an ordinary
MarketOrdersale of the same security a week earlier settles T+1 correctly, so this is specific to the delisting path.Cause
AlgorithmManager.Runperforms the hourly settlement scan by iteratingalgorithm.Securities.Values(Engine/AlgorithmManager.cs,foreach (var security in algorithm.Securities.Values) { ... security.SettlementModel.Scan(...) }).SecurityManager.Valuesis documented as "Excludes non active or delisted securities" (Common/Securities/SecurityManager.cs).UniverseSelection.HandleDelisting(Engine/DataFeeds/UniverseSelection.cs) setsIsDelisted = trueand calls_algorithm.Securities.Remove(data.Symbol)on theDelistedevent, in the same time step in whichBrokerageTransactionHandler.HandleDelistingNotificationfills the synthetic liquidation order and enqueues the proceeds on that security'sDelayedSettlementModel.From that point on the scan loop never visits the delisted security, so
DelayedSettlementModel.Scanis never called for it and its_unsettledCashAmountsqueue is never drained. CallingScanby hand from the algorithm drains it immediately (see the log below), which shows the settlement time and the queued amount are both correct and only the scan is unreachable.Securities.ContainsKeystill returnstruefor the symbol (it reads the complete collection), which hides the removal from a casual check.Margin accounts are unaffected:
DefaultBrokerageModel.GetSettlementModelonly assignsDelayedSettlementModelto equities whenAccountType == Cash.Reproduce
QC Cloud, LEAN
v2.5.0.0.18130(master). Backtestaff9fc27e21470a270605e2967c91ea1. FMR (First Mercury Financial) was acquired for cash and its map file ends 2011-02-09; any equity held through a delisting reproduces it.Log (trimmed):
Without the manual scan (same algorithm, the
MANUALSCANblock removed) the 16,500 is still unsettled at the end of the run on 2011-03-15.Control (same setup, ordinary
market_order(self.fmr, -1000)on 2011-02-01 instead of holding through the delisting): unsettled 16,450 at the 2011-02-02 fill,cash=99990.00 unsettled=0.00on 2011-02-03. T+1 settlement works on the normal path.Proposed change
Run the settlement scan over the securities that can still hold unsettled funds rather than only the active ones. The smallest change is to iterate
algorithm.Securities.Totalfor theSettlementModel.Scancall inAlgorithmManager.Run(keepingValuesfor the margin-interest call if delisted securities should not accrue interest), so a delisted security keeps being scanned until its queue is empty.An alternative is to move pending settlements off the per-security model on delisting (e.g.
UniverseSelection.HandleDelistingorHandleDelistingNotificationforcing a final scan at the settlement time), but the queued item's settlement time is in the future at that point, so a scan-on-removal alone does not settle it.Open questions
Reported via Intercom conversation 215476138524644.