Backtest illusion: model trained on a delayed external feed
The model was trained on an external feed and trades in a market with a 5% fee and 8-second delay. Only 11% of orders were filled. Here is the test that exposes the fake edge.
English · Original discussion in Spanish · Published
The backtest doesn't lie through a coding error: it measures something else. It was trained and validated on a reference feed that is not the market where execution takes place, with a 5% fee on profits and roughly 8 seconds of matching delay. With a log-loss of 0.4914 versus 0.4902 for the market price, the model adds no information: it selects entries on a price that no longer exists by the time the order arrives.
There comes a moment at every quant desk when the backtest numbers and the real account numbers stare at each other and fail to recognize one another. Here the backtest showed returns between 30% and 60% per market with 183 trades in total and a per-trade Sharpe of 2 to 6.7. Actual trading with earlier versions produced about 260 signals, a return of roughly -11%, and a hit ratio pinned exactly at the price's implied probability. Only 11% of orders were filled. That's not bad luck. It's a system that measures one thing and trades another.
## A log-loss that merely matches the market already says a lot
The most uncomfortable figure is the log-loss: 0.4914 for the model versus 0.4902 for the market price, with no margin over the same universe. The model predicts worse than the price itself. And yes, it is mathematically possible to have an edge trading only when the model disagrees strongly, even if the overall log-loss is equal or worse. But it requires a very specific condition: the model must be better than the market precisely in the disagreement tail, not on average. That is checked with conditional calibration on the selection region and with a log-loss test restricted to the observations where EV exceeds alpha, compared against the log-loss of the price on that same subsample. If it doesn't win there, there is no edge to select.
The mechanism behind the overestimation is selection bias on a global calibration. Calibrating on all observations and then trading only in the EV > alpha tail overestimates p because the calibration was fitted to a different population from the one being traded. The practical fix is to calibrate conditional on the selection: local isotonic regression or a Bayesian shrinkage estimated only from the observations above the threshold. With little data, a strong prior with few parameters beats a loose isotonic fit.
## The external feed reacts late and the market has already priced it in
The reference source shows a clear pattern: after a sharp jump, the next tick still moves 2.3 times more than normal. That is delay, not information. The minimal experiment to separate them is cheap: replicate the signal using exclusively data from the execution market, with its 8-second delay and 5% fee, and compare the performance of both versions on the same set of events. If the edge disappears when using the execution price, the model was exploiting the feed's lag, not a real inefficiency.
The data doesn't help either: about 3,800 events over 16 days, some 25 observations per event every 200 seconds, and only 33% with auxiliary state variables, imputing the rest as zero. With that granularity there is no high frequency, and GroupKFold by event prevents leakage between folds, but it doesn't fix the fact that the training price is not the execution price.
## What to check before putting money back in
Prices from the execution market were never stored. That is the root flaw. Without your own order book series including queue, fees and delay, any backtest is a simulation of a different market. The first step is not to improve the GBDT or the beta calibration: it is to record the executable price and redo the validation against it.
After that, the stopping criterion must be set before trading: if the log-loss conditioned on the selection region doesn't beat the price's, don't trade. And the event time-window filter must be validated out of sample with complete events, not with loose observations. A per-trade Sharpe of 6.7 over 183 trades is a red flag, not an endorsement.
This is an analysis of a specific case, not financial advice. Before risking capital, it is wise to review the process with an independent professional.
There comes a moment at every quant desk when the backtest numbers and the real account numbers stare at each other and fail to recognize one another. Here the backtest showed returns between 30% and 60% per market with 183 trades in total and a per-trade Sharpe of 2 to 6.7. Actual trading with earlier versions produced about 260 signals, a return of roughly -11%, and a hit ratio pinned exactly at the price's implied probability. Only 11% of orders were filled. That's not bad luck. It's a system that measures one thing and trades another.
## A log-loss that merely matches the market already says a lot
The most uncomfortable figure is the log-loss: 0.4914 for the model versus 0.4902 for the market price, with no margin over the same universe. The model predicts worse than the price itself. And yes, it is mathematically possible to have an edge trading only when the model disagrees strongly, even if the overall log-loss is equal or worse. But it requires a very specific condition: the model must be better than the market precisely in the disagreement tail, not on average. That is checked with conditional calibration on the selection region and with a log-loss test restricted to the observations where EV exceeds alpha, compared against the log-loss of the price on that same subsample. If it doesn't win there, there is no edge to select.
The mechanism behind the overestimation is selection bias on a global calibration. Calibrating on all observations and then trading only in the EV > alpha tail overestimates p because the calibration was fitted to a different population from the one being traded. The practical fix is to calibrate conditional on the selection: local isotonic regression or a Bayesian shrinkage estimated only from the observations above the threshold. With little data, a strong prior with few parameters beats a loose isotonic fit.
## The external feed reacts late and the market has already priced it in
The reference source shows a clear pattern: after a sharp jump, the next tick still moves 2.3 times more than normal. That is delay, not information. The minimal experiment to separate them is cheap: replicate the signal using exclusively data from the execution market, with its 8-second delay and 5% fee, and compare the performance of both versions on the same set of events. If the edge disappears when using the execution price, the model was exploiting the feed's lag, not a real inefficiency.
The data doesn't help either: about 3,800 events over 16 days, some 25 observations per event every 200 seconds, and only 33% with auxiliary state variables, imputing the rest as zero. With that granularity there is no high frequency, and GroupKFold by event prevents leakage between folds, but it doesn't fix the fact that the training price is not the execution price.
## What to check before putting money back in
Prices from the execution market were never stored. That is the root flaw. Without your own order book series including queue, fees and delay, any backtest is a simulation of a different market. The first step is not to improve the GBDT or the beta calibration: it is to record the executable price and redo the validation against it.
After that, the stopping criterion must be set before trading: if the log-loss conditioned on the selection region doesn't beat the price's, don't trade. And the event time-window filter must be validated out of sample with complete events, not with loose observations. A per-trade Sharpe of 6.7 over 183 trades is a red flag, not an endorsement.
This is an analysis of a specific case, not financial advice. Before risking capital, it is wise to review the process with an independent professional.
Summary of a discussion on Burbuja.info - Foro de economía, actualidad y política., translated from Spanish and reviewed before publication. Read the full discussion (0 replies).
More summaries
-
1804 dollar: worth $3.8 million if genuine, pocket change if not
A genuine 1804 silver dollar sold for over $3.8 million. Most of the ones found at home are fakes: weight and diameter give them away.
-
Buying gold without middlemen in Spain: SEMPSA requires self-employed status
Buying gold bars at factory price in Spain requires being self-employed or a company. The gap with jewellery retail exceeds 260 euros per kilo of gold.
-
Amending your Spanish tax return: four years to recover 800 euros
A wrong box on a couple's pension plan cost 800 euros in refunds. Spanish law allows taxpayers to amend their return up to four years later.
-
Hotel101 sells rooms in Valdebebas for €208,800
Hotel101 sells rooms in Valdebebas for €208,800 plus VAT with monthly income distribution and doubts about what happens if the operator goes bankrupt
-
Spain's shadow economy: neither 24.6% of GDP nor nothing
The 24.6% of GDP attributed to Spain's shadow economy clashes with a reality where everything goes through the bank, card and VAT. The math doesn't add up.
-
From market timing to all in: dividend investors change tack
A investor tired of holding cash plans to go all in on 5% to 10% dividend stocks, with Enagas at 10%. DCA and market timing warnings dominate the debate.