Quantitative auto-trading system: backtest promised 6.7 Sharpe, live trading lost 11%

The backtest was a gold mine and reality was noise, because the model was trained on a reference feed different from the market where it executes, and the actual execution prices were never recorded. With a log-loss of 0.4914 versus the 0.4902 of the market price, the model adds no information: the real result confirms it.

There is a scene that repeats itself in homemade quantitative trading: someone builds a system, tests it against years of data, sees green numbers and decides they have found the goose that lays the golden eggs. Then they put money in and the goose lays stones. The case at hand has all the pieces of the puzzle: **3,800 events in 16 days**, about **25 observations per event** every **200 seconds**, a logistic anchor model plus residual GBDT with beta calibration, and a GroupKFold validation by event with 5 folds. The out-of-fold backtest showed between **+30% and +60%** per market with 183 total trades and a Sharpe per trade of **2 to 6.7**. Real trading with earlier versions: about **260 signals**, a return of **-11%**, a hit ratio equal to the price's implied probability and only **11%** of orders filled.

## The market you execute in is not the market you train on

Here lies the original sin. The system was trained and validated on an external reference source, a third-party feed with its own latency and spread. But orders are executed in another book, with a queue, a **5% fee on profits** and a fixed **8-second** delay in matching. And the prices of that second market were never recorded. This means the backtest measures a reality that does not exist for the trader. The model learns patterns from a price that is not the one being paid. The 5% fee on profits, on its own, eats up a good part of the margin the backtest took for granted. The 8-second delay turns any signal that depends on microstructure into fiction.

## A log-loss that doesn't beat the market is no edge

The most uncomfortable figure: the model's log-loss is **0.4914**, and the market price without margin sits at **0.4902**, over the same universe. The model predicts worse than the price itself. Can there be an edge trading only when the model disagrees strongly? Yes, in theory, but only if the model has a better log-loss conditioned on that region of disagreement and is conditionally calibrated. The exact test: compare the model's log-loss and the price's log-loss restricted to the observations where EV exceeds alpha. If the model doesn't win there, there is no edge. Calibrating on all observations and then trading only in the tail overestimates probability, because global calibration does not guarantee local calibration. The fix is to calibrate only in the selected region, with local isotonic regression or Bayesian shrinkage, and to estimate the parameter with little data using cross-validation by event.

## The feed lag that looks like information

The reference source reacts with a lag to jumps: after a sudden move, the next tick still moves **2.3 times** more than usual. This can look like the model sees information the market has not yet priced in, when in reality it exploits the lag of a feed that no longer exists in the execution market. The minimal experiment: simultaneously record the prices of the feed and of the execution book for a sufficient period, and measure whether the model's predictive signal survives when real prices are used. If it disappears, it was latency, not information.

## What to do with 16 days and 3,800 events

Very little. With that sample and that frequency, any backtest with a Sharpe of 6.7 is suspicious by construction. Honest validation requires recording the execution market's prices from day one, paper trading with the real delay and fee, and comparing the log-loss conditioned on the selection. Until that is done, the system has no demonstrated edge. And with real money, caution is not optional: this is not financial advice.