HOW WE TEST
Why you can trust these numbers.
This page explains how a strategy gets tested here, and defines every word we use to describe the result. It is written for someone who has never done any of this before. If a sentence needs a word you might not know, the word is explained in the same breath — and every one of them is in the dictionary at the bottom.
THE PROBLEM
First, the problem with every backtest you have ever seen.
A backtest is a strategy replayed over old market data. They are almost always too good, usually for one of five reasons:
FIVE REASONS
- 01Someone tried fifty versions and showed you the one that worked.
- 02The test used information that was not available at the time.
- 03Trading costs were ignored, or hidden in a footnote — the commission you pay, and the gap between the price you wanted and the price you actually got.
- 04It measured profit, when a funded-account challenge — the test a trading firm sets before it will let you trade its money — measures something else entirely.
- 05It got lucky, and nobody counted how many tries it took.
Every rule below exists to block one of those five.
RULE ONE
Every strategy has to beat a coin flip.
You can pass a funded-account challenge on pure luck. On a typical $50,000 challenge you have to make $3,000 before the account ever falls $2,000 below its best point. Flip a coin to decide buy or sell, size your trades sensibly, and you still pass sometimes — in one of our tests a coin-flipper passed 38 times out of 100 attempts. The target sits closer than the limit, so the rulebook hands out free passes.
So “my strategy passed 68 of 100” means nothing until you ask what a coin flip scored on the same test. If the coin flip scored 38, then 38 of those passes came from the rules and 30 came from the strategy.
That is why every strategy is tested beside random guessing: the same trading days, the same number of trades and the same sizes, but a coin decides whether each day’s result stands or gets turned upside-down. We run 2,000 simulated attempts for both, and only the gap between them counts.
Why a coin flip, and not simply reversing the trades
Reversing a good strategy gives you a reliably bad one, and beating a bad opponent proves nothing. We want an opponent with no opinion — right as often as it is wrong.
Making sure the coin is fair
A few hundred coin flips never land exactly half and half. If they happen to favour the winning days, random guessing turns up with a small accidental edge of its own. So we measure whatever profit is left in it and take that back out. It ends at exactly break-even, every time.
The trap almost everyone falls into
Accounts do not blow up at the closing price. They blow up at the worst moment of the day — the spike that hits the loss limit before price recovers. Our records store that worst moment. The tempting shortcut is to build the random version by putting a minus sign in front of those numbers, which turns every worst moment into a best moment: the random version would then almost never break a rule, and our edge would look far bigger than it is. What we do instead is keep a second recording of every trade — what the worst moment would have been had the trade gone the other way — and let the random version use that.
PASSES OUT OF 100 ATTEMPTS
Coin flip
Example strategy
38 of those passes came from the rules. 30 came from the strategy, and only the gap counts.
RULE TWO
We decide what counts as success before we look.
Every threshold, every minimum and every setting is written into a dated record before the test is allowed anywhere near real market data. If a number later turns out to be wrong, the change is dated, signed off, explained in writing — and the original result stays on the shelf next to the new one.
The lottery-ticket tax
Test a lot of ideas and a few will look brilliant purely by chance. The only defence is to count your tries and charge yourself for them. We keep a running tally of every idea we have ever tested, and the bar a new result has to clear rises as that tally grows. Trying more ideas makes it harder for us to claim a winner, not easier. We do not print the tally here, because it changes every time the lab registers another trial — the live figure is on the strategies page.
We keep our failures visible
A real example. Our first year-by-year test would only judge a six-month period if the strategy had traded on at least 30 days inside it. Our strategies are picky and often trade fewer days than that, so nothing could be judged at all — the test came back completely empty. We could have deleted it and nobody would ever have known. Instead we lowered the floor in two dated steps, to 15 trading days and then to 8, where every idea finally got a real verdict, and some of the ideas we had hoped for genuinely lost. All three versions of that test are still on the shelf, the empty one included. Deleting it would also delete the evidence that we moved the goalposts.
One exam we can only sit once
All market data from 2026 onward is sealed, and no test we have built has ever touched it. That one future result will be worth more than any number of tests on data we have already seen — and we can spend it exactly once.
THE RECORD
at least 200 firings
at least 100 trading days
costs of 2 ticks a side
→ written before any data was touched
RULE THREE
We test against the firm’s actual rules, not profit.
A prop firm — a company that lets you trade its money and keeps a share of what you make — does not pay for profit. It pays for surviving a specific rulebook. On a typical $50,000 account: reach $3,000 in profit; never let the account fall $2,000 below its best end-of-day value; trade on a minimum number of days; do not make more than about half your profit in a single day; and keep paying the monthly fee the whole time you are trying.
Under those rules a profitable strategy can fail and an unremarkable one can pass. So we do not score profit. We walk every simulated attempt day by day through the firm’s own published rulebook, fees included, with each firm’s rules written down beside links to the pages we took them from.
Quiet days still cost money
Our strategies trade only when they see their setup, often one day in four. On every other day the monthly fee still charges, the rules about trading often enough still tick along, and where the firm sets a deadline the clock still runs. Counting only the days you traded hands a picky strategy unlimited time and free rent.
The order of your days matters
Each simulated attempt is built from blocks of 20 consecutive real market days, never a shuffled bag of days. Accounts are killed by losing streaks, not by bad averages, and shuffling the days apart breaks the streaks up and makes everything look survivable.
We check the worst price, not the convenient one
The account is tested against the worst price reached inside each fifteen-minute period, not the price at the end of it. We measured the sloppy way too: using only the price at the end of each period made one firm’s results come out far too generous.
Costs, and then costs doubled
Every trade pays the real commission plus the real gap between the price we wanted and the price we got. Anything that survives is then run again with those costs doubled, and we report whether it still passes. A strategy that only works at perfect prices is not a strategy.
Flat before the close, always
Every strategy closes its trades before the trading day ends, so nothing is ever carried overnight. This is not a policy we check afterwards — the software physically cannot build a strategy that holds a position overnight.
A TYPICAL $50,000 ACCOUNT
- Profit target
- $3,000
- Floor
- $2,000 below the best end-of-day value
- Trading days
- a minimum number of days
- One day
- no more than about half the profit
- Monthly fee
- paid the whole time you are trying
RULE FOUR
We spend real effort trying to break our own results.
A mistake in the software that flatters you looks exactly like skill, and you cannot tell the difference by staring at the numbers.
Build it twice, independently
Everything rests on a database of what price did next after each market moment we recorded. We rebuilt that database from scratch with a second program, written from the written rules by a developer who was not allowed to look at the first one. The two agreed — and they still agreed when we re-checked them against much finer price data. Checking the first program against itself would only have let a shared mistake agree with itself perfectly.
We break the code on purpose
A test suite that passes proves nothing by itself, so we sabotage our own code — flip a comparison, delete a safety check — and demand that the tests notice. Every fault we have planted so far has been caught. Tests that survive sabotage are the only ones worth quoting.
Two reviewers who re-run everything
Every piece of work is checked by someone who did not build it, then checked again for quality, and reviewers verify by re-running the numbers rather than reading the author’s summary. In one evening that caught a calculation that understated an account’s worst loss, and a planning flaw that would have crashed on real data.
BEFORE YOU SEE A NUMBER
- Build it twice, independently
- We break the code on purpose
- Two reviewers who re-run everything
What this costs us, and what you get.
Our numbers look smaller than the ones you will see elsewhere, and that is deliberate. Testing against random guessing eats most of every headline. Charging ourselves for every idea we try penalises us for being thorough. Doubled costs kill strategies we would love to keep. Our failed tests stay visible. And sometimes the answer is “we do not know yet”, which never looks as good as a confident chart.
In exchange: every number traces back to the market data that produced it, you see the full list of what we tried rather than only what worked, our failures are still on the shelf, and the method is one you can argue with.
Anyone can show you a chart. We would rather show you the test and let you check it.
THE ACTUAL GAUNTLET
Seven checks, in this order, every time.
Everything above is the thinking. This is the machinery: the checks every idea runs through, in order. An idea that fails a check stops there, and we keep the result. These are the same seven you will see on every strategy page.
- 01
Does it happen often enough?
It has to have fired at least 200 times, on at least 100 different trading days, spread over at least 4 years. All three, not one of three.
- 02
Does it beat the plain version of itself?
The comparison is the same idea with its extra condition stripped out — not the market. A filter has to earn its place, and the wins have to travel further than the losses.
- 03
Does it still work after costs?
We ask how much cost per trade the advantage could absorb before it vanishes. Under about double what we actually pay, and it never reaches the expensive tests.
- 04
Would it pass a funded-account challenge?
We deal 2,000 simulated attempts and walk each one day by day through a real firm’s published rulebook, fees included — then do the identical thing to random guesses and report the gap.
- 05
Did it keep working year after year?
We judge each six-month period on its own, not just the whole history at once.
- 06NOT YET RUN
What size, at which firm?
Exact position size under each firm’s own rules.
- 07NOT YET RUN
The sealed final test
One run of market data nobody has touched, faced once, with no retakes.
The last two are marked not yet run because they genuinely have not been. We will not show an untaken test as a pass.
Before check 1: the idea gets written down as a recipe
An idea is not code here, it is a short description a machine can read: buy when price breaks through a level that has already stopped the market six times, approached from below, but only while the fast average is above the slow one. We generate these in large batches so that a great many are tested in exactly the same way, and nobody can quietly hand-tune a favourite.
Between check 3 and check 4: how the trade gets out
Knowing when to get in says nothing about how to get out, so anything that survives the first three checks is paired with the settings that decide the exit: where the stop loss goes, where the take profit goes, when the stop moves to break-even, a time limit, and always flat before the close. They come from a frozen catalog of 620 combinations, and which ones a test is allowed to use is named in advance — so “try more settings until one of them works” is not available to us.
The shape of it
Most ideas die at check 1, which is the cheapest possible place to die. A smaller share of the survivors beat the plain version of themselves, and only a handful of those get through costs, so by the time anything reaches the expensive tests the pile is small. We deliberately do not put numbers on that funnel here: the counts move every evening, and the live ones are on the strategies page.
We also re-check our own blind spots
Our main screen asks whether price reached the next level up before it reached the next one down. That suits a breakout, which is already moving. It is much less kind to an idea that turns at a line and grinds along slowly, which can make money all afternoon and still be scored a failure. So we re-graded every idea the screen had rejected, on the trade it actually implies — in at the signal, out flat at the close, real costs charged. A few were genuinely profitable that way, survived the cost check, and went on to pass the funded-account test as well. The screen is not broken, but its blind spot is now measured rather than assumed.
What is not finished, stated plainly.
Several strategies on one account
The machinery for running a few proven strategies side by side — adding up their positions, respecting the firm’s cap on contracts — is built and reviewed, but it has not formed any real combinations yet. It was waiting for a varied enough set of survivors.
The sealed year
Untouched, on purpose.
Putting a dollar value on a strategy
Turning a result into an expected dollar figure runs through a separate calculation that currently disagrees with its own cross-check. So we quote results and never valuations.
One indicator so far
The engine takes any indicator’s signals and a second one is designed, but everything you see today runs on our own price-level engine. A new indicator gets its own round of testing rather than inheriting this one’s.
THE DICTIONARY
Every word we had to use.
This is the one place each of these is defined. If you find one of them used anywhere else on the site in a way this page does not cover, that is our mistake, not yours.
- Prop firm
- A company that lets you trade its money and keeps a share of what you make. Before it funds you, it sets a test. Short for proprietary trading firm.
- Challenge (also called an evaluation)
- That test. Hit a profit target without ever breaking the firm’s loss rules, while paying a monthly fee. Pass it and the firm funds an account for you.
- Drawdown
- How far the account has fallen from its best point. Firms set a hard limit on it: cross the limit and the account is gone, however well you were doing an hour earlier.
- Trailing drawdown
- A loss limit that follows you upward as the account grows. A good run shrinks the room you have left to be wrong, which is why a strategy can be profitable and still fail the challenge.
- Stop loss
- The price where a losing trade is closed automatically, so that one bad trade cannot run away with the account.
- Take profit
- The price where a winning trade is closed automatically, instead of being held in the hope of more.
- Backtest
- A strategy replayed over old market data to see what it would have done. Useful, and very easy to flatter — which is what most of this page is about.
- Simulation
- Every number on this site comes from running the rules over real recorded market prices. None of it is live trading, and none of it is a forecast.
- Random guessing
- The same trading days, the same number of trades and the same sizes as the real strategy, but a coin decides whether each day’s result stands or is turned upside-down. It has no opinion about the market, so it is the honest thing to measure a strategy against. If a strategy cannot beat it, it has no edge.
- Out of 100
- How many times out of a hundred attempts something happened. We use it instead of percentages because it is easier to picture — a count out of 100 is a thing you can hold in your head, where a decimal is a statistic.
- Period
- A six-month span of market history. We judge each one on its own, so a strategy cannot hide a bad run of market inside a good average.
- Too few trades to judge
- In that period the strategy barely traded, so there is not enough evidence to call it either way. It is not the same as losing, and we never show it as one.
- Out-of-sample
- Data a strategy has never been tested on. It is the only data that can genuinely surprise you, which is why we keep a year of it sealed and unopened.
Soon you can check the work.
Every strategy in the shop will show which of these checks it passed, which it failed, and why — in the same words this page just defined. The shop is not open yet; in the meantime, watch one of our strategies run a challenge.
Watch a run