Builder Journal · Mars Environmental Dynamics Analyzer (MEDA) Virtual Sensor Recovery
Ten times in a row I predicted what my next submission would score before I uploaded it. The worst miss was 0.0025 on a number around nineteen. I took that as confirmation that the physics underneath was correct.
It was confirmation that I can do arithmetic.
Two days before this competition closed I pointed a review at my own endgame, expecting notes about the code. It came back with three errors and none of them were in the code. All three were in my reasoning, and all three had the same shape: I had run something that felt like a measurement and was not one.
This is the fourth entry in this series and the one I would keep if I had to burn the other three. The models are competition-specific. This part is not.
The competition in one breath
Perseverance carries an environmental station called MEDA. Some of its surface pressure readings are missing, and the competition is to reconstruct them. Scored on mean squared error.
The wrinkle is the split. Training covers sols 1 through 100, when pressure is climbing toward its seasonal peak. Test covers sols 201 through 300, when it is falling hard toward the aphelion minimum. Sols 101 through 200 do not exist in either file. Every prediction is outside the range the model was fit on.
The first entry covers the first submission, which contained no machine learning at all and took the top of the board at 61.04. Six weeks and seven versions later the public score was 18.99.
Almost everything in between was selected by one signal. Not cross-validation. Cross-validation here can only hold out sols from the rising limb, so it is structurally blind to the regime I am scored on. The leaderboard was the only thing that could see the falling limb, so the leaderboard picked every scalar that mattered: the residual shrink, the blend weight, a constant seasonal offset, a diurnal scaling.
Hold onto that. It becomes the joke about four hundred words from now.
The review that was supposed to be about code
I ran twenty-two agents over the whole endgame in parallel. Fresh context each, no memory of how any of it had been argued for, pointed at the scripts and the notes and told to attack.
Why that works is not mysterious and has nothing to do with the agents being clever. Self-review fails on the class of error that sits upstream of the code. You cannot re-read your way out of a wrong premise, because the premise is what you are reading with. A reviewer arriving cold has no such loyalty. They see a claim and a piece of evidence and they check whether one supports the other, which is exactly the check I had stopped performing.
It found three real errors in my reasoning. Here they are in the order they hurt.
One: the experiment that could not have failed
I had started to suspect the public leaderboard was not scoring the whole test set. So I designed a diagnostic. Add plus two pascals to every row with sol greater than or equal to 255, resubmit, and watch. If the public half contains those sols, the score has to move. It is a clean idea.
The score came back bit-identical. I wrote down the conclusion: the public leaderboard is sols 201 through 254.
The real cut is at row 1,974,995 in spacecraft-clock order, exactly fifty percent of the file, and it lands inside sol 253. The lowest row I perturbed sat at index 1,986,773.
row index: 0 .................. 1,974,995 ...................... end
|<------ public ------>|<-------- private --------->|
^
1,986,773 = lowest row I touched
Every row I perturbed was already private. Under my hypothesis and under its negation, both. The null result was guaranteed by construction before I spent a submission slot on it. It was consistent with my conclusion and equally consistent with the opposite of my conclusion, which is a long way of saying it carried zero information.
The note I wrote that night is the shortest thing in six weeks of notes. A test that cannot come out the other way is not a test.
What the diagnostic actually established was narrower than what I recorded. It proved the public half contains no sol at or past 255. It never addressed whether the public half equals sols 201 through 254. I read a bound as an equality, because the equality was the answer I was already carrying and the bound was compatible with it.
The repair is one question, asked before the experiment instead of after: what result would falsify this, and can this experiment produce that result? If the answer is no, what you have is not a diagnostic. It is a ritual with a submission slot attached.
I already knew this. On a wellbore-geology competition I made a habit of writing the killing condition down in advance, before running anything, precisely so that a null result would mean something. Then I came to Mars and designed a probe that could only return the answer I wanted.
Two: the disconfirming number I filed as a match
The mask question was eventually settled properly, and the method is worth a paragraph because it is the same machinery that produces failure number three.
Mean squared error is not a black box. If you perturb your prediction vector along some direction and resubmit, the score moves by an amount that depends on the projection of your hidden error onto that direction. Which means a submission is a measuring instrument pointed at labels you are not allowed to see. Pick two directions, compute what each inner product should be under a candidate mask, and compare against what the leaderboard implies.
Under the correct row-count mask, the self-inner-product of the sol direction came out as 0.335887303844 from the leaderboard and 0.335887303844 from my own files. Twelve significant figures. The diurnal direction agreed to 25.52751 against 25.527508.
That is what agreement looks like in this pipeline. Now go back to the sol-based mask I had believed for a day. Same quantity, 0.3339 against 0.3359. Off by six tenths of a percent.
Six tenths of a percent is nothing in most contexts. It is inside the error bars of practically any real measurement, and every instinct trained on real measurements says round it off and move on. But nothing else in this pipeline behaves that way. Everything correct here matches to twelve digits. In a system with that property, a 0.6 percent gap is not noise and it is not rounding. It is a different number wearing a similar shirt.
And it was sitting in my own notes. Written down. Labeled “match.”
When a number is close but not equal, and everything else in your pipeline is exact, the gap is the signal. Close-but-not-equal is its own category. Filing it under “equal” is how a piece of disconfirming evidence becomes a piece of confirming evidence without anybody lying to anybody.
Three: the validation that validated nothing
This is the one I would have defended hardest, and it is the reason the whole session ended up being about epistemics rather than about pressure.
Start with the machinery. If v is my prediction vector, e is some direction, and c a scalar:
MSE(v + c·e) = MSE(v) + 2c·⟨err, e⟩/N + c²·⟨e, e⟩/N
A parabola in c. Normalize e to unit root-mean-square and the quadratic coefficient is exactly one, which means a single submission at c = 1 hands you the projection of your unknown error onto any direction you choose. One number per submission, measured against hidden truth.
The pleasant consequence is that I could compute, in advance, what a candidate submission would score. And I did, ten times. Never off by more than 0.0025. Every prediction landed. It felt like an independent oracle confirming my physical model to four decimal places, over and over, on data I could not see.
Here is what was really happening. Given the projections I had already measured, the predicted score of any correction living in the span of one, the sol coordinate, and the sol coordinate squared follows by arithmetic. It is an algebraic identity, not a forecast. The prediction cannot come out wrong, because the prediction is not about the world. It is about the linear algebra I did on my own numbers.
The demonstration is brutal and it took the reviewers about a paragraph. Earlier that same day I had built a six-term polynomial correction, fitted on the public half, that gained 0.80 MSE where it was fitted and reached minus 1,397 pascals by sol 300. That is not a model of anything. Martian surface pressure does not go negative, let alone fourteen hundred pascals negative. I rejected it on sight, correctly, and my prediction machinery would have called its public score just as precisely as it called everything I shipped.
A test that assigns the same passing grade to the thing you shipped and the thing you threw away for being physically impossible is not grading the thing you think it is grading.
It validated my arithmetic, not my physics.
So the second question, alongside the first one: when a prediction succeeds, ask what else it would have predicted equally well. If the answer is “anything in this family,” it discriminates nothing. Evidence earns its name by ruling something out. A test that rules nothing out is a very sophisticated way of nodding along with yourself.
Four: the justification I wrote before reading my own code
Smaller than the other three, and different in kind. The three above are experiments that returned nothing while looking like measurements. This one is not a measurement at all. It is a paragraph of reasoning I wrote without opening the file it was about, and it is the one I am least proud of.
The correction I ended up shipping is keyed to how much I trust different parts of the published climatology curve. Some of its knots come from a paper’s text and are solid. Most were digitized off a figure and carry a couple of pascals of slop. So the model’s error should be smallest at the trustworthy anchors and largest in between them, which gives a bias curve with a physical shape rather than a fitted one.
I wrote a confident paragraph justifying a particular pin in that scheme. Technical, specific, wrong. The argument depended on the climatology having one long interpolation span across the test window, so error would swell smoothly toward its middle. I opened the file. There are seven knots inside the test range.
Ninety seconds of reading a file that had been sitting in my own repository for six weeks killed a paragraph I had already written down as reasoning. I had not consulted the code. I had consulted my memory of the code, and memory does not retrieve, it reconstructs, and what it reconstructed was an argument rather than a file.
Read the code before writing the justification. I have shipped this mistake before. On a hyperspectral tracking problem I was sure about a contribution right up until the data said otherwise, and the confidence was doing real work in keeping me from checking.
The thing all four have in common
Now put them next to the earlier entries in this series, because the shape does not start here.
The first entry ends on a polynomial degree. Degree four scored better in cross-validation than degree three, and degree four bent the seasonal curve back upward inside the test window, which is physically wrong on the only sols being scored. The cross-validation number was real. It was correctly computed. It answered the question “which curve fits the rising limb more snugly,” and I read it as an answer to “which curve is right.”
The second entry does it twice. First on a feature importance: the residual model’s top feature carried thirty-one percent of its measured gain, and every one of its test values sat outside the training range, so a tree returns a flat constant for it on every test row. Importance was a real number too. It measures in-training contribution. I read it as test value. Then on the folds themselves, which walk forward, always training on earlier sols and validating on later ones. That is correct methodology, and it still chose every magnitude too cautiously. The residual shrunk to half strength when full strength was better. A blend weight of 0.1 when 0.7 was better. The scheme was right. It was still only ever measuring the rising limb.
The third entry ends on the finding that reframes all of it, which is where this session started before it turned into a review of my own head. The leaderboard I had switched to trusting scores the first half of the test set in time order and nothing else. So every leaderboard-calibrated constant in this project was fit on the first half of the test season and applied, unmeasured, to the second. The shrink, the blend weight, the minus 2.33 pascal offset, the diurnal scaling, all of them. I had spent six weeks moving away from a validation signal I knew was partial toward a leaderboard signal I had decided was total, and the leaderboard was half.
It even explains something I had filed and forgotten. Back in June a probe for a linear trend across the test sols came back at plus 0.06, essentially nothing, and I declared that lever exhausted and moved on. It was not exhausted. The public window happens to sit near the peak of the baseline’s error bump, where the slope really is flat. All the decay lives in the half the board cannot see. I had a measurement that was locally true and globally misleading, and I used it to close a line of work.
Add them up across four entries and they are one failure wearing a different costume each time. Every one of them is a number that answered a question I had not asked.
That is what makes them hard. A wrong number is easy. Wrong numbers get caught by tests, by sanity checks, by a colleague squinting at a plot. These were all correct, precise, reproducible, and about something adjacent to the thing I cared about. There is no assertion that fires. The only defense is a habit of asking what a number is a number of, and that habit is the first thing that goes when the number agrees with you.
On an ARC-AGI competition I eventually had to admit the bug was in my beliefs rather than anywhere I could set a breakpoint. This is the same admission, made four times, by someone who had already written the entry about it.
What a real test looks like
Three questions, all of them free, none of them requiring a submission slot or a training run.
Before running a diagnostic: what result would falsify my hypothesis, and can this experiment produce that result? If it cannot, redesign it before you spend anything on it. This is the whole difference between a test and a ceremony.
When a prediction succeeds: what else would this have predicted equally well? If it would have blessed a model you already rejected as nonsense, it discriminated nothing, no matter how many decimal places it hit.
When a number is close but not equal: is close the same as equal in this pipeline? Calibrate against how exact your correct answers usually are. In a twelve-digit system, half a percent is a scream.
Then a fourth, which is not a question but a practice. Get eyes that have no history with your reasoning. Not because they know more than you do about the problem, they usually know far less, but because they have not spent six weeks building the scaffolding that makes your conclusion feel obvious. That is the entire asset. Fresh context is cheap now, and it buys the one thing that self-review structurally cannot supply.
There is a fifth thing worth saying, less about epistemics and more about consequences. Kaggle auto-selects your final submissions by best public score. When the public score is the trap, doing nothing is close to worst case. Defaults are decisions, made by someone who has never seen your problem, and they execute whether or not you thought about them.
What I am actually taking with me
Six weeks on this problem produced one model and a handful of notes about how I fool myself, and the notes are the more valuable artifact.
The model does not transfer. It decomposes Martian surface pressure into a seasonal baseline, a set of thermal tides, and a residual anchored to a 2022 climatology paper, and I will probably never use any of it again. The notes transfer to everything. Every problem after this one will hand me a number that looks like an answer, and the number will be correct, and the question it answers will be one I did not ask, and I will want very badly to accept it because it agrees with what I already built.
Writing this down is not a confession, it is instrumentation. The four failures above got caught because they were written somewhere a stranger could read them and ask whether the evidence supported the claim. The ones I never wrote down are still in here.
I am writing this two days before the competition closes, with five submissions locked in and nothing left to do but wait. As of right now the private half of the test set has never been scored by anybody but the organizers. By the time you read this it will have been. Whatever it says, this part is already banked.
Frequently asked questions
How do I know whether an experiment can actually falsify my hypothesis?
Before running it, write down what result would prove you wrong, then check whether the experiment is physically capable of producing that result. If every possible outcome is compatible with your hypothesis, the experiment carries no information regardless of how it comes out. This is what happened to my leaderboard split diagnostic: every row I perturbed was in the hidden half under both the hypothesis and its negation, so the null result was guaranteed before I ran it.
Does the Kaggle public leaderboard score the whole test set?
No. The public score is computed on a subset, with the rest held back for the private leaderboard revealed at the end. In this competition the split turned out to be the first fifty percent of rows in spacecraft-clock order, cutting inside a single sol rather than on a clean boundary. Any constant you tune against the public score is fit on that subset and extrapolated onto the rest, which matters enormously when the two halves are not exchangeable.
If I can predict my model’s score accurately, does that validate the model?
Not by itself. Ask what else the same prediction method would have predicted equally well. In my case the prediction was an algebraic identity given projections I had already measured, so it was equally accurate for a correction I had rejected as physically impossible. A prediction is evidence only to the extent that it could have come out wrong for a bad model.
What is leaderboard probing, and what does it actually measure?
For a mean squared error metric, shifting your predictions along a chosen direction changes the score by a known quadratic function of the shift. Normalize the direction to unit root-mean-square and a single submission recovers the projection of your hidden error onto that direction. It measures the error structure of your predictions against labels you cannot see, one number per submission. It does not measure whether your reasoning about the underlying physics is correct.
How do you catch errors in your own reasoning rather than your code?
Hand the work to a reviewer with no history with it. Self-review fails on upstream errors because the flawed premise is the lens you are re-reading through, while a reviewer arriving cold simply checks whether the stated evidence supports the stated claim. I ran twenty-two independent reviews over my own endgame and every finding was in the argument, not the implementation.
Standing: rank 2 of 15, best public score 18.8441, behind a perfect zero posted by someone submitting published ground truth rather than a model.
More in this series
- I Took First Place Without Training a Model · how the physics decomposition took the top of the board with no ML.
- The Green Test Suite That Hid a Broken Model · three ways a passing test lied to me on a different competition.
- The Builder Journal · the live log across every competition I’m in.
- Every entry from this competition · the full MEDA Virtual Sensor Recovery thread.
This is part of an ongoing builder’s log written from inside live competitions. You’re reading where I was, not where I am.
