From 1,880th to 8th: A Six-Day, Validation-First Kaggle Sprint

From 1,880th to 8th: A Six-Day, Validation-First Kaggle Sprint

How a disciplined experiment framework, leakage-safe target statistics, class-aware decision rules, and a GPU-trained FT-Transformer produced an 8th-place private leader board finish in Playground Series S6E7.

The result

I finished Kaggle's Playground Series S6E7: Predicting Student Health Risk at 8th on the private leaderboard and 63rd on the public leaderboard, out of 3,356 teams.

Result	Rank	Score	Approximate percentile
Final private leaderboard	8 / 3,356	0.95084	Top 0.24%
Final public leaderboard	63 / 3,356	0.95306	Top 1.88%

The first repository commit was made on 26 July 2026. The final public submission landed on 31 July, less than five hours before the deadline. In roughly five and a half days, I recorded 161 completed experiments across 28 model and ensemble families and made 10 scoring Kaggle submissions.

The headline is exciting, but the most useful part of the story is the distinction between the two final scores:

• The private score of 0.95084, which produced the 8th-place finish, came from the independently trained FT-Transformer pipeline. This is the result that best represents model generalization.
• The public score of 0.95306 came from a late, leaderboard-guided 29-row cleanup of a public 0.95303 anchor. That artifact scored only 0.95018 privately. It was a legitimate, reproducible competition experiment, but it was not a universal model improvement and it clearly overfit the 20% public split.

That separation matters. The public leader board rewarded the final clean-up; the private leader board rewarded the model.

The competition in one page

The task was a three-class classification problem. For every test row, the model had to predict one of:

• at-risk;
• fit;
• unhealthy.

The training set contained 690,088 rows, the test set 295,753 rows, and the model received 13 health and behaviour features: seven numeric and six categorical. Missingness was part of the problem—there were 449,496 missing training cells, with per-feature missing rates ranging from roughly 1% to 12%.

The class distribution was extremely uneven:

Class	Training rows	Share
at-risk	592,561	85.87%
unhealthy	57,724	8.36%
fit	39,803	5.77%

Kaggle evaluated submissions with balanced accuracy, the unweighted mean of the three class recalls:

$$\mathrm{BA}=\frac{1}{3}({\mathrm{Recall}}_{\text{at-risk}}+{\mathrm{Recall}}_{\text{fit}}+{\mathrm{Recall}}_{\text{unhealthy}})\mathrm{.}$$

That metric changes the strategy completely. A model that predicts the 85.87% majority class for every row has high ordinary accuracy but a balanced accuracy of only one third. The model must recover the two small classes without sacrificing too much majority-class recall.

The data audit found no duplicate IDs, no exact train/test overlap, and no automatic leakage flags. Adversarial validation did, however, produce an AUC of 0.5850 in the initial audit and 0.6199 with the expanded diagnostic feature set. In other words, train and test were similar, but not interchangeable. That moderate drift was one reason not to trust tiny cross-validation gains or public-leaderboard movement on its own.

Build the laboratory before chasing the score

The project started as more than a single notebook. I built a reusable, configuration-driven framework around the competition:

• append-only experiment and submission registries;
• fold-aligned out-of-fold and test probabilities;
• fold-local feature fitting and target encoding;
• stratified cross-validation with fixed, reproducible seeds;
• candidate comparison with fold wins, bootstrap intervals, per-class recall, and disagreement analysis;
• submission schema, ID-order, label-domain, duplicate, and missing-value checks;
• SHA-256 hashes for important notebook inputs, probability files, and submissions;
• guarded upload commands, so training and submission remained separate actions;
• unit, integration, lint, formatting, and type-checking gates.

This investment paid off when the experimentation became frantic. By the last two days, several long-running GPU notebooks were in flight and many candidate scores differed by less than 0.0002. Without stable folds, stored OOF probabilities, and lineage, it would have been impossible to tell a real improvement from noise.

The promotion rule for a full candidate was deliberately strict. Relative to the verified incumbent, it had to:

1. improve balanced accuracy by at least 0.00020;
2. win at least five of seven folds;
3. avoid reducing any class recall by more than 0.00050.

That rule rejected some attractive-looking results—including the best XGBoost/FT blend—and protected the campaign from spending submissions on fragile OOF wins.

The path from 0.92933 to 0.95054

The first scoring model was a five-fold LightGBM baseline. It established that the pipeline worked, but also exposed the central issue: the default probability argmax was optimized for ordinary accuracy, not balanced accuracy.

Milestone	Honest local OOF BA	Public score	Private score
LightGBM baseline	0.933069	0.92933	0.92979
LightGBM + cross-fitted class decisions	0.939377	0.93520	0.93597
CatStat XGBoost	0.949837	0.95006	0.95031
Two-seed XGBoost blend	0.949802	0.94972	0.95012
Exact-statistic XGBoost	0.950398	0.95054	0.95027
FT-Transformer v2	0.950549	0.95094	0.95084
Public-anchor 29-row cleanup	not a trained model	0.95306	0.95018

The biggest early gain was the decision rule

For each class, I added a learned offset to the log probability:

$$\widehat{y}=\arg \underset{k}{\max }(\log p_k+b_k)\mathrm{.}$$

The offsets were cross-fitted. When scoring fold $f$, its offset vector was learned from all other folds, never from the labels being evaluated. A final vector was then fitted on all OOF rows and applied to test predictions.

On the LightGBM model, this increased OOF balanced accuracy from 0.933069 to 0.939377 and improved all five folds. On the stronger tree and neural models, the same idea remained the dominant post-processing lever.

The FT notebook expressed the equivalent intuition as prior correction:

$$\widehat{y}=\arg \underset{k}{\max }\frac{p_k}{{\pi }_k^{\beta }},$$

where ${\pi }_k$ is the training prior for class $k$. Setting $\beta$ near one asks the classifier to behave as if the classes were closer to equally likely, which aligns the decision rule with balanced accuracy.

Exact values were more informative than they looked

The next breakthrough was fold-local multiclass target statistics. The data was synthetic, but many numeric values repeated exactly. A value such as a step count or heart rate could therefore behave like a high-cardinality category.

For each selected feature value, I estimated the conditional class distribution on the training portion of a fold and transformed only the held-out portion. The tree pipeline expanded exact-value statistics across all seven numeric features and three ordinal feature interactions. The enhanced XGBoost model improved every corrected fold over its predecessor and reached 0.950398 honest OOF and 0.95054 public.

This feature family also taught an expensive lesson: do not screen a sample-size-dependent encoder only on a small sample. Exact numeric target encoding looked weak on tens of thousands of rows and became valuable at 690,088 rows, where repeated values finally had enough support.

The model that delivered 8th place: FT-Transformer v2

Most strong solutions in this competition were tree-based. The final private result came from adding a genuinely different family: an FT-Transformer trained with masamlp on Kaggle's T4 GPU.

The final v2 specification was:

Component	Setting
Outer validation	7-fold shuffled StratifiedKFold, seed 42
Raw inputs	13 competition features
Additional inputs	39 fold-local exact-value target-statistic features
Total model inputs	52
Categorical handling	learned embeddings for six categorical columns
Numeric handling	quantile scaling and plr-lite embeddings
Transformer	128-dimensional blocks, 2 blocks, 8 attention heads
Training	AdamW, learning rate 0.001, batch size 4,096
Schedule	cosine, 16 fixed epochs, no early stopping
In-model ensemble	4 independently initialized members per fold
Test prediction	mean probabilities across seven fold models

The target encoder was fitted inside every outer training fold. Its training-side values were themselves generated out of fold, while validation and test were only transformed. Quantile scaling, categorical vocabularies, imputation, embeddings, and model weights were also fold-local.

Fixed epochs were deliberate. Restoring the checkpoint that performed best on the same validation fold would have added roughly 0.0003 of optimistic validation score, not necessarily 0.0003 of generalizable skill.

The neural model's raw argmax OOF score was 0.891869. After honest cross-fitted decision correction, it reached 0.950549 ± 0.001740. It beat the corrected XGBoost incumbent by 0.000151 and won four of seven folds. Its raw probabilities had classwise correlations of 0.9984–0.9988 with XGBoost: similar, but different enough to create useful disagreement.

Under the predeclared local gate, that was not enough to claim a confirmed standalone improvement: the gain was below 0.00020, only four folds improved, and unhealthy recall fell by 0.000624. Near the deadline, however, the independently trained neural submission was still a rational model-family hedge. The distinction is important: the repository recorded it as a valid full model without overstating the OOF gain.

The Kaggle notebook submission scored 0.95094 public and 0.95084 private. The private score was the best of the campaign and placed the team 8th on the final leaderboard.

Why the notebook infrastructure mattered

The local full FT run had to be stopped after more than six hours. The RTX 2060 was pinned at full utilization and reached 91°C, while other experiments still needed the machine. Smaller local FT screens also underestimated the exact-value feature family because they lacked enough rows.

The practical solution was to move the full neural confirmation to a private Kaggle Notebook, save raw OOF and test probabilities, download them, verify their hashes and row alignment, and import them into the local experiment registry. This let the T4 do the expensive training while the repository remained the source of truth for honest cross-fitting, blending, comparison, and lineage.

The completed v2 run trained 28 neural networks in total: four ensemble members for each of seven folds. Under the deadline, the notebook was not just a convenient UI; it was the only realistic way to complete the full-scale neural experiment.

The blend that almost passed

The FT model was introduced partly for diversity. I evaluated a predeclared grid of XGBoost/FT probability blends, correcting every blend with fold-external class decisions.

The best blend used 75% XGBoost and 25% FT. It reached 0.950595, improved six of seven folds, and had a paired bootstrap interval of +0.000035 to +0.000357 versus the incumbent.

It still failed the promotion rule:

• the exact gain was +0.000197, missing the +0.00020 threshold by about three millionths;
• unhealthy recall fell by 0.001351, well beyond the 0.00050 guardrail.

That is the kind of candidate the public leaderboard tempts you to submit. The framework rejected it. The final private result vindicated the value of having a standalone neural submission without claiming that every small OOF blend gain was real.

What did not work

The negative results were as useful as the winners because they narrowed the search under severe time and compute constraints.

Experiment family	Best relevant result	Decision
Native CatBoost screen	0.948367	Below matched XGBoost
Hand-built health-range features	−0.000565 vs screen control	Rejected
Independent exact-stat XGBoost seed	0.950362	Confirmed family, not a gain
Missing-count features, full run	0.950293	Screen gain did not transfer
Structural XGBoost trial	0.950452	Only +0.000054; recall guards failed
RealMLP-TD screen	0.948076	Weaker and insufficiently diverse
Class weighting, best corrected screen	+0.000082 vs control	Below promotion threshold
Richer temperature scaling	temperatures converged to 1.0	No value beyond offsets
Two newly probed interactions, full run	0.950303	Two fold wins; unhealthy recall failed

The common pattern was important: many ideas improved raw argmax accuracy or a three-fold screen, then disappeared after class-aware correction or full seven-fold confirmation. Balanced accuracy was won at the interaction between probabilities and decisions, not by maximizing a convenient proxy.

How the 0.95306 public score was produced

The final public submission was a separate, late competition experiment. It began from the public V98 anchor, which scored 0.95303, and applied an explicit ledger of 29 row-level label corrections recovered through controlled small-block ablations.

Transition	Rows
unhealthy → at-risk	20
fit → at-risk	5
at-risk → unhealthy	4
Total	29

The notebook checked the anchor's exact SHA-256, schema, row count, row order, label domain, and every expected source label before changing a row. It then round-trip validated the CSV and asserted the final output hash:

4171D46147B36CEA16AF087FED7BE36BFD9B4BD48098B3D5D20A129123AFBB81

That tiny patch raised the public score from 0.95303 to 0.95306, producing public rank 63. Its private score was 0.95018, substantially below the FT model's 0.95084. The result is therefore best understood as a reproducible public-leader board clean-up, not evidence of a better classifier.

I am including this detail because a useful competition retrospective should explain both what worked and what merely scored. The 29-row artifact explains the latest public score; the FT-Transformer explains the private finish.

What I would carry into the next competition

1. Optimize the competition metric at the decision layer. On an imbalanced multiclass task, probability argmax can leave a huge amount of balanced accuracy unused.
2. Cross-fit every learned post-processing step. Offsets, thresholds, target statistics, calibration, and ensemble weights can all leak if fitted and scored on the same OOF labels.
3. Test scale-dependent features at the scale where they should work. Exact-value statistics were a false negative on small samples and a real gain on the full data.
4. Use a different model family for real diversity. A second tree seed barely helped; the FT-Transformer created different errors and delivered the best private score.
5. Keep raw probabilities, not just labels. Stored OOF and test probabilities made later calibration, disagreement analysis, and blending possible without retraining.
6. Predeclare promotion gates. A +0.000197 blend can look irresistible until its class-recall regression is measured.
7. Treat public leaderboard engineering as a separate layer. It may be a valid competition tactic, but it should never be confused with private generalization.
8. Use hosted notebooks strategically. Under a six-day deadline, Kaggle's T4 was essential for experiments that could not be completed safely on local hardware.

Final reflection

Moving from a 0.92933 first submission and public rank around 1,880 to 8th private and 63rd public was not the result of one magic hyperparameter. It came from a sequence of compounding decisions:

• audit the data;
• align validation with balanced accuracy;
• fit target statistics inside folds;
• preserve OOF probabilities;
• add class-aware decisions;
• demand repeated evidence;
• use Kaggle Notebooks when local compute became the bottleneck;
• record negative results instead of repeating them;
• separate a generalized model gain from public leaderboard post-processing.

The most satisfying number is not 0.95306. It is 8th on the private leaderboard. That is the result produced by the modelling pipeline when the hidden 80% of the test labels finally decided the competition.

Reproducibility and links

• Competition: Predicting Student Health Risk
• Project repository
• Validation strategy
• Full experiment log
• Post-0.950 research results
• Deep-learning and class-balance results
• Training-only feature probing
• Public 0.95306 cleanup notebook
• Public 0.95301 solution pack credited by the cleanup notebook
• Clean-up experiment discussion

All local validation figures in this post come from aligned stored OOF predictions. Final public/private submission scores and leader board positions were reconciled against the authenticated Kaggle API after the competition closed on 31 July 2026.