Training a MiFID reporting model on an RTX 2060: the journey from v0.1 failures to v0.5

Can a four-billion-parameter language model be adapted for UK MiFIR transaction-reporting work on an ordinary Windows PC with a 6 GiB graphics card?

The practical answer is yes for the engineering workflow, but not yet for standalone regulatory reliability.

MiFIDBrain is an auditable research experiment comparing untouched Qwen3-4B, QLoRA adapters, retrieval, deterministic rules, and a hybrid architecture. It preserves sources, synthetic scenarios, model manifests, checkpoints, raw outputs, and benchmark failures. Its most useful result is not that a small adapter became a compliance expert; it is that the experiment made the adapter’s failure modes visible.

The result in one minute

September 2026 update: v0.5 completed training and achieved 80/80 on its synthetic holdout, versus 28/80 for the untouched base using the same prompt. On eight previously examined manual regression cases it achieved 7/8, versus 5/8 for the base. The remaining error treats investment advice as reportable. This is a meaningful improvement in the narrow experiment; neither test establishes production regulatory reliability. The history below explains why that qualification matters. The live Hub repository now serves v0.5; tag v0.2 preserves the previous weights.

Item	v0.1	v0.2
Base model	Qwen/Qwen3-4B, immutable revision 1cfa9a7208912126459214e8b04321603b3df60c	Same base model and revision
Hardware	Windows, RTX 2060, 6,144 MB VRAM	Same local profile
Training	3 epochs, 75 optimiser steps, 3.06 hours	3 epochs, 75 optimiser steps, 4.98 hours trainer runtime
Synthetic data finding	No NOT_REPORTABLE examples existed	Balanced reportable, non-reportable, unknown, and future-policy cases
Generated 50-case LoRA holdout	0/50; all outputs unparseable	37/50 (74%) exact determinations
Frozen manually authored comparison	Not available	3/8 exact determinations
Deterministic hybrid	50/50 on generator-derived holdout	8/8 on frozen rule-constrained comparison

The v0.2 result is an improvement in structured synthetic classification. The frozen comparison shows that it is still not suitable as an independent transaction-reporting decision-maker.

What was built

The trained deliverable is a LoRA adapter, not a second copy of Qwen3-4B:

Qwen3-4B base weights
        +
adapter_model.safetensors
        =
MiFIDBrain adapter model

The adapter is approximately 11.83 MB and is published at thunderjordi/mifidbrain-qwen3-4b-lora-v0. It requires the Apache-2.0 Qwen base model. The adapter is also Apache-2.0; MiFIDBrain source code is MIT.

Both local runs used QLoRA with 4-bit NF4 quantization, double quantization, float16 compute, LoRA rank 8, alpha 16, dropout 0.05, and q_proj, k_proj, v_proj, and o_proj as target modules. The micro-batch was one, gradient accumulation was 16, sequence length was 512, learning rate was 0.0002, and the optimizer was paged AdamW 8-bit.

The v0.2 run trained on 400 synthetic scenarios with 50 validation scenarios. Its final training loss was 0.3848095949 and validation loss was 0.0443963632. These are token-prediction losses, not regulatory-accuracy scores.

Why v0.1 failed

The v0.1 adapter really trained, but its dataset was seriously deficient. It contained no NOT_REPORTABLE examples. Its 50-case adapter benchmark produced 50 unparseable outputs, and a six-question verification achieved only 1/6 full determination-contract passes.

The lesson was important: a smooth loss curve and low validation loss can coexist with obvious domain failure. Validation sampled the same skewed synthetic distribution, so it could not demonstrate that the model understood the distinction between reportable and non-reportable transactions.

What v0.2 changed

The v0.2 data generator introduced balanced constrained scenario families:

• reportable UK-firm execution;
• a reportable instrument on a non-UK venue;
• principal execution;
• an entity outside the simplified supplied UK firm scope;
• advice rather than execution;
• an explicitly non-reportable instrument;
• missing instrument-reportability information; and
• the announced 2028 regime, where technical implementation details remain unpublished.

Venue country and decision/execution location are retained as auditable facts. They are not used as standalone reportability shortcuts. In this experiment, instrument_reportable is an explicit authoritative-reference-data input rather than an inference from venue geography.

On the corrected generator-derived 50-case holdout, v0.2 achieved 37/50 exact determinations: all 17 reportable and all 20 non-reportable current binary cases were correct. It failed all future-regime and insufficient-information cases. The deterministic hybrid achieved 50/50 on the same holdout, but these labels came from the same deterministic logic that generated the data, so this is not independent regulatory validation.

The frozen comparison: the test that matters more

To challenge the apparent 74% gain, MiFIDBrain added eight manually authored scenarios outside the generator. They cover a UK-firm execution, a reportable instrument traded on a non-UK venue, an entity outside the simplified supplied UK firm scope, advice without execution, a non-reportable instrument, missing instrument scope, the announced 2028 regime, and off-venue execution.

The labels are still constrained to the project’s documented simplified rules; this is not external legal review or legal advice. The key difference is that the cases were not generated by the v0.2 scenario generator.

System	Exact determinations	What happened
Untouched Qwen3-4B	0/8	Returned noncanonical regime text rather than the required determination contract.
v0.1 LoRA	0/8	Returned noncanonical reportability phrases for every case.
v0.2 LoRA	3/8	Correct on two reportable cases and one explicitly non-reportable instrument.
Deterministic hybrid	8/8	Expected within this rule-constrained test because deterministic rules provide the decision.

The v0.2 adapter failed firm scope, advice/non-execution, missing authoritative instrument data, future-regime handling, and one noncanonical reportable label. Those are substantive failures. An out-of-scope firm must not be treated as in scope; advice must not become execution; missing data must remain insufficient; and announced future policy must not become a current technical determination.

The frozen test is reproducible:

powershell -ExecutionPolicy Bypass -File scripts/run_frozen_v0_2_comparison.ps1

The fixture lives at tests/fixtures/frozen_v0_2_comparison.jsonl, and the generated machine-readable results are written under artifacts/frozen-v0.2-comparison/.

What v0.3 and v0.4 taught us

v0.3 achieved 37/80 on its generated holdout and 3/8 on the manual set. v0.4 added early stopping, but its selected checkpoint reached only 30/80 synthetic. Training had completed; the models continued to fail missing-information and future-regime cases. In v0.4, 156 examples per safety class reduced to just one semantic input after identifiers were removed. Unique IDs were not diverse data.

The trainer also concatenated raw prompt and answer text, while inference used Qwen's chat template. Loss was calculated across input text as well as answers. Thus a low loss and high token accuracy could reward reproduction of the input without demonstrating correct generated determinations. Early stopping exposed the failure sooner but could not repair the training contract.

The v0.5 repair and measured result

v0.5 uses the exact non-thinking Qwen inference prefix during training, masks input tokens from loss, and supervises the JSON answer plus its stop token. It removes IDs, category, difficulty, expected answers and rule IDs from model inputs. The future label becomes FUTURE_RULES_UNPUBLISHED; strict JSON parsing rejects invalid outputs. There is no deterministic substitution for model answers in these standalone adapter evaluations.

The revised data has 200 contrastive families, each with four outcomes. Whole families are split into 640 training, 80 validation and 80 test rows. Each training class has 160 distinct model inputs. The checks found no duplicate model inputs or family overlap across splits, and the completed run's data hashes match preflight. The longest complete training example occupies only 157 of the 512 allowed tokens. These are still variations of simplified rule-generated supervision.

Same v0.5 prompt and cases	Untouched base	Selected adapter
Synthetic holdout	28/80 (35%)	80/80 (100%)
Previously examined manual regression	5/8 (62.5%)	7/8 (87.5%)
Synthetic insufficient information	6/20	20/20
Synthetic future regime	0/20	20/20

Every adapter answer in these 88 cases was valid contract JSON. The manual failure remains visible: advice without execution returned {"determination":"REPORTABLE"} when NOT_REPORTABLE was expected. This motivates broader activity examples and a new blind evaluation, not a claim that transaction-reporting decisions can now be delegated to the adapter.

The run 20260910-200943 used learning rate 0.0001, seed 45, rank 8, alpha 16, dropout 0.05, batch one, accumulation 16 and paged AdamW 8-bit. A balanced 40-case validation gate ran every 25 steps. It reached 100% at step 75, stayed there at 100 and 125, and stopped after two checks without improvement. The published adapter is step 75, not the last checkpoint. Runtime was 11973.0642 seconds (about 3.33 hours). The adapter contains 11,834,776 bytes.

Historical versions used different prompts, data distributions and macro-F1 definitions. The controlled comparison is base versus adapter in the table above. The manual cases had already been reviewed during development, so 7/8 is regression evidence rather than independent validation. Temporal labels reflect the project's frozen regulatory assumptions; this article does not reverify current legal rules.

Reproduce v0.5 using the existing compatible CUDA environment:

powershell -ExecutionPolicy Bypass -File .\scripts\run_v0_5.ps1 -PrepareOnly
powershell -ExecutionPolicy Bypass -File .\scripts\run_v0_5.ps1

Each launch writes its own timestamped artifacts/v0.5/ folder. The runbook and release evidence include the configuration, raw answers, checkpoint selection and provenance. The model card supplies the exact loading and structured prompting example.

Why deterministic controls remain necessary

MiFIDBrain should not be presented as a model that decides UK MiFIR reportability by itself. The credible architecture is:

Natural-language question
  → model-assisted candidate fact extraction
  → deterministic validation and missing-fact checks
  → deterministic temporal and regulatory controls
  → model-assisted explanation with source citations

Hard controls remain deterministic:

• explicit as_of_date and current/future regime selection;
• firm, execution, transmission, and instrument-scope predicates;
• required-fact and uncertainty handling;
• future-policy safeguards that do not invent unpublished technical rules.

The language model can help normalize terminology, extract candidate facts, summarize retrieved passages, and explain an already-controlled outcome. It must not silently override the rules engine.

Running the adapter without reloading it every question

The following API example documents the earlier general-question interface. Its prompting differs from v0.5's evaluated structured contract; the v0.5 scores must not be attributed to it. For v0.5, retain the model loaded by the model-card example and send additional structured prompts through that same instance.

The normal ask CLI ends after each request, so it reloads the quantized base model and adapter every time. For repeated local experimentation, start the persistent API once:

mifidbrain serve --adapter thunderjordi/mifidbrain-qwen3-4b-lora-v0

Then call it from another PowerShell window:

Invoke-RestMethod http://127.0.0.1:8000/health
Invoke-RestMethod http://127.0.0.1:8000/ask -Method Post -ContentType 'application/json' -Body '{"question":"Is this reportable?","as_of_date":"2026-08-31"}'

The service keeps one model-plus-adapter resident in GPU memory and serializes generations to remain within the RTX 2060’s VRAM budget.

Reproduce the local workflow

These are the original workflow commands. For the v0.5 experiment use the versioned runner above; the default training configuration is historical.

python -m pip install -e ".[training,dev]"
mifidbrain doctor
mifidbrain sources update
mifidbrain knowledge build
mifidbrain dataset generate
mifidbrain dataset validate
mifidbrain evaluate --system base
mifidbrain train

The source pipeline preserves retrieval timestamps, source hashes, parser versions, effective dates, and current-versus-future regime provenance. The project intentionally keeps runtime models, raw source downloads, training artifacts, and generated data out of Git while retaining reproducible manifests and scripts.

Final takeaway

This experiment demonstrates that a real QLoRA adapter can be trained locally on a Windows machine with an RTX 2060, saved as a portable PEFT artifact, benchmarked, and served from a GPU-resident process.

It also demonstrates a more valuable engineering principle: an apparent improvement on a generator-derived benchmark is not enough. The frozen comparison exposed that v0.2, while better than v0.1, remains unreliable as a standalone decision-maker.

v0.5 adds evidence that repairing supervision and the input/output contract can produce a substantial gain on the same consumer GPU. Its 80/80 synthetic and 7/8 regression results support that engineering claim, while the advice error and limited evaluation scope keep the broader reliability claim open.

The showcase is therefore an auditable process: train a real model, measure it, challenge the measurement, preserve the failures, and use deterministic controls wherever the consequence of a wrong answer is unacceptable.