Improving training data for magnetic property prediction

1. The premise and the observable

For ions near a spin-state crossover, the crystal-field splitting and the Hund's coupling are comparable in magnitude, so which one wins is set by a parameter fitted to a different property entirely. The database records the resulting number with no indication of how far it would move under a different but equally defensible U.

If that is true, a model trained end-to-end on those labels inherits the ambiguity, and a standard random split hides it, because the affected materials cluster in chemistry space and a random split leaks them across the boundary. So every result here is reported as skill against a trivial predictor rather than raw MAE, which is dominated by each stratum's native moment scale.

Zero means no better than guessing the training mean, and negative means worse. Every structure containing a 3d/4d/5d magnetic ion is assigned to one of twelve strata by clustering on features chosen from theory rather than from the data, namely coordination geometry, d-electron count, M–X–M superexchange angle, bond-length anisotropy as a Jahn–Teller proxy, and site-resolved mixed valence.

2. Three lines of evidence

Three observables on three different systems. Only the third confirms the refined hypothesis at scale, and I have kept them separate rather than averaging them into one number, because they do not carry equal weight.

3. What each line actually supports

The three claims are not equally strong, and collapsing them would hide that.

4. The mechanism, measured directly

One material carries the first-principles side of the charge-state claim. Li₃CoNi₃O₈ is Co⁴⁺ with three Ni³⁺. At U = 4 eV, two starting densities converge to two different self-consistent solutions, a bistable ground state obtained by seeding rather than inferred from SCF oscillation. The ground-state trajectory runs between two integer-pinned plateaus.

The clean electron-counting story, one Ni³⁺ flipping from low to high spin, did not survive the per-site output. All three Ni moments barely move (1.015 → 1.140). What flips is cobalt's local moment, in both sign and magnitude, from −0.21, antiparallel and quenched, to +1.06, parallel and localized, 68% of the projected change.

That is a charge-transfer signature rather than a single-ion crossover, and the oxygen sites can tell the two apart. If Co⁴⁺ here is really Co³⁺ plus a ligand hole, then localizing that hole onto cobalt should strip hole character from the oxygens coordinating Co, and only those. Splitting the eight oxygens by nearest-neighbour distance gives exactly that, with no overlap.

Two limits on how far to lean on it. The charge that redistributes is small (roughly 0.07 e off Co, 0.008 e per oxygen), so this is a modest hole redistribution accompanying a much larger spin reorientation, not a whole hole migrating from ligand to metal. And these are projected-sphere moments capturing 1.856 of the cell's actual 2.00 μ_B change, so the 68 / 20 / 13 percent shares partition what projection sees, not the physical total.

5. Where the U-scan design broke

The group comparison in panel 2 runs against the hypothesis, and for a while I assumed the culprit was the anion-chemistry mismatch the design already acknowledged. The real problem is more basic. The two groups differ in whether the observable is continuously variable at all.

A gapped system's total moment is pinned to an integer by the gap. It cannot drift with U and can only jump when a level crosses. A metal's moment is a Fermi-surface property that moves continuously under any parameter change. So the swing statistic is discrete on one side of the comparison and continuous on the other. The controls are near-guaranteed nonzero swings by band structure alone, before any crossover physics enters. An anion filter would not fix this, since a metallic oxide would still drift.

6. A failure mode I guessed and surprisingly found

The scan's k-point density formula divided by cell size rather than scaling with it, silently giving every material a Γ-only mesh. An earlier revision had already flagged, as a risk, that Γ-only sampling with cold smearing could manufacture spurious exact-integer moment jumps through discrete level occupation. That is what happened. Li₅Mn₂CoO₈'s clean-looking jump to exactly 10.0 μ_B at U = 6, one of three headline "spin-state transitions" supporting the original hypothesis, is flat under a correct mesh, with a swing of 0.11.

The rerun then hands over a free test of the diagnosis, with no additional calculation. If the bug is what I claimed it was, fixing it should barely move gapped oxides and should move metals a great deal.

Five of the seven test oxides did not move at all. FeCoB moved 2.00 → 6.08. That asymmetry is the signature of the specific bug diagnosed, not generic post-rerun drift.

7. Asking the model what it knows about its own error

Everything so far treats the models as things to measure. I wanted to turn that around. Does CHGNet already represent the thing it gets wrong, and if the failure is written somewhere in its own activations, could you read it back out of a single forward pass, with no ensemble and no retraining to pay for? That would be a free uncertainty estimate. Worth chasing.

I set out to probe for error magnitude. That target cannot work. Stratum 0 is, of all things, CHGNet's most accurate stratum by absolute error.

Every stratum under-predicts the moment. Stratum 0 just under-predicts least, so this is a bias effect and not an accuracy one. One alternative had to be cleared away first, because abs_error partly cancels what signed_error isolates, and a stratum whose bias already sits near zero will show a smaller |error| for that reason alone. But it survives the check. Removing each stratum's own bias before recomparing still leaves stratum 0 the tighter group (0.229 vs 0.332, δ = −0.131 against −0.153 raw), so the ordering is real and only a sliver of it is cancellation. A direction fit to |error| has no reason to point at stratum 0 either, and it does not. AUC 0.487, cosine −0.61 with a stratum-0 label direction, actively anti-aligned.

On signed error, though, the probe does find a direction. A ridge fit on the frozen 64-dimensional crystal embedding survives a stability gate of 40 bootstrap refits at a mean pairwise cosine of 0.913, so it is not noise. And it travels. Held out of training on all 1,682 octahedral spin-crossover materials, it still ranks them above the rest, and it ranks error inside that unseen regime about as well as inside the strata it actually saw.

One scope note, because the ablation section below turns on it. That is the pooled crystal vector, which sits downstream of the moment head. It is the wrong tensor for a causal claim, and I do not make one here. This is a legibility result, and legibility does not require causality. Whether the model computes this feature or reads it off the chemical formula is a separate question, and one I did not think to ask until later.

Three things keep this from being a clean headline. The effect is modest. AUC 0.592 against a supervised stratum-0 detector's 0.826 means the direction recovers under half of what is linearly available. It is not low-rank. One component carries 32% of the predictive power and twenty are needed for 88%, so "one or two directions carry most of it" is false here. And its positive extreme is dominated by layered cobalt oxides (15 of the top 25 are Co–O against a 14% base rate), but those are largely the same materials as the stratum-0 enrichment, so it is one observation reported twice, not two confirmations.

8. A circuit I found, and the control that dissolved it

The more interesting target is not model error at all. DFT+U with a Hubbard gap drives d-shell occupancies toward integers, so the recorded moment lands on an integer far more often than real magnetism would produce, 80.5% of labels here against 11.7% of CHGNet's predictions. That is a fingerprint of the labelling protocol, not of accuracy, and a model that represents it has internalised the method's signature. It is the same family as the Li₃CoNi₃O₈ case above, scaled from one material to 12,133.

And it does, remarkably cleanly. Reading the atom features the moment head actually consumes, not the pooled crystal vector that sits downstream of the head and cannot be causal, the feature shows up at AUC 0.923, with one coordinate out of 64 carrying 83% of it. That coordinate sorts elements by how electropositive they are, Cs, Na, K and Rb at the top, Rh, Ir and Pd at the bottom. That is exactly the precondition for Hubbard quantization. And the readout weights it third-heaviest of all 64.

That is a circuit story, and I spent most of this study's effort on it. Three controls, run in order, took it apart.

The first control was the null. The moment head is rank-1, so a control has to match a direction's overlap with the readout, or anything with a readout component looks special by geometry. Matched that way across all 64 coordinates, wiring and predictiveness explain 44% of the apparent selectivity, and dimension 24's residual drops to z = +1.99 (p ≈ 0.047), short of the z = 3.23 a 64-dimension scan needs. The energy pathway showed the same thing cheaply. One control read 1.66× and looked real, eight read chance.

The second control was pre-registration, and it half-failed. One dimension did clear the scan, dim 31, an anion-versus-cation axis carrying no information about a site's own moment (ρ = −0.003). But it emerged from that same scan, so I wrote the hypothesis into PREREGISTRATION.md before testing it on held-out data. Discovery replicates. An untouched half re-selects dim 31 as rank 1 of 64, and the pre-registered primary endpoint passes. That endpoint was the wrong one, because overlap-matched random directions pass it too and dimension 24 scores higher. On the statistic that does discriminate, dim 31 reaches the 92nd percentile of its control distribution, short of the 95th I had committed to.

The third control was the formula itself. Mean-pooling an atom embedding averages a lookup table over sites, which makes it a linear function of the chemical formula. I had assumed that was roughly true and never measured it. Measured, the pooled embedding is recoverable from element mole fractions at R² = 1.0000. So the question is not whether a probe succeeds but whether it beats the formula, and for integer-pinning it does not.

This does not make the observation false. DFT+U integer-pinning really is largely set by which elements are present, which is chemically sensible, and CHGNet's embedding really does reflect it. It makes it weak, and it makes the composition baseline the control that should have been in place from the first probe onward.

The same test rescues the other target. Composition barely touches signed error at R² = 0.110, and message passing more than doubles that to 0.243, leaving 55% of the reachable signal outside the formula. It climbs through conv0, conv1, conv2, and then it stops, right at the tensor the moment head reads. So the network does compute something about its own bias, and it finishes exactly where the readout picks up. That is the target the circuit work should have been aimed at all along.

9. Re-running the protocol on the claim that survived

So I ran the ablation protocol again on signed error at conv2, with three directions instead of one. The first is the raw probe. The second is the probe on composition-residualised features, the computed component only. The third is the probe on the composition-predicted part, the control the layerwise check had exposed as missing.

Nothing reached the 95th percentile against matched random directions. The computed component was, if anything, the weakest of the three. MAE barely moved.

A null means nothing unless the intervention landed, and mine had not. Projecting out a single direction removes only 27% of the signed-error signal, so three-quarters of the target survives and a null from it says nothing. Removing an orthogonal subspace and refitting the probe at each step takes eight of 64 directions to remove 72%, so the feature is distributed, unlike the pinning target. Re-run at rank 8 against rank-matched random subspaces, the damage is large (MAE 0.619 → 1.730) and not stratum-specific. That null holds.

So the end state is split. The network computes something about its own signed error, finishing at the layer the moment head reads, but I cannot show by ablation that the head uses it. Probing and ablation disagree, and after four rounds of controls they still disagree. The obstacle is not statistics but extensivity. A per-formula-unit prediction is a sum over sites through an absolute value, so crystal-level pooled directions are the wrong object for a causal test at a per-site readout.

10. What the DFT runs can label, and what they cannot

That points at site-resolved targets, which Materials Project's cell-total magnetisation cannot supply but this project's own Quantum ESPRESSO runs can. 66 converged runs across 17 materials carry a final per-site moment block. Atom ordering between each scf.in and the structure CHGNet is fed was checked element-by-element, and all 13 align exactly with none mismatched, giving 561 matched (site, U) records.

It is not enough, and not for the reason I expected. Raw site count overstates the sample. Symmetry-equivalent sites share a CHGNet feature vector, and re-running one structure at five U values reuses that vector with different labels, so extra U points add no independent rows. That leaves 27 distinct transition-metal environments against 64 dimensions, 0.42 samples per dimension, underdetermined rather than underpowered. A reduced probe would fix the dimensionality. The top 11 principal components carry 77.9% of site variance and 59.5% of the readout direction, and CHGNet's site moments reconstruct from them at r = 0.983.

One thing the data does support needs no probe and had not been checked anywhere in the project, namely how accurate CHGNet's site-resolved moments are. On transition-metal sites they track DFT at ρ = 0.905, with MAE 0.270 μ_B against a mean magnitude of 1.81, about 15%. Cobalt is unbiased on the largest sample (n = 45, +0.02), and the d¹⁰ cases are correctly zero. Iron is the outlier, under-predicted by 0.64 μ_B, a quarter of its DFT value. That is a specific, checkable claim, and a new one. The same caveat as the U-scan applies. These runs sweep a uniform diagonal U rather than Materials Project's differentiated values, so they are not the labels CHGNet was trained toward, and the per-element biases pool across U.

11. What the interpretability half supports

Two of those are retractions of my own headline results, and they are the part of this work I would most want to be judged on. Probing located the information, but only ablation could test whether the head used it, and the two disagreed. Each of the three collapses came from a control that had not been there before, whether an overlap-matched null, a pre-registered held-out split, or a composition-only baseline. In each case the control was cheaper than the result it overturned.

12. What this establishes, and what it does not

What the work establishes is narrower than what it set out to test, and that narrowing is the finding. The link that holds runs from a structure-only descriptor, through a first-principles measurement of how fragile the label is, to a failure mode in the models that go on to eat those labels. What the descriptor has to capture is whether Δ_oct at an ion's actual charge state competes with Hund's coupling, and a plain d-count cannot express that. Co³⁺ and Ni³⁺ get there. Co²⁺ does not, despite sharing d⁷ with Ni³⁺.

The interpretability half narrowed the same way. I went looking for the circuit that computes the labelling artifact and found that nothing computes it, because the formula already carries it. What survived was the target I had treated as secondary. CHGNet builds a real, non-compositional representation of its own signed error and finishes it at exactly the layer the moment head reads. Whether the head then uses it, I still could not show.

This is a preliminary study, not a paper. The split table is a single seed at a single epoch budget. The feature correlations rest on twelve clusters. The DFT side of the charge-state claim rests on one material, so CHGNet's 360-material result carries most of the weight. The causal interpretability test is blocked on stratum coverage in the DFT sample, not on anything I can compute from what exists. And the motivating premise, that DFT+U moments disagree systematically with neutron diffraction, is taken from the literature rather than demonstrated here, and the comparison module against curated experimental moments is not yet built.

Data from the Materials Project (materialsproject.org). Hubbard U values follow the fitted oxide scheme of Wang et al. (2006). CHGNet is the pretrained universal potential of Deng et al. (2023). The interpretability half follows standard linear-probing and activation-ablation practice, and the contribution there is the control design, not the method. Physics-guided data selection for materials ML is established research, and this report develops and stress-tests one specific descriptor in a small, fully documented study.