Validation · one of three core parts · earliest stage

Which candidate materials can be synthesised?

Generative models now propose candidate crystals faster than they can be tested. The bottleneck is no longer producing candidates but identifying which merit a laboratory attempt.

Why the usual check is not enough

The standard screen is a computed stability check. Quantum-mechanical simulation (density functional theory, DFT) estimates a compound’s energy, and compares it with every mix of competing compounds it could break down into. The gap is called the energy above the hull; zero means nothing it could decompose into is more stable. It is a useful filter but an unreliable verdict:

• —The errors of standard DFT methods are often about as large as the energy margins that decide whether a compound counts as stable.
• —It says nothing about kinetics, precursors, temperature, atmosphere, or which competing phase forms first.
• —Many materials that have been made are metastable, above the hull. Many compounds computed to be stable have never been made.

The true label is experimental: whether the compound was made, by which route, and whether the intended phase formed. Public data suffers from survivorship bias: successful syntheses are published and failed attempts rarely are, so a model trained on public data sees almost only successes. Many laboratories keep records of failed attempts that they have little reason or freedom to publish. Validation therefore depends on the other two components: the data it needs is private by nature, so it can be gathered only through pooling, and only if the security holds.

The design

A screening pipeline, where each stage is cheaper than the one after it and a candidate’s score discounts its value rather than acting as a hard pass or fail:

“Feasible” must not mean “already known”

A model trained only on reported successes can score well by learning “looks like something already made”. A generator optimised against that score drifts back towards known crystals. The design therefore keeps four separate scores and combines them only at ranking:

Results would be reported as a funnel per thousand candidates (valid, novel, stable, synthesisable, useful, confirmed, made) rather than as a single “percent feasible”, which can be inflated by generating near-copies.

The model, as designed

A prior trained centrally on public data, plus a route-dependent correction trained only on lab attempts, where both successes and failures come with their route. The correction is the only part trained across labs, under secure aggregation. Each lab also keeps a private offset for its own equipment and technique. Routes enter only the correction, because in public data only successes have routes, so a route feature there would leak the label. The intended output is a distribution over which phases form, rather than a binary outcome, because a failure that records which competing phase won carries more information than a zero.

None of the neural components has been built. Each must outperform a gradient-boosting baseline on the same splits before it is.

What has been measured

Public data has no failed syntheses, so there are no true “cannot be made” examples to learn from. Both experiments below instead compare two groups. The first is about 7,000 compositions that papers report having made, from a public dataset of synthesis recipes extracted automatically from the literature. The second is compounds a simulation says are close to stable (within 50 meV per atom of the hull), from one file of the Alexandria database, with no record of being made. Some of the second group have in fact been made and are simply missing from the recipe dataset. This is called positive-unlabeled learning, and it means every score here is approximate.

1 · Does pooling successes help?

The same test as the pooling benchmark: simulated labs each hold one slice of chemistry, and the model is scored on families of compounds none of them trained on. Scores are AUC: the chance that the model ranks a randomly chosen made compound above a randomly chosen unlabeled one. 0.5 is chance level and 1.0 is perfect. Threshold set in advance: at eight labs, the pooled model must make at least 10% fewer ranking errors than the best single lab, with the lower end of the 95% confidence interval above zero.

It passes in every setting. However, this pools successes only; it does not yet test whether pooling failures helps, which is the central claim and requires a partner laboratory. Part of the gain reflects more data: each single lab at eight holds only a few hundred rows.

2 · Is the model more than a similarity detector?

This test addresses that concern directly. Compounds get a first-reported year from the publication date of the papers they appear in. The model is trained on compounds reported up to a cutoff (2015 or 2016, set by rule at the 80th percentile) and tested on compounds first reported after it. The hardest version puts those later compounds into training as unlabeled, reflecting what a model built at the cutoff would have seen. Test candidates are binned by distance from the nearest known compound. The test counts only the farthest third, since near-copies inflate any average.

There are two baselines. The first scores a candidate only by its closeness to the nearest known compound, so it learns nothing but similarity. The second sees only element count and formula size. To pass, the model must beat both by at least 0.05 AUC in the far bin, with the lower 95% bound above zero. Bounds come from resampling whole chemical systems.

The model is gradient boosting on composition features: element-property statistics, charge-balance rules, and a Magpie-style descriptor set. Which features and settings to use is chosen on an earlier cutoff using training data only, and a candidate that sat in training as unlabeled is scored only by a copy of the model that never saw it. We ran the whole test on three different random splits.

In both size-matched settings it passes on every split, 6 runs out of 6, beating both baselines by 0.15 to 0.31 AUC far from known materials. This indicates that it has learned more than similarity to known compounds. It does not pass when the size cue is left in: across all compounds, formula size alone is nearly as predictive as the model, which reflects the public benchmark as much as the model. Among unmatched oxides it remains ahead of similarity on average, but not by a statistically significant margin on two of the three splits.

Outstanding work

• —Failed-synthesis data: everything above compares successes with unlabeled compounds.
• —Physics features (hull energy, decomposition, formation energy). In progress: the full Alexandria database is being downloaded so the same physics can be computed for both groups. Reaction-driving-force features will follow.
• —“First reported” means first appearance in the text-mined corpus, not first synthesis. Some compounds counted as new after the cutoff are older, and filtering them needs an experimental structure database.
• —The novelty check, the property model and the ranking are designed but not yet built.
• —Laboratory validation: no candidate has yet been synthesised on the basis of this work.
• —A literature review sufficient to establish which parts of the design, if any, are novel. Synthesizability prediction is an active field, and we claim no priority.