What we are working on now
The lab’s active work is the four research areas: interpretability, evaluation and measurement, robustness, and efficient architectures. Those are questions about method, and they are what we are doing day to day. You can read them in detail on the Research page.
Where we intend to apply it
The nine domains below are not nine running projects. They are the fields whose measurement problems we think are the most underserved, and where we are looking for collaborators who already have the data and the problem.
Each entry is a survey of where that field currently stands, and what we think the hard measurement question in it is. None of it reports results of our own. If one of these is your field, that is exactly the conversation we want to have — the contact page is here.
Robotics and embodied learning
Robotics is where a model’s errors stop being text and start being physical. It is also the hardest place to evaluate anything, because the test distribution is the world.
Where the edge is
- Vision-language-action models. Single policies trained on large, heterogeneous robot datasets that take a camera feed and an instruction and emit motor commands, rather than a hand-built stack per task.
- Sim-to-real transfer. Training in simulation at enormous scale, then closing the reality gap with domain randomisation and real-world fine-tuning. How much transfers, and which parts do not, is still argued over.
- Dexterous manipulation. Contact-rich tasks with multi-fingered hands, where the control problem is dominated by friction and deformation that simulators model badly.
- Learned world models for control. Predicting future observations well enough to plan in latent space instead of acting greedily.
The measurement problem here: success rates are reported on task suites the authors chose. There is no accepted way to say how a policy will behave in a kitchen it has never seen.
Genomics and protein design
Biology gave machine learning its clearest recent win and its clearest reminder that a benchmark is not the thing you care about.
Where the edge is
- Structure prediction beyond single chains. Complexes, nucleic acids, ligands and post-translational modifications — the cases where confidence scores are least well calibrated.
- Generative protein design. Diffusion and flow-based models that propose backbones for a specified function, with wet-lab success rates as the only honest metric.
- Variant effect prediction. Estimating whether a specific human mutation matters clinically, where the label distribution is badly skewed by what has historically been studied.
- Single-cell and genomic foundation models. Large models trained on expression atlases, with an active debate about whether they beat much simpler baselines on downstream tasks.
The measurement problem here: held-out structural accuracy is not biological usefulness, and the datasets carry decades of selection bias about which proteins anyone bothered to solve.
Mathematics and formal reasoning
Mathematics is the one domain where correctness is machine-checkable, which makes it unusually informative about what models can actually do.
Where the edge is
- Neural theorem proving. Models that search for proofs inside a proof assistant, where the kernel verifies every step and a wrong answer cannot be talked past.
- Autoformalisation. Translating informal mathematical prose into formal statements — currently the bottleneck, and much harder than proving once formalised.
- Conjecture and counterexample generation. Using search and learned heuristics to propose objects that refute a plausible statement.
- Competition-style reasoning. Olympiad problems as a probe, with the caveat that training-set contamination makes published scores hard to interpret.
The measurement problem here: almost none, for formal proof — and that is exactly why it is worth studying. Informal benchmarks have the contamination problem badly.
Materials and chemistry
A field where model predictions are increasingly used to decide which physical experiment to run next, so calibration matters more than average accuracy.
Where the edge is
- Machine-learned interatomic potentials. Universal models that approximate quantum-chemical forces at a fraction of the cost, extending molecular dynamics by orders of magnitude.
- Crystal structure and stability prediction. Generating candidate inorganic materials and estimating whether they are synthesisable, not merely energetically plausible.
- Retrosynthesis planning. Search over reaction templates and learned scoring to propose routes a chemist would actually attempt.
- Self-driving laboratories. Closed-loop systems where the model proposes, robotics executes, and the result updates the model.
The measurement problem here: large claimed yields of “new materials” often shrink substantially once synthesisability and novelty are checked by hand.
Climate and earth systems
Weather forecasting is the rare domain with a continuous, independent ground truth arriving every six hours, which makes it a natural laboratory for evaluation research.
Where the edge is
- Learned forecast emulators. Models trained on reanalysis data that match or beat numerical weather prediction on many headline scores at a tiny fraction of the compute.
- Extremes and tails. Skill on the mean is not skill on the events people care about; blurry predictions score well and forecast badly.
- Downscaling. Turning coarse climate projections into local, decision-relevant fields without inventing structure that is not there.
- Emissions monitoring from remote sensing. Detecting point-source plumes and land-use change at scale.
The measurement problem here: a model optimised for average error learns to hedge, which systematically understates extremes. The scoring rule is doing the damage.
Neuroscience and neural interfaces
Two directions at once: using models to decode neural activity, and using neural data to test whether model representations resemble biological ones.
Where the edge is
- Speech and handwriting decoding. Restoring communication for people with paralysis by decoding intended output from cortical recordings.
- Representational alignment. Comparing learned features against recorded neural responses, and arguing about what similarity metrics actually establish.
- Cross-subject and long-term stability. Decoders drift as electrodes shift and tissue responds; recalibration burden decides whether a system is usable.
- Connectomics at scale. Automated reconstruction of wiring diagrams from volume electron microscopy.
The measurement problem here: two systems can match on a similarity score while computing very different things. The metric needs its own validation.
Clinical and biomedical decision support
The domain with the widest gap between benchmark performance and deployed usefulness, and the one where that gap is most expensive.
Where the edge is
- Multimodal clinical models. Combining imaging, notes, labs and time series rather than treating each in isolation.
- Foundation models over health records. Pretraining on longitudinal patient data, with serious questions about privacy, shortcut learning and site-specific artefacts.
- Distribution shift between hospitals. Models that transfer poorly across scanners, populations and coding practices — the canonical robustness failure.
- Evaluation that reflects care. Moving from retrospective accuracy toward prospective, clinician-in-the-loop measurement of decisions and outcomes.
The measurement problem here: retrospective accuracy on a curated cohort predicts deployed performance poorly, and the failures concentrate in under-represented groups.
Software engineering and program synthesis
Code is unusually easy to evaluate, which has made it the proving ground for agentic systems — and the place where benchmark contamination is most obvious.
Where the edge is
- Repository-scale agents. Systems that navigate a real codebase, run tests and iterate, rather than completing a single function.
- Verification and specification. Pairing generation with type systems, property testing and proof so correctness is checked rather than assumed.
- Long-horizon reliability. Failure rates compound across a multi-step task; the interesting metric is where the trajectory goes wrong, not whether it did.
- Contamination-resistant benchmarks. Building test sets from work created after a model’s training cutoff.
The measurement problem here: a passing test suite is a weak proxy for a correct change, and public benchmarks leak into training data quickly.
Multi-agent systems and mechanism design
When learning systems interact with each other and with people, the object of study stops being a model and becomes an economy.
Where the edge is
- Learning agents in markets. Whether independently trained pricing or bidding agents converge on outcomes no participant chose.
- Mechanism design with learned participants. Rules that stay robust when the agents playing them are optimisers rather than the idealised actors in the proof.
- Agent-based simulation for policy. Using populations of learned agents to stress-test an intervention before it is deployed.
- Cooperation and negotiation protocols. How agents from different principals reach agreements, and what they can misrepresent.
The measurement problem here: equilibrium results assume rational play. Trained agents are neither rational nor stationary, so the guarantees do not carry over.
If you work in one of these domains and have a measurement problem you cannot get traction on, that is exactly the kind of collaboration we are looking for. Get in touch.