We are a small group of AI researchers, and the projects we are furthest along on are ones we are keeping quiet about for now — we are in stealth while the work matures. We would rather say more, and in time we will. If you want to know where things actually stand, write to us. We read everything written by humans and we answer.
Why the lab exists
Machine learning systems are now deployed in places where being wrong is expensive, and the field’s ability to say why a system behaves as it does has not kept pace with its ability to make systems behave impressively. That gap is not a communication problem. It is a measurement problem, and measurement problems are tractable.
We think a small group working on a narrow agenda can make real progress on parts of it: on reading the internal structure of trained networks, on building evaluations that keep meaning something after they have been optimised against, and on characterising how behaviour changes between the test set and the world.
The useful question is not whether a model is impressive. It is what we would have to observe to conclude we were wrong about it.
What we commit to
Negative results get published
An approach that failed, written up clearly with the conditions under which it failed, saves other people months. We treat those write-ups as first-class output, not as an appendix.
Artefacts ship with papers
Code, evaluation harnesses, probes and configuration are released under a permissive licence at the time of publication. Where a dataset cannot be released, we say so and explain why rather than omitting the question.
Limitations are stated, not implied
Every piece of work carries an explicit section on the range over which we verified the result and where we expect it to break. We would rather publish a narrow claim that holds than a broad one that reads well.
Independence
We publish what we find, including when it is inconvenient. Any organisation that supports a piece of work is named on the write-up that work produces. No funder sees a result before it is final, and none has the right to change or withhold one.
Where a piece of work touches a commercial interest of ours or of a partner, we say so in the paper rather than leaving a reader to discover it.
Collaboration
We work with academic groups, with people inside industry labs acting in a personal capacity, and with practitioners who have a measurement problem and real data. If you have a question that overlaps with ours, the fastest route is to write to us with the question rather than a proposal.
Where we are
The lab is based in New Jersey and works with collaborators remotely.