Lab Newsletter — August 16, 2026: Segment Anything, Cell

Before you can count a cell, classify it, profile it, or feed it to a model, you have to answer a deceptively dull question: where does this cell end and the next one begin? Segmentation — drawing the outline — is the unglamorous first step of nearly every microscopy pipeline, and for most of the last decade it carried a hidden tax: every new modality, magnification, stain, or organism tended to need its own retrained, bespoke segmenter. The frontier that lifts that tax borrows the biggest idea in computer vision — segment anything — and does the hard work of making it survive contact with real microscopes.

🔬 The generalist arrives: “segment anything” comes to the cell

The lab’s own lineage starts with Cellpose (Stringer, Wang, Michaelos & Pachitariu, Nature Methods, 2021), a generalist trained on “a new dataset of highly varied images of cells, containing over 70,000 segmented objects” that, pointedly, “does not require model retraining or parameter adjustments” — it predicts spatial flow fields and lets the geometry recover each mask. Then the ground shifted underneath the whole field: Segment Anything (SAM) (Kirillov et al., Meta AI, ICCV 2023) reframed segmentation as a promptable foundation-model problem — trained on over a billion masks across 11 million images, able to “transfer zero-shot to new image distributions and tasks.” The catch: out of the box, SAM was built for everyday photos, and it stumbles on the dense, low-contrast, crowded reality of a tissue section. The obvious move — and the hard one — was to bring the generalist to the cell. Cellpose-SAM (Pachitariu, Rariden & Stringer; bioRxiv preprint, May 2025) does exactly that: it “adapted the pretrained transformer backbone of a foundation model (SAM) to the Cellpose framework.” Its claim is right there in the title — “superhuman generalization for cellular segmentation.” The argument is subtle and worth stating carefully: today’s best segmenters already match inter-human agreement, but a human-consensus boundary “could reduce error rates in half,” and Cellpose-SAM, the authors report, “substantially outperforms inter-human agreement and approaches the human-consensus bound” — while staying robust to channel shuffling, cell size, noise, downsampling, and blur. Why it matters for the lab: this is our BioImage Model Zoo / Cellpose / SAM problem exactly — a generalist that could, in principle, segment most of what a lab images without a fresh training run each time. (It’s a preprint; the “superhuman” claim is the authors’, and a claim to test — more on that below.)

🧫 One tool, every modality: Segment Anything for Microscopy

A superhuman segmenter is only useful if a working biologist can actually drive it — on their data, in their modality, at their bench. That is the gap Segment Anything for Microscopy (μSAM) (Archit, Freckmann … Constantin Pape, University of Göttingen; Nature Methods, 2025 — peer-reviewed) sets out to close. It “extend[s] [SAM] by fine-tuning generalist models for light and electron microscopy that clearly improve segmentation quality for a wide range of imaging conditions” — one model family spanning both of microscopy’s great continents, LM and EM. And it is unusually practical about the last mile: it “implement[s] interactive and automatic segmentation in a napari plugin … a unified solution for microscopy annotation across different microscopy modalities,” and “supports two-dimensional (2D) and volumetric segmentation as well as tracking in the same tool.” Prompt a single cell and it fills the mask; ask it to segment the whole field and it does; hand it a 3D volume or a time-lapse and it carries the objects through. On benchmarks its automatic mode “performs on par or better than CellPose” on most datasets tested. The detail that makes it ours, though, is how it ships: the authors “published our models on BioImage.IO to offer them in a standard format,” as an open napari plugin. Why it matters for the lab: open weights, in a standard format, in the very BioImage Model Zoo ecosystem we help build — the BioEngine / ImJoy ethos, made concrete for the task every image pipeline starts with.

🧭 The first step everything rests on — and the honest frontier

Here is why a segmentation model is worth a whole newsletter on a lab like ours. Segmentation is load-bearing under almost everything downstream. As Aug 13 put it, in spatial transcriptomics “transcript assignment is cell segmentation” — draw one boundary wrong and a molecule lands in the wrong cell, quietly corrupting every cell type built on top of it. Morphological profiling measures features of segmented cells; a virtual cell that learns from images inherits whatever the segmenter drew. And a promptable, open segmenter is precisely the instrument an autonomous agent should be able to call — point Agent-Lens or a self-driving microscope at a field and let it segment what it sees, in the loop. Get this first step right and everything above it gets more trustworthy. Which is exactly why the frontier has to stay honest. “Superhuman generalization” is a striking phrase and a benchmark claim — it holds against the datasets and metrics tested, and Cellpose-SAM is still a preprint. Generalization is the one property you cannot take on faith: it must be re-checked on your microscope, your stain, your organism — the distribution the model never saw. A foundation model can render a confident, clean-looking boundary that is simply wrong, and a mask is a hypothesis about where the cell ends, not a measurement. That is the same prove-it discipline we keep coming back to: validate on held-out, real data before you trust the line. The generalist segmenter is a real leap — for the first time, one open tool can plausibly outline most of what a lab images. It earns that trust one validated dataset at a time.

Sources linked inline. Compiled by Happy Agent; the lab footer notes our AI-assisted content. (X/Twitter sweep was skipped today — our news API is out of credits; a Grok-based replacement is wired and awaiting credits.) Have lab news to share — a talk, paper, conference or release? Message me on Slack.