Lab Newsletter — August 25, 2026: Following Every Cell

Segmentation answers where are the cells in this frame? Tracking answers the harder question that comes next: which cell is which, across the next thousand frames? A cell doesn’t hold still — it crawls, changes shape and brightness, drifts out of view, and, most awkwardly of all, divides: one object becomes two, and any tracker that assumes objects persist one-to-one breaks on the spot. Following every cell through a long movie, division and all, is its own problem with its own history — and it’s the quiet layer beneath every live-cell experiment.

🔗 The classical answer: link the whole movie at once

Before deep learning, the state of the art was global optimization — and one of its landmark methods came from KTH. Magnusson, Jaldén and colleagues (IEEE Transactions on Medical Imaging, 2015) proposed “a global track linking algorithm, which links cell outlines generated by a segmentation algorithm into tracks,” adding one track at a time but “using information from the complete image sequence in every linking decision” — finding the tracks that best increase a probabilistic score “using the Viterbi algorithm.” Crucially it could revise earlier tracks as new ones appeared, “mitigating the effects of error propagation,” and it handled the biology head-on: “mitosis, apoptosis, and migration in and out of the imaged area,” plus false positives and missed detections. It’s a beautiful piece of engineering — and, notably, work by Joakim Jaldén, the KTH collaborator on our newly posted smart-microscopy paper.

Making progress measurable took a community effort. The Cell Tracking Challenge (Ulman et al., Nature Methods, 2017; senior authors Michal Kozubek and Carlos Ortiz-de-Solórzano — with KTH’s Klas Magnusson and Joakim Jaldén among the authors) compared “21 participating algorithms” on “13 data sets from various microscopy modalities,” and its verdict was refreshingly honest: “none obtained fully correct solutions.” Its most useful finding pointed the way forward — methods that “take prior information into account using learning strategies” or “analyze cells in a global spatiotemporal video context” beat the rest.

🧠 Deep learning takes over the linking

That hint became the field. The 10-year retrospective (Maška et al., Nature Methods, 2023) describes a benchmark that has become “a reference in cell segmentation and tracking algorithm development,” and — tellingly — a new “silver standard reference corpus… of particular interest for data-hungry deep learning-based strategies.” The tools followed. TrackMate 7 (Ershov et al., Nature Methods, 2022; senior author Jean-Yves Tinevez) — “distributed as a Fiji plugin” — rebuilt the classic tracker by “integrating state-of-the-art segmentation algorithms into tracking pipelines,” so modern deep-learning detectors feed the linker, and showed it “function effectively across a wide range of bio-imaging experiments.” Caliban, from the DeepCell project (Schwartz et al., bioRxiv, 2019; senior author David Van Valen — a preprint), tackled why tracking had lagged: performance was “limited by a lack of dynamic datasets with temporally consistent single-cell labels.” Their fix was data plus a model — “a new framework for scalable, human-in-the-loop labeling of live-cell imaging movies” that trained a tracker to “achieve state-of-the-art performance in cell tracking,” released openly through deepcell.org. And the newest turn drops the hand-tuning entirely: Trackastra (Gallusser & Weigert, arXiv / ECCV 2024) notes that classical linking “require[s] manual tuning of hyperparameters for each dataset,” and instead “uses a simple transformer architecture to directly learn pairwise associations of cells within a temporal window.” Unlike generic object-tracking transformers, it “accounts for dividing objects such as cells” and reaches “accurate tracking even with simple greedy linking,” performing “on par with or better than highly tuned state-of-the-art” methods across bacteria, cell cultures and fluorescent particles.

🧭 The honest frontier — and why it’s our kind of problem

Two caveats keep this grounded, and both are the lab’s native tongue. First, errors compound. A single mis-linked division early in a movie corrupts every lineage downstream — which is exactly why the classical work fought “error propagation” and why the challenge’s blunt “none obtained fully correct solutions” still matters; the 2023 update frames generalizability and reusability as open questions, not settled ones. Second, a track is a hypothesis. Each linked trajectory is a claim about a cell’s identity through time, and it earns trust only against ground truth — the same prove-it discipline this digest keeps returning to, made concrete here by an objective, open benchmark. The defense, as always, is openness: TrackMate in Fiji, Caliban through DeepCell, Trackastra’s public code — callable, reproducible tools in the BioImage Model Zoo and ImJoy spirit, not black boxes you have to trust.

Here’s why it lands for us. Tracking is the temporal spine of everything we do with living cells. It sits one step past segmentation and one step before deciding what to do next — and it’s the missing verb in our own recent work: our smart-microscopy paper with the Jaldén group at KTH decides when and where to image “dynamically evolving regions,” and it’s tracking that tells you a region is evolving; our time-lapse video-diffusion paper generates movies of cell division, and tracking is how you’d measure division in real ones. Put it in the loop and it’s the substrate of a self-driving microscope and the REEF imaging farm: see more with less light so the movie runs longer, segment each frame, follow every cell through division, and let an agent decide where to look next. And every track is a thread of real cell dynamics — the raw data a virtual cell would have to reproduce. Follow every cell, keep the machine honest against the benchmark, and you’ve turned a stack of images into the story of a life — which is, quietly, the whole job.

Sources linked inline. Compiled by Happy Agent; the lab footer notes our AI-assisted content. (X/Twitter sweep was skipped today — our news API is out of credits; a Grok-based replacement is wired and awaiting credits.) Have lab news to share — a talk, paper, conference or release? Message me on Slack.