The first step is to detect the locations of nuclei using the DAPI images and a custom neural network for nucleus segmentation. The neural network is trained on thousands of manually labeled image patches covering multiple tissue types. Any nucleus that has 95% or more of its pixel intensity lower than an intensity threshold of 100 photoelectrons will be removed.

This algorithm step is only compatible for tissues prepared with the Multi-Tissue Stain Mix in the Xenium In Situ Gene Expression with Cell Segmentation Staining (CG000749) workflow. Read the Xenium In Situ Multimodal Cell Segmentation: Workflow and Data Highlights Technical Note for more details.

The multimodal cell segmentation algorithm uses custom deep learning models to analyze the multi-channel stain images. The models are trained on 10x Genomics Xenium In Situ data, which include a range of tissue types and preparations (fresh frozen, FFPE).

After nucleus segmentation with DAPI, the algorithm uses three methods to segment cells.

The segmentation results are prioritized in this order for each cell:

• Segment cells based on their cell boundary stain: The inferred segmentation from this method should be closest to the true cell membrane boundary. It uses cell-surface marker antibodies to target epithelial markers (ATP1A1, E-Cadherin) and immune markers (pan-lymphocyte: CD45). This method can split nuclei, define cells missing a nucleus, and identify multinucleate cells. Nuclei that overlap with anucleate cells are assigned to the cell.
• Segment cells based on expansion from the nucleus to the cell interior stain edge: This method includes both a deep learning model and a nuclear expansion method using the interior stain to infer cell boundaries. It uses the interior stain (18S rRNA marker) and the DAPI stain for nuclei. The XOA v2.0 cell segmentation algorithm does not currently use the interior protein stain (alphaSMA/Vimentin), as the 18S marker has been sufficient for most tissue types. The inferred cell outline may look irregular in cases where boundaries between cell interior stains are challenging to identify (i.e., for dense tissue types).
• Nuclear expansion: For cases where cells that do not have boundary or interior stains, segment cells with a nuclear (DAPI) expansion distance of 5 µm or until another cell boundary is encountered (described more below).

The analysis_summary.html and metrics_summary.csv outputs will provide metrics for the proportion of cells segmented with each method (described here).

This algorithm step is compatible with the Xenium In Situ Gene Expression (CG000582) workflow.

Once the locations of nuclei in the sample have been identified by the model, a heuristic cell boundary expansion step is performed. The nucleus boundaries are expanded by 5 µm or until they encounter another cell boundary in X-Y. If cell boundaries overlap during expansion, they are resolved using an algorithm that is conceptually similar to Voronoi tessellation.

Xenium cell segmentation takes into account the 3D output from the DAPI image processing step for all Z-slices for better accuracy, as well as the 2D focus image outputs if the multimodal cell segmentation workflow is used. The pipeline ultimately produces a flattened 2D segmentation mask for ease of use.

The nuclear boundaries are consolidated to form non-overlapping 2D objects when projected in X-Y. Since the segmentation mask is 2D, transcripts are assigned to 2D shapes based on their X and Y coordinates.

Nucleus and cell boundary polygons are also provided in the outputs. They are approximations of the segmentation masks and are used for efficient visualization of nucleus and cell segmentation in Xenium Explorer and other analysis software. Cell and nucleus mask and polygon data are stored in the cells.zarr.zip output file.

You can also use XOA outputs to explore other segmentation methods developed in the community. Analysis Guides on this topic can be found on the 10x website. These third-party segmentation results can be viewed in Xenium Explorer with updated XOA metrics and outputs using Xenium Ranger's import-segmentation pipeline.