Xenium Ranger's notion of a pipeline is very flexible in that a pipeline can be composed of stages that run stage code or sub-pipelines that may themselves contain stages or sub-pipelines.

Xenium Ranger pipelines follow the convention that stages are named with verbs (such as, MERGE_METRICS, SELECT_CELLS_DATASET) and sub-pipelines are named with nouns and prefixed with an underscore (e.g., _CELL_SEGMENTOR_FINALIZER). Each stage runs in its own directory bearing its name, and each stage's directory is contained within its parent pipeline's directory.

For example, the xeniumranger resegment pipeline begins with the following process graph:

where

• XR_PREFLIGHT is a preflight stage, which validates inputs prior to running the other stages.
• XENIUM_RANGER_CS is the top-level pipeline stage
• XR_RESEGMENT is a sub-pipeline contained in XENIUM_RANGER_CS. The SEGMENT_NUCLEI, SETUP_CELL_SEGMENTOR_FINALIZER stages are contained in the XR_RESEGMENT sub-pipeline.

Every pipestance operates wholly inside of its pipeline output directory. When the pipestance completes, this pipestance output directory contains three outputs: metadata files, the pipestance output file directory, and the top-level pipeline stage directory.

• Metadata files are files prefixed with an underscore (_) and usually contain unstructured text or JSON-encoded arrays and hashes.
• The pipestance output file directory is a directory called outs/ that contains the pipestance's output files.
• The top-level pipeline stage directory is a directory named according to the top-level pipeline stage that contains the child stage directories that compose this pipestance.

The top-level pipeline stage directory is a stage directory that contains any number of child stage directories as well as one stage output directory for each fork run by that stage. There is one possible top-level pipeline stage, Xenium_Ranger_CS.

All Xenium Ranger pipelines contain only single-fork stages, so there is only one fork0 stage output directory within each stage directory. Chunk output directories are a subset of stage output directories that additionally contain runtime information specific to the job or process being run by that chunk such as a process ID or cluster job ID.

For example, any of the Xenium Ranger pipelines' pipeline output directory contains the following directory structure:

The metadata contained in the pipeline output directory includes:

Stage directories contain stage output directories, stage output files, and the stage directories of any child stages or pipelines.

Stage output directories typically contain:

Chunk output directories are a subset of stage output directories that, in addition to the aforementioned stage output, may contain:

Metadata files should be treated as read-only. Altering the contents of metadata files is not recommended.

Pipestance output directories can demonstrate very complicated structures, and re-attaching the Xenium Ranger UI is the easiest way to quickly navigate to a pipeline stage of interest and examine its metadata. In the absence of being able to access the UI, the standard find command can quickly return high-level information about a pipestance.

For example, to find the stages that resulted in the overall failure of a pipestance whose output directory is sample345/,

Copy
find sample345/ -name _errors

# Example output
# sample345/XENIUM_RANGER_CS/INSITU_COUNTER_CS/INSITU_COUNTER_CORE/INSITU_ANALYZER/SUMMARIZE_ANALYSIS/fork0/_errors
...

This indicates that the failed stage was SUMMARIZE_ANALYSIS.

It is helpful to view all _errors files' contents at once by piping to xargs cat:

Copy
find sample345/ -name _errors | xargs cat

In the above case, the error is an unhandled exception without obvious cause. Failures such as this should be reported to the 10x Genomics software support team for assistance with diagnosis.