
Cluster mode is one of three primary ways of running Cell Ranger ARC. To learn about the other approaches, see the computing options page.

Cell Ranger ARC can be run in cluster mode, using SGE or LSF to run the stages on multiple nodes via batch scheduling. This allows highly parallel stages to utilize hundreds or thousands of cores concurrently, thereby significantly reducing time to solution.

<SupportCallout level={"info"}>
10x Genomics does not officially support _Slurm_ or _Torque/PBS_. While some customers have successfully used Cell Ranger ARC with those job schedulers in _cluster mode_, it is unsupported and may require trial and error.
</SupportCallout>

<Image src={"https://cdn.10xgenomics.com/image/upload/v1694818856/software-support/Cell%20Ranger%20ARC/cluster_mode.png"} />

Running pipelines in *cluster mode* requires the following:

1. Cell Ranger ARC is installed in the same location on all nodes of the cluster. For example, `/opt/cellranger-arc-2.0.2` or `/net/apps/cellranger-arc-2.0.2`
2. Cell Ranger ARC pipelines are run on a shared file system accessible to all nodes of the cluster. NFS-mounted directories are the most common solution for this requirement.
3. The cluster accepts both single-core and multi-threaded (shared-memory) jobs.

<LinkableSection id={"configuring-cluster-template"} title={"Configuring cluster template"}>

Installing the Cell Ranger ARC software on a cluster is identical to installation on a local server. After confirming that the cellranger-arc commands can run in single server mode, configure the job submission template that Cell Ranger ARC uses to submit jobs to the cluster. Assuming Cell Ranger ARC is installed to /opt/cellranger-arc-2.0.2, the process is as follows:

Step 1. Navigate to the Martian runtime's jobmanagers/ directory which contains example jobmanager templates.

```
cd /opt/cellranger-arc-2.0.2/external/martian/jobmanagers
ls
config.json  lsf.template.example  sge.template.example  slurm.template.example
```

**Step 2.** Make a copy of the cluster's example template (LSF, SGE, or Slurm) to either `lsf.template`, `sge.template` or `slurm.template` in the `jobmanagers/` directory.

```
cp -v sge.template.example sge.template
`sge.template.example' -> `sge.template'
ls
config.json  lsf.template.example  sge.template  sge.template.example  slurm.template.example
```

The job submission templates contain a number of special variables that are substituted by the Martian runtime when each stage is submitted. Specifically, the following variables are expanded when a pipeline is submitting jobs to the cluster:

| Variable                                                | Must be present? | Description                                                                  |
| ------------------------------------------------------- | ---------------- | ---------------------------------------------------------------------------- |
| `__MRO_JOB_NAME__`                                      | Yes              | Job name composed of the sample ID and stage being executed                  |
| `__MRO_THREADS__`                                       | No               | Number of threads required by the stage                                      |
| `__MRO_MEM_GB__` `__MRO_MEM_MB__`                       | No               | Amount of memory (in GB or MB) required by the stage                         |
| `__MRO_MEM_GB_PER_THREAD__` `__MRO_MEM_MB_PER_THREAD__` | No               | Amount of memory (in GB or MB) required per thread in multi-threaded stages. |
| `__MRO_STDOUT__` `__MRO_STDERR__`                       | Yes              | Paths to the `_stdout` and `_stderr` metadata files for the stage            |
| `__MRO_CMD__`                                           | Yes              | Bourne shell command to run the stage code                                   |

**It is critical that the special variables listed as required are present in the final template you create.** For more information on how the template should appear for a cluster, consult your cluster administrator or help desk.

</LinkableSection>

<LinkableSection id={"additional-template-considerations"} title={"Additional template considerations"}>

Depending on which job scheduler you have, please select a tab below.

</LinkableSection>

<LinkableSection id={"lsf"} nested={true} title={"LSF"}>

The most common modifications to the job submission template include adding additional lines to specify:

1. Your research group's queue. For example, `#BSUB -q smith_queue`
2. The account to which your jobs will charge. For example, `#BSUB -P smith_lab`

</LinkableSection>

<LinkableSection id={"sge"} nested={true} title={"SGE"}>

For SGE cluster, you MUST replace `<pe_name>` within the example template to reflect the name of the cluster's multi-threaded parallel environment. To view a list of the cluster's parallel environments, use the `qconf -spl` command.

The most common modifications to the job submission template include adding additional lines to specify:

1. The research group's queue. For example, `#$ -q smith.q`
2. The account to which jobs will charge. For example, `#$ -A smith_lab`

</LinkableSection>

<LinkableSection id={"slurm"} nested={true} title={"Slurm"}>

The most common modifications to the job submission template include adding additional lines to specify:

1. Your research group's account. For example, `#SBATCH -A smith_lab`
2. Time limit for the cluster job. For example, `#SBATCH -t=24:00:00` to set a 24 hours limit.

</LinkableSection>

<LinkableSection id={"validating-template-configuration"} title={"Validating template configuration"}>

To run a Cell Ranger ARC pipeline in cluster mode, add *one* of `--jobmode=lsf`, `--jobmode=sge` or `--jobmode=slurm` command-line option when using the `cellranger-arc` commands. It is also possible to use `--jobmode=<PATH>`, where is the full path to the cluster template file.

To validate that cluster mode is properly configured, follow the same validation instructions given for `cellranger-arc` in the [Installation](/support/software/cell-ranger-arc/downloads) page and specify the appropriate cluster via `--jobmode`.

```
cellranger-arc mkfastq --run=./tiny-bcl --samplesheet=./tiny-sheet.csv --jobmode=sge

Martian Runtime - 2.0.2

Running preflight checks (please wait)...
2016-09-13 12:00:00 [runtime] (ready)           ID.HAWT7ADXX.MAKE_FASTQS_CS.MAKE_FASTQS.PREPARE_SAMPLESHEET
2016-09-13 12:00:00 [runtime] (split_complete)  ID.HAWT7ADXX.MAKE_FASTQS_CS.MAKE_FASTQS.PREPARE_SAMPLESHEET
...
```

Once the preflight checks are finished, check the job queue to see stages queuing up:

```
qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
8675309 0.56000 ID.HAWT7AD jdoe         qw    09/13/2016 12:00:00 all.q@cluster.university.edu       1
8675310 0.55500 ID.HAWT7AD jdoe         qw    09/13/2016 12:00:00 all.q@cluster.university.edu       1
```

In the event of a pipeline failure, an error message is displayed:

```
[error] Pipestance failed. Please see log at:
HAWT7ADXX/MAKE_FASTQS_CS/MAKE_FASTQS/MAKE_FASTQS_PREFLIGHT/fork0/chnk0/_errors

Saving diagnostics to HAWT7ADXX/HAWT7ADXX.debug.tgz
For assistance, upload this file to 10x Genomics by running:

uploadto10x <your_email> HAWT7ADXX/HAWT7ADXX.debug.tgz
```

The `_errors` file contains a `jobcmd error`:

```
cat HAWT7ADXX/MAKE_FASTQS_CS/MAKE_FASTQS/MAKE_FASTQS_PREFLIGHT/fork0/chnk0/_errors

jobcmd error:
exit status 1
```

The most likely reason for this failure is an invalid job submission template. This occurs when the job submission via `bsub`, `qsub`, or `sbatch` commands failed.

The "tiny" dataset does not stress the cluster, so it is worthwhile to follow up with a more realistic test using one of our [sample datasets](/resources/datasets?menu%5Bproducts.name%5D=Single+Cell+Multiome+ATAC+%2B+Gene+Expression)

</LinkableSection>

<LinkableSection id={"cluster-mode-mechanics"} title={"Cluster mode mechanics"}>

There are two subtle variants of running `cellranger-arc` pipelines in cluster mode, each with their own pitfalls. Check with your cluster administrator to see which approach is compatible with your institution's setup.

1. **Run `cellranger-arc` with `--jobmode=sge` on the head node**. Cluster mode was originally designed for this use case. However, this approach leaves `mrp` and `mrjob` running on the head node for the duration of the pipeline and some clusters impose time limits to prevent long running processes.
2. **Use a job script to submit a `cellranger-arc` command with `--jobmode=sge`**: With this approach, `mrp` and `mrjob` run on a cluster mode. However, the cluster must allow jobs to be submitted from a compute node to make this viable.

When `cellranger-arc` is run in cluster mode, a single library analysis is partitioned into *hundreds* and potentially *thousands* of smaller jobs. The underlying Martian pipeline framework launches each stage job using the `bsub`, `qsub` or `sbatch` commands when running on LSF, SGE, or Slurm cluster, respectively. As stage jobs are queued, launched, and completed, the pipeline framework tracks their status using the metadata files that each stage maintains in the pipeline output directory.

Like *single server* pipelines, cluster-mode pipelines can be restarted after a failure. They maintain the same order of execution for the dependent stages of the pipeline. All executed stage code is identical to *single server* mode and the quantitative results are identical to the limit of each stage's reproducibility.

<SupportCallout level={"warning"}>
Cluster-mode pipelines that are stopped (either by you or due to a stage failure) do _not_ delete pending stages that have already been submitted to the cluster queue. As a result, some pipeline stages may continue to execute after the cellranger-arc commands have exited.
</SupportCallout>

In addition, the [Cell Ranger ARC UI](/support/software/cell-ranger-arc/latest/advanced/the-cell-ranger-arc-user-interface) can still be used with cluster mode. Because the Martian pipeline framework runs on the node from which the command was issued, the UI will also run from that node.

</LinkableSection>

<LinkableSection id={"memory-requests-and-consumption"} title={"Memory requests and consumption"}>

Stages in the Cell Ranger ARC pipelines each request a specific number of cores and memory to aid with resource management. These values are used to prevent oversubscription of the computing system when running pipelines in single server mode. The way CPU and memory requests are handled in cluster mode is defined by the following:

- How the `__MRO_THREADS__` and `__MRO_MEM_GB__` variables are used within the job template.
- How your specific cluster's job manager schedules resources.

Depending on which job scheduler you have, select a tab below:

</LinkableSection>

<LinkableSection id={"lsf-2"} nested={true} title={"LSF"}>

LSF supports job memory requests through the `-M` and `-R [mem=...]` options, but these requests generally must be expressed in MB, not GB. As such, your LSF job template should use the `__MRO_MEM_MB__` variable rather than `__MRO_MEM_GB__`. For example,

```
cat bsub.template
#BSUB -J __MRO_JOB_NAME__
#BSUB -n __MRO_THREADS__
#BSUB -o __MRO_STDOUT__
#BSUB -e __MRO_STDERR__
#BSUB -R "rusage[mem=__MRO_MEM_MB__]"
#BSUB -R span[hosts=1]

__MRO_CMD__
```

</LinkableSection>

<LinkableSection id={"sge-2"} nested={true} title={"SGE"}>

SGE supports requesting memory via the mem_free resource natively, although your cluster may have another mechanism for requesting memory. To pass each stage's memory request through to SGE, add an additional line to your sge.template that requests mem_free, h_vmem, h_rss, or the custom memory resource defined by your cluster:

```
cat sge.template
# -N __MRO_JOB_NAME__
# -V
# -pe threads __MRO_THREADS__
# -l mem_free=__MRO_MEM_GB__G
# -cwd
# -o __MRO_STDOUT__
# -e __MRO_STDERR__

__MRO_CMD__
```

<SupportCallout level={"info"}>
In the above example, the trailing G in the highlighted MRO_MEM_GB is required by SGE to denote that mem_free is being expressed in GB units.
</SupportCallout>

Note that the `h_vmem` (virtual memory) and `mem_free`/`h_rss` (physical memory) represent two different quantities and that Cell Ranger ARC stages' `__MRO_MEM_GB__` requests are expressed as physical memory. Using `h_vmem` in your job template may cause certain stages to be unduly killed if their virtual memory consumption is substantially larger than their physical memory consumption. It follows that we do not recommend using `h_vmem`.

<SupportCallout level={"warning"}>
If you do use h_vmem in a template, it is recommended that you use the MRO_MEM_GB_PER_THREAD or MRO_MEM_MB_PER_THREAD variables instead of MRO_MEM_GB and MRO_MEM_MB. To determine memory limits for a multicore job, SGE will multiply the number of threads by the value in h_vmem. The MRO_MEM_GB and MRO_MEM_MB variables already reflect the sum amount of memory across all threads needed to run the job. Using those variables as h_vmem will inflate the memory required for multi-threaded jobs.
</SupportCallout>

</LinkableSection>

<LinkableSection id={"slurm-2"} nested={true} title={"Slurm"}>

Slurm supports job memory requests through the `-mem` option, but the default units are megabytes, so in order to specify the memory in GB, it's important to keep the trailing 'G' suffix after the `__MRO_MEM_GB__` variable, as shown below.

```
cat slurm.template
#SBATCH -J __MRO_JOB_NAME__
#SBATCH --export=ALL
#SBATCH --nodes=1 --ntasks-per-node=__MRO_THREADS__
#SBATCH --signal=2
#SBATCH --no-requeue
### Alternatively: --ntasks=1 --cpus-per-task=__MRO_THREADS__
###   Consult with your cluster administrators to find the combination that
###   works best for single-node, multi-threaded applications on your system.
#SBATCH --mem=__MRO_MEM_GB__G
#SBATCH -o __MRO_STDOUT__
#SBATCH -e __MRO_STDERR__

__MRO_CMD__
```

For clusters whose job managers do not support memory requests, it is possible to request memory in the form of cores via the `--mempercore` command-line option. This option scales up the number of threads requested via the `__MRO_THREADS__` variable according to how much memory a stage requires.

For example, given a cluster with nodes that have 16 cores and 128 GB of memory (8 GB per core), the following pipeline invocation command

```
cellranger-arc mkfastq --run=./tiny-bcl --samplesheet=./tiny-sheet.csv --jobmode=sge --mempercore=8
```

will issue the following resource requests:

- A stage that requires 1 thread and less than 8 GB of memory will pass `__MRO_THREADS__` of `1` to the job template.
- A stage that requires 1 thread and 12 GB of memory will pass `__MRO_THREADS__` of `2` to the job template because (12 GB) / (8 GB/core) = 2 cores.
- A stage that requires 2 threads and less than 16 GB of memory will pass `__MRO_THREADS__` of `2` to the job template.
- A stage that requires 2 threads and 40 GB of memory will pass `__MRO_THREADS__` of `5` to the job template because (40 GB) / (8 GB/core) = 5 cores.

As the final bullet point illustrates, this mode can result in wasted CPU cycles and is only provided for clusters that cannot allocate memory as an independent resource.

Every cluster configuration is different. If you are unsure of how your cluster resource management is configured, contact your cluster administrator or help desk.

</LinkableSection>

<LinkableSection id={"controlling-rate-of-job-submissions"} title={"Controlling rate of job submissions"}>

Some Cell Ranger ARC pipeline stages are divided into hundreds of jobs. By default, the rate at which these jobs are submitted to the cluster is throttled to at most 64 at a time and at least 100ms between each submission to avoid running into limits on clusters which impose quotas on the total number of pending jobs a user can submit.

If your cluster does not have these limits or is not shared with other users, you can control how the Martian pipeline runner sends job submissions to the cluster by using the `--maxjobs` and `--jobinterval` parameters.

To increase the cap on the number of concurrent jobs to 200, use the `--maxjobs` parameter:

```
cellranger-arc count --id=sample ... --jobmode=sge --maxjobs=200
```

You can also change the rate limit on how often the Martian pipeline runner sends submissions to the cluster. To add a five-second pause between job submissions, use the `--jobinterval` parameter:

```
cellranger-arc count --id=sample ... --jobmode=sge --jobinterval=5000
```

The job interval parameter is in milliseconds. The minimum allowable value is 1.

</LinkableSection>


Cluster Mode


Job submission mode is one of three primary ways of running Cell Ranger ARC. To learn about the other approaches, refer to the [computing options](/support/software/cell-ranger-arc/downloads) page.

Cell Ranger ARC can be run in *job submission* mode, by treating a *single* node from the cluster like a local server. This leverages existing institutional hardware for pipeline analysis.

<Image src={"https://cdn.10xgenomics.com/image/upload/v1694817814/software-support/Cell%20Ranger%20ARC/hybrid_mode.png"} />

Running `cellranger-arc` commands in this mode is typically achieved by submitting a job script to the cluster. The process is as follows:

**Step 1.** Select a job scheduler tab below to see a sample job script.

<SupportCallout level={"info"}>
10x Genomics does not officially support Slurm or Torque/PBS. However, many customers have successfully used Cell Ranger ARC with those job schedulers in job submission mode. Sample job scripts are shown below.
</SupportCallout>

**Step 2.** Copy the sample job script to a text editor and replace all the variables enclosed within braces (e.g. `{JOB_NAME}` or `{STDOUT_LOG}`) with literal names, paths, and values. The last line, `{PIPELINE_CMD}` , is replaced by a Cell Ranger ARC command, specifying `--jobmode=local`, `--localcores={NUM_THREADS}` and `--localmem=0.9*{MEM_GB}` to restrict CPU and memory use that is consistent with the directives in the job script. For example, if the job script requests 8 cores and 64 GB of memory from the cluster, then the Cell Ranger ARC command should include `--jobmode=local --localcores=8 --localmem=57`.

<SupportCallout level={"info"}>
Since cluster setup and policy vary greatly from one institution to another, the sample job scripts provided below should be treated as a starting point. Consult with your local cluster administrator to see if additional directives are required.
</SupportCallout>

<LinkableSection id={"sge"} nested={true} title={"SGE"}>

```
#!/usr/bin/env bash
#
# =============================================================================
# Setup Instructions
# =============================================================================
#
# 1. Substitute {PE_NAME} below with name of your cluster's shared-memory
#    parallel environment. If your cluster does not have a parallel environment,
#    delete this line. However, the job will run with only 1 thread.
#
# =============================================================================
# Job Script
# =============================================================================
#
#$ -N {JOB_NAME}
#$ -V
#$ -pe {PE_NAME} {NUM_THREADS}
#$ -cwd
#$ -l mem_free={MEM_GB}G
#$ -o {STDOUT_LOG}
#$ -e {STDERR_LOG}
#$ -S "/usr/bin/env bash"

{PIPELINE_CMD}
```

</LinkableSection>

<LinkableSection id={"lsf"} nested={true} title={"LSF"}>

```
#!/usr/bin/env bash
#
# =============================================================================
# Job Script
# =============================================================================
#
#BSUB -J {JOB_NAME}
#BSUB -n {NUM_THREADS}
#BSUB -o {STDOUT_LOG}
#BSUB -e {STDERR_LOG}
#BSUB -R "rusage[mem={MEM_MB}]"
#BSUB -R span[hosts=1]

{PIPELINE_CMD}
```

</LinkableSection>

<LinkableSection id={"slurm"} nested={true} title={"Slurm"}>

```
#!/usr/bin/env bash
#
# =============================================================================
# Job Script
# =============================================================================
#
#SBATCH -J {JOB_NAME}
#SBATCH --export=ALL
#SBATCH --nodes=1 --ntasks-per-node={NUM_THREADS}
#SBATCH --signal=2
#SBATCH --no-requeue
### Alternatively: --ntasks=1 --cpus-per-task={NUM_THREADS}
###   Consult with your cluster administrators to find the combination that
###   works best for single-node, multi-threaded applications on your system.
#SBATCH --mem={MEM_GB}G
#SBATCH -o {STDOUT_LOG}
#SBATCH -e {STDERR_LOG}

{PIPELINE_CMD}
```

</LinkableSection>

<LinkableSection id={"torque-pbs"} nested={true} title={"Torque/PBS"}>

```
#!/usr/bin/env bash
#
# =============================================================================
# Job Script
# =============================================================================
#
#SBATCH -J {JOB_NAME}
#SBATCH --export=ALL
#SBATCH --nodes=1 --ntasks-per-node={NUM_THREADS}
#SBATCH --signal=2
#SBATCH --no-requeue
### Alternatively: --ntasks=1 --cpus-per-task={NUM_THREADS}
###   Consult with your cluster administrators to find the combination that
###   works best for single-node, multi-threaded applications on your system.
#SBATCH --mem={MEM_GB}G
#SBATCH -o {STDOUT_LOG}
#SBATCH -e {STDERR_LOG}

{PIPELINE_CMD}
```

**Step 3.** If this job script is saved as **runJob.bash**, then the following command submits the job to the cluster:

```
sbatch runJob.bash
```

</LinkableSection>


Job Submission Mode


The *pipeline output directory*, described in [Understanding Output](/support/software/cell-ranger-arc/latest/analysis/single-library-analysis#command-line-argument-reference), contains all of the data produced by one invocation of a pipeline (a *pipestance*) as well as rich metadata describing the characteristics of each stage. This directory contains a specific structure that is used by the [Martian](http://martian-lang.org) pipeline framework to track the state of the pipeline as execution proceeds.

<LinkableSection id={"pipeline-structure"} title={"Pipeline structure"}>

Cell Ranger's notion of a pipeline is very flexible in that a pipeline can be composed of *stages* that run *stage code* or *sub-pipelines* that may themselves contain stages or sub-pipelines.

Cell Ranger pipelines follow the convention that stages are named with verbs (e.g., `ALIGN_READS`, `MARK_DUPLICATES`, `FILTER_BARCODES`) and sub-pipelines are named with nouns and prefixed with an underscore (e.g., `_BCSORTER`). Each stage runs in its own directory bearing its name, and each stage's directory is contained within its parent pipeline's directory.

For example, the `cellranger-arc mkfastq` pipeline has the following process graph:

<Image src={"https://cdn.10xgenomics.com/image/upload/v1694810763/software-support/Cell%20Ranger%20ARC/mkfastq-in-flight.png"} />

where

- `MAKE_FASTQS_CS` is the *top-level pipeline stage*
- `MAKE_FASTQS` is a *sub-pipeline* contained in `MAKE_FASTQS_CS`
- `PREPARE_SAMPLESHEET`, `BCL2FASTQ_WITH_SAMPLESHEET`, `MAKE_QC_SUMMARY`, and `MERGE_FASTQS_BY_LANE_SAMPLE` are *stages* contained in the `MAKE_FASTQS` *sub-pipeline*.
- `MAKE_FASTQS_PREFLIGHT` and `MAKE_FASTQS_PREFLIGHT_LOCAL` are *preflight stages*, which validate inputs prior to running the other stages. These also belong to `MAKE_FASTQS`, but have no connections to other stages because they don't produce any outputs.

<SupportCallout level={"info"}>
The `MAKE_FASTQS_CS` stage is not strictly necessary since it contains no stages and only one child pipeline (`MAKE_FASTQS`); however, it serves to mask some of the low-level inputs required by the `MAKE_FASTQS` pipeline.
</SupportCallout>

</LinkableSection>

<LinkableSection id={"directory-structure"} title={"Directory structure"}>

Every pipestance operates wholly inside of its *pipeline output directory*. When the pipestance completes, this *pipestance output directory* contains three outputs: metadata files, the pipestance output file directory, and the top-level pipeline stage directory.

- **Metadata files** are files prefixed with an underscore (`_`) and usually contain unstructured text or JSON-encoded arrays and hashes.
- The **pipestance output file directory** is a directory called `outs/` that contains the pipestance's output files.
- The **top-level pipeline stage directory** is a directory named according to the top-level pipeline stage that contains the child *stage directories* that compose this pipestance.

The *top-level pipeline stage* directory is a *stage directory* that contains any number of child stage directories as well as one *stage output directory* for each *fork* run by that stage. The top-level pipeline stages for Cell Ranger ARC are:

- `MAKE_FASTQS_CS` for [cellranger-arc mkfastq](/support/software/cell-ranger-arc/latest/analysis/generating-fastqs-mkfastq)
- `SC_ATAC_GEX_COUNTER_CS` for [cellranger-arc count](/support/software/cell-ranger-arc/latest/analysis/single-library-analysis)

Most of the Cell Ranger ARC pipelines contain single-fork stages, which means there is one `fork0` *stage output directory* within each *stage directory*. *Chunk output directories* are a subset of *stage output directories* that additionally contain runtime information specific to the job or process being run by that chunk (e.g., a process ID or cluster job ID).

For example, the `cellranger-arc mkfastq` pipeline's *pipeline output directory* contains the following directory structure:

|                                                                    |                                    |
| ------------------------------------------------------------------ | ---------------------------------- |
| \_log                                                              | Metadata file                      |
| outs/                                                              | Pipestance output file directory   |
| MAKE_FASTQS_CS/                                                    | Top-level pipeline stage directory |
| MAKE_FASTQS_CS/fork0/                                              | Stage output directory             |
| MAKE_FASTQS_CS/fork0/files/                                        | Stage output files                 |
| MAKE_FASTQS_CS/MAKE_FASTQS/                                        | Stage directory                    |
| MAKE_FASTQS_CS/MAKE_FASTQS/fork0/                                  | Stage output directory             |
| MAKE_FASTQS_CS/MAKE_FASTQS/fork0/files/                            | Stage output files                 |
| MAKE_FASTQS_CS/MAKE_FASTQS/BCL2FASTQ_WITH_SAMPLESHEET/             | Stage directory                    |
| MAKE_FASTQS_CS/MAKE_FASTQS/BCL2FASTQ_WITH_SAMPLESHEET/fork0/       | Stage output directory             |
| MAKE_FASTQS_CS/MAKE_FASTQS/BCL2FASTQ_WITH_SAMPLESHEET/fork0/chnk0/ | Chunk output directory             |

</LinkableSection>

<LinkableSection id={"commonly-generated-metadata"} title={"Commonly generated metadata"}>

The metadata contained in the *pipeline output directory* includes

| File Name    | Description                                                                                                                               |
| ------------ | ----------------------------------------------------------------------------------------------------------------------------------------- |
|              | Metadata cache that is populated when a pipestance completes to minimize re-aggregation of metadata                                       |
|              | The MRO call used to invoke this pipestance                                                                                               |
|              | The log messages that are reported to your terminal window when running `cellranger-arc` commands                                         |
| `_mrosource` | The entire MRO describing the pipeline with all `@include` statements dereferenced                                                        |
| `_perf`      | Detailed runtime performance data for every stage in the pipestance                                                                       |
| `_timestamp` | The start and finish time for this pipestance                                                                                             |
| `_vdrkill`   | A list of all of the volatile data (temporary files) removed during pipeline execution as well as total number of files and bytes deleted |
| `_versions`  | Versions of the components used by the pipeline                                                                                           |

*Stage directories* contain *stage output directories*, *stage output files*, and the *stage directories* of any child stages or pipelines.

*Stage output directories* typically contain:

| File Name     | Contents                                                                                                                                  |
| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `files/`      | Directory containing any files created by this stage that were not considered volatile (temporary)                                        |
| `split/`      | A special *stage output directory* for the step that divided this stage's input into parallel chunks                                      |
| `chnkN/`      | A *chunk output directory* for the Nth parallel chunk executed                                                                            |
| `join/`       | A special *stage output directory* for the step that recombined this stage's parallel output chunks into a single output dataset again    |
| `_complete`   | A file that, when present, signifies that this stage has successfully completed                                                           |
| `_errors`     | A file that, when present, signifies that this stage failed. Contains the errors that resulted in stage failure.                          |
| `_invocation` | The MRO call used to execute this stage by the Martian framework                                                                          |
| `_outs`       | The output files generated by this stage                                                                                                  |
| `_vdrkill`    | A list of all of the volatile data (temporary files) removed during pipeline execution as well as total number of files and bytes deleted |

*Chunk output directories* are a subset of *stage output directories* that, in addition to the aforementioned stage output, may contain:

| File Name    | Contents                                                                                                                        |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------- |
| `_args`      | The arguments passed to the stage's *stage code*                                                                                |
| `_jobinfo`   | Metadata describing the stage's execution, including performance metrics, job manager jobid and jobname, and process ID         |
| `_jobscript` | The script submitted to the cluster job manager ([cluster mode](/support/software/cell-ranger-arc/latest/advanced/cluster-mode) |
| `_stdout`    | Any *stage code* output that was printed to the stdout stream                                                                   |
| `_stderr`    | Any *stage code* output that was printed to the stderr stream                                                                   |

These metadata files should be treated as read-only, and altering the contents of metadata files is not recommended.

</LinkableSection>

<LinkableSection id={"navigating-pipestances"} title={"Navigating pipestances"}>

Pipestance output directories can demonstrate very complicated structures, and re-attaching the [Cell Ranger ARC UI](/support/software/cell-ranger-arc/latest/advanced/the-cell-ranger-arc-user-interface) is the easiest way to quickly navigate to a pipeline stage of interest and examine its metadata. In the absence of being able to access the UI, the standard `find` command can quickly return high-level information about a pipestance.

For example, to find the stages that resulted in the overall failure of a pipestance whose output directory is `sample345/`,

```
$ find sample345/ -name _errors
sample345/SC_ATAC_GEX_COUNTER_CS/SC_ATAC_GEX_COUNTER/_ATAC_MATRIX_COMPUTER/_PEAK_CALLER/COUNT_CUT_SITES/fork0/chnk0-u308e57a713/_errors
```

This tells us that the failed stage was `COUNT_CUT_SITES`.

It can be helpful to view all `_errors` files' contents at once by piping to `xargs cat`:

```
$ find sample345/ -name _errors | xargs cat

Traceback (most recent call last):
  File "/home/jdoe/cellranger-arc-1.0.0/external/martian/adapters/python/martian_shell.py", line 659, in _main
    stage.main()
  File "/home/jdoe/cellranger-arc-1.0.0/external/martian/adapters/python/martian_shell.py", line 618, in main
    self._run(lambda: self._module.main(args, outs))
  File "/home/jdoe/cellranger-arc-1.0.0/external/martian/adapters/python/martian_shell.py", line 589, in _run
    cmd()
  File "/home/jdoe/cellranger-arc-1.0.0/external/martian/adapters/python/martian_shell.py", line 618, in
    self._run(lambda: self._module.main(args, outs))
  File "/home/jdoe/cellranger-arc-1.0.0/mro/atac/stages/processing/count_cut_sites/__init__.py", line 119, in main
    contig=args.contig, filename=args.fragments, index=args.fragments_index
  File "/home/jdoe/cellranger-arc-1.0.0/lib/python/atac/tools/io.py", line 136, in parsed_fragments_from_contig
    fragments = pysam.TabixFile(filename, index=index)
  File "pysam/libctabix.pyx", line 351, in pysam.libctabix.TabixFile.__cinit__
  File "pysam/libctabix.pyx", line 404, in pysam.libctabix.TabixFile._open
  File "pysam/libchtslib.pyx", line 516, in pysam.libchtslib.HTSFile.tell
NotImplementedError: seek not implemented in files compressed by method 1
```

In the above case, the error is an unhandled exception that results in a stack trace and whose cause is not obvious; these sorts of failures should be reported to [the 10x Genomics software support team](support@10xgenomics.com) for assistance with diagnosis. (Note: this failure was induced by truncating the fragments file while the pipestance was running.)

Stages whose stage code run external binaries (for example, the `ALIGN_AND_COUNT` stage which runs STAR) often generate output to their stdout and stderr streams. These messages are captured in the `_stdout` and `_stderr` metadata files within the chunk output directories, and combining `find` and `xargs cat` to examine their contents can also assist with troubleshooting.

</LinkableSection>


Pipestance Structure


Cell Ranger ARC's pipelines expose a user interface (UI) for monitoring progress through a web browser. By default, this UI is exposed at an operating system assigned port with a randomly-generated authentication token to restrict access. Specifying the `--uiport=3600` option when using `cellranger` will force the UI to be exposed on port 3600 and the `--disable-ui` option will turn off the UI.

This UI is accessible through a web interface that runs on the given port on the machine where the pipeline was started. The URL to use with the web browser is printed to the pipeline standard output and log files and can also be found in the `[sampleid]/_uiport` file where the pipeline was launched.

<LinkableSection as={"h4"} id={"understanding-the-ui"} title={"Understanding the UI"}>

When the pipeline runs, it will display the URL for the UI.

```
$ cellranger-arc count --id=sample345 \
                       --reference=/opt/refdata-cellranger-arc-GRCh38-2020-A-2.0.0 \
                       --libraries=/home/jdoe/runs/libraries.csv \
                       --localcores=16 \
                       --localmem=64


Martian Runtime - v4.0.5
Serving UI at http://host.example.com:5603/?auth=mcV3MKANWfNTERRGASgYV8aXskx-rSH7hxynAdsTieA
2012-01-01 12:00:00 [runtime] Reattaching in local mode.

Running preflight checks (please wait)...
```

The UI will become unavailable once the pipeline completes unless the `--noexit` flag is passed.

While a pipeline is running, you can open the url (`http://host.example.com:5603/?auth=mcV3MKANWfNTERRGASgYV8aXskx-rSH7hxynAdsTieA`) in your web browser to view the pipeline process graph (See [the full graph here](/support/software/cell-ranger-arc/latest)):

<Image src={"https://cdn.10xgenomics.com/image/upload/v1694807540/software-support/Cell%20Ranger%20ARC/ui_pipeline_graph.png"} />

Clicking on any of the graph's nodes will reveal more information about that stage in the right pane. This information pane is broken into several sections and the topmost shows high-level details about the stage's execution. For example, the `CLUSTER_CELLS` stage would show:

<Image src={"https://cdn.10xgenomics.com/image/upload/v1694807653/software-support/Cell%20Ranger%20ARC/ui_details.png"} />

This includes the state of the stage (running, failed, or complete), the fully-qualified stage name (`FQName`), the directory in which the stage is running (`Path`), and the stage code that is being run (the location of the Python package being run in the above example).

Clicking the arrow next to the stage name (`← CLUSTER_CELLS` in the above example) will return to the pipeline-level metadata view.

The **Sweeping** section allows you to view the MRO call used to invoke this stage (`invocation`) and information about the split and join components of the stage if it was parallelized.

<Image src={"https://cdn.10xgenomics.com/image/upload/v1694807718/software-support/Cell%20Ranger%20ARC/ui_sweeping.png"} />

The **Argument Bindings** and **Return Bindings** sections display the inputs and outputs of the stage:

<Image src={"https://cdn.10xgenomics.com/image/upload/v1694807778/software-support/Cell%20Ranger%20ARC/ui_arg_binding.png"} />

In general, only top-level pipeline stages (those represented by rectangular nodes in the process graph) contain return bindings.

The **Chunking** section displays information about the parallel execution of the stage:

<Image src={"https://cdn.10xgenomics.com/image/upload/v1694807818/software-support/Cell%20Ranger%20ARC/meta-chunks.png"} />

Many stages are automatically parallelized and process different pieces (chunks) of the same input dataset in parallel. In the above example, the `CLUSTER_CELLS` was split into 9 chunks for this stage. Chunks 0-3 already completed, 4-6 are in flight, and 7-8 are waiting for CPU and/or memory to become available before running.

Clicking on an individual chunk exposes additional options for viewing metadata about that chunk's execution, including any errors, standard output, and in cluster mode the job script used to queue the job to the cluster. The `jobinfo` includes information about how a chunk was executed, selected environment variables, and for completed chunks various performance statistics such as peak memory usage and CPU time used. As with the **Sweeping** section, additional metadata associated with the chunk execution can also be viewed.

</LinkableSection>

<LinkableSection as={"h4"} id={"launching-the-ui-for-existing-pipelines"} title={"Launching the UI for existing pipelines"}>

You can also examine pipestances that have already completed using the Cell Ranger UI. Assuming your pipestance output directory was `/home/jdoe/runs/sample345`, simply re-run the `cellranger-arc` command with the `--noexit` option to re-attach the UI:

```
$ cellranger-arc count --id=sample345 \
                       --reference=/opt/refdata-cellranger-arc-GRCh38-2020-A-2.0.0 \
                       --libraries=/home/jdoe/runs/libraries.csv \
                       --localcores=16 \
                       --localmem=64
                       --noexit


Martian Runtime - v4.0.5
Serving UI at http://host.example.com:3600
2012-01-01 12:00:00 [runtime] Reattaching in local mode.

Running preflight checks (please wait)...
Pipestance completed successfully, staying alive because --noexit given.
```

Because `cellranger-arc` assumes it is resuming an incomplete pipeline job when re-attaching, the pipeline must be valid and preflight checks must still be passed. As such, relocating a complete pipeline may prevent the UI from reattaching.

</LinkableSection>

<LinkableSection as={"h4"} id={"accessing-the-ui-through-a-firewall"} title={"Accessing the UI through a firewall"}>

If you run pipelines on a server that blocks access to all ports except SSH, you can still access the Cell Ranger UI using SSH forwarding. Assuming you have a `cellranger-arc` pipeline running on port `3600` on `cluster.university.edu`, you can type the following from your computer:

```
$ ssh -NT -L 9000:cluster.university.edu:3600 jdoe@cluster.university.edu
jdoe@cluster.university.edu's password:
```

Upon entering your password (assuming you are `jdoe@cluster.university.edu`), the command will appear to hang. However, in the background it has mapped port 9000 on your laptop to port 3600 on `cluster.university.edu` through the ssh connection for which you just entered the password.

This allows you to go to http://localhost:9000/ in your web browser and access the UI running on `cluster.university.edu:3600`. Once you are done examining the UI, use Ctrl+C in your `ssh -NT -L ...` terminal window to terminate this SSH forwarder.

A full explanation of SSH forwarding is beyond the scope of this guide. Refer to [SSH/OpenSSH/PortForwarding](https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding) for more information.

</LinkableSection>


The Cell Ranger ARC User Interface

Cell Ranger ARC is an advanced analytical suite designed for the Chromium Single Cell Multiome ATAC + Gene Expression sequencing. It provides in-depth analysis of gene expression and chromatin accessibility at a single cell level, uniquely linking these aspects for enhanced genomic understanding.

Overview

Getting Started

Downloads & Installation

Analysis

Tutorials

Algorithms

Advanced

Troubleshooting

Release Notes

Datasets

Loupe Browser

Cell Ranger ARC

A tutorial on using Cell Ranger ARC mkref to create a reference for Rattus norvegicus

Custom Reference for Rat

Learn more about input files required to run Cell Ranger ARC

Inputs

Generating FASTQs (mkfastq)

Specifying FASTQs

FASTQs with Illumina Software

Custom Reference (mkref)

Learn more about how to run Cell Ranger ARC

Running Pipelines

Analyzing GEX + ATAC (count)

Aggregating Multiple Samples (aggr)

Custom Analysis (reanalyze)

Learn more about the output files generated by Cell Ranger

Outputs

Web Summary

ATAC + Gene Expression Analysis

Feature-Barcode Matrices

Per Barcode Metrics

Per Barcode QC Metrics

ATAC Barcoded BAM

ATAC Fragments

ATAC Peaks

ATAC Peak Annotations

ATAC Transposition Counts

GEX Barcoded BAM

GEX Molecule Info (H5)

HDF5 Feature Barcode Matrix Format

What is Cell Ranger ARC?

We provide resources from protocols, user guides, and free-to-use analysis tools

Algorithms overview

Advanced pipeline content for Cell Ranger ARC.

View the latest updates to Cell Ranger ARC.

Browse Q&As from the 10x Genomics support team.

Access free public Chromium data from 10x Genomics.

Loupe Browser is a desktop application that allows you to interactively visualize data.