When the xeniumranger pipelines fail, they will automatically generate a "debug tarball" that contains the logs and metadata generated by the pipestance leading up to failure. This file, named sample_id.mri.tgz, can be e-mailed to the 10x Genomics software team to help resolve any issues with using Xenium Ranger.

You may also use the xeniumranger upload command to send the tarball to 10x Genomics. Customize the code with your email and sample_id.mri.tgz file name:

Copy
xeniumranger upload your@email.edu sample_id.mri.tgz

If you wish to troubleshoot a pipeline failure yourself, it is important to identify if you are experiencing a preflight failure or an in-flight failure.

The remainder of this guide uses the term pipestance to refer to a specific instance of a pipeline running.

Preflight failures are the most common and are the result of invalid input data or runtime parameters. Because they occur before the pipeline actually runs, there will be no pipeline output and the error is reported directly to your terminal.

In-flight failures are generally the result of factors external to the pipeline such as running out of system memory or disk space. Different stages may fail in different ways so the specific error messages vary widely.

There are a few important files that are saved to your pipeline output directory which, by default, is named according to the --id name (output_dir in the examples below).

1. The pipeline execution log that is output to your terminal during pipeline execution is also saved to _log.

2. Stages that experience a hard failure generate an _errors file containing the precise error that caused a stage to halt. You view these error logs, if they exist, using:

   Copy
   find output_dir -name _errors | xargs cat
   

3. Each stage also logs its stderr and stdout streams to _stderr and _stdout files. These logs can be listed using:

   Copy
   find output_dir -name _stderr
   find output_dir -name _stdout
   

A more detailed description of the pipeline output directory and its contents is available in the Pipestance Structure page.

If you are unable to diagnose a failure yourself, you can always contact the 10x Genomics software support team for help.

Once you have determined the reason for failure and are ready to continue running the pipeline, you can typically issue the same xeniumranger command to continue execution of the pipestance from the stage that originally failed.

At the start of any xeniumranger pipeline, the software looks for its intended output directory. If such a directory exists, this directory/run is treated as an incomplete pipestance and xeniumranger resumes pipeline execution. While this feature allows xeniumranger pipelines to be stopped and resumed with great flexibility, it can also result in errors such as:

• The following error indicates that you specified an --id that corresponds to an existing directory that was not created by Xenium Ranger:

  Copy
  RuntimeError: /home/jdoe/runs/sample345 is not a pipestance directory
  

• The following error indicates that you may already have a copy of xeniumranger running that is using the same output directory:

  Copy
  RuntimeError: pipestance 'HAWT7ADXX' already exists and is locked by another Martian instance. If you are sure no other Martian instance is running, delete the _lock file in /home/jdoe/runs/HAWT7ADXX and start Martian again.
  

  If you are sure that there is no pipestance running in the given output directory, you can either rename that output directory entirely (mv HAWT7ADXX HAWT7ADXX.old) to restart the pipestance from the beginning, or you can remove the pipestance's _lock file (rm HAWT7ADXX/_lock) and re-run the xeniumranger command to resume pipeline execution.

• If you encounter the following error when attempting to resume a pipestance, you are attempting to resume a pipestance using command line arguments that differ from those used to first run it:

  Copy
  RuntimeError: pipestance 'sample345' already exists with different invocation file /home/jdoe/runs/sample345/_invocation
  

  You can view the input parameters for the existing pipeline by examining the _log file located in the output directory (e.g., head -n20 /home/jdoe/runs/sample345/_log).