How to: Use Parallel Loop in Bash

· 5 min read · Tutorials

How to: Use Parallel Loop in Bash

TL;DR: Run bash loops in parallel with & or GNU parallel to process multiple files simultaneously. Reduces runtime for independent tasks like FASTQ processing. Remember: N parallel jobs × memory per job = total RAM needed.

The Problem

Processing files sequentially wastes time when tasks are independent:

# Sequential: processes one file at a time
for file in *.fastq.gz; do
    process_sample "$file"  # Wait for each to finish
done
# Total time: 10 files × 30 min = 5 hours

The Solution: Parallel Processing

Run multiple tasks simultaneously:

# Parallel: processes all files at once
for file in *.fastq.gz; do
    process_sample "$file" &  # & = run in background
done
wait  # Wait for all background jobs to finish
# Total time: 30 min (if you have enough cores/RAM)

Sequential vs Parallel

Method 1: Background Jobs (&)

Basic Pattern

# Define task function
task() {
    input=$1
    output="${input%.fastq.gz}.bam"
    echo "Processing $input"
    bwa mem ref.fa "$input" > "$output"
    echo "Done: $output"
}

# Run in parallel
for file in *.fastq.gz; do
    task "$file" &  # Launch in background
done

wait  # Wait for all jobs to complete
echo "All files processed"

Real Example: RNA-seq Quantification

#!/bin/bash

# Configuration
data="/path/to/fastqs"
output="/path/to/results"
cores_per_job=4

# Task function
quantify_sample() {
    r1=$1
    r2=${r1%_R1.fastq.gz}_R2.fastq.gz
    sample=$(basename ${r1%_R1.fastq.gz})
    
    echo "[$(date)] Starting: $sample"
    
    kallisto quant \
        -i transcripts.idx \
        -o $output/$sample \
        -t $cores_per_job \
        $r1 $r2
    
    echo "[$(date)] Finished: $sample"
}

# Export function for subshells
export -f quantify_sample
export data output cores_per_job

# Run all samples in parallel
for r1 in $data/*_R1.fastq.gz; do
    quantify_sample "$r1" &
done

wait
echo "All samples quantified"

Limiting Concurrent Jobs

Prevent overwhelming the system:

max_jobs=8
running_jobs=0

for file in *.fastq.gz; do
    # Wait if max jobs reached
    while [ $(jobs -r | wc -l) -ge $max_jobs ]; do
        sleep 1
    done
    
    process_sample "$file" &
done

wait

Method 2: GNU Parallel (Recommended)

GNU parallel offers better control and monitoring.

Installation

# Ubuntu/Debian
sudo apt install parallel

# macOS
brew install parallel

# conda
conda install -c conda-forge parallel

Basic Usage

# Process files with automatic job management
parallel process_sample ::: *.fastq.gz

# Control number of concurrent jobs
parallel -j 4 process_sample ::: *.fastq.gz

# Pair R1/R2 files
parallel -j 8 \
    'kallisto quant -i idx -o {1/.} {1} {2}' \
    ::: *_R1.fastq.gz \
    :::+ *_R2.fastq.gz

Advanced: Progress & Logging

# Show progress bar
parallel --bar -j 8 process_sample ::: *.fastq.gz

# Log output per job
parallel -j 8 --joblog run.log --results logs/ \
    process_sample ::: *.fastq.gz

# Resume failed jobs
parallel --resume --joblog run.log -j 8 \
    process_sample ::: *.fastq.gz

Paired-End FASTQ Example

#!/bin/bash

# Function to process paired-end reads
process_pair() {
    r1=$1
    sample=$(basename ${r1%_R1.fastq.gz})
    r2=${r1%_R1.fastq.gz}_R2.fastq.gz
    
    mixcr analyze rna-seq \
        --species mmu \
        --threads 4 \
        $r1 $r2 \
        output/$sample
}

export -f process_pair

# Run with GNU parallel (8 samples × 4 threads = 32 cores)
parallel -j 8 process_pair ::: data/*_R1.fastq.gz

Resource Management

Calculate Required Resources

Scenario Calculation
Cores parallel_jobs × cores_per_job
Memory parallel_jobs × mem_per_job
Time total_files / parallel_jobs × time_per_file

Example: SLURM Cluster

#!/bin/bash
#SBATCH --job-name=parallel_mixcr
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=32      # 8 jobs × 4 cores
#SBATCH --mem=300G              # 8 jobs × ~35G
#SBATCH --time=06:00:00

# Rule of thumb: 1 core per 10GB RAM
# 300G / 10 = 30 cores (requested 32, good)

parallel -j 8 process_sample ::: *.fastq.gz

Memory-Intensive Tasks

# Limit jobs based on memory, not cores
# If each job needs 50GB and you have 256GB total:
max_mem_jobs=$((256 / 50))  # = 5 jobs

parallel -j $max_mem_jobs memory_intensive_task ::: files

Comparison: & vs GNU Parallel

Feature Background & GNU Parallel
Setup Built-in Requires install
Job control Manual Automatic
Progress None --bar, --eta
Logging Manual redirect --joblog
Resume No --resume
Complexity Simple Feature-rich
Best for Quick scripts Production workflows

Best Practices

1. Test on Subset First

# Test with 2 samples before full run
parallel -j 2 process_sample ::: $(ls *.fastq.gz | head -2)

2. Monitor Resource Usage

# In another terminal
watch -n 5 'ps aux | grep process_sample | wc -l'
htop  # Interactive process viewer

3. Handle Errors Gracefully

process_with_error_handling() {
    input=$1
    output="${input%.fastq.gz}.bam"
    
    if bwa mem ref.fa "$input" > "$output" 2>&1; then
        echo "SUCCESS: $input"
    else
        echo "FAILED: $input" >> failed.log
        return 1
    fi
}

parallel -j 8 --halt soon,fail=3 \  # Stop if 3 jobs fail
    process_with_error_handling ::: *.fastq.gz

4. Organize Output

# Create output dirs automatically
parallel -j 8 \
    'mkdir -p output/{/.} && process {/} output/{/.}' \
    ::: data/*.fastq.gz

Common Use Cases

Compress Multiple Files

parallel -j 4 gzip ::: *.fastq

Download Multiple URLs

parallel -j 10 wget ::: $(cat urls.txt)

Convert File Formats

parallel -j 8 \
    'samtools view -bS {}.sam > {}.bam' \
    ::: sample1 sample2 sample3

Run Parameter Sweep

# Test multiple parameter combinations
parallel -j 16 \
    'run_analysis --param1 {1} --param2 {2} > {1}_{2}.out' \
    ::: 0.1 0.5 1.0 \
    ::: 10 50 100

Troubleshooting

"Too many open files"

Cause: Parallel jobs exceed file descriptor limit
Fix:

ulimit -n 4096  # Increase limit

Jobs fail silently

Cause: No error checking
Fix: Use set -e and check exit codes:

task() {
    set -e  # Exit on error
    command || { echo "FAILED: $1"; return 1; }
}

Memory overflow

Symptoms: System becomes unresponsive, OOM killer activates
Fix: Reduce -j parameter or increase RAM request on cluster

Performance Tips

  1. I/O-bound tasks: Use more parallel jobs than cores (e.g., -j 32 on 16 cores)
  2. CPU-bound tasks: Use parallel jobs = cores / threads_per_task
  3. Mixed workloads: Start long jobs first, short jobs last:
    # Sort by file size (largest first)
ls -S *.fastq.gz | parallel -j 8 process_sample

See also:

Originally posted on WeChat Official Account: bioinfo_studyroom

Parallel Processing