Environment variables

Environment variables are used to configure deployment-specific settings for processing jobs, like file paths and Slurm settings. These variables are set automatically when sourcing the environment scripts, but you can modify them to change settings like which Slurm partition a job is submitted to, or to configure a new environment.

Quick reference

• Sandbox
• Staging
• Production

CAQRN_ENV

Purpose: Identifies the active environment.

Usage: Used by Slurm job scripts to determine which Python virtual environment to activate at job execution time.

The following flowchart summarizes how the active environment’s associated Python virtual environment is reactivated on the compute node(s).

flowchart LR
    subgraph LN ["Login Node"]
        direction TB
        A1["Source Environment Script<br/>Set Environment Variables"] --> A2
        A2["Activate Python venv<br/>(current user environment)"] --> A3["Submit Slurm Job<br/>Export Environment Variables<br/>(via sbatch argument)"]
    end
    
    subgraph CN ["Compute Node(s)"]
        direction TB
        B1["Slurm Job Script Executes<br/>(processing_job.sh)"] --> B2["Use $CAQRN_ENV to Determine Job Context"]
        B2 --> B3["Activate Python venv<br/>(job execution environment)"] 
    end
    
    LN --> CN

CAQRN_CODE_ROOT

Purpose: Root directory containing project source code repositories.

Usage: Referenced by scripts to locate project modules and executables. Used in PYTHONPATH construction.

CAQRN_CONFIG_PATH

Purpose: Directory containing environment-specific configuration files.

Usage: Passed to processing jobs via --config parameter to load application configuration settings.

CAQRN_DATA_ROOT

Purpose: Root directory for all data storage including downloaded data directories, temporary workspaces for Slurm jobs, and output product directories.

Usage: Base path for constructing data directories. Contains the following subdirectories:

• incoming/ - Date-based directories created by download scripts
• processing/ - Temporary working directories for Slurm jobs
• products/ - Date-based output directories for Slurm jobs

Slurm jobs create date-based subdirectories in the format $CAQRN_DATA_ROOT/products/YYYY/MM/DD/ for product outputs.

CAQRN_LOG_ROOT

Purpose: Root directory for all log file storage.

Usage: Referenced by job submission scripts for Slurm log file output paths (--output and --error parameters). Used to construct paths to application logs at $CAQRN_LOG_ROOT/processing/.

CAQRN_INCOMING_DATA

Purpose: Directory containing downloaded sensor data.

Usage: Processing jobs read input files from date-based subdirectories under this path. By default, sandbox environments use the staging environment’s downloaded sensor data, shared at /projects/caqrn/shared/data/incoming.

CAQRN_TEMP_DIR

Purpose: Directory for temporary files created during processing.

Usage: Working directory for intermediate files created during processing.

CAQRN_SLURM_PARTITION

Purpose: Slurm partition for job submission.

Usage: Passed to sbatch --partition parameter. Scheduled production and staging jobs use the cluster’s general-purpose partition, while development jobs use the dedicated development partition.

CAQRN_SLURM_ACCOUNT

Purpose: Slurm account for job submission.

Usage: Passed to sbatch --account parameter. Separates development resource usage from production/staging.

CAQRN_JOB_PREFIX

Purpose: Prefix applied to Slurm job names.

Usage: Prepended to job names in submission scripts. Helps identify job source in queue listings and log files.