HIBU — New User Guide

Cluster Details

HIBU is a high-performance computing (HPC) system consisting of four compute nodes organized into two partitions. Job scheduling is managed by the Slurm scheduler. A dedicated head (login) node allows users to prepare scripts, submit and cancel jobs, and transfer data. HIBU uses a NAS with a total capacity of up to 42 TB.

• Default partition: long
• Available partitions: long, short, smallhm
• Filesystems: HOME /home/{USER}/ · SCRATCH /data/{USER}/
• Data transfer: scp, rsync, FTP, wget, curl

Partitions

1. Access & First Login

New account setup

1. Request an account by emailing hpc-support@imbb.forth.gr. Include your affiliation, intended workloads, and software needs.
2. Set up the IMBB VPN — required to access HIBU from outside the IMBB/FORTH network.
3. Download Google Authenticator on your smartphone and scan the QR code sent by the administrator.

Windows users

1. Download PuTTY on your personal computer.
2. Connect to the IMBB VPN (when working outside the office).
3. Enter your username, the correct hostname/IP (hibu.imbb.gr or 139.91.75.141), and port in PuTTY.
4. When connected, enter your password and the verification code from Google Authenticator.

Linux / macOS users

Open a terminal and connect via SSH:

ssh my-username@139.91.75.141

When connected you will be prompted for your password and a one-time code from Google Authenticator.

2. Filesystems & Data Storage

• $HOME — /home/{USER}: small quota, for configuration files and scripts only. Not backed up.
• SCRATCH — /data/{USER}: fast, large (42 TB total). Use this for all analysis input/output. Not backed up.

Data transfers

scp   local_file.fastq.gz  username@139.91.75.141:/data/username/

rsync -avP local_data/  username@139.91.75.141:/data/username/data/

wget  https://example.org/reference_genome.fa -P /data/username/refs/

3. SLURM Job Scheduler

3.2 Core commands

• sinfo — list the current status of all partitions and nodes.
• squeue — list all currently running and queued jobs.
• sbatch <script> — submit a job script to the queue.
• scancel <jobid> — cancel a running or queued job. Use squeue to find your job ID.
• sacct — retrieve accounting information for completed jobs.
• scontrol show job <jobid> — inspect a specific job's details.

Common #SBATCH arguments:

• --partition=long — target queue (long / short / smallhm)
• --nodes=1, --ntasks=1, --cpus-per-task=4 — CPU allocation
• --mem=8G — memory per node; or --mem-per-cpu=4G
• --time=24:00:00 — wall-clock time limit (HH:MM:SS)
• --array=1-100%10 — job array (100 tasks, max 10 concurrent)
• -o slurm-%j.out — stdout file (%j = job ID)

3.3 Basic job types

A) Single node — serial or multicore

Request only the cores and memory you need, leaving the rest available to other users.

#!/bin/bash
#SBATCH --job-name="hello"
#SBATCH --partition=long
#SBATCH --nodes=1
#SBATCH --cpus-per-task=4   # 4 cores out of 24
#SBATCH --mem=8G           # 8 GB out of 256 GB
#SBATCH --time=12:00:00
#SBATCH -o slurm-%j.out

python myscript.py \
  --input  /data/${USER}/data/input.dat \
  --output /data/${USER}/results/out.txt

B) Job array — many similar tasks in parallel

Ideal when you can split input into independent chunks processed across nodes.

#!/bin/bash
#SBATCH --job-name="array_job"
#SBATCH --partition=long
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=2G
#SBATCH --array=1-10       # 10 tasks
#SBATCH -o logs/array_%A_%a.out

PARAMS=$(sed -n "${SLURM_ARRAY_TASK_ID}p" params.txt)

python simulate.py $PARAMS \
  --seed ${SLURM_ARRAY_TASK_ID} \
  --out  results/${SLURM_ARRAY_TASK_ID}.json

C) Job dependencies — run B after A finishes successfully

# Submit first job and capture its ID
jid1=$(sbatch jobA.slurm | awk '{print $4}')

# Submit second job only if jobA exits cleanly
sbatch --dependency=afterok:${jid1} jobB.slurm

3.4 Requesting the right resources

• Time: be realistic — shorter jobs queue faster. Use previous sacct records to calibrate.
• CPUs: for pure Python/R without native multi-threading, 1–2 cores are often enough. Use --cpus-per-task for OpenMP, --ntasks for MPI.
• Memory: start modestly; inspect MaxRSS from sacct after a test run and adjust.
• Single-node shared memory: -N 1 --cpus-per-task=X.
• Multi-node MPI: -N N --ntasks-per-node=T.

3.5 Monitoring & Logs

# Cluster & queue status
sinfo
squeue -u $USER

# Inspect a specific job
scontrol show job <jobid>

# Accounting history for a job
sacct -j <jobid> \
  --format=JobID,JobName%20,Partition,State,Elapsed, \
           Timelimit,AllocTRES%30,ReqMem,MaxRSS,ExitCode

Stdout and stderr go to slurm-%j.out by default; customise with -o and -e in your script header.

Best Practices & Etiquette

• Develop small, run big: always test with small inputs before submitting large jobs.
• Use SCRATCH for I/O: read input from and write output to /data/{USER}/, never from $HOME.
• Keep environments portable: activate conda environments or load modules inside your job script.
• Clean up SCRATCH: copy important results to your local storage and delete intermediate files.
• One process per CPU unless threaded: do not oversubscribe cores.
• Version control your scripts: keep job scripts in a personal Git repository; include --version outputs in logs.
• Implement checkpointing for long runs so you can resume from the last checkpoint if a job is interrupted.
• Cite the cluster in publications (ask the support team for the citation text).

Troubleshooting

• Job stuck in pending (PD): check squeue -u $USER — the REASON column indicates the cause (e.g., Resources = waiting for resources, Priority = other jobs queued ahead).
• Job fails immediately: check slurm-<jobid>.out for error messages. Common causes: wrong paths, missing modules, exceeded memory.
• Out of memory: inspect MaxRSS with sacct and increase --mem in your script.
• Cannot connect via SSH: ensure you are on the IMBB VPN, then try ssh username@139.91.75.141.
• MFA issues: sync your device clock — Google Authenticator codes are time-sensitive.

Quick Reference (Cheat Sheet)

Job Templates

Copy and adapt the templates below. Remember to replace {USER} with your actual username and adjust resources to match your workload.

RNA-seq alignment (STAR, 16 cores)

#!/bin/bash
#SBATCH --job-name="star_align"
#SBATCH --partition=long
#SBATCH --nodes=1
#SBATCH --cpus-per-task=16
#SBATCH --mem=64G
#SBATCH --time=08:00:00
#SBATCH -o logs/star_%j.out

SAMPLE=$1
GENOME=/data/${USER}/refs/genome_index/

STAR --runThreadN 16 \
     --genomeDir    ${GENOME} \
     --readFilesIn  /data/${USER}/fastq/${SAMPLE}_R1.fq.gz \
                    /data/${USER}/fastq/${SAMPLE}_R2.fq.gz \
     --readFilesCommand zcat \
     --outSAMtype   BAM SortedByCoordinate \
     --outFileNamePrefix /data/${USER}/aligned/${SAMPLE}_

Generic Python script (4 cores, 16 GB)

#!/bin/bash
#SBATCH --job-name="py_analysis"
#SBATCH --partition=long
#SBATCH --nodes=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --time=06:00:00
#SBATCH -o slurm-%j.out

conda activate myenv

python myscript.py \
  --input  /data/${USER}/data/input.csv \
  --output /data/${USER}/results/