This course is organised in collaboration with MAX CoE, EuroCC Czechia and EuroCC Germany.

Annotation

Many scientific workflows consist of huge numbers of small tasks, which traditional HPC schedulers such as SLURM and PBS handle poorly. This tutorial, presented by the core developers of HyperQueue, introduces this open-source scheduler and meta-scheduler for large pipelines.

Starting from the basics, we show how to run millions of tasks on SLURM- or PBS-managed clusters. HyperQueue requests allocations automatically and distributes tasks into them with minimal overhead. It also works on systems without any batch scheduler. It installs as a single binary without admin rights and is already preinstalled on many supercomputers.

Participants will learn to submit and monitor tasks and to define complex resource requests for heterogeneous clusters. We conclude with best practices and tips drawn from our experience supporting real-world pipelines.

Benefits for the attendees, what will they learn:

  • Recognise when the “many small tasks” pattern overwhelms SLURM or PBS, and how a meta-scheduler solves it.

  • Install HyperQueue on their own machine or use an existing installation on a supercomputer.

  • Submit, monitor, and manage large numbers of tasks, including task dependencies.

  • Let HyperQueue request SLURM/PBS allocations automatically instead of submitting jobs by hand.

  • Define complex resource requests (e.g. GPUs, CPU counts, custom resources) and use heterogeneous clusters effectively.

  • Keep scheduling overhead low when scaling to millions of tasks.
  • Apply best practices and avoid common pitfalls, with tips from real-world pipelines shared by the HyperQueue developers.

 

Level

beginner, intermediate 

Prerequisites

No prior HyperQueue experience is required, only basic familiarity with the command line and HPC clusters.

Language

English

Tutor

Ada Böhm holds a Ph.D. in Computer Science and is a researcher at the IT4Innovations National Supercomputing Center, Czech Republic. She works on task scheduling and distributed computing for HPC and is a core author of HyperQueue. Her work also covers Rust systems programming, and infrastructure for quantum computing. She has a background in formal methods and automata theory.

Conference information

Date/Time

Starts

Ends

All times are in Europe/Budapest

Location

HYBRID
Zoom/IT4Innovations