Back to All Events
Presenter: Ramses van Zon, SciNet
Many research computing projects involve workloads of large amounts of relatively short, independent computations. Such workloads are a common reason for researchers to move to shared resources. Despite being called an "embarrasingly parallel" problem, orchestrating large amounts of small computational jobs is surprisingly subtle, particularly when dealing with shared resources with their own usage policies and implementations. To deal with this, many tools have been created, used for a while, and then abandoned. Here, we will focus on a tool for job orchestration that is very versatile and actively maintained, GNU Parallel, and show how it helps for a number of use cases.