No recording available.

Presenter: Ching-Hsing Yu, SciNet

Managing large amounts of data can be a challenging task. Processing large numbers of files incurs heavy overhead of I/O communications. In this talk, we will explore several options such as using Apptainer Overlay and SQLite to pack and reduce a large number of files to few files, and hence, improving IO performance. Python scripts will be used throughout the presentation.