Data Management
Choosing the optimal storage system
Quoting Choosing the optimal storage system from the scientific computing wiki:
When working on an HPC cluster that provides different storage categories/systems, the choice of which system to use can have a big influence of the performance of your workflow. In the best case you can speedup your workflow by quite a lot, whereas in the worst case the system administrator has to kill all your jobs and has to limit the number of concurrent jobs that you can run because your jobs slow down the entire storage system and this can affect other users jobs.
The same holds for the clusters and storage infrastructure as maintained in the D-ITET environment.
When thus working (in particular from compute jobs in one of the clusters) with storage systems please consider the following guidelines, those are largely inspired by the mentioned scientific computing guidelines.
- Use local "scratch" disks as whenever possible. Many of the nodes (but not all) have in meanwhile in particular SSD disks available as scratch storage further improving the performance.
[Applicable to BIWI] For working in parallel from a cluster with large files consider using the scale-out scratch place (beegfs02).
Do not create large number of small files in the D-ITET NetScratch service storage (or the BeeGFS scratch or project filesystem), this can slow down not only you but the whole system. Consider the first item when working on data stored on those services whenever possible.
- If on the data you need to perform very high I/O (e.g. opening and closing files at high rate, reading many small files per second, do short appends to files from various locations), then this will have a severe impact o the network attached storages or the scale out filesystems. Use as much as possible data copied to local storage for those and only move back results to appropriate places.
- If working a lot with large amount of small files, then keep those sensibly grouped in bigger archives which you can move to local storage in the job (as big files) and unpack those there and work on the local storage with them. Do not work on the large amount of small files on network attached storage or the cluster filesystems. Process them and group the results again in archives and move the results to appropriate places.
Respecting this guidelines can improve your own work performance and at same time do not severely impact the performance of the storage systems in a bad way (and for other users).