Differences between revisions 1 and 2
Revision 1 as of 2018-06-18 13:09:13
Size: 2381
Editor: davidsch
Comment:
Revision 2 as of 2018-06-18 15:01:09
Size: 2662
Editor: davidsch
Comment:
Deletions are marked like this. Additions are marked like this.
Line 24: Line 24:
 * Do not store small files in the archive, especially avoid storing many of such small files.
 * The ideal size of these archive files is between 5 and 10 Gigabytes.
 * Don't create archive files bigger than 20 Gigabytes. (???)
 * Do not store small files in the archive, especially avoid storing many (from thousands to millions) of such small files.
Line 30: Line 28:
==== Usage examples ====
Line 31: Line 30:
 * Create new file tarball on the archive system: {{{
tar cvfz /itet-stor/USERNAME/lagoon_backup/phd_thesis_files.tar.gz /phd/thesis/directory
}}}
 * Or use zip, if you like it: {{{
zip -r /itet-stor/USERNAME/lagoon_backup/phd_thesis_file.zip /phd/thesis/directory
}}}
 * It is no more necessary to split large archives (e.g. 50 GB+) into smaller chunks.

Data archiving on lagoon

Overwiew

The archive system is thought to be used for long-term storage of privte data (e.g. home data). The archive system file space is not a working place, e.g. like a standard user home, where you operate on the files, create, read, write and delete them frequently. Instead the archive space should be used as a "quiet" location where you keep backup copies or data which is likekly not to be used for a longer period of time. A typical use of the archive system might be storing larger output data of scientific applications which is no more frequently used or modified after a certain deadline/ project end, but you think parts of the data may be reused one day in the future.

Usage

Obtaining access to the archive

The archive is not automatically accessible for every user. Instead, access must be obtained from ISG.EE (write an e-mail request to support@ee.ethz.ch). All D-ITET users are allowed to obtain an archive access (excluding students). The first 5 GB per user are free of cost.

How to access the archive

Once access has been granted by ISG.EE, you may access the archive in the ways shown below:

  • NFS (preferred on Linux): /itet-stor/USERNAME/lagoon_backup or /usr/beegfs01/archive/USERNAME

  • Samba or MS Windows Network Map: \\itet-stor\USERNAME\lagoon_backup or S:\lagoon_backup

Notes, read carefully

Please follow these rules when storing data in the lagoon archive:

  • Summarize small files (e.g. only a few kB per file) in to larger archives (e.g. zip-archives, tarballs (.tar.gz) etc.)
  • Do not store small files in the archive, especially avoid storing many (from thousands to millions) of such small files.
  • Take care when you delete files. You are working on the backup system and there is no further backup of that data.
  • The quota applied to the files in the archive is shared among all users of the same laboratory. If one user eats up all the free space, other users won't have enough space to add their data to the archive. If you think the amount of data you plan to add to the archive is especially large, please first contact your local administrator or ISG.EE.

Usage examples

  • Create new file tarball on the archive system:

    tar cvfz /itet-stor/USERNAME/lagoon_backup/phd_thesis_files.tar.gz /phd/thesis/directory
  • Or use zip, if you like it:

    zip -r /itet-stor/USERNAME/lagoon_backup/phd_thesis_file.zip /phd/thesis/directory
  • It is no more necessary to split large archives (e.g. 50 GB+) into smaller chunks.

CategoryBKUP

Services/DataArchivingNG (last edited 2023-10-16 11:33:32 by alders)