|
Size: 3664
Comment:
|
← Revision 81 as of 2020-09-08 11:56:17 ⇥
Size: 192
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| ## page was renamed from DataArchiving = Data Archiving on Jabba = |
#rev 2020-09-03 davidsch #rev 2018-11-19 davidsch #redirect Services/DataArchivingNG This article has been moved to [[Services/DataArchivingNG|DataArchivingNG]]. |
| Line 4: | Line 6: |
| <<TableOfContents(3)>> == About our data archiving system Jabba == All data you put on our data archiving system will we stored on tape drives. Like this the archive system can hold a large amount of data. The downside is that read and write operations are slow since the tape roboter has to physically grab the right tapes to handle your data. Thus, you should not use Jabba for data you use daily. But if you have a huge amount of simulation data that you might need in some weeks, Jabba would be a good place to take care of them. == Jabba Usage == ==== Access rights ==== We will have to give the proper rights to your account before you will be able to use Jabba. Please write an email to support@ee.ethz.ch if you need access to our data archiving system. ==== How to access Jabba ==== As soon as access is granted, you have the following options to get/put data from/to the archiving system: * NFS (preferred on Linux): `/usr/jabba/archive/INSTITUTE/USERNAME` or `/usr/jabba/backup/INSTITUTE/USERNAME` * Samba: `\\jabba\USERNAME` * OpenSSH: `scp` ==== Important notes ==== Please follow these rules when storing data on Jabba: * '''Only write big archive files (`.tar.gz, .zip, ...`) to jabba, not many small files'''. * The ideal size of these archive files is between 5 and 10 Gigabytes. * Don't create archive files bigger than 10 Gigabytes. * Take care when you delete files. You are working on the backup system and there is no further backup of these data. * Only put data into the `archive` or `backup` folder. If you don't obey these rules, backing up and restoring your data will take ''very, very'' long. ==== Differences between storage areas ==== * '''archive''' : Files within the archive directory will be stored on '''two tapes''', five minutes after the last modification. * '''backup''' : Files within the backup directory will be saved to '''one tape''', 30 minutes after their last modification on harddisk. In both cases the cached files on harddisk will be deleted after storage on tape only if the threshold of free disk space on the partition is reached. == Some usage examples == * Backup `Pictures` folder via NFS: {{{ tar cfv - Pictures | /usr/isgtc/bin/tubasplit --size 5056 /usr/jabba/backup/tik/hans/Pictures.tar }}} * Archive `Pictures` folder with ssh: {{{ tar cfv - Pictures | ssh jabba "/usr/isgtc/bin/tubasplit --size 5056 /usr/jabba/archive/tik/hans/Pictures.tar }}} '''Note:''' The usage through NFS/Samba is preferred since above command creates excessive CPU load on the archive system. * To retreive your data from jabba, change to the directory where the archived files need to end up (e.g. /scratch/<username>): {{{ cat </usr/archive/biwi/<username>/tar-a> </usr/archive/biwi/<usernam>/tar-b> </usr/archive/biwi/<username>/tar-c> | tar xvf - }}} == User commands == The data transfer between harddisk and tape storage can be influenced with the following commands: '''-sls '''''an extended version of GNU ls for listing directory information'' '''-sdu '''''an extended version of GNU du'' '''-archive '''''set archive attributes and archive files'' '''-release '''''release disk space and set release attributes'' '''-stage '''set staging attributes and copy off-line files to harddisk '''-ssum '''''set file checksum attributes'' Note: You must be logged in on Jabba to use these commands. Certain options are only available to the super user. Please consult the man pages on Jabba for more detailed information. |
---- [[CategoryBKUP]] |