|
Size: 3245
Comment:
|
Size: 4161
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 6: | Line 6: |
| == About our data archiving system Jabba == All data you put on our data archiving system will we stored on tape drives. Like this the archive system can hold a large amount of data. The downside is that read and write operations are slow since the tape roboter has to physically grab the right tapes to handle your data. Thus, you should not use Jabba for data you use daily. But if you have a huge amount of simulation data that you might need in some weeks, Jabba would be a good place to take care of them. |
|
| Line 8: | Line 11: |
| For using the system, you must have an account on Jabba. Contact your system manager and ask him for such an account. If you have an account, Jabba can be accessed via NFS: {{{ /usr/jabba/backup/<institute>/<username> resp. /usr/jabba/archive/<institute>/<username> }}} |
We will have to give the proper rights to your account before you will be able to use Jabba. Please write an email to support@ee.ethz.ch if you need access to our data archiving system. |
| Line 14: | Line 13: |
| Please note the following: * Jabba is a huge file system and should not be used as a one-one copy of your files. Please use tar or zip to store your backups and archives. * Files should not be bigger then 5 to 10 GB. (Under Windows you can use winzip to split the files.) |
As soon as access is granted, you have the following options to get/put data from/to the archiving system: * NFS (Linux only): `/usr/jabba/archive/INSTITUTE/USERNAME` or `/usr/jabba/backup/INSTITUTE/USERNAME` * Samba: `\\jabba\USERNAME` * OpenSSH: `scp` ==== Important note ==== Since data you put on Jabba are stored on tape drives, please follow these rules: * '''Only write big archive files''' (`.tar.gz, .zip, ...`) to jabba, not many small files'''. * The ideal size of these archive files is between 5 and 10 Gigabytes. * Don't create archive files bigger than 10 Gigabytes. If you don't obey these rules, backing up and restoring your data will take ''very, very'' long. |
| Line 23: | Line 33: |
| To archive/backup your files, you can use the following alternatives: * Via Samba ('''\\jabba\[username]''') * By remote-copy ('''scp''') * Via ftp (not recommended) * and via ssh Via ssh, archiving your files can done the following way: |
To archive your files utilizing ssh: |
| Line 31: | Line 35: |
| '''Note:''' If you have an official jabba-account you should prefer the NFS-way. Because the ssh-way causes to much load on 'jabba'! | '''Note:''' If you have a local jabba-account you should preferably use the NFS-mount. Using ssh creates excessive load on 'jabba'! |
| Line 35: | Line 39: |
| <<BR>> <<BR>> To retreive your data from jabba, change to the directory where the archived files need to end up (e.g. /scratch/<username>): {{{ $ cat </usr/archive/biwi/<username>/tar-a> </usr/archive/biwi/<usernam>/tar-b> </usr/archive/biwi/<username>/tar-c> | tar xvf - }}} |
|
| Line 37: | Line 46: |
| * '''archive''' : Every file you put into the archive directory (partition) will be stored twice on two tapes, 5 minutes after the last modification. | * '''archive''' : Every file you put into the archive directory (partition) will be stored on two tapes, 5 minutes after the last modification. |
Data Archiving on Jabba
Contents
About our data archiving system Jabba
All data you put on our data archiving system will we stored on tape drives. Like this the archive system can hold a large amount of data. The downside is that read and write operations are slow since the tape roboter has to physically grab the right tapes to handle your data. Thus, you should not use Jabba for data you use daily. But if you have a huge amount of simulation data that you might need in some weeks, Jabba would be a good place to take care of them.
Jabba Usage
We will have to give the proper rights to your account before you will be able to use Jabba. Please write an email to support@ee.ethz.ch if you need access to our data archiving system.
As soon as access is granted, you have the following options to get/put data from/to the archiving system:
NFS (Linux only): /usr/jabba/archive/INSTITUTE/USERNAME or /usr/jabba/backup/INSTITUTE/USERNAME
Samba: \\jabba\USERNAME
OpenSSH: scp
Important note
Since data you put on Jabba are stored on tape drives, please follow these rules:
Only write big archive files (.tar.gz, .zip, ...) to jabba, not many small files.
- The ideal size of these archive files is between 5 and 10 Gigabytes.
- Don't create archive files bigger than 10 Gigabytes.
If you don't obey these rules, backing up and restoring your data will take very, very long.
To backup/archive your files via NFS, do the following:
tar cfv - <Directory> | /usr/isgtc/bin/tubasplit --size 5056 /usr/jabba/backup/tik/hans/file.tar
To archive your files utilizing ssh:
* * In both cases the files on harddisk will be deleted (after storage on tape) only if the threshold of free disk space on the partition is reached. The idea of this two methods of storage is, that you can define how much security you need for your data by choosing the name of the directory. There are several methods to transport the files to the jabba. You can use
The data transfer between harddisk and tape storage can be influenced with the following commands: Note: You must be logged in on Jabba to use these commands. Certain options are only available to the super user. Please consult the man pages on Jabba for more detailed information.
Note: If you have a local jabba-account you should preferably use the NFS-mount. Using ssh creates excessive load on 'jabba'! tar cfv - [Directory] | ssh jabba "/usr/isgtc/bin/tubasplit --size 5056 /usr/jabba/backup/tik/hans/file.tar"
To retreive your data from jabba, change to the directory where the archived files need to end up (e.g. /scratch/<username>): $ cat </usr/archive/biwi/<username>/tar-a> </usr/archive/biwi/<usernam>/tar-b> </usr/archive/biwi/<username>/tar-c> | tar xvf -
Two different storage areas are provided
User commands