Differences between revisions 39 and 58 (spanning 19 versions)

Data Archiving on Jabba

Contents

Data Archiving on Jabba

About our data archiving system Jabba

All data you put on our data archiving system will we stored on tape drives. Like this the archive system can hold a large amount of data. The downside is that read and write operations are slow since the tape roboter has to physically grab the right tapes to handle your data. Thus, you should not use Jabba for data you use daily. But if you have a huge amount of simulation data that you might need in some weeks, Jabba would be a good place to take care of them.

Jabba Usage

Access rights

We will have to give the proper rights to your account before you will be able to use Jabba. Please write an email to support@ee.ethz.ch if you need access to our data archiving system.

How to access Jabba

As soon as access is granted, you have the following options to get/put data from/to the archiving system:

NFS (preferred on Linux): /usr/jabba/archive/INSTITUTE/USERNAME or /usr/jabba/backup/INSTITUTE/USERNAME
Samba: \\jabba\USERNAME
OpenSSH: scp

Important notes

Please follow these rules when storing data on Jabba:

Only write big archive files (.tar.gz, .zip, ...) to jabba, not many small files.
The ideal size of these archive files is between 5 and 10 Gigabytes.
Don't create archive files bigger than 10 Gigabytes.
Take care when you delete files. You are working on the backup system and there is no further backup of these data.
Only put data into the archive or backup folder.

If you don't obey these rules, backing up and restoring your data will take very, very long.

Differences between storage areas

archive : Files within the archive directory will be stored on two tapes, five minutes after the last modification.
backup : Files within the backup directory will be saved to one tape, 30 minutes after their last modification on harddisk.

In both cases the cached files on harddisk will be deleted after storage on tape only if the threshold of free disk space on the partition is reached.

Some usage examples

Backup Pictures folder via NFS:

 tar cfv - Pictures | /usr/isgtc/bin/tubasplit --size 5056 /usr/jabba/backup/tik/hans/Pictures.tar

Archive Pictures folder with ssh:
```
 tar cfv - Pictures | ssh jabba "/usr/isgtc/bin/tubasplit --size 5056  /usr/jabba/archive/tik/hans/Pictures.tar
```
Note: The usage through NFS/Samba is preferred since above command creates excessive CPU load on the archive system.

Get two tar archives via NFS and unpack them in scratch:

 cd /scratch/hans
 cat  /usr/archive/biwi/hans/Pictures_1.tar /usr/archive/biwi/hans/Pictures_2.tar | tar xvf -

Advanced usage

The data transfer between harddisk and tape storage can be influenced with the commands below. You have to be logged in on jabbe through SSH to use these commands. Please consult the man pages for further details.

Command	Description
`sls`	An extended version of GNU ls for listing directory information
`sdu`	An extended version of GNU du
`archive`	Set archive attributes and archive files
`release`	Release disk space and set release attributes
`stage`	Set staging attributes and copy off-line files to harddisk
`ssum`	Set file checksum attributes

-  ⇤ ← Revision 39 as of 2010-03-08 07:32:09 → 
  Size: 3557
  Editor: hgiger
  Comment:
+   ← Revision 58 as of 2012-03-23 11:27:03 → ⇥
  Size: 3541
  Editor: alders
  Comment:
-Deletions are marked like this.
+Additions are marked like this.
 Line 6:
+== About our data archiving system Jabba ==
All data you put on our data archiving system will we stored on tape drives. Like this the archive system can hold a large amount of data. The downside is that read and write operations are slow since the tape roboter has to physically grab the right tapes to handle your data. Thus, you should not use Jabba for data you use daily. But if you have a huge amount of simulation data that you might need in some weeks, Jabba would be a good place to take care of them.
-Line 8:
+Line 11:
-For using the system, you must have an account on Jabba. Contact your system manager and ask him for such an account. If you have an account, Jabba can be accessed via NFS:
{{{
/usr/jabba/backup/<institute>/<username> resp.
/usr/jabba/archive/<institute>/<username>
}}}
-Line 14:
+Line 12:
-Please note the following:
 * Jabba is a huge file system and should not be used as a one-one copy of your files. Please use tar or zip to store your backups and archives.
 * Files should not be bigger then 5 to 10 GB. (Under Windows you can use winzip to split the files.)
+==== Access rights ====
We will have to give the proper rights to your account before you will be able to use Jabba. Please write an email to support@ee.ethz.ch if you need access to our data archiving system.
-Line 18:
+Line 15:
-To backup/archive your files via NFS, do the following:
{{{
tar cfv - <Directory> | /usr/isgtc/bin/tubasplit --size 5056 /usr/jabba/backup/tik/hans/file.tar
}}}
+==== How to access Jabba ====
As soon as access is granted, you have the following options to get/put data from/to the archiving system:
-Line 23:
+Line 18:
-To archive/backup your files, you can use the following alternatives:
 * Via Samba ('''\\jabba\[username]''')
 * By remote-copy ('''scp''')
 * Via ftp (not recommended)
 * and via ssh
+ * NFS (preferred on Linux): `/usr/jabba/archive/INSTITUTE/USERNAME` or `/usr/jabba/backup/INSTITUTE/USERNAME`
 * Samba: `\\jabba\USERNAME`
 * OpenSSH: `scp`
-Line 29:
+Line 22:
-Via ssh, archiving your files can done the following way:
<<BR>>
'''Note:''' If you have an official jabba-account you should prefer the NFS-way. Because the ssh-way causes to much load on 'jabba'!
{{{
tar cfv - [Directory] | ssh jabba "/usr/isgtc/bin/tubasplit --size 5056  /usr/jabba/backup/tik/hans/file.tar"
}}}
<<BR>>
<<BR>>
To get the files back from the archive, change to the directory where the archived files should be dropped (e.g. /scratch/<username>) and do the following:
{{{
$ cat  </usr/archive/biwi/<username>/tar-a> </usr/archive/biwi/<usernam>/tar-b>  </usr/archive/biwi/<username>/tar-c> | tar xvf -
}}}
== Two different storage areas are provided ==
* '''archive''' : Every file you put into the archive directory (partition) will be stored twice on two tapes, 5 minutes after the last modification.
+==== Important notes ====
Please follow these rules when storing data on Jabba:
-Line 44:
+Line 25:
-* '''backup''' : Files write into the backup directory (partition) will only be saved to one tape and 30 minutes after their last modification on harddisk.
+ * '''Only write big archive files (`.tar.gz, .zip, ...`) to jabba, not many small files'''.
 * The ideal size of these archive files is between 5 and 10 Gigabytes.
 * Don't create archive files bigger than 10 Gigabytes.
 * Take care when you delete files. You are working on the backup system and there is no further backup of these data.
 * Only put data into the `archive` or `backup` folder.
-Line 46:
+Line 31:
-In both cases the files on harddisk will be deleted (after storage on tape) only if the threshold of free disk space on the partition is reached. The idea of this two methods of storage is, that you can define how much security you need for your data by choosing the name of the directory. There are several methods to transport the files to the jabba. You can use '''ftp''', '''ssh''' (secure shell) or '''scp''' (secure copy) or your system manager can mount the directories on jabba using NFS on your own machine. If you want retrieve your files, its only necessary that you call it in the appropriate directory. If the inode leaves on the harddisk (even when the file is stored on tape and deleted on harddisk), the filename leave in the directory and will recopied automatically from tape to the harddisk when you call it.
+If you don't obey these rules, backing up and restoring your data will take ''very, very'' long.
-Line 48:
+Line 33:
-== User commands ==
The data transfer between harddisk and tape storage can be influenced with the following commands:
+==== Differences between storage areas ====
-Line 51:
+Line 35:
-'''-sls  '''''an extended version of GNU ls for listing directory information''
+ * '''archive''' : Files within the archive directory will be stored on '''two tapes''', five minutes after the last modification.
 * '''backup''' : Files within the backup directory will be saved to '''one tape''', 30 minutes after their last modification on harddisk.
-Line 53:
+Line 38:
-'''-sdu   '''''an extended version of GNU du''
+In both cases the cached files on harddisk will be deleted after storage on tape only if the threshold of free disk space on the partition is reached.
-Line 55:
+Line 40:
-'''-archive '''''set archive attributes and archive files''
-Line 57:
+Line 41:
-'''-release  '''''release disk space and set release attributes''
+== Some usage examples ==
-Line 59:
+Line 43:
-'''-stage  '''set staging attributes and copy off-line files to harddisk
+ * Backup `Pictures` folder via NFS:
 {{{
 tar cfv - Pictures | /usr/isgtc/bin/tubasplit --size 5056 /usr/jabba/backup/tik/hans/Pictures.tar
 }}}
-Line 61:
+Line 48:
-'''-ssum   '''''set file checksum attributes''
+ * Archive `Pictures` folder with ssh:
 {{{
 tar cfv - Pictures | ssh jabba "/usr/isgtc/bin/tubasplit --size 5056  /usr/jabba/archive/tik/hans/Pictures.tar
 }}}
 '''Note:''' The usage through NFS/Samba is preferred since above command creates excessive CPU load on the archive system.
-Line 63:
+Line 54:
-Note: You must be logged in on Jabba to use these commands. Certain options are only available to the super user. Please consult the man pages on Jabba for more detailed information.
+ * Get two tar archives via NFS and unpack them in scratch:
 {{{
 cd /scratch/hans
 cat  /usr/archive/biwi/hans/Pictures_1.tar /usr/archive/biwi/hans/Pictures_2.tar | tar xvf -
 }}}


== Advanced usage ==
The data transfer between harddisk and tape storage can be influenced with the commands below. You have to be logged in on jabbe through SSH to use these commands. Please consult the man pages for further details.

 || ''Command'' || ''Description'' ||
 || `sls` || An extended version of GNU ls for listing directory information ||
 || `sdu` || An extended version of GNU du ||
 || `archive` || Set archive attributes and archive files ||
 || `release` || Release disk space and set release attributes ||
 || `stage` || Set staging attributes and copy off-line files to harddisk ||
 || `ssum` || Set file checksum attributes ||

Wiki

Page