Large Volume SLS/SwissFEL Data Transfer
This service targets users that need to export big volumes of data from experimental stations at SLS and SwissFEL.
To use the data transfer service you need to have a PSI account. If you don't have one, follow the procedure .
Authorisation to use the data transfer service
Contact your beamline manager or IT responsible (SwissFEL/Photonics group member only) with the following information
- your PSI account
- data identifier (Proposal ID or e-account used to collect the data) for the data you need to access
In case you have internal firewall, access to the following IP's should be allowed for the outbound connections, TCP port 22:
Data accessible via the data transfer service
The following directories are accessible with the data transfer service:
|/sls||/sls/MX/Data10/e15874||yes||Raw data from SLS|
|/das||/das/work/p15/p15874||Working area of the Ra cluster, please note a structure of subdirectories: pAB/pABCDEF|
|/swissfel||/swissfel/photonics/data||Working directory of SwissFEL/Photonics group|
|e.g. to copy data from another facility,like LCLS|
How to transfer data
Use the generic hostname ra-export.psi.ch for the transfer.
The following examples use ext-name, e15874 and MX as PSI account, e-account and beamline name correspondingly.
Listing the content
$ ssh firstname.lastname@example.org "ls /sls/MX/Data10/e15874"
or the recursive listing (might be slow)
$ ssh email@example.com "ls -R /sls/MX/Data10/e15874"
To copy directory "data_exchange/processing_output" to your computer (to the DESTINATION_DIRECTORY, like ./ ):
- with the rsync
$ rsync -av firstname.lastname@example.org:/sls/MX/Data10/e15874/data_exchange/processing_output DESTINATION_DIRECTORY
- with the scp
scp -r email@example.com:/sls/MX/Data10/e15874/data_exchange/processing_output DESTINATION_DIRECTORY
SSH keys (recommended option)
To avoid typing password for every connection to the data transfer server, upload your ssh-key to the server machine (this command will overwrite existing authorized_keys file):
scp .ssh/id_rsa.pub firstname.lastname@example.org:.ssh/authorized_keys
If you don't have ssh key, you may generate it (example for linux machine):
- generate a key-pair:
ssh-keygen -C "PSI data export" -f $HOME/.ssh/datatransfer(You'll be asked for the password, invent one, don't leave it empty, you'll use this password in next step)
- copy the public key to the data transfer server
scp .ssh/datatransfer.pub email@example.com:.ssh/authorized_keys
- add your key to the ssh-agent:
ssh-add .ssh/datatransfer(here you'll need to provide password you invented at first step)
Please note that DSA keys is considered insecure and disabled in the most recent ssh version. ssh-keygen by default will generate RSA key, which one can steer with the -t option.
GlobusOnline is a web service, which allows one to use the GridFTP file transfer protocol in an easy and intuitive way. This protocol is well suited for transfers of big files (>100 MB).
GlobusOnline has a number of attractive features:
- Automatic network optimization
- Parallel/multistream transfers (up to 4 transfers/streams)
- Automatic retry in case of failure
- Online task monitor
- Summary email sent at the end of the transfer
- Usually is firewall safe, as it uses only outgoing connections. If your firewall blocks also outgoing connections, then you need some special rules to be set up, contact your local IT support
Requirements for using GlobusOnline:
- A GlobusID account (free), or an account recognized by GlobusOnline (like Google, XSEDE, US Universities), see a very detailed description (PSI staff may use psi account to login to GlobusOnline, "ext-" accounts can't be used to login to GlobusOnline)
- GlobusConnect Personal client installed (available for Win, Mac, Linux here) or the GlobusOnline endpoint in your organisation
- a web browser to use the Globus web app: https://app.globus.org
When you login to the GlobusOnline, choose endpoint "PSI" in the Transfer Page. When you are asked to authenticate, provide you PSI account and password (your credentials will not leave PSI computers, the authentication server is located at PSI). After successful login you should see a page like this:
In the transfer window on a right side, choose either your GlobusConnect endpoint (they look like username#endpoint, e.g. slac#lcls) or endpoint of your organisation.
For the comprehensive description of the GlobusOnline GUI, please visit this page