Large Volume SLS/SwissFEL Data Transfer
PSI accountTo use the data transfer service you need to have a PSI account. If you don't have one, follow the procedure .
Authorisation to use the data transfer serviceContact your beamline manager or IT responsible (SwissFEL/Photonics group member only) with the following information
- your PSI account
- data identifier (Proposal ID or e-account used to collect the data) for the data you need to access
Firewall ConfigurationIn case you have internal firewall, access to the following IP's should be allowed for the outbound connections, TCP port 22:
Special Case: WinSCP clientIf you use a WinSCP client, contact service administrators.
Data accessible via the data transfer serviceThe following directories are accessible with the data transfer service:
|/sls||/sls/MX/Data10/e15874||yes||Raw data from SLS|
|/das||/das/work/p15/p15874||Working area of the Ra cluster, please note a structure of subdirectories: pAB/pABCDEF|
|/swissfel||/swissfel/photonics/data||Working directory of SwissFEL/Photonics group|
|e.g. to copy data from another facility,like LCLS|
How to transfer data
rsync/sshUse the generic hostname ra-export.psi.ch for the transfer.
The following examples use ext-name, e15874 and MX as PSI account, e-account and beamline name correspondingly.
Listing the contentExample:
$ ssh firstname.lastname@example.org "ls /sls/MX/Data10/e15874"or the recursive listing (might be slow)
$ ssh email@example.com "ls -R /sls/MX/Data10/e15874"
TransferTo copy directory 2016-05-23 to your computer (to the DESTINATION_DIRECTORY, like ./ ):
- with the rsync
$ rsync -av firstname.lastname@example.org:/sls/MX/Data10/e15874/2016-05-23 DESTINATION_DIRECTORY
- with the scp
scp -r email@example.com:/sls/MX/Data10/e15874/2016-05-23 DESTINATION_DIRECTORY
- if you have patched version of the SSH (HPN) (for the better network performance compared to standard ssh):
rsync -av -e "PATH_to_OpenSSH_HPN/bin/ssh -o TCPRcvBufPoll=yes" firstname.lastname@example.org:/sls/MX/Data10/e15874/2016-05-23 DESTINATION_DIRECTORY ssh -t PATH_to_OpenSSH_HPN/bin/scp -oTcpRcvBufPoll=no -oNoneSwitch=yes -oNoneEnabled=yes email@example.com:/sls/MX/Data10/e15874 DESTINATION_DIRECTORY
SSH keys (recommended option)To avoid typing password for every connection to the data transfer server, upload your ssh-key to the server machine (this command will overwrite existing authorized_keys file):
scp .ssh/id_rsa.pub firstname.lastname@example.org:.ssh/authorized_keysIf you don't have ssh key, you may generate it (example for linux machine):
- generate a key-pair:
ssh-keygen -C "PSI data export" -f $HOME/.ssh/datatransfer(You'll be asked for the password, invent one, don't leave it empty, you'll use this password in next step)
- copy the public key to the data transfer server
scp .ssh/datatransfer.pub email@example.com:.ssh/authorized_keys
- add your key to the ssh-agent:
ssh-add .ssh/datatransfer(here you'll need to provide password you invented at first step)
GlobusOnlineGlobusOnline is a web service, which allows one to use the GridFTP file transfer protocol in an easy and intuitive way. This protocol is well suited for transfers of big files (>100 MB).
GlobusOnline has a number of attractive features:
- Automatic network optimization
- Parallel/multistream transfers (up to 4 transfers/streams)
- Automatic retry in case of failure
- Online task monitor
- Summary email sent at the end of the transfer
- Usually is firewall safe, as it uses only outgoing connections. If your firewall blocks also outgoing connections, then you need some special rules to be set up, contact your local IT support
- A GlobusID account(free), or an account recognized by GlobusOnline (like Google, XSEDE, US Universities), see a very detailed description
- GlobusConnect client installed (available for Win, Mac, Linux) link or the GlobusOnline endpoint in your organisation
For the comprehensive description of the GlobusOnline GUI, please visit this page