Large Volume SLS/SwissFEL Data Transfer

This service targets users that need to export big volumes of data from experimental stations at SLS and SwissFEL.

Prerequisites

PSI account

To use the data transfer service you need to have a PSI account. If you don't have one, follow the procedure .

Authorisation to use the data transfer service

Contact your beamline manager or IT responsible (SwissFEL/Photonics group member only) with the following information
  • your PSI account
  • data identifier (Proposal ID or e-account used to collect the data) for the data you need to access

Firewall Configuration

In case you have internal firewall, access to the following IP's should be allowed for the outbound connections, TCP port 22:
  • 192.33.126.68
  • 192.33.126.69

Special Case: WinSCP client

If you use a WinSCP client, contact service administrators.

Data accessible via the data transfer service

The following directories are accessible with the data transfer service:
Directory Name Example Read-Only Comments
/sls /sls/MX/Data10/e15874 yes Raw data from SLS
/das /das/work/p15/p15874   Working area of the Ra cluster, please note a structure of subdirectories: pAB/pABCDEF
/swissfel /swissfel/photonics/data   Working directory of SwissFEL/Photonics group
e.g. to copy data from another facility,like LCLS

How to transfer data

rsync/ssh

Use the generic hostname ra-export.psi.ch for the transfer.

The following examples use ext-name, e15874 and MX as PSI account, e-account and beamline name correspondingly.

Listing the content

Example:
$ ssh ext-name@ra-export.psi.ch "ls /sls/MX/Data10/e15874" 
or the recursive listing (might be slow)
$ ssh ext-name@ra-export.psi.ch "ls -R /sls/MX/Data10/e15874" 

Transfer

To copy directory 2016-05-23 to your computer (to the DESTINATION_DIRECTORY, like ./ ):
  • with the rsync
$ rsync -av ext-name@ra-export.psi.ch:/sls/MX/Data10/e15874/2016-05-23 DESTINATION_DIRECTORY
  • with the scp
scp -r ext-name@ra-export.psi.ch:/sls/MX/Data10/e15874/2016-05-23 DESTINATION_DIRECTORY
  • if you have patched version of the SSH (HPN) (for the better network performance compared to standard ssh):
rsync -av -e "PATH_to_OpenSSH_HPN/bin/ssh -o TCPRcvBufPoll=yes" ext-name@ra-export.psi.ch:/sls/MX/Data10/e15874/2016-05-23 DESTINATION_DIRECTORY
ssh -t PATH_to_OpenSSH_HPN/bin/scp -oTcpRcvBufPoll=no -oNoneSwitch=yes -oNoneEnabled=yes ext-name@ra-export.psi.ch:/sls/MX/Data10/e15874 DESTINATION_DIRECTORY

SSH keys (recommended option)

To avoid typing password for every connection to the data transfer server, upload your ssh-key to the server machine (this command will overwrite existing authorized_keys file):
scp .ssh/id_rsa.pub ext-name@ra-export.psi.ch:.ssh/authorized_keys 
If you don't have ssh key, you may generate it (example for linux machine):
  • generate a key-pair:
    ssh-keygen -C "PSI data export" -f $HOME/.ssh/datatransfer
    (You'll be asked for the password, invent one, don't leave it empty, you'll use this password in next step)
  • copy the public key to the data transfer server
    scp .ssh/datatransfer.pub ext-name@ra-export.psi.ch:.ssh/authorized_keys
  • add your key to the ssh-agent:
    ssh-add .ssh/datatransfer
    (here you'll need to provide password you invented at first step)
Please note that DSA keys is considered insecure and disabled in the most recent ssh version. ssh-keygen by default will generate RSA key, which one can steer with the -t option.

GlobusOnline

GlobusOnline is a web service, which allows one to use the GridFTP file transfer protocol in an easy and intuitive way. This protocol is well suited for transfers of big files (>100 MB).

GlobusOnline has a number of attractive features:
  • Automatic network optimization
  • Parallel/multistream transfers (up to 4 transfers/streams)
  • Automatic retry in case of failure
  • Online task monitor
  • Summary email sent at the end of the transfer
  • Usually is firewall safe, as it uses only outgoing connections. If your firewall blocks also outgoing connections, then you need some special rules to be set up, contact your local IT support
Requirements for using GlobusOnline:
  • A GlobusID account(free), or an account recognized by GlobusOnline (like Google, XSEDE, US Universities), see a very detailed description
  • GlobusConnect client installed (available for Win, Mac, Linux) link or the GlobusOnline endpoint in your organisation
When you login to the GlobusOnline, choose endpoint "PSI" in the Transfer Page. When you are asked to authenticate, provide you PSI account and password (your credentials will not leave PSI computers, the authentication server is located at PSI). After successful login you should see a page like this: go2.png In the transfer window on a right side, choose either your GlobusConnect endpoint (they look like username#endpoint, e.g. slac#lcls) or endpoint of your organisation.

For the comprehensive description of the GlobusOnline GUI, please visit this page

Contacts

  • Your beamline manager
  • SwissFEL/Photonics IT responsibles (accessible by the mail below)
  • Admins of the data transfer service: Mail