Large Volume SLS/SwissFEL Data Transfer

This service targets users that need to export big volumes of data from experimental stations at SLS and SwissFEL.

PSI account

To use the data transfer service you need to have a PSI account. If you don't have one, follow the procedure .

Contact your beamline manager or IT responsible (SwissFEL/Photonics group member only) with the following information

  • your PSI account
  • data identifier (Proposal ID or e-account used to collect the data) for the data you need to access

In case you have internal firewall, to be able to use ssh, access to the following IP's should be allowed for the outbound connections, TCP port 22:

  • 192.33.126.61
  • 192.33.126.62
  • 192.33.126.55

The following directories are accessible with the data transfer service:

Directory Name Example Read-Only Comments
/sls /sls/MX/Data10/e15874 yes Raw data from SLS
/das /das/work/p15/p15874   Working area of the Ra cluster, please note a structure of subdirectories: pAB/pABCDEF
/swissfel /swissfel/photonics/data   Working directory of SwissFEL/Photonics group
e.g. to copy data from another facility,like LCLS

Use the generic hostname ra-export.psi.ch for the transfer.

The following examples use ext-name, e15874 and MX as PSI account, e-account and beamline name correspondingly.

Listing the content

Example:

$ ssh ext-name@ra-export.psi.ch "ls /sls/MX/Data10/e15874"

or the recursive listing (might be slow)

$ ssh ext-name@ra-export.psi.ch "ls -R /sls/MX/Data10/e15874"

Transfer

To copy directory "data_exchange/processing_output" to your computer (to the DESTINATION_DIRECTORY, like ./ ):

  • with the rsync
$ rsync -av ext-name@ra-export.psi.ch:/sls/MX/Data10/e15874/data_exchange/processing_output DESTINATION_DIRECTORY
  • with the scp
scp -r ext-name@ra-export.psi.ch:/sls/MX/Data10/e15874/data_exchange/processing_output DESTINATION_DIRECTORY

SSH keys (recommended option)

To avoid typing password for every connection to the data transfer server, upload your ssh-key to the server machine (this command will overwrite existing authorized_keys file):

scp .ssh/id_rsa.pub ext-name@ra-export.psi.ch:.ssh/authorized_keys

If you don't have ssh key, you may generate it (example for linux machine):

  • generate a key-pair: 
    ssh-keygen -C "PSI data export" -f $HOME/.ssh/datatransfer
    (You'll be asked for the password, invent one, don't leave it empty, you'll use this password in next step)
  • copy the public key to the data transfer server 
    scp .ssh/datatransfer.pub ext-name@ra-export.psi.ch:.ssh/authorized_keys
  • add your key to the ssh-agent: 
    ssh-add .ssh/datatransfer
    (here you'll need to provide password you invented at first step)

Please note that DSA keys is considered insecure and disabled in the most recent ssh version. ssh-keygen by default will generate RSA key, which one can steer with the -t option.

GlobusOnline is a web service, which allows one to use the GridFTP file transfer protocol in an easy and intuitive way. This protocol is well suited for transfers of big files (>100 MB).

GlobusOnline has a number of attractive features:

  • Automatic network optimization
  • Parallel/multistream transfers (up to 4 transfers/streams)
  • Automatic retry in case of failure
  • Online task monitor
  • Summary email sent at the end of the transfer
  • Usually is firewall safe, as it uses only outgoing connections. If your firewall blocks also outgoing connections, then you need some special rules to be set up, contact your local IT support

Requirements for using GlobusOnline:

  • A GlobusID account (free), or an account recognized by GlobusOnline (like Google, XSEDE, US Universities), see a very detailed description (PSI staff may use psi account to login to GlobusOnline, "ext-" accounts can't be used to login to GlobusOnline)
  • GlobusConnect Personal client installed (available for Win, Mac, Linux here) or the GlobusOnline endpoint in your organisation
  • a web browser to use the Globus web app: https://app.globus.org

When you login to the GlobusOnline, choose endpoint "PSI" in the Transfer Page. When you are asked to authenticate, provide you PSI account and password (your credentials will not leave PSI computers, the authentication server is located at PSI). After successful login you should see a page like this:

go2.png

In the transfer window on a right side, choose either your GlobusConnect endpoint (they look like username#endpoint, e.g. slac#lcls) or endpoint of your organisation.

For the comprehensive description of the GlobusOnline GUI, please visit this page

  • Your beamline manager
  • SwissFEL/Photonics IT responsibles (accessible by the mail below)
  • Admins of the data transfer service: Mail