Large Volume SLS/SwissFEL Data Transfer

IMPORTANT INFO: Access to data transfer service require Microsoft MFA (multi factor authentication)


https://www.psi.ch/en/computing/change-to-mfa


If your account isn`t already enabled for Microsoft MFA, please
contact the IT Servicedesk during the business hours
using helpmfa@psi.ch

Access using ssh-key is disabled

This service targets users that need to export big volumes of data from experimental stations at SLS and SwissFEL.

PSI account

To use the data transfer service you need to have a PSI account. If you don't have one, follow the procedure .

Contact your beamline manager or IT responsible (SwissFEL/Photonics group member only) with the following information

  • your PSI account
  • data identifier (Proposal ID or e-account used to collect the data) for the data you need to access

In case you have internal firewall, to be able to use ssh, access to the following IP's should be allowed for the outbound connections, TCP port 22:

  • 192.33.126.61
  • 192.33.126.62
  • 192.33.126.55

The following directories are accessible with the data transfer service:

Directory Name Example Read-Only Comments
/sls /sls/MX/Data10/e15874 yes Raw data from SLS
/das /das/work/p15/p15874   Working area of the Ra cluster, please note a structure of subdirectories: p{AB}/p{ABCDEF}
/sf /sf/alvra/data/p17502/{raw,res,work} yes(raw) Raw data and working area for the data taken a SwissFEL facility
e.g. to copy data from another facility,like LCLS

Use the generic hostname ra-export.psi.ch for the transfer.

The following examples use ext-name, e15874 and MX as PSI account, e-account and beamline name correspondingly.

Listing the content

Example:

$ ssh ext-name@ra-export.psi.ch "ls /sls/MX/Data10/e15874"

or the recursive listing (might be slow)

$ ssh ext-name@ra-export.psi.ch "ls -R /sls/MX/Data10/e15874"

Transfer

To copy directory "data_exchange/processing_output" to your computer (to the DESTINATION_DIRECTORY, like ./ ):

  • with the rsync
$ rsync -av ext-name@ra-export.psi.ch:/sls/MX/Data10/e15874/data_exchange/processing_output DESTINATION_DIRECTORY
  • with the scp
scp -r ext-name@ra-export.psi.ch:/sls/MX/Data10/e15874/data_exchange/processing_output DESTINATION_DIRECTORY

Using SSH multiplexing

With using SSH multiplexing, one can efficiently re-use created with the MFA secure connection for the subsequent connections, without the need to authenticate every time with MFA.

Create a corresponding to the data transfer server configuration in .ssh/config file (here ra-export-3.psi.ch is used as an example):

$ cat .ssh/config

Host ra-export-3
  ControlMaster auto
  ControlPath ~/.ssh/sockets/%r@%h-%p
  ControlPersist 86400
  hostname ra-export-3.psi.ch
  User ext-name

(one time action) Create .ssh/sockets/ directory

$ mkdir ~/.ssh/sockets

Initialise master connection (will be prompted for password and would need to confirm with MFA Authenticator)

$ ssh ra-export-3 "ls /sls/MX"
password:

In case everything is successful with authentication and creation of master multiplex connection - all following ssh commands will be run over master connection, without additional MFA authentication:

$ ssh ra-export-3 "ls /sls/MX"

(no password)

To check the state of master connection:

$ ssh ra-export-3 -O check

To terminate master connection:

$ ssh ra-export-3 -O exit

Master connection lifetime is determined by ConrolPersist value. Please note that very long lived master connections (more than few days) will be terminated on our side

 

GlobusOnline is a web service, which allows one to use the GridFTP file transfer protocol in an easy and intuitive way. This protocol is well suited for transfers of big files (>100 MB).

GlobusOnline has a number of attractive features:

  • Automatic network optimization
  • Parallel/multistream transfers (up to 4 transfers/streams)
  • Automatic retry in case of failure
  • Online task monitor
  • Summary email sent at the end of the transfer
  • Usually is firewall safe, as it uses only outgoing connections. If your firewall blocks also outgoing connections, then you need some special rules to be set up, contact your local IT support

Requirements for using GlobusOnline:

  • A GlobusID account (free), or an account recognized by GlobusOnline (like Google, XSEDE, US Universities), see a very detailed description (PSI staff may use psi account to login to GlobusOnline, "ext-" accounts can't be used to login to GlobusOnline)
  • GlobusConnect Personal client installed (available for Win, Mac, Linux here) or the GlobusOnline endpoint in your organisation
  • a web browser to use the Globus web app: https://app.globus.org

When you login to the GlobusOnline, choose endpoint "PSI" in the Transfer Page. When you are asked to authenticate, provide you PSI account and password (your credentials will not leave PSI computers, the authentication server is located at PSI). After successful login you should see a page like this:

go2.png

In the transfer window on a right side, choose either your GlobusConnect endpoint (they look like username#endpoint, e.g. slac#lcls) or endpoint of your organisation.

For the comprehensive description of the GlobusOnline GUI, please visit this page

  • Your beamline manager
  • SwissFEL/Photonics IT responsibles (accessible by the mail below)
  • Admins of the data transfer service: Mail