Data Transfer Services

Transferring data into and out of CHPC resources is a critical portion of many science workflows. The CHPC offers a number of options for moving data to and from CHPC storage resources, both for the transfer of a few small files to the transfer of multiple terabytes, or even petabytes, of data. In some cases, the data may not need to be moved, as there are options to mount some CHPC file systems from local resources.

A good resource for information on data transfer considerations is the ESnet Faster Data site. Specifically, for setting expectations regarding transfer times, see the page about expected time to transfer data. For understanding the impacts of dropped packets on network protocols and the corresponding impacts to large science transfers, see the page regarding tcp for long range data transfers.

Large transfers are very dependent on the characteristics of the resources on both ends of the transfer. If you need assistance in initiating, monitoring, or troubleshooting large transfers, you can reach out to CHPC via helpdesk@chpc.utah.edu.

On this page

The table of contents requires JavaScript to load.

Data Transfers for Small Data Sets

Your research workflow may require the transfer of a few small files to CHPC resources for storage, final reporting, or analysis. In cases like these, the CHPC provides four easy options for the transfer of small datasets:

Open OnDemand Recommended

You can connect to CHPC resources all from a web link, making the transfer of small data sets to and from your local computer to the CHPC simple.

WinSCP (Windows), CyberDuck (Macs)

These tools provide a graphical interface of simple graphical tools to move data to/from a local machine. In these cases you can "drag and drop" files from one system to the other

Authenticated File Transfers (scp, rsnyc)

Authenticated file transfer tools like SCP and rsync offer the significant advantage of secure data transmission through encryption and the necessity of user authentication. All through the simple and quick use of the command line, tools like this ensure that data remains confidential during transfer and that only authorized individuals can access or modify files at the destination.

Unauthenticated File Transfers (wget, curl)

Unauthenticated command-line file transfer tools like wget and curl excel at efficiently downloading publicly available data directly from web servers without requiring logins or credentials. This makes them ideal for scripting automated downloads and retrieving resources that are intended for open access.

Data Transfers for Large Data Sets

For workflows that require you to transfer very large files or datasets between national labs, industry partners, peer institutions, or colleauges, you will require more advanced parallel transfer tools running on tuned endpoint devices such as Data Transfer Nodes (DTNs).

CHPC supports various parallel transfer tools that support the known science workflows at the University of Utah. If a research group requires the support of an additional transfer tool, the group may request it through helpdesk@chpc.utah.edu.

Data Transfer Nodes

Campus network traffic usually routes through a firewall, which is sufficient for small transfers. However, research computing requires high bandwidth connections, which can overwhelm the firewall and impact general campus use. To address this, a Science DMZ was created, which is a separate network with optimized performance and security. CHPC providesData Transfer Nodes (DTNs) utilizing this DMZ for efficient research data transfers.

For more information, visit "What makes a Data Transfer Node?".

DTNs can be used with standard CLI or GUI data transfer tools (e.g., rsync, wget, WinSCP, etc.) to move data to and from CHPC resources. Addresses to use for these tools can be found below.

The CHPC also supports a number of group-owned DTNs. If you need any information about an existing group owned DTN or are interested in having a DTN dedicated for your group, please contact us through helpdesk@chpc.utah.edu.

Both internal and external DTNs can be used via Slurm on notchpeak (General Environment) and redwood (Protected Environment).

General Environment DTNs

General Environment DTNs are divided into internal and external groups. Internal DTNs (intdtn) offer 40 Gbps connectivity within the campus firewall, ideal for on-campus transfers. External DTNs (dtn) provide 100 Gbps connectivity to the Science DMZ, better suited for off-campus transfers and large datasets.

CHPC General Environment DTNs:

intdtn01.chpc.utah.edu
intdtn02.chpc.utah.edu
intdtn03.chpc.utah.edu
intdtn04.chpc.utah.edu
dtn05.chpc.utah.edu
dtn06.chpc.utah.edu
dtn07.chpc.utah.edu
dtn08.chpc.utah.edu

Protected Environment DTNs

The CHPC's Protected Environment (PE) DTNs operate at 40 Gbps and require the University of Utah VPN for off-campus access. While offering the same data transfer tools as the General Environment, the PE DTNs have stricter security for remote connections. Tools like screen and tmux allow background transfers to continue even if the SSH connection is lost.

CHPC Protected Environment DTNs:

pe-dtn03.chpc.utah.edu
pe-dtn04.chpc.utah.edu
pe-dtn.chpc.utah.edu (Can be used to connect to the servers with round-robin balancing)

See Data Transfer Node Access via Slurm for details.

Parallel Transfer Tools

The DTNs are also set up to leverage all parallel transfer tools that CHPC supports. See our Software page for more information on these tools.

The CHPC also supports additional specialized tools for moving data to/from cloud storage. Some of these tools are specific to a single cloud storage provider (such as s3cmd for Amazon cloud services), whereas others, such as rclone, work with different cloud storage providers.

Globus

Recommended

Globus is a great option because it transfers data in the background and in parallel, maximizing the use of the network's full bandwidth for optimized and efficient data movement. This allows users to initiate transfers without interrupting their workflow and experience significantly faster transfer speeds.

Rclone

Recommended

Rclone is a versatile command-line tool that facilitates file transfers and synchronization between local storage and various cloud services like Google Drive, Dropbox, and OneDrive, as well as other storage options. Offering optimization features, Rclone can achieve faster transfer speeds compared to tools like scp and rsync

Fpsync

Fpsync is a robust file-migration tool that parallelizes data transfers using tools such as rsync, cpio, pax, and tar. It supports efficient data migrations using an SSH connection and starts transfers while the file system is still being crawled. This allows users to initiate transfers without interrupting their workflow and experience significantly faster transfer speeds.

To use:

module load fpsync

fpsync -h

Aspera

Aspera is IBM's high-performance file transfer software which allows for the transfer of large files and data sets with predictable, reliable and secure delivery regardless of file size or transfer distance. The NCBI recommends the use of Aspera for the transfer of data sets from their site. The command line client is available by loading the ascp module.

To use:

module load aspera

ascp --help

Fast Data Transport

Fast Data Transport (FDT) is a Java-based file transfer tool installed at the CHPC. It operates in three modes: Server (listens for connections), Client (connects to a server to send/receive files), and SCP (uses SSH for secure server/client initiation with credential-based security, allowing server connections only from the "SCP" client).

To use:

java -jar /uufs/chpc.utah.edu/sys/pkg/fdt/0.9.20/fdt.jar --help

or visit the FDT site.

BBCP

BBCP is a point-to-point network file copy application developed at SLAC National Accelerator Laboratory that allows for sending data in multiple simultaneous streams, developed for the BABAR as a particle physics experiment. Complete documentation on bbcp usage can be found at the BBCP website.

To use:

/uufs/chpc.utah.edu/sys/pkg/bbcp/std/bin/bbcp --help

or visit the BBCP site.

Direct Mounts of CHPC File Systems

The CHPC doesn't offer direct mounts of PE file servers outside of the PE. For transfers of data to the Protected Envrionment, we recommend that users consider Globus, SFTP (FileZilla), SCP (WinSCP), rclone, and the PE Data Transfer Nodes.

CHPC allows home directories and group owned file systems that exist in the General Environment to be mounted onto local machines. We do not allow mounts of the general CHPC scratch file systems such as /scratch/general/nfs1 and /scratch/general/vast.

To mount home or group directories from the CHPC general environment onto your personal machine, you must be on the University of Utah network. If not on campus, this will require the University of Utah VPN. Please follow the steps below based on if you are wanting to mount your home directory or your group storage space. In all of the following, you must replace <uNID> with your unid and be on University network:

Note that there is also a short training video that covers this topic.

Home Directories

1. Obtain the correct network drive information by issuing this command on CHPC resources: df

2. If the df command returns eth.vast.chpc.utah.edu:/home/XX/, use the value of XX to map the network drive in the next step

3. Mount an external drive onto your personal laptop using the address formatted for your laptop OS below, replacing XX with the value obtained in step 2.

On Windows: \\samba.chpc.utah.edu\XX-home\<uNID>
- (NOTE - when authenticating, your username is ad\)
On a Mac: smb://samba.chpc.utah.edu/XX-home/<uNID>
On Linux: If you have root, you can cifs mount CHPC file spaces by (you will be prompted for your password): mount.cifs //samba.chpc.utah.edu/XX-home/<uNID> /mnt -o domain=AD,user=<uNID>

Group Directories

You can mount group spaces on your personal laptop by using the drive formatting appropriate for your use case, replacing <name-groupnumber> with the assigned group directory name:

On Windows: \\samba.chpc.utah.edu\<name-groupnumber>
- (NOTE - when authenticating, your username is ad\)
On a MAC: smb://samba.chpc.utah.edu/<name-groupnumber>
On Linux: If you have root, you can cifs mount CHPC file spaces by (you will be prompted for your password): mount.cifs //samba.chpc.utah.edu/<name-groupnumber> /mnt -o domain=AD,user=<uNID>

Temporary Guest Transfer Service

CHPC provides a mechanism for our users to transfer files to and/or from individuals without CHPC accounts. This service is called guest-transfer.

What is the temporary Guest Transfer Service for?

At times, CHPC users need to transfer files to or from individuals that don't have CHPC accounts. These files are often quite large (many gigabytes), and thus unsuitable for other transport mechanisms (email, DVD).
These file transfers often need to happen with little or no warning. They may also need to occur outside CHPC's support hours. Thus, the guest-transfer service must function without intervention or assistance from CHPC staff.

What is the Temporary Guest Transfer Service not for?

The guest transfer service is not for repeated events.
The guest transfer service is not for long-term data storage.
The guest transfer service is not for restricted (e.g. PHI/HIPAA/HITECH/FERPA) or sensitive data.
The guest transfer service is not for huge data transfers (it's currently restricted to approximately 5 terabytes).

How do I get a guest account?

When you need to use the guest transfer service, visit https://guest-transfer.chpc.utah.edu/ and fill out the form. This form creates a guest transfer account. You then give the guest account username and password to your colleague. You and your colleague can now share files.

How do I use the service?

Once you have created a guest-transfer account and given it to your colleague, you and your colleague can copy your files to guest-transfer.chpc.utah.edu with your scp/sftp client (scp, sftp, WinSCP, etc). Files should be transferred to /scratch on guest-transfer.chpc.utah.edu.
The CHPC user can then transfer the files from /scratch on guest-transfer.chpc.utah.edu to other CHPC resources.

Things to remember:

The process is completely automatic. You fill out the form, it immediately gives you a guest account.
Only CHPC users can create guest accounts.
The person who creates the guest account is responsible for all activity of the guest account.
This guest account is only usable for the guest transfer service. It provides no access to any other CHPC or University resources.
Files are transferred via scp/sftp. Interactive logins are disabled.
Files are automatically removed after 90 days (based on last-access time).
Files in the guest-transfer service can be read or changed by any other user.
Consider using encryption to protect and verify your files.
DO NOT USE THIS SERVICE FOR SENSITIVE OR RESTRICTED DATA.