Partnerships and Technology Transfer

Technology Summary

The emergence of distributed, extreme-scale science applications is generating significant challenges for data transfer. Ideally, high-performance data transfer should reach terabit/s throughput to make full use of the underlying networks and provide real-time and deadline-bound data transfer.

Although significant improvements have been made in the area of bulk data transfer, currently-available data transfer tools and services cannot successfully meet these challenges, for the following reasons:

• Existing data transfer tools and services lack a data-transfer-centric approach to seamlessly and effectively integrate and coordinate the various entities in an end-to- end data transfer loop.

• Existing data transfer tools and services lack effective mechanisms to minimize cross-interference between data transfers.

• Existing data transfer tools and services are oblivious to user (or user application) requirements (e.g., deadlines and QoS requirements).

• Inefficiencies arise when existing data transfer tools are run on DTNs.

These are common and fundamental problems for bulk data transfer in the extreme-scale era.

The Solution

BigData Express provides:

• A data-transfer-centric architecture to seamlessly integrate and effectively coordinate the various resources in an end-to-end data transfer loop

• SDN and SDS to improve network and storage I/O performance

• A time-constraint-based scheduler to schedule data transfer tasks

• An admission control mechanism to provide guaranteed resources for admitted data transfer tasks

• A rate control mechanism to improve data transfer schedulability and reduce cross- interference between data transfers

Benefit

• BigData Express is a Schedulable, Predictable, and High-performance data transfer service for large-scale scientific computing facilities.

Applications and Industries

• BigData Express is useful for large-scale scientific computing facilities like LCF, NERSC, US-LHC, and their many collaborators in industry and academia.