BigData Express

Technology Summary

The emergence of distributed, extreme-scale science applications is generating significant challenges for data transfer. Ideally, high-performance data transfer should reach terabit/s throughput to make full use of the underlying networks and provide real-time and deadline-bound data transfer.

Although significant improvements have been made in the area of bulk data transfer, currently-available data transfer tools and services cannot successfully meet these challenges, for the following reasons:

  • Existing data transfer tools and services lack a data-transfer-centric approach to seamlessly and effectively integrate and coordinate the various entities in an end-to- end data transfer loop.
  • Existing data transfer tools and services lack effective mechanisms to minimize cross-interference between data transfers.
  • Existing data transfer tools and services are oblivious to user (or user application) requirements (e.g., deadlines and QoS requirements).
  • Inefficiencies arise when existing data transfer tools are run on DTNs.

These are common and fundamental problems for bulk data transfer in the extreme-scale era.

The Solution

BigData Express provides:

  • A data-transfer-centric architecture to seamlessly integrate and effectively coordinate the various resources in an end-to-end data transfer loop
  • SDN and SDS to improve network and storage I/O performance
  • A time-constraint-based scheduler to schedule data transfer tasks
  • An admission control mechanism to provide guaranteed resources for admitted data transfer tasks
  • A rate control mechanism to improve data transfer schedulability and reduce cross- interference between data transfers


  • BigData Express is a Schedulable, Predictable, and High-performance data transfer service for large-scale scientific computing facilities.

Applications and Industries

  • BigData Express is useful for large-scale scientific computing facilities like LCF, NERSC, US-LHC, and their many collaborators in industry and academia.


Category: Computers & Information Science

Tags and keywords: Big Data, Data Transfer, Distributed Computing


Invention Details

Copyright Status: Apache License 2.0

Aaron G Sauers, CLP
Fermilab, MS 312- PO Box 500
Batavia, IL 60510