Networking Materials Data: Accelerating Discovery at Experimental Facilities

Advances in both sensor and computing technologies promise new approaches to discovery in materials science and engineering. For example, it appears possible to integrate theoretical modeling and experiment in new ways, test existing models with unprecedented rigor, and infer entirely new models from first principles. But, before these new approaches can become useful in practice, practitioners must be able to work with petabytes and petaflops as intuitively and interactively as they do with gigabytes and gigaflops today. The Discovery Engines for Big Data project at Argonne National Laboratory is tackling key bottlenecks along the end-to-end discovery path, focusing in particular on opportunities at Argonne's Advanced Photon Source. Here, we describe results relating to data acquisition, management, and analysis. For acquisition, we describe automated pipelines based on Globus services that link instruments, computations, and people for rapid and reliable data exchange. For management, we describe digital asset management solutions that enable the capture, management, sharing, publication, and discovery of large quantities of complex and diverse data, along with associated metadata and programs. For analysis, we describe the use of 100K+ supercomputer cores to enable new research modalities based on near-real-time processing and feedback, and the use of Swift parallel scripting to facilitate authoring, understanding, and reuse of data generation, transformation, and analysis software.