Dr. Marrinan Publishes Article in Computational Science
CISC Assistant Professor Thomas Marinnan recently published a research article in the Journal of Computational Science on the benefits of in-transit streaming between HPC applications with different data layouts (available at https://doi.org/10.1016/j.jocs.2019.01.003).
The paper detailed joint research conducted by Thomas Marinnan, Dr. Greg Eisenhauer from the Georgia Institute of Technology, Dr. Matthew Wolf from the Oak Ridge National Laboratory, and Joseph Insley and Drs. Silvio Rizzi and Michael Papka from the Argonne National Laboratory.
Parallel streaming between heterogeneous HPC resources for real-time analysis
Performing analysis or generating visualizations concurrently with high performance simulations can yield great benefits compared to post-processing data. Writing and reading large volumes of data can be reduced or eliminated, thereby producing an I/O cost savings. One such method for concurrent simulation and analysis is in transit – streaming data from the resource running the simulation to a separate resource running the analysis.
In transit analysis can be beneficial since computational resources may not have certain resources needed for visualization and analysis (e.g. GPUs) and to reduce the impact of performing analysis tasks to the run time of the simulation. When sending and receiving data in transit, data redistribution mechanisms are needed in order to support heterogeneous data layouts that may be required by the simulation and analysis applications.
The work described in this paper compares two mechanisms for on-the-fly data redistribution when streaming data in parallel between two distributed memory applications. Our results show that it is often more advantageous to stream data in the same layout as the sender and redistribute data amongst processes on the receiving end than to stream data in the final layout needed by the receiver.