Paraflow: A dataflow distributed data-computing system

Abstract

We describe the Paraflow system for connecting heterogeneous computing services into a flexible and efficient data-mining metacomputer. There are three levels of parallelism: a dataflow paradigm provides functional parallelism, connecting services by channels; each service may be a SPMD parallel program; and each node of the SPMD service may be multithreaded. Paraflow is easily integrated with web-computing, using MIME types, HTTP forms, a Java GUI for setting up the network of services, and a simple API to allow web servers to be utilized in the computation. Emphasis is on providing a bridge from straightforward, public, unauthenticated browsing of data to collaborative, high-performance data-computing.

Additional details