An active distributed storage architecture.

Abstract

This thesis presents the Distributed Active Resource Architecture (DARC), a modular, dynamic system for the flexible construction of distributed storage and I/O infrastructure. DARC is comprised of a component-oriented framework and distributed runtime system, inspired by the Actor Model. Within DARC, components form the fundamental unit through which all functionality is provided and all communication occurs via asynchronous message-passing. This component model extends to the mechanisms through which DARC exposes its services and makes use of external storage resources. Components are modular, lightweight and can be dynamically deployed, configured and composed. The composition of DARC components is based on directed graphs which enable arbitrary topologies of distributed I/O down to per-message granularity. These graphs are mutable through the lifecycle of each message and thus allow distributed, nested compositions. Composition is facilitated at a high level by an embedded domain-specific language for the generation and modification of graphs. The system is implemented in Java, which provides the code mobility required for dynamic component deployment. Network communication occurs via an open platformindependent protocol, allowing interoperability. Prototype interfaces to the system include a REST-oriented web service and user-level filesystem module. Interfaces to external storage services include filesystems, object stores and cloud storage. Example functional component implementations include name-and content-based data distribution, write-back caching, parallel transfers, optimistic replication and aggregated I/O. The implementation and evaluation described in this thesis show that the overhead of graph processing is not an impediment to high performance and illustrate the feasibility and flexibility of the design. The contribution of this work is widely applicable to a range of systems and especially dynamic cloud computing infrastructure.