Skip to content

Documentation

Welcome to the general documentation for the Aruna Object Storage (AOS).

This documentation includes theoretical concepts, the internal data structure, the entity-relationship model, a basic usage guide, some generic user story playbooks, and much more in the future.

Deeper technical documentation can be found in the implementation repositories for the API, Server or CLI. Details on the individual structures can be found in the API documentation and/or the Data Structure part of this documentation.

UML diagram of the Aruna Object Storage data structure

UML diagram of the Aruna Object Storage data structure. A more detailed description of the individual parts can be found here.

AOS Components

AOS API

Main Aruna API repository

The API is designed generally with Google Protocol Buffers. With the release of a new API version, the client libraries are automatically compiled and updated to the latest version. The API is fundamentally backwards compatible, which means that users' applications will continue to work as usual before they also decide to move to the new version.

AOS Server

Main Aruna Server repository

The implementation of the API that handles the incoming requests.

Aruna Object Storage is data lake application with API that manages scientific data objects according to FAIR principles. It interacts with multiple data storage backends (e.g. S3, File ...) via the DataProxy application and stores a rich and queryable set of metadata about stored objects in a postgres compatible database (e.g. CockroachDB). Aruna is conceptually geo-redundant with multiple dataset and database locations. Users can choose where and how to store their data.

AOS Data Proxy

Main Aruna Data Proxy repository

This is the internal server implementation handling the communication between the data storage backend used for the specific AOS instance.

The DataProxy service is designed to handle data transfers AOS. It starts two server: One server implements the internal proxy api from the official AOS gRPC API. The other server handles the transfer of the actual data. All data access is handled through presigned URLs.

AOS CLI

Main Aruna CLI repository

This is a simple CLI application for the ScienceObjectsDB API. Its currently work in progress and will be developed along with the API. Neither concept nor implementation are final.

Notification system

The storage system has a notification system that can be used to receive change notification on specific resources.

The notification system is currently still in the implementation phase but will be available soon.

Implementation Design Trivia

  • An RDBMS will be used as database backend for the AOS Server
  • The AOS Server, Data Proxy nd CLI will be implemented in Rust
  • The base API interface will be defined using Protocol Buffers
  • All endpoints work with JSON over HTTP just as they do with requests made via gRPC from individual clients
  • Clients stubs will be generated for major programming languages on every API release
  • A basic CLI client will be offered to simplify the usage entry barrier
  • A web UI is available for demonstration purposes