Prenacs

Prenacs is a system, which allows to run batch computations of attribute values for sets of entities, and to store the computation results in a database, alongside metadata which allows to track the data provenance.

The name is an acronym for “PRovenance-tracking ENtity Attribute Computation and Storage system”.

Key concepts

Entities are thereby object of any kind, characterized by an identifier, which allows distinguishing the object from other objects of the same kind. For each entity, ProBatch allows to compute the values of attributes.

Attribute are properties of the entity. Anything whose value can be determined using a computation (in a general sense, thus e.g. also obtaining the value from an external data source) is an attribute. Each attribute may consist of a single value (scalar attribute) or or multiple values (composite attribute). The attribute system is open and flexible, i.e. new attributes can be added at any moment.

ProvBatch allows to store a single instance of each attribute for an entity: that is, it is not meant to store multiple measurements of a value or to store a journal of previous attribute computation results.

Alongside the computation results, ProBatch allows to track the provenance of attribute values. For achieving this, the code for attribute value computation must be implemented in form of a computation plugin. Multiple plugins may be available for computing the same attribute. A plugin may compute one or multiple attributes. Plugins contain the code for the attribute computation, as well as metadata, describing the input, output, supported paramaters and methods and implementation notes. Whenever a plugin code changes, a new plugin version number is assigned.

Batch computation metadata instances are identified by an unique identifier (UUID) and include a reference to a given plugin (plugin identifier and version number), alongside information such as the computation parameters, timestamps, identifier of the user starting the computation, and key system data. Thus, storing the batch computation UUIDs alongside with computation results, ProvBatch allows to keep track of the data provenance.

Related libraries

Prenacs has been developed as part of an ecosystem of Python libraries, including:

• multiplug, which implements the infrastructure for the plugin system

• attrtables, which implements the infrastructure for the storage of attribute values and metadata

• snacli, which is used for implementing double-purpose scripts, callable interactively from the command line, as well as inside a Snakemake pipeline

Usage

The usage of the library is explained in the user manual.

Acknowledgements

This software has been originally created in context of the DFG project GO 3192/1-1 “Automated characterization of microbial genomes and metagenomes by collection and verification of association rules”. The funders had no role in study design, data collection and analysis.