Archiver Appliance documentation

This is an implementation of an archiver for EPICS control systems that aims to archive millions of PVs.

Here are the main features.

• Ability to cluster appliances and to scale by adding appliances to the cluster.

  • Limited support for redundancy.

  • Ability to dynamically re-assign PVs to appliances.

• Multiple stages and an inbuilt process to move data between the stages.

  • This supports the ability to use faster storage (which is perhaps limited in size) to improve performance.

  • Ability to reduce (decimate) the data as it moves into a store.

• Focus on data retrieval performance.

• A management interface giving you the ability to manage and monitor the system using a browser. This includes

  1. The ability to add PVs to a cluster of appliances using a browser (perhaps by users).

  2. Various metrics to help with capacity planning.

  3. Ability to define system-wide defaults for archiving parameters using policies.

• Ability to script the business processes in the appliances using an external process in a language like Python.

• Ability to configure various archiving parameters on a per PV basis.

• Ability to customize the appliance to suit a different set of requirements. This includes

  • The ability to use alternate storage technologies that may better suit your needs or perform better in your environment.

  • The ability to define your own data reduction algorithms and to optionally cache data generated by these algorithms on a per PV basis.

  • Simple ways to add support for new MIME types in data retrieval responses.

• Support for EPICS aliases.

• Support for EPICS 7/PVAccess/Structured data.

• Support for retrieval of data using CS-Studio, the ArchiveViewer and Matlab.

• Limited integration with existing Channel Archiver data sources.

Motivation

Archiving millions of PVs requires a few additional features beyond a simple archiver. In addition to focusing on the EPICS side of the equation, we also need to address the various IT aspects like flexibility, scriptability, and policy enforcement. An integrated multi-head, multi-stage solution was called for; hence the appliance.

EPICS 7

The archiver appliance has built in support for EPICS 7 and archiving PVs over PVAccess. NTScalars and NTScalarArrays are stored as their channel access counterparts. For example, PVDoubles will be stored as DBR_SCALAR_DOUBLEs. This makes it possible to use standard archive viewers to view NTScalars and NTScalarArrays archived thru PVAccess. Other PVData types are stored as a bunch of bits using PVAccess serialization. While this is probably not the most efficient, it does allow for archiving of arbitrary structured data. There is support for retrieving of structured data using the RAW and JSON formats.

Last published on Mar 06, 2026. This project is part of the AccelUtils collaboration.

Operator documentation

• Tutorials
• How-to guides
  • Configuring CS-Studio
    • Configuring CS-Studio
  • Retrieving data using the ArchiveViewer
  • Retrieving data using Matlab
• Explanations
  • Timezones
• References
  • Post-processors

Integrator documentation

• Tutorials
  • Submit a PV for archiving
    • Check the status of a PV
• How-to guides
• Explanations
  • Policies
  • SCAN vs MONITOR
  • EPICS aliases
• References

System admin documentation

• Tutorials
  • Quickstart
    • Quickstart using predefined VMs
  • Install Guide
    • Customize preexisting VM’s
    • Site specific installs
    • Using an install script
    • Manual installation overview
• How-to guides
  • Create your policies file
    • Setting up storage folders
  • Deployment
    • Create individual Tomcat containers for each web app
    • Deploy the WAR files onto their respective containers
    • Stopping and starting the individual Tomcats
  • Tomcat setup
  • MySQL setup
  • Backing up your config databases
  • Channel Archiver integration
  • Deployment guides
    • Deploying
    • Deploying on bare-metal (WIP)
    • Deploying with Docker (WIP)
    • Deploying with Docker Swarm (WIP)
  • Recovering from a lost appliance
  • Inspecting the Channel Access ( also PVAccess ) protocol
  • Temporarily suspend using a store
  • Dynamic (re)-assignment of PVs to appliances
  • Customization
    • Site specific builds
    • Policies
  • Making simple site-specific content changes
  • Using SQLite as a configuration database
• Explanations
  • Configuration
    • Appliances XML
    • archappl.properties
    • Key Mapping
    • Persistence
  • Log file locations
  • Monitoring your EPICS archiver appliance
  • Scripting for bulk operations
  • Storage
  • Redundancy in the EPICS Archiver Appliance
    • Case study
  • Reports and metrics
    • Reports
    • Metrics
    • Storage
    • Appliances
• References
  • Storage Plugins
    • NIO2
    • Type systems
    • Configuration
    • Conversion
  • Screenshots
  • System requirements
  • Environment variables

Developer documentation

• Tutorials
• How-to guides
  • Scripting
  • Fetch data
    • Constructing a data retrieval URL
    • Example: fetching data in Python
• Explanations
• References
  • Management Scriptables
  • Side Projects
  • Parquet format
    • Implementation Details
    • Configuration and Management
    • Tools for Analyzing Parquet Files
  • pbraw format
    • The .pb file format
    • Configuration
    • The PB/HTTP protocol
  • Retrieval API
    • Optional request parameters
    • Response format
    • Point-in-time retrieval (Save/Restore API)

Contributor documentation

• Tutorials
• How-to guides
  • Prerequisites
  • Building and testing
  • Running Tomcat
  • Unit testing
  • Formatting
• Explanations
  • Architecture
  • Clustering
  • Information useful to developers
  • ConfigService
• References