Skip to content

Archive Operations

As described in Running Aeron Archive, Aeron Archive is an application that has a piece of Client code (the AeronArchive class) that a User Application can use to interact with it.

Overall Workflow

At a high level, the workflow for using Aeron Archive is:

  1. Run the Aeron Archive application
  2. The User Application calls AeronArchive.connect() to connect to the Aeron Archive application and establish a Control Session
  3. The User Application calls other methods on the AeronArchive Client, which sends Control Request messages to the Aeron Archive application over the Control Session and receives responses on a Control Response return channel

The remainder of this page lists, and then provides some high level details about the main operations (Control Requests) that a User Application can request, after it has connected to Aeron Archive.

The next page describes Connecting to Aeron Archive in a lot more detail, then subsequent pages cover how Recording and Replaying actually work.

Archive Operations

A summary of the Aeron Archive operations:

  • Start Recording - starts a new Recording of a Publication, independent of any previous Recordings. This adds a new RecordingDescriptor to the Catalog that contains metadata about the Publication (termLength, streamId, etc.) and the Recording (startTimestamp, startPosition, etc.)
  • Stop Recording - in addition to stopping a recording, this records its stopPosition in the Catalog
  • Extend Recording - used to reopen a stopped Recording to allow new messages to be appended to the end of it (must continue from where it stopped - no gaps are allowed). The stopPosition and stopTimestamp are cleared in the Catalog, as the Recording has become active again
  • List Recordings - query the Catalog to retrieve information about Recordings. This is useful during User Application startup, to find an existing Recording that the application wants to continue using
  • Replay Recording - starts Replay of a Recording, reading messages from its Segment files and publishing them on a Publication, which a User Application receives via a Subscription
  • Stop Replay - stops a Replay session
  • Detach Segments - update a Recording's startPosition in the Catalog so it no longer refers to one or more of the oldest Segment files. The user is then free to move them, compress them, etc.
  • Attach Segments - opposite of Detach - update a Recording's startPosition in the Catalog to add some previously detached Segment files back onto the start of a Recording (no gaps are allowed between the new Segment files and the existing ones)
  • Purge Segments - same as Detach Segments, but also delete them
  • Purge Recording - delete a whole Recording - remove its RecordingDescriptor from the Catalog and delete all its Segment files
  • Truncate Recording - set a new stopPosition in the Catalog and remove everything that was ahead of it, zeroing the remainder of the current Segment file and/or deleting any other Segment files ahead of the new stopPosition
  • Replication - replicate a Recording from one instance of Aeron Archive to another

Replay Merge

Another useful Aeron Archive feature is Replay Merge, which is where an application wants to follow a live Publication, but needs to catch up on older, missed messages first. It starts by Replaying an active Recording of the live Publication. When the application is within a window of the live Publication position, it switches over to it and stops following the Replay Publication. Once it is following the live Publication directly, it will not be subject to any delay in reading live messages via the Recording.

This is an application level concern in how Aeron Archive can be used and is provided as part of the Aeron Archive Client (see ReplayMerge). It is not something that you ask Aeron Archive to perform itself, so it is not listed as an operation above.

Start Recording

Starts a new Recording.

It takes Channel, StreamId and SourceLocation parameters. It creates a Subscription to the Channel and StreamId and records any messages it receives to a new set of Segment files.

The Recording is allocated a recordingId (just a number that increments from zero), and a RecordingDescriptor is created for it in the Catalog.

When starting a Recording for a Network Publication, the User Application needs to tell Aeron Archive whether it is on the Sending End or Receiving End of the Publication. This is what the SourceLocation parameter is for:

  • SourceLocation.LOCAL - use when recording on the sending end of a Network Publication. This makes Aeron Archive add aeron-spy: to the start of the Channel, so it creates a Spy Subscription to read messages from the Publication Log Buffer
  • SourceLocation.REMOTE - use when recording on the receiving end of a Network Publication. This makes Aeron Archive create a normal Subscription that reads messages from the Image Log Buffer

The sourceLocation parameter is not used for IPC Publications, so it can be left set to SourceLocation.NULL_VAL.

Stop Recording

Closes the Subscription to the Publication, closes the active Segment file, and records the stopPosition and stopTimestamp in the Catalog.

Extend Recording

Extending a Recording means reopening it and appending new messages to the end. A typical use case is when your application restarts and wants to resume writing to the same Recording from where it left off.

There are some conditions for extending a Recording:

  • the Recording must not be active, so it must have stopped and have a stopPosition
  • the request to extend the Recording must use the same streamId
  • the start position of the extension must equal the Recording's stopPosition, i.e. there can be no gaps in the Recording
  • the initialTermId, termLength and mtuLength of the new Publication must be the same as the original Publication that was recorded (these values are stored in the Catalog)

In order to extend the Recording without any gaps, when Aeron Archive creates a Subscription to the Publication to be recorded, the Subscription must have a join position that matches the Recording stopPosition. In other words, the Subscription's start position should match the Recording's stopPosition. This can be achieved by setting the initial position on the Publication and starting the Recording before publishing any new messages on it. When the Recording starts, it will Subscribe to the Publication before any messages have been published on it, so the join position will be its initial position.

The initial position, initialTermId, termLength and mtuLength can be set on the new Publication by assigning values to its Channel, using values from the Recording:

final String publicationExtendChannel = new ChannelUriStringBuilder()
    .initialPosition(recording.stopPosition(), recording.initialTermId(), recording.termBufferLength())
    .mtu(recording.mtuLength())
    // other channel params...
    .build();

List Recordings

There are a few methods for listing Recordings on AeronArchive, which query the Catalog. They all return data via a callback that returns all the RecordingDescriptor parameters.

When trying to find an existing Recording, it can be useful to set an alias parameter on the Publication's Channel. The alias parameter is a string value that is not used by Aeron, so you can set it to something descriptive. When listing Recordings, you can then filter on the alias parameter on the RecordingDescriptor's originalChannel or strippedChannel field.

Replay Recordings

Replaying a recording replays recorded messages back to a User Application. Replay is started using one of the various AeronArchive.replay() or startReplay() methods. The replay() methods also create a Subscription object that a User Application can use to receive the replayed data. The startReplay() methods require the User Application to create the Subscription itself.

Replay requires the following parameters:

  • recordingId, which can be looked up from the Catalog
  • replayPosition - the position to replay from, or NULL_POSITION to replay from the start
  • length - how much of the recording to replay, or Long.MAX_VALUE to tail the recording once at the end
  • replayChannel and replayStreamId - the channel and streamId that Aeron Archive will publish the recorded data to, which the User Application needs to subscribe to

There is also a startBoundedReplay() method that behaves like startReplay(), but the User Application can also pass in a limitCounterId. This the id of a Counter (in cnc.dat), which will be queried by the replay to determine how much it can read up to, and obviously the counter can advance while the replay is happening.

Stop Replay

Stop Replay stops a Replay session. It takes a replaySessionId, which is returned from the startReplay() methods.

Detach Segments

Takes recordingId and startPosition parameters. The new start position must be:

  • the start of a Segment file
  • higher than the current start position, so it detaches at least one Segment file
  • not higher than the stop position of a stopped Recording, or rec-pos of an active Recording
  • not higher than any Replay positions

If the new startPosition is valid, the startPosition in the Catalog is updated to it. Nothing else is changed. The detached Segment files will no longer be accessed by Aeron Archive, so the user is free to do whatever they want with them, such as move them to another machine, compress them, etc.

Attach Segments

The opposite of Detach Segments - allows you to add Segment files to the start of a Recording. These would almost certainly be Segment files that have previously been Detached and need re-Attaching. The current startPosition in the Catalog must be at the start of the first Segment file. The new Segment files must immediately precede the current startPosition, so that when they are attached, there are no gaps in the Recording.

This requires you to manually add Segment files into the archive directory before asking Aeron Archive to attach the files. You just pass in the recordingId parameter, and it automatically searches backwards from what is currently the first Segment file, looking for earlier files that can be added to the Recording.

For each file found, it checks that the file's length is the Segment file length. It then looks for the first fragment in the first Term's worth of data in the file. If the first fragment is at the start of the file, some basic checks are done (the termId and streamId are correct), then the startPosition in the Catalog is updated, and it looks for the next earlier file.

If the first fragment is not at the start of the file, i.e. the Frame length at the start of the file is zero, it walks forwards and checks each alignment boundary (every 32 bytes), looking for a non-zero Frame length. If it finds one, it does the same basic checks, then sets the startPosition in the Catalog to that position and stops looking for further files. This handles re-Attaching the first file in a Recording, where the Recording didn't start at the start of the file.

For example, if you had the following Segment files:

0-393216.rec  <-- startPosition = 393216
0-524288.rec

and added two more to the directory:

0-131072.rec  <-- needs attaching
0-262144.rec  <-- needs attaching
0-393216.rec  <-- startPosition = 393216
0-524288.rec

then called AeronArchive.attachSegments() with recordingId 0 as a parameter, Aeron Archive would work backwards from 0-393216.rec. The Segment file length here is 128 KB, so it would subtract 128 KB from 393216 and look for a file named 0-262144.rec. It would find the file, see that it starts with a fragment and set startPosition in the Catalog to 262144. It would then subtract 128 KB from that filename and find 0-131072.rec, then set startPosition to 131072. It would subtract a further 128 KB and look for 0-0.rec but wouldn't find that file, so it would stop.

Purge Segments

The same as Detach Segments, but also deletes the detached Segment files.

Purge Recording

Takes a recordingId parameter. The Recording must:

  • be stopped
  • not be being replayed

If that is the case, the RecordingDescriptor is removed from the Catalog and all the Segment files for the recordingId are deleted.

Truncate Recording

Takes recordingId and newStopPosition parameters. The truncate request is validated:

  • there cannot be any active replays
  • the Recording must have a stopPosition (must not be active)
  • newStopPosition must be between the startPosition and stopPosition (inclusive), and be on a Frame boundary

If the request is valid, the stopPosition in the Catalog is set to newStopPosition.

If newStopPosition is not at the start of a Segment file, then everything after newStopPosition in that file is set to zeros. Any whole Segment files after newStopPosition are deleted.

If the newStopPosition == startPosition, this results in all the Segment files being deleted, but the Recording is left in the Catalog.