Skip to content

Recording

Here's the same diagram from the previous page. Within Aeron Archive, we have a RecordingSession that is responsible for reading messages from a Log Buffer (circled 'Pub') and writing them to a Recording. A RecordingWriter is used to write the Recording Segment files.

Media Driver Sender Conductor Receiver Market Data Collector Pub Pub Machine 1 Aeron Archive Spy Sub Recording

The RecordingSession uses its Image object to poll for new fragments from the Log Buffer. When polling, it uses a blockPoll() method with a BlockHandler to poll for blocks of fragments, rather than the regular poll() that returns fragments one at a time.

In this case, the BlockHandler is the RecordingWriter, which writes each block of fragments to disk. The RecordingWriter knows the length of each Segment file and creates a new file each time the current one is full. BlockHandlers are never called with a block that spans a Term boundary. The Segment file length is a multiple of Term length, so there's never an issue with having to create a new Segment file in the middle of processing a block.

After each block of fragments is written to disk, the rec-pos counter is updated with the new position.

Recording Detail

TermLength: 2 MB Metadata: InitialTermId: 1005 ActiveTermCount: 3 ActiveTermCount: 2 ActiveTermCount: 1 ActiveTermCount: 0 TermId: 1007 TermId: 1004 TermId: 1006 TermId: 1008 TermId: 1005 TermId: 1003 Term 2 Term 1 Term 0 0-4194304.rec 4 MB 6 MB 8 MB TermId: 1008 TermId: 1007 0-0.rec 0 MB 2 MB 4 MB TermId: 1006 TermId: 1005 RecordingSession rec-pos cnc.dat

Use the tabs above to step through the animation.


Above is an empty Log Buffer containing three Terms, each with a Term Length of 2 MB. Below it is the first Segment file from a Recording that's about to record from the Log Buffer. Segment file length here is 2 * Term Length, so 4 MB. The Recording is going to start from the beginning of the Publication, so rec-pos is 0 and the filename is 0-0.rec.

As the Publication writes messages to Term 0, the RecordingSession receives the messages from its Image and writes them into the Segment file. The diagram shows a snapshot after this has been happening for a while and is not intended to depict one large batch operation. Messages are copied as they are published into the Log Buffer. rec-pos is advanced after each write.

As more messages are written to the Log Buffer, the RecordingSession continues to copy them to the Segment file. They are copied verbatim - nothing is changed (unless Checksums are enabled), so if you were to diff the first Term in the Log Buffer with the first Term in the Segment file, they would be identical. Even the PAD frame at the end of Term 1 is copied (small sliver of grey in the diagram).

When the RecordingSession gets to the end of Term 1, the Segment file is full, so it creates a new one. The new Segment file starts from position 4 MB (4194304 bytes) into the Recording (which is also the Term Offset of the start of Term 2), so its filename is 0-4194304.rec. Messages written to the start of Term 2 are then copied to the start of the second Segment file.

Recording continues. When the Publication wraps around the Log Buffer and starts using Term 0 again, the RecordingSession continues copying to the second Segment file. This process continues, with new Segment files being created as necessary.

Recording Events Channel

When a User Application starts or stops a Recording, a RecordingSignalEvent message is sent to it on its Control Response Channel. Aeron Archive also contains a Recording Events Channel (which is disabled by default), which can be configured to broadcast recording events to multiple clients (on multiple hosts) if required.

If enabled, the Recording Events Channel can be set to multicast (if supported on the underlying network) or MDC (Multi-Destination Cast). Recording Events comprise:

  • RecordingStarted (recordingId, channel, streamId, sessionId, startPosition, sourceIdentity (e.g. 'aeron:ipc' or host:port))
  • RecordingProgress (recordingId, startPosition, position) - sent on every write, after rec-pos is updated
  • RecordingStopped (recordingId, startPosition, stopPosition)

Recording Events are sent by the RecordingSession.

Counters

The Recorder can contain multiple RecordingSessions, each of which has a RecordingWriter. Whenever a RecordingWriter writes to a Segment file, it reports the number of bytes written and time taken to write them (in nanos) back to the Recorder. The Recorder uses this information to maintain three Counters in the cnc.dat file (example from AeronStat):

  44:              120,048 - archive-recorder max write time in ns - archiveId=1
  45:               31,744 - archive-recorder total write bytes - archiveId=1
  46:            1,353,831 - archive-recorder total write time in ns - archiveId=1

The archiveId above is taken from the clientId in the Aeron Client that Aeron Archive uses to talk to the Media Driver (described in cnc.dat). If there was more than one instance of Aeron Archive running on the same machine, using the same Media Driver, it would have a different archiveId.