Command and Control file (cnc.dat)
As stated in the Detailed Overview, the cnc.dat
file is the Command and Control file that is central to Aeron.
Again, remember that this file is memory mapped into the Aeron Clients and the Media Driver, so they can all read /
write different sections just by accessing different parts of a buffer, even if they're in different processes.
The file contains three unrelated sections: buffers for communicating with the Media Driver, counters, and an error log. Two of these sections are further divided and a metadata section is added, but it essentially holds three sets of data.
The sections have different lengths. Each section's length is configurable and is calculated when the Media Driver starts, after which it cannot be changed. This means each section will have a fixed position within the file, so it's possible to overlay classes like ManyToOneRingBuffer over a section.
The sections are as follows:
Metadata
The Metadata section is further broken down into the following layout, as described in CncFileDescriptor:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Aeron CnC Version |
+---------------------------------------------------------------+
| to-driver buffer length |
+---------------------------------------------------------------+
| to-clients buffer length |
+---------------------------------------------------------------+
| Counters Metadata buffer length |
+---------------------------------------------------------------+
| Counters Values buffer length |
+---------------------------------------------------------------+
| Error Log buffer length |
+---------------------------------------------------------------+
| Client Liveness Timeout |
| |
+---------------------------------------------------------------+
| Driver Start Timestamp |
| |
+---------------------------------------------------------------+
| Driver PID |
| |
+---------------------------------------------------------------+
If you start the Media Driver with default settings, the first 48 bytes of the cnc.dat file (the Metadata section) contains:
00000000 00 02 00 00 00 03 10 00 80 00 10 00 00 00 40 00 |..............@.|
00000010 00 00 10 00 00 00 10 00 00 e4 0b 54 02 00 00 00 |...........T....|
00000020 c1 46 51 d4 8b 01 00 00 99 ec 04 00 00 00 00 00 |.FQ.............|
This decodes to the following metadata (further details below):
- Aeron CnC Version (0x00000200): major: 0, minor: 2, patch: 0 (most significant byte unused)
- to-driver buffer length (0x00100300): the to-driver buffer is used as a ManyToOneRingBuffer with 1MB for data and 0x300 bytes for metadata
- to-clients buffer length (0x00100080): the to-clients buffer is used for a BroadcastTransmitter/Receiver with 1MB for data and 0x80 bytes for metadata
- Counters Metadata buffer length (0x00400000): 4MB, which is 4 x Counters Values buffer length
- Counters Values buffer length (0x00100000): 1MB, which is enough for 32768 counters, each consuming 128 bytes
- Error Log buffer length (0x00100000): 1MB
- Client Liveness Timeout (0x00000002540be400): 10,000,000,000 nanos = 10 seconds
- Driver Start Timestamp (0x0000018bd45146c1): 1700074178241 epoch time millis (15 Nov 2023 18:49:38.241) - when the Media Driver was started
- Driver PID (0x000000000004ec99): 322713, which was the process id of the Media Driver
The default cnc.dat file is 8392704 bytes long:
- metadata: 48 bytes, aligned (rounded up) to 128 bytes = 128 bytes
- to-driver buffer: 0x00100300 (1049344 bytes)
- to-clients buffer: 0x00100080 (1048704 bytes)
- Counters Metadata buffer: 4MB (4194304 bytes)
- Counters Values buffer: 1MB (1048576 bytes)
- Error Log buffer: 1MB (1048576 bytes)
- Total: 8389632, which is then rounded up to the next file page size (4KB): 8392704
to-driver / to-clients Buffers
When an Aeron Client wants to send a command to the Media Driver, e.g. to ask it to create a Publication, it writes it to the to-driver buffer. The Media Driver writes response messages to the to-clients buffer.
Note that there can be more than one Aeron Client using the Media Driver on a machine. This means multiple Clients need to be able to write commands to the to-driver buffer and Clients need to identify the correct response in the to-clients buffer, ignoring responses for other Clients.
The to-driver buffer is wrapped by a ManyToOneRingBuffer, which manages this section of the cnc file. This allows multiple Clients (in different processes) to write commands into the RingBuffer safely, without overwriting each other.
Question
Hold on, if the Media Driver and Aeron Clients are all in different processes, how do they share an instance of the ManyToOneRingBuffer class?
They don't!
They each have their own instance of the ManyToOneRingBuffer class. The instances are flyweights - their state is in the shared memory buffer, not in instance fields. The buffer contains an area for messages and another area for the ring buffer metadata, which includes the head and tail positions, next correlationId, etc. ManyToOneRingBuffer uses atomic 'compare and set' instructions to modify the metadata values in a thread-safe way.
The RingBuffer has a nextCorrelationId()
method for generating a correlationId (just an incrementing number), which
Clients add to their commands. When the Media Driver writes a response message into the to-clients buffer, it includes
the correlationId, so Clients can identify the correct response.
The nextCorrelationId()
method is used whenever an id that is unique within a given Media Driver is required. For
instance, whenever a client connects to the Media Driver, it calls nextCorrelationId()
to generate a clientId
for itself. Clients add the clientId
to every message they send to the Media Driver to identify themselves.
The to-clients buffer needs to support a single producer (the Media Driver) and many consumers (Aeron Clients). There is no RingBuffer implementation that supports one-to-many semantics, so the to-clients buffer uses a broadcast abstraction, mapped over the buffer. The Media Driver uses a BroadcastTransmitter to write to the to-clients buffer and each Aeron Client API uses a BroadcastReceiver to poll messages from the buffer.
Counters Buffers
The Media Driver maintains a lot of counters (64-bit numbers) that it writes to the cnc.dat file.
For example, when a message is sent from one Aeron Client to another, the message is first written to a Publication
log buffer file. Each Publication has a publication position counter pub-pos
in the cnc.dat file, which tracks how
far into the log buffer the Aeron Client has written. For a Network Publication, there is another counter for the
Sender's position snd-pos
in the log buffer. The Receiver and any Subscriptions add more counters and if you are
using Aeron Cluster, it adds yet more counters.
Position Counters
Aeron Docs describe the position counters well on the Understanding Position page, so I won't cover that again here. Note that the Aeron Docs description is for Network Publications. IPC Publications don't have counters for the Sender and Receiver as they are not used.
The content of the Counters Buffers is described in CountersReader.
The counters are split across two sections in the cnc.dat file. Each counter has a fixed length record in the Counters Metadata Buffer to store its name, id and other metadata. Each counter has a counterId, which can be multiplied by the length of a Metadata record to find its offset in the Metadata Buffer. Similarly, each counter has a fixed length area in the Counters Values Buffer to store its 64-bit value. Each value area is actually 128 bits long, with 64 bits of unused padding to spread them out. The reason for this organisation is to avoid false sharing.
False Sharing
False sharing is where independent variables in different memory locations fall on the same cache line. This negatively impacts performance when one thread is writing to one variable and another thread is writing to another variable on the same cache line, causing cache line contention.
See False Sharing.
Error Log
Finally, at the end of cnc.dat lies the Error Log section. It contains a record of any errors that occur within Aeron components like the Media Driver, Aeron Archive or Aeron Cluster. This part of the cnc file is wrapped in a DistinctErrorLog, so if an exception occurs multiple times, the error log will contain one copy of the exception, plus the time of the first and last occurrence and the number of occurrences.