Network Publications - Connecting to a Subscription
On the sending side, when a Network Publication is created, the Sender will repeatedly (every 100 ms) send a SETUP message to the receiving side until the receiving side responds with a Status Message (SM).
On the receiving side, when a Subscription is created for a Network Publication, the Receiver will start listening for a SETUP message for that Subscription.
It doesn't matter which order they are created in. Once they both exist and the Receiver receives the SETUP message, an Image is created for the Subscription, then the Receiver replies with an SM to the Sender. At this point, the connection is said to have been established and the sending side can start sending messages.
Let's dig into the details.
1) Creating a Network Publication
Here's how a Network Publication and its associated log buffer file is created. This also serves as a good example of how the CNC file is used.
Use the tabs above to step through the animation.
The application calls into the Aeron Client API using Aeron.addPublication(channel, streamId) to create a
Concurrent Publication or Aeron.addExclusivePublication(channel, streamId) for an Exclusive Publication.
This is given to the Client Conductor part of the Client API.
The Client Conductor gets the next correlationId from the to-driver buffer (a ManyToOneRingBuffer) in the cnc.dat file. Let's say it returns 23. It then writes a PublicationMessage (i.e. 'add publication') into the to-driver buffer, which contains the Channel, StreamId, and correlationId 23.
The application thread then loops in the Client Conductor (until a timeout), polling for a response in the to-clients buffer that contains correlationId 23.
The Driver Conductor's duty cycle includes polling the to-driver buffer, where it finds the PublicationMessage and starts to process it.
The Driver Conductor creates a SendChannelEndpoint for the Channel, if no other Publications have created one for the Channel already. This manages sending messages on the underlying UdpChannel. It is also used to receive SM, NAK and RTTM response messages from the Receiver. This is given to the Sender.
The Driver Conductor starts using the correlationId as a registrationId. A registrationId is just a unique id for
something created within Aeron. It creates a Publication log buffer, using the registrationId in the filename.
On Linux, this might appear as a file like /dev/shm/.../publications/23.logbuffer.
The Driver Conductor then creates counters in the cnc.dat file that are specific to the publishing part of this
Publication (pub-pos and pub-lmt) and the sending part of this Publication (snd-pos, snd-lmt and snd-bpe).
The Driver Conductor creates a NetworkPublication object to manage everything to do with this Publication within the Media Driver. It is given a reference to the Publication's counters and log buffer.
The Driver Conductor tells the Sender about it. The Sender adds it to the list of Network Publications that it needs to service as part of its duty cycle. From this point, when the Network Publication is serviced, it sends a SETUP message on its Channel every 100 ms, trying to establish a connection with the receiving end.
The Driver Conductor writes a PublicationBuffersReady message to the to-clients buffer using a BroadcastTransmitter. This contains, among other things, correlationId 23, a registrationId (also 23), and the log buffer filename.
When the Client Conductor next polls the to-clients buffer, it reads the PublicationBuffersReady response containing correlationId 23, which it was waiting for. There may be response messages for other clients in the to-clients buffer, which this Aeron client would ignore.
The Client Conductor creates a ConcurrentPublication or ExclusivePublication API object (depending on the method invoked in step 1) with registrationId 23. The Publication API object opens the log buffer file and maps it into memory. The Publication API object is returned to the application, which can use it to publish messages (once connected to the Subscription).
2) Creating a Subscription
On the receiving side, here's how a Subscription is created. Note there is only one kind of Subscription, which handles all Publication types. Subscriptions are not thread safe and should only be used by one thread, but it is possible to create multiple Subscriptions for the same Channel and StreamId.
Use the tabs above to step through the animation.
Creating a Subscription starts as per creating a Publication. The application calls into the Aeron Client API
using Aeron.addSubscription(channel, streamId), the Client Conductor puts this into a message in the to-driver
buffer, then waits for a response. Let's say the correlationId is 47.
The Driver Conductor polls the to-driver buffer, reads the SubscriptionMessage and starts to process it.
As the Channel in the SubscriptionMessage is for a Network Publication, it needs a ReceiveChannelEndpoint to handle messages read from the underlying UDP channel. The Driver Conductor creates one if one doesn't already exist. Messages for different StreamIds can pass over the same Channel, so the ReceiveChannelEndpoint may have been created by an earlier Subscription.
The Driver Conductor creates a NetworkSubscriptionLink. Whenever a new Publication is created, this is used to find matching Subscribers. The correlationId 47 is used as the registrationId for the NetworkSubscriptionLink.
The Driver Conductor tells the Receiver to start looking out for messages on the ReceiveChannelEndpoint for the Subscription's StreamId.
The Driver Conductor sends a SubscriptionReady response to the ClientConductor to say the Subscription is ready.
It contains registrationId 47 and the id of a
ReceiveChannelStatus
counter rcv-channel for the 'connected' status of the ReceiveChannelEndpoint.
If other Subscriptions already exist for the same Channel and StreamId and there are Image files, the Image files would be linked to this Subscription at this point. This is not the case in this example. Image files are discussed in the section on Setup Connection below.
The Client Conductor polls the to-clients buffer and reads the SubscriptionReady response.
The Client Conductor creates a Subscription API object and returns it to the application. The application can poll the Subscription for messages, but it won't receive anything. The Subscription still needs linking to one or more Image log buffer files, which happens after the Publication connects.
3) Setup Connection - receiving side
Once both the Publication and Subscription have been created, when the sending side sends a SETUP message for the new Network Publication, the receiving side will be ready to process it. Let's see the final pieces on the receiving side.
Note that this is a simple example. It is possible for there to be many Network Publications on one or more sending machines, targeting the same Channel and StreamId on the receiving machine. Each Network Publication would have its own Image log buffer on the receiving machine. Also, it's possible for the receiving machine to have more than one Subscription for the Channel and StreamId. Each Subscription would share access to the same Image log buffer files.
Use the tabs above to step through the animation.
When the Receiver receives the SETUP message from the sending machine, it asks the Driver Conductor to create a Publication Image. This will become a replica of the Network Publication log buffer on the sending machine.
The Driver Conductor creates a sub-pos counter for every SubscriptionLink that matches the Channel and StreamId
in the SETUP message. If there was more than one Subscription created before the SETUP message was processed, each
one would get a sub-pos counter to track its position in the Image. In this example, there was only one. It has
the Subscription's registrationId 47.
The Driver Conductor creates an Image log buffer file. An Image log buffer can be read by more than one Subscription, so it needs its own registrationId. This is created like the others: by getting the next correlationId from the to-drivers RingBuffer. This single source of registrationIds can be useful in understanding the creation order of different objects within Aeron.
The Driver Conductor creates rcv-pos and rcv-hwm counters. These have the Image's registrationId as they are
tracking the Receiver's positions in the Image that it will be rebuilding.
The Driver Conductor creates a PublicationImage object to keep track of everything. A reference to the PublicationImage is given to the Receiver, so the Receiver can forward data packets to it. Note that the PublicationImage is used by the Driver Conductor and Receiver, which may run on different threads, so careful consideration is given to which parts are used by each.
Part of the Receiver's duty cycle is to ask each PublicationImage to send a Status Message (SM) if it needs to. The PublicationImage does so, as it hasn't sent one before. The SM contains a window length field, which tells the Sender how many more bytes the Receiver will accept for this PublicationImage at this moment in time.
The Driver Conductor tells the Client Conductor that Image 51 is available for Subscription 47. If there were other
Subscriptions interested in the Image, the Driver Conductor would send an ImageReady message for each. The message
also contains the Image log buffer filename and the id of the sub-pos counter.
The Client Conductor polls the to-clients buffer for unsolicited messages and reads the ImageReady message.
It creates an Image object to manage the Image log buffer and sub-pos counter. Note that
this Image object is specific to Subscription 47. If there were more than one Subscription, there would be an
Image object for each of them, even though there's only one Image log buffer.
The Image is then added to the Subscription. Note that the sub-pos counter is for the Subscriber's position
within Image 51. If the Subscription had more than one Image, it would have a sub-pos counter for each
of them. When the application called addSubscription(), it could have provided an
AvailableImageHandler
and an
UnavailableImageHandler.
If it did, the AvailableImageHandler would be called at this point.
4) Setup Connection - sending side
Use the tabs above to step through the animation.
Part of the Sender's duty cycle is to poll each SendChannelEndpoint for response messages from the Receiver. The Receiver can send SM, NAK and RTTM messages. When the Sender receives an SM, it it gives it to the NetworkPublication (still on the Sender's thread), which it looks up by SessionId and StreamId in the message.
The SM contains a receiver window field to indicate how much data the receiving side can receive at present. The
NetworkPublication (still on the Sender thread) uses that to set snd-lmt to control how much data the Sender
can send. If the Sender hits snd-lmt, it increments snd-bpe (Sender back-pressure events). The
NetworkPublication also sets isConnected=true in the log buffer metadata.
Once connected, next time the Driver Conductor performs its duty cycle and checks on the NetworkPublication, the
NetworkPublication sets pub-lmt to snd-pos + termWindowLength (zero + half a term).
At this point, a connection has been established and the client can start publishing messages. After connecting, the
Receiver will continue to send SMs periodically, which will allow the Sender to increase snd-lmt.
Multiple Publications
The above scenario is for a single Publication connecting to a single Subscription. If a second Publication was created for the same Channel and StreamId, and if it had a different SessionId (i.e. it's an Exclusive Publication, or it is coming from a different machine), it will send a SETUP message. On the receiving side, it will create a new Image, as it's for a different SessionId, and the Image will be linked to the existing Subscription. When the application polls the Subscription, it polls each of the connected Images (polling in a round-robin fashion, so the first one is different each time).