Replay Merge

todo: work in progress. I think it's accurate, but it needs diagrams and refinement

Multi-Destination Subscriptions are used in replay merge. This is where a live stream of messages is being recorded in Aeron Archive, and a client wants to listen to the live stream, but wants to start from an earlier position in the Recording. It starts listening to the Recording, then when it is near to the Live position (which is a moving target), it listens to both the Recording and Live streams, which are written into the same underlying Image log buffer. When the Recording stream catches up with the Live stream, the Recording stream is stopped and the client continues on just the Live stream. At that point, it has merged from the Recording stream into the Live stream. This technique is also used in Aeron Cluster, when a follower wants to catch up to live Log messages from the leader, but needs to start from an earlier position in the Log Recording.

So what's the problem that Multi-Destination Subscriptions solves?

When a message is received by the Aeron Transport Receiver, the Frame Header at the start of the message contains the StreamId and SessionId that identifies which Image log buffer the message needs writing to. The Frame Header also contains the position that the message needs writing at. Both senders - the Archive and the Live stream sender - need to use the same StreamId and SessionId.

Both senders also want to use their own Publication log buffers. The Live stream sender has a log buffer of a certain size that contains the latest Live messages. The Archive needs to stream messages off disk that may be far behind the current Live position, so it cannot use the Live log buffer.

The problem is that Aeron Transport will not allow more than one Publication log buffer to be created for a given Channel, StreamId and SessionId combination. Once the Live sender has created the Live Publication, the Archive would not be able to create its Publication to replay the Recording to the same Channel, StreamId and SessionId. ConcurrentPublications do allow this, but that would interleave messages from each sender in the same Publication log buffer, which is not what is required.

The solution is to use different Channels. The Subscription listens on two Receive Channel endpoints (ports) for the same StreamId and SessionId. The senders use different Channels when creating their Publications, which creates a separate Publication log buffer for each of them.

The Receive Channel endpoints are referred to as destinations, which is from the point of view of the Publication. A Multi-Destination Subscription is created without an endpoint, and the endpoints are added later. The Subscription needs the control-mode parameter, e.g. aeron:udp?control-mode=manual.

todo: diagram

When I do a diagram: The Live log buffer will have a certain term length. Each log buffer contains 3 terms. The client may want to start listening to the Recording more than 3 terms behind the Live position. The ReplayMerge class and Aeron Cluster don't add the Live stream until the Recording is within 1/4 of a term length of it.