Skip to content

Log Messages

Clustered Service Consensus Module Archive log Clustered Service Consensus Module Archive log Client B Client A Network cnc.dat cnc.dat cnc.dat cnc.dat cons-module service control-response control-request control-response control-request cons-module service consensus0-1 consensus2-1 consensus1-0 consensus1-2 consensus1-0 consensus2-0 consensus0-1 consensus0-2 log log egressB ingressB egressA ingressA ingressB ingressA egressA egressB Conductor Receiver Sender Conductor Receiver Sender Receiver Conductor Sender Receiver Conductor Sender StreamId: 102 StreamId: 104 StreamId: 105 StreamId: 100 StreamId: 108 StreamId: 101 LEADER FOLLOWER

Log messages are written to the Log by the cluster leader. This is the Raft Log. The main source of Log messages is clients of the Cluster, which send messages to the leader Consensus Module on the Ingress. The leader sequences these into a single sequence, which is the Log. Log messages can also originate from the Clustered Service, which can publish messages back into the cluster, and timers - when a timer fires on the leader, it adds a TimerEvent to the Log, so it is processed by all Clustered Services at the same position in the Log.

Log messages are consumed by the Consensus Modules, as well as the Clustered Services. The Consensus Modules maintain a list of client sessions, which need their timeOfLastActivityNs updating. The leader Consensus Module does not process messages by reading them from the Log - it adds messages to the Log and actions them immediately. This does not break consensus as it's more to do with maintaining the live Ingress and Egress connections to clients.

In the follower Consensus Modules, messages are received by LogAdapter, which forwards them to ConsensusModuleAgent. Followers use the Log messages to maintain a list of client sessions like on the leader, but they do not have live connections. The followers are ready to take over the connections if the leader dies.

In the Clustered Service, messages are received by BoundedLogAdapter, which forwards them to ClusteredServiceAgent, which forwards them to the ClusteredService application. The BoundedLogAdapter uses the commit-pos Counter as an upper bound when polling the Log.

The Log Publication is an MDC (multi-destination cast) channel, which means the Sender is set up to send to multiple recipients - the followers. It does not have a separate Publication for each follower.

The Log on each cluster member is also recorded by the Archive to the Log Recording.

The Log messages are described below.

SessionMessageHeader

This is the same SessionMessageHeader as on the Ingress, but copied onto the Log. When the leader Consensus Module uses the IngressAdapter to poll the Ingress Subscription, that iterates over all the underlying Ingress log buffers from different clients (in a different order each time). For each SessionMessageHeader it read, it calls onIngressMessage(). That appends it to the Log using the LogPublisher, also setting the timestamp to the current cluster time.

SessionMessageHeader {
    leadershipTermId  long  // current leadership term
    clusterSessionId  long  // identifies the session
    timestamp         long  // set from the cluster clock (epoch time in cluster time units)
}

It is received in the follower Consensus Modules by the LogAdapter, which calls onReplaySessionMessage(). All that does is update the ClusterSession's last activity timestamp.

It is received in all the Clustered Services by the BoundedLogAdapter, which calls onSessionMessage(), which delivers it to the ClusteredService. Note that this is a BoundedLogAdapter, which means it will only deliver messages up to the bound, which is the commit-pos Counter.

SessionOpenEvent

Appended to the Log by the leader Consensus Module when a client successfully connects. It tells the follower Consensus Modules and the Clustered Services about the new session.

Appended by the ConsensusModuleAgent in processPendingSessions() when a pending session becomes authenticated.

SessionOpenEvent {
    leadershipTermId  long    // current leadership term on the leader
    correlationId     long    // from the client's SessionConnectRequest
    clusterSessionId  long    // session id of the client connection
    timestamp         long    // set from the cluster clock (epoch time in cluster time units)
    responseStreamId  int     // egress streamId for the cluster to connect back to the client
    responseChannel   String  // egress channel for the cluster to connect back to the client
    encodedPrincipal  byte[]  // from the Authenticator
}

In the follower Consensus Modules, the message is received by LogAdapter, which calls onReplaySessionOpen(). That adds a new ClusterSession, bringing the follower's list of ClusterSessions in sync with the leader.

In all the Clustered Services, the message is received by the BoundedLogAdapter, which calls onSessionOpen(), which calls ClusteredService.onSessionOpen(), notifying the application of the new client connection.

SessionCloseEvent

Appended to the Log by the leader Consensus Module when a client session has been closed. It tells the follower Consensus Modules and the Clustered Services. It could have been closed by the client, the Clustered Service, or an inactivity timeout.

It is appended in a number of places by the ConsensusModuleAgent.

SessionCloseEvent {
    leadershipTermId  long         // current leadership term on the leader
    clusterSessionId  long         // session id of the client connection
    timestamp         long         // set from the cluster clock (epoch time in cluster time units)
    closeReason       CloseReason  // enum: CLIENT_ACTION, SERVICE_ACTION, TIMEOUT
}

In the follower Consensus Modules, the message is received by LogAdapter, which calls onReplaySessionClose(). That removes the ClusterSession, bringing the follower back in sync with the leader.

In all the Clustered Services, the message is received by the BoundedLogAdapter, which calls onSessionClose(), which calls ClusteredService.onSessionClose(), notifying the application that the client has disconnected.

TimerEvent

Appended to the Log by the leader Consensus Module when a timer expires (fires). It tells the follower Consensus Modules and the Clustered Services that the timer has fired. The Clustered Services will all process the timer at the same position in the Log, relative to other messages.

It is appended by the ConsensusModuleAgent in onTimerEvent(), which is called by the TimerService.

TimerEvent {
    leadershipTermId  long  // current leadership term on the leader
    correlationId     long  // timer id used when adding the timer
    timestamp         long  // set from the cluster clock (epoch time in cluster time units)
}

In the follower Consensus Modules, the message is received by LogAdapter, which calls onReplayTimerEvent(). That just cancels the timer (because it has fired), bringing the follower's TimerService in sync with the leader.

In all the Clustered Services, the message is received by the BoundedLogAdapter, which calls onTimerEvent(), which calls ClusteredService.onTimerEvent(), notifying the application.

ClusterActionRequest

Appended to the Log by the leader Consensus Module when it checks the Cluster Control Toggle Counter and finds that ClusterTool has given it an action to perform. A user can use ClusterTool to set the Counter to one of the following actions (SNAPSHOT can also be set by an AdminRequest on the Ingress):

  • SUSPEND - suspend Ingress and Timers, i.e. stop appending to the Log
  • RESUME - undo SUSPEND
  • SNAPSHOT - take a snapshot
  • SHUTDOWN - take a snapshot and perform an orderly shutdown
  • ABORT - perform an orderly shutdown
  • STANDBY_SNAPSHOT - trigger a snapshot that will only occur on a cluster standby member

The Counter can also hold the values:

  • INACTIVE - not accepting new actions
  • NEUTRAL - ready to accept a new action

It is appended to the Log so that all cluster members perform the action at the same point in the Log.

It is appended from checkClusterControlToggle().

ClusterActionRequest {
    leadershipTermId  long          // current leadership term on the leader
    logPosition       long          // position of this message, which is where the snapshot should be taken at
    timestamp         long          // set from the cluster clock (epoch time in cluster time units)
    action            ClusterAction // enum, values listed above
    flags             int           // only set for STANDBY_SNAPSHOT
}

In the follower Consensus Modules, the message is received by LogAdapter, which calls onReplayClusterAction(). That sets the ConsensusModule state to SUSPENDED, ACTIVE or SNAPSHOT, to make the follower's state match the leader's.

In all the Clustered Services, the message is received by the BoundedLogAdapter, which calls onServiceAction(), which calls executeAction(). If the action is SNAPSHOT, it takes a snapshot, and once complete, sends a ServiceAck to the Consensus Module, confirming it has taken the snapshot at the logPosition.

NewLeadershipTermEvent

Appended by the leader Consensus Module when it's in the LEADER_READY Election state, i.e. the Election is about to complete. This is not to be confused with the NewLeadershipTerm Consensus message, which can be sent multiple times earlier in the Election, or even outside an Election. This message is a permanent record of the new leadership term on the Log, which will be replayed whenever the Log is replayed.

Appended by the ConsensusModuleAgent in appendNewLeadershipTermEvent(), which is called from the Election object in leaderReady().

NewLeadershipTermEvent {
    leadershipTermId     long             // current leadership term on the leader
    logPosition          long             // position of this message, 
    timestamp            long             // set from the cluster clock (epoch time in cluster time units)
    termBaseLogPosition  long             // start position of the leadership term in the Log
    leaderMemberId       int              // leader's id
    logSessionId         int              // Aeron Transport sessionId for the Log Publication
    timeUnit             ClusterTimeUnit  // enum: MILLIS, MICROS or NANOS
    appVersion           int              // user assigned application version
}

In the follower Consensus Modules, the message is received by LogAdapter, which calls onReplayNewLeadershipTermEvent(). That stores the leadershipTermId and updates the Election object.

In all the Clustered Services, the message is received by the BoundedLogAdapter, which calls onNewLeadershipTermEvent(), which calls ClusteredService.onNewLeadershipTermEvent(), notifying the application of the new leadership term.