Message Handling

This page looks at how the Consensus Module handles inbound messages. Here's another copy of the final structure of the Consensus Module after starting:

LeaderFollower

The leader has a Log Publication for publishing Log messages, and Egress Publications for publishing Egress messages to any connected clients. It has a LogAdapter, but it isn't used as it doesn't subscribe to Log messages.

The follower has a Log Subscription for its LogAdapter, which it uses to receive Log messages from the leader.

ConsensusAdapter¶

The ConsensusAdapter is used to receive Consensus messages from other members of the Cluster. During an Election, this will be from all of the other members, but after an Election, it will only be between the leader and followers. Followers don't talk to each other after the Election is complete.

The ConsensusModuleAgent polls the ConsensusAdapter, which polls the Consensus Subscription. For each message, the ConsensusAdapter unpacks the field values and calls a corresponding method on the ConsensusModuleAgent, e.g. CanvassPosition goes to onCanvassPosition(). This is all the ConsensusAdapter does.

It handles the following messages:

CanvassPosition
RequestVote
Vote
NewLeadershipTerm
AppendPosition
CommitPosition
CatchupPosition
StopCatchup
TerminationPosition
TerminationAck
BackupQuery (from ClusterBackupAgent)
ChallengeResponse (from ClusterBackupAgent)
HeartbeatRequest (from Cluster Standby)
StandbySnapshot (part of Cluster Standby)

The following messages are in the schema, but are now ignored as dynamic join has been removed (1.41.0: Deprecate cluster dynamic join feature. This is to be replaced with a more robust and user-friendly premium offering - PR)

SnapshotRecordingQuery
AddPassiveMember
ClusterMembersChange
SnapshotRecordings
JoinCluster

IngressAdapter¶

All Cluster members use the IngressAdapter to receive Ingress messages from Cluster clients. The ConsensusModuleAgent polls the IngressAdapter, which polls the Ingress Subscription. For each message, the IngressAdapter unpacks the field values and calls a corresponding method on the ConsensusModuleAgent, e.g. SessionConnectRequest goes to onSessionConnect(). This is all the IngressAdapter does.

The method on the ConsensusModuleAgent checks whether the Cluster member is the leader or a follower and takes the appropriate action.

Application level messages from Cluster clients are wrapped in a SessionMessageHeader. These are passed to the ConsensusModuleAgent via onIngressMessage(). It ignores them if it's a follower. On the leader, it looks up the ClusterSession using the clusterSessionId in the message and checks that it's still OPEN (i.e. not in the process of closing). It then reads the current time off the cluster clock (the leader has the 'official' clock). It publishes the same type of SessionMessageHeader to the Log using the LogPublisher, copying all the values from the Ingress, but setting the timestamp to the current time, ignoring the timestamp on the Ingress message. This time will be used as the clock time when the message is processed, whether that's by the leader's Clustered Service, a follower's Clustered Service, or if the message is replayed some time later.

It handles the following messages:

LogAdapter¶

Log messages are processed by both the Consensus Modules and the Clustered Services. The Consensus Modules deal with session handling and the Clustered Services deal with application messages. The LogAdapter is only used in the Consensus Modules. The Clustered Service handling of Log messages is discussed later, here.

The LogAdapter is used by the Consensus Modules to receive Log messages in two places:

in the follower Consensus Modules to receive live Log messages from the leader's Log Publication
in any Consensus Module to receive Log messages from a Log Recording when replaying the Log during an Election

It is not used on the leader to receive live Log messages from its own Log Publication. The leader does whatever processing it needs to perform before publishing the message to the Log. Note that this does not subvert the Raft principle of only processing Log messages once quorum has been achieved on them, because the processing in the Consensus Module is only related to a client's connection, e.g setting up a new session, updating the last activity timestamp, etc. It is not executing application business logic.

The LogAdapter is polled in 3 places:

in a follower ConsensusModuleAgent's duty cycle to process live messages
in a follower or leader Election, when replaying a Log Recording (covered later in Elections)
in a follower Election, when catching up from the leader's Log Recording (covered later in Elections)

When the LogAdapter is polled, it polls the underlying log buffer, which can be the follower's Log Image, or a replication log buffer (covered later in Elections). For each message, the LogAdapter unpacks the field values and calls a corresponding method on the ConsensusModuleAgent, e.g. SessionMessageHeader goes to onReplaySessionMessage(). The methods called by the LogAdapter are all prefixed onReplay.

It handles the following messages:

ConsensusModuleAdapter¶

The ConsensusModuleAdapter is used to receive Consensus Module messages from its Clustered Service(s). These are not to be confused with Consensus messages that are sent between Consensus Modules. There is one Consensus Module message that can also come from ClusterTool. These are requests (or an ack) from the Clustered Service:

the Clustered Service can request that a message is published onto the Log, as a consequence of processing another message, or on a timer
it can request that a timer is added or removed
it can request that a ClusterSession is closed

The ConsensusModuleAgent polls the ConsensusModuleAdapter, which polls the Consensus Module Subscription. For each message, the ConsensusModuleAdapter unpacks the field values and calls a corresponding method on the ConsensusModuleAgent, e.g. ScheduleTimer goes to onScheduleTimer(). This is all the ConsensusModuleAdapter does.

The ConsensusModuleAdapter is polled in the regular duty cycle of the ConsensusModuleAgent, and at several points during an Election to check for a ServiceAck from the Clustered Service.

It handles the following messages:

SessionMessageHeader (from ClusteredService)
ScheduleTimer (from ClusteredService)
CancelTimer (from ClusteredService)
ServiceAck (from ClusteredService)
CloseSession (from ClusteredService)
ClusterMembersQuery (from ClusterTool)