Skip to content

Consensus Module Messages

Clustered Service Consensus Module Archive log Clustered Service Consensus Module Archive log Client B Client A Network cnc.dat cnc.dat cnc.dat cnc.dat cons-module service control-response control-request control-response control-request cons-module service consensus0-1 consensus2-1 consensus1-0 consensus1-2 consensus1-0 consensus2-0 consensus0-1 consensus0-2 log log egressB ingressB egressA ingressA ingressB ingressA egressA egressB Conductor Receiver Sender Conductor Receiver Sender Receiver Conductor Sender Receiver Conductor Sender StreamId: 102 StreamId: 104 StreamId: 105 StreamId: 100 StreamId: 108 StreamId: 101 LEADER FOLLOWER

Consensus Module messages are sent from the Clustered Service to the Consensus Module via an IPC Publication. Within the Consensus Module, the ConsensusModuleAgent polls its ConsensusModuleAdapter to check for new Consensus Module messages. The ConsensusModuleAdapter polls the ConsensusModule Subscription, decodes any messages and calls methods on the ConsensusModuleAgent.

Cluster Tool Consensus Module cons-module service

In addition, when ClusterTool is used, it connects to the Consensus Module using the same IPC streams, so it can also send Consensus Module messages (see ClusterTool for more details). It sends a different set of messages than the Clustered Service.

The Consensus Module messages are described below.

SessionMessageHeader

This is the same SessionMessageHeader as on the Ingress, but this is an application level message that the Clustered Service wants publishing to the Log. This might be useful if you are running more than one Clustered Service and need to send a message from one to another. All Clustered Services would receive it, so the sender would have to ignore its own message.

ScheduleTimer

If the ClusteredService calls Cluster.scheduleTimer(), the ClusteredServiceAgent (which implements the Cluster interface) sends a ScheduleTimer message to the Consensus Module. Each timer is identified by a correlationId. It is up to the ClusteredService to map this to the work that it wants to perform at the scheduled time. See Time for cluster time units.

ScheduleTimer {
    correlationId  long
    deadline       long (epoch time in cluster time units)
}

When the ConsensusModuleAgent performs its consensus work, it polls the Consensus Module Subscription using the ConsensusModuleAdapter. When it finds the ScheduleTimer message, the ConsensusModuleAdapter calls ConsensusModuleAgent.onScheduleTimer(), which adds the timer to its TimerService. This happens on all cluster members, so their TimerServices should eventually have the same contents. It doesn't matter if the followers have a different set of timers momentarily, because only the leader acts on them.

The leader polls its TimerService for timers that have elapsed (again, in consensus work). When a timer elapses, the TimerService calls ConsensusModuleAgent.onTimerEvent(), which appends a TimerEvent to the Log. This is on the Log, so it will be replicated to the followers in the same way as Ingress messages. See TimerEvent to continue the description of how timers work.

CancelTimer

If the ClusteredService calls Cluster.cancelTimer(), the ClusteredServiceAgent (which implements the Cluster interface) sends a CancelTimer message to the Consensus Module.

CancelTimer {
    correlationId  long
}

When the ConsensusModuleAgent performs its consensus work, it polls the Consensus Module Subscription using the ConsensusModuleAdapter. When it finds the CancelTimer message, the ConsensusModuleAdapter calls ConsensusModuleAgent.onCancelTimer(), which cancels the timer from its TimerService. This happens on all cluster members, so their TimerServices should eventually have the same contents. It doesn't matter if the followers have a different set of timers momentarily, because only the leader acts on them.

ServiceAck

The Clustered Service sends a ServiceAck to the Consensus Module to acknowledge that it has reached a particular position in the Log. It does this in a number of places:

  • during startup, after it has recovered state (loaded a snapshot if there was one) and reached the logPosition in the RecoveryState Counter set by the Consensus Module (the snapshot position or zero)
  • in response to the Consensus Module sending a JoinLog message, acknowledging the Log position that it joined at
  • after processing a ClusterActionRequest on the Log that requests the cluster member take a snapshot, it acknowledges the Log position that the snapshot was taken at
  • in response to the Consensus Module sending a RequestServiceAck, explicitly requesting a ServiceAck when the Clustered Service has reached the requested Log position
  • in response to the Consensus Module sending a ServiceTerminationPosition, asking the Clustered Service to terminate when it reaches the requested termination Log position
ServiceAck {
    logPosition  long
    timestamp    long (epoch time in cluster time units)
    ackId        long
    relevantId   long
    serviceId    int
}

When the ConsensusModuleAgent performs its consensus work, it polls the Consensus Module Subscription using the ConsensusModuleAdapter. When it finds the ServiceAck message, the ConsensusModuleAdapter calls ConsensusModuleAgent.onServiceAck(), which stores the ack so it can be found by the task that was waiting for it. It may also perform subsequent actions, such as snapshotting the Consensus Module if the ServiceAck was the Clustered Service acknowledging that it has taken its snapshot.

CloseSession

If the ClusteredService calls Cluster.closeClientSession() or ClientSession.close() to close a client session (connection), these are both implemented in the ClusteredServiceAgent, which sends a CloseSession message to the Consensus Module.

CloseSession {
    clusterSessionId  long
}

When the ConsensusModuleAgent performs its consensus work, it polls the Consensus Module Subscription using the ConsensusModuleAdapter. When it finds the CloseSession message, the ConsensusModuleAdapter calls ConsensusModuleAgent.onServiceCloseSession(), which marks the ClusterSession as CLOSING (this happens on all cluster members).

Then, if it's the leader, it:

  • appends a SessionCloseEvent to the Log. See that message for details, but it essentially closes its Egress Publication if it has one, then removes the ClusterSession
  • sends a SessionEvent on the Egress with EventCode CLOSED to the client, to say the session is being closed by the Clustered Service
  • adds it to uncommittedClosedSessions
  • closes and removes the ClusterSession

ClusterMembersQuery

This is sent by ClusterTool when invoked with the list-members parameter. ClusterTools reads the Cluster mark file to find the Channel and StreamId of the Consensus Module and Service streams to connect to them, then sends the ClusterMembersQuery message.

ClusterMembersQuery {
    correlationId  long
    extended       Boolean
}

When the ConsensusModuleAgent performs its consensus work, it polls the Consensus Module Subscription using the ConsensusModuleAdapter. When it finds the ClusterMembersQuery message, the ConsensusModuleAdapter calls ConsensusModuleAgent.onClusterMembersQuery(). If the extended field is true, it returns a ClusterMembersExtendedResponse on the Service stream, otherwise it returns a ClusterMembersResponse.

Note

At present, ClusterTool hardcodes extended to true