Skip to content

Clients

This page covers the Consensus Module end of a client connection. See also the Client side and we'll eventually get to a later page on the Clustered Service end.

Archive log Archive log Client B Client A Network cnc.dat cnc.dat cnc.dat cnc.dat cons-module service control-response control-request control-response control-request cons-module service consensus0-1 consensus2-1 consensus1-0 consensus1-2 consensus1-0 consensus2-0 consensus0-1 consensus0-2 log log egressB ingressB egressA ingressA ingressB ingressA egressA egressB Conductor Receiver Sender Conductor Receiver Sender Receiver Conductor Sender Receiver Conductor Sender ClusterSession Cons Mod ClusterSession Cons Mod ClientSession ClientSession Clustd Svc Clustd Svc Cons Mod ClusterSession ClusterSession Cons Mod ClusterSession Cons Mod ClientSession ClientSession Clustd Svc Clustd Svc Consensus Module ClientSession ClientSession ClusterSession Clustd Svc Clustd Svc Cons Mod ClusterSession Consensus Module Cons Mod Clustered Service Clustered Service Consensus Module Consensus Module Clustered Service Clustered Service LEADER FOLLOWER

Use the tabs above to step through the animation.



Clients can connect to any of the Consensus Modules by sending a SessionConnectRequest on the Ingress, and the Consensus Module will establish a connection back to the client on its Egress. However, followers respond by redirecting the client to the leader, then close the Egress and don't create a client session. Client sessions are only created on the leader. The Consensus Modules represent a session using a ClusterSession, which contains the Egress Publication.

The Clustered Services also need to know about the session, but they only take input from the Log. When a new client connects, the leader Consensus Module publishes a SessionOpenEvent to the Log, which all the Clustered Services read, at the same position relative to other Log events. The Clustered Services represent a session using a ClientSession.

If the leader dies and a new one is elected, the new leader needs to be able to switch all the client connections to itself, to minimise disruption. For this reason, the follower Consensus Modules also process the SessionOpenEvent. This contains enough information for the followers to build their own collection of ClusterSessions, including Egress connection info. This means followers are ready to take over, if they are elected leader. Followers do not connect the Egress at this stage.

The leader Clustered Service creates an Egress Publication with the same details as the Consensus Module. They both use a ConcurrentPublication, so the Clustered Service's Publication ends up sharing the Egress Publication log buffer with the Consensus Module's Publication. The end result is that all Consensus Modules and Clustered Services are aware of all connections and the leader Consensus Module and Clustered Service share the Egress Connection back to the client.

Both the Consensus Module and Clustered Service can send messages back to the client.

Consensus Module

The Consensus Module manages the Egress connection, i.e. creating it, checking for inactivity, and closing it. The only messages that the Consensus Module sends are related to the connection itself, or responses to short-lived connections that don't involve the Clustered Service:

  • SessionEvent, in relation to the connection itself (OK (connected), CLOSED, ERROR, or REDIRECT)
  • NewLeaderEvent, sent by the new leader after an election, to get the client to switch its Ingress to it
  • Challenge, for additional authentication when connecting
  • AdminResponse, in response to a snapshot request (on a short-lived connection)
  • BackupResponse, in response to a backup request (on a short-lived connection)

Whenever a message is sent on the Ingress, the Consensus Module sets the connection's timeOfLastActivityNs. If it hasn't received a message for 10 seconds (SESSION_TIMEOUT_DEFAULT_NS), the connection is closed.

Clustered Service

The Clustered Service only uses the Egress to send application level messages back to the Client:

It doesn't do anything else with the Egress. It can close the Egress, but that is done via a CloseSession request to the Consensus Module.

ClusterSession

The Consensus Module represents a client session using a ClusterSession object. On the leader, this contains the Egress Publication back to the client.

How does the Consensus Module use the ClusterSessions?

  • when the Consensus Module shuts down, it closes all the sessions
  • when the client sends a SessionCloseRequest, it closes the session
  • when the client sends an AdminRequest, the session is used to return the response on its Egress
  • when the client sends an Ingress message (a SessionMessageHeader or SessionKeepalive), the lastActivityTimestamp is updated, which stops the session from timing out
  • it closes the session if the Clustered Service sends the Consensus Module a CloseSession
  • the new leader sends a NewLeaderEvent after an Election
  • slow tick work (covered later, but it's work that happens in the Consensus Module every 10 ms) checks if the session has timed out, progresses closing sessions, etc

State

Each ClusterSession has a State, which is an enum:

enum State
{
    INIT, CONNECTING, CONNECTED, CHALLENGED, AUTHENTICATED, REJECTED, OPEN, CLOSING, INVALID, CLOSED
}

ActionType

Each ClusterSession has an ActionType, which is an enum:

enum Action
{
    CLIENT, BACKUP, HEARTBEAT, STANDBY_SNAPSHOT
}

When a client application connects, its ClusterSession has a CLIENT ActionType. The main list of sessions in the ConsensusModuleAgent only contains this kind of ClusterSession. Other tools can connect to the Consensus Module to issue one-off commands, but they create a temporary ClusterSession with a different ActionType.

Establishing a connection - details

IngressAdapter poll()

The ConsensusModuleAgent receives the SessionConnectRequest when it polls its IngressAdapter, which calls ConsensusModuleAgent.onSessionConnect().

That starts by creating a ClusterSession, but only assigns it a clusterSessionId if the ConsensusModuleAgent is the leader. Followers only keep the ClusterSession long enough to return a REDIRECT to it, so they don't assign a clusterSessionId that might conflict with one assigned by the leader.

The ClusterSession contains the Egress channel and streamId from the SessionConnectRequest, and the ConsensusModuleAgent asks it to start the async creation of the Egress Publication.

If the ConsensusModuleAgent is a follower:

  • the ClusterSession is added to a redirectUserSessions collection, which will be picked up later

If the ConsensusModuleAgent is the leader:

  • if the client version in the SessionConnectRequest is not compatible with the Consensus Module's version, or if the ConsensusModuleAgent already has the maximum number of ClusterSessions, an error message is added to the ClusterSession, and it's added to rejectedUserSessions, which will be picked up later
  • otherwise, at this point, the Authenticator is notified of the new connection request (Authentication is covered on the next page, but the DefaultAuthenticator does nothing with this info). The ClusterSession is then added to pendingUserSessions, which will be picked up later

This completes the work on the IngressAdapter poll. At this point, creation of the Egress Publication is in progress, and the ClusterSession is held in one of the temporary collections in the ConsensusModuleAgent.

Slow tick work

The ConsensusModuleAgent performs slow tick work every 10 ms. This starts by invoking the Aeron Transport Client to do any work it needs to, which includes progressing the creation of the Egress Publication (it's created asynchronously).

For any redirectUserSessions, once the Egress Publication has connected, a REDIRECT SessionEvent is sent on it, then the Egress is closed. The ClusterSession is then removed from the collection and discarded.

rejectedUserSessions are handled in the same way, but with an ERROR SessionEvent.

pendingUserSessions are processed in processPendingSessions(). Once the Egress has connected, it moves the ClusterSession to the CONNECTED state and calls Authenticator.onConnectedSession(). At this point, the DefaultAuthenticator (which fakes authentication) moves the ClusterSession to the AUTHENTICATED state. Authentication is covered on the next page.

The next slow tick work checks pendingUserSessions again and finds the ClusterSession in the AUTHENTICATED state. Now it performs the ActionType that the ClusterSession was created for, which for a client connection is CLIENT.

For CLIENT ClusterSessions, it attempts to append a SessionOpenEvent to the Log.

When that succeeds, it moves the ClusterSession to the OPEN state and sends a SessionEvent on the Egress with EventCode OK, which tells the client that it has successfully connected. The SessionEvent contains clusterSessionId and leadershipTermId fields that the client needs to include in all subsequent Ingress messages. If the clusterSessionId in an Ingress message doesn't map to an open ClusterSession within the ConsensusModuleAdapter, the message is ignored. If the leadershipTermId in an Ingress message doesn't match the current leadership term, the message is ignored.

The ClusterSession is moved from pendingUserSessions into the main sessions collection.

This completes slow tick work handling of ClusterSessions. At this point, redirected or rejected ClusterSessions will have been dealt with. Successful ClusterSessions will be in the main list of sessions, their Egress will be connected, and there will be a SessionOpenEvent on the Log.

SessionOpenEvent

At this point, the leader's ConsensusModuleAgent has the ClusterSession in its main list of sessions, but the follower ConsensusModuleAgents don't and none of the Clustered Services know about it.

Each follower's ConsensusModuleAgent polls its LogAdapter to read the Log, and when it finds the SessionOpenEvent, calls ConsensusModuleAgent.onReplaySessionOpen(). This just creates a ClusterSession object and adds it to the sessions collection, so it matches the leader. The ClusterSession objects on the followers don't connect to the Egress.

The followers also process SessionCloseEvent messages to remove ClusterSessions.