ServiceComb Pack 0.4.0 Cluster : Implementation

5 minute read

Now you can run multiple Alpha server in High Availability (HA) mode which support to run the event scanning in the master node only. You can also turn off event table scanning for some nodes with parameter alpha.event.scanner.enabled=false . However, transaction compensation will fail when the node that enabled the event table scan is crashed. We implemented database-based distributed lock in version 0.4.0 version, event scanning only runs on the master node in the cluster, When the master node is down, other nodes in the cluster will elect a new master node.

Quick Starts

Enable cluster support by parameter alpha.cluster.master.enabled=true.

  • Start Two Alphas.
java -jar alpha-server-0.4.0-SNAPSHOT-exec.jar \
--server.port=8090 \
--alpha.server.port=8080 \
--spring.datasource.url="jdbc:postgresql://" \
--spring.datasource.username=saga-user \
--spring.datasource.password=saga-password \
java -jar alpha-server-0.4.0-SNAPSHOT-exec.jar \
--server.port=8091 \
--alpha.server.port=8081 \
--spring.datasource.url="jdbc:postgresql://" \
--spring.datasource.username=saga-user \
--spring.datasource.password=saga-password \
  • Who Is Master

Master Node This alpha is the master

01:31:07.032 [pool-3-thread-1] INFO  org.apache.servicecomb.pack.alpha.server.cluster.master.ClusterLockService - Master Node

Slave Node This alpha is the slave

01:31:31.059 [pool-3-thread-1] INFO  org.apache.servicecomb.pack.alpha.server.cluster.master.ClusterLockService - Slave Node
  • Switch Master Nodes

After the master node is off line, other nodes in the cluster will elect a new master node

How to make event scanning only run on the master node

Event scanning is implemented by , initialize it in , you can enable it by setting the parameter ‘alpha.event.scanner.enabled=true’, it will be instantiated EventScanner. The variable nodeStatus is the node type(Master or Slave). Later, I will introduce how nodeStatus is initialized.

TxConsistentService txConsistentService(
  @Value("${alpha.event.pollingInterval:500}") int eventPollingInterval,
  @Value("${alpha.event.scanner.enabled:true}") boolean eventScannerEnabled,
  ScheduledExecutorService scheduler,
  TxEventRepository eventRepository,
  CommandRepository commandRepository,
  TxTimeoutRepository timeoutRepository,
  OmegaCallback omegaCallback) {
    if (eventScannerEnabled) {
      new EventScanner(scheduler,
          eventRepository, commandRepository, timeoutRepository,
          omegaCallback, eventPollingInterval, nodeStatus).run();"Starting the EventScanner.");
    TxConsistentService consistentService = new TxConsistentService(eventRepository);
    return consistentService;

The pollEvents method of for scanning events, If the master node nodeStatus.isMaster() returns true.

private void pollEvents() {
    () -> {
      // only pull the events when working in the master mode

Construct NodeStatus in by the following to ensure that the event scan will work regardless of whether you have configured the alpha.cluster.master.enabled parameter, Here you can see the node is a slave just after startup when cluster mode is enabled. Later, I will introduce how to switch to master.

  NodeStatus nodeStatus (){
      return new NodeStatus(NodeStatus.TypeEnum.SLAVE);
      return new NodeStatus(NodeStatus.TypeEnum.MASTER);

  NodeStatus nodeStatus; is in charge of node state switching, It periodically perform lock preemption and set the node as a master after successful preemption.

  LockProvider lockProvider;
  @Scheduled(cron = "0/1 * * * * ?")
  public void masterCheck() {
    if (applicationReady) { = lockProvider.lock(this.getMasterLock());
      if ( {
        if (!this.locked) {
          this.locked = true;
"Master Node");
        //Keep locked
      } else {
        if (this.locked || !lockExecuted) {
          locked = false;
"Slave Node");
      lockExecuted = true;

Lock architecture

As you can see from the previous description, in the masterCheck method of, get a lock by = lockProvider.lock(this.getMasterLock()); and determine if the lock was successful. is an interface. Currently, we provide an implementation based on JDBC. The package structure and class dependencies are as follows:


Dependencies are as follows



    The interface defines the lock methods


    The interface defines the following three persistence methods

    • initLock Create a lock, try to lock and return whether the lock was successful
    • updateLock Update the lock, lock it again and return it successfully (the interface design of the update lock is designed for non-long connection lock design, for example, for locking according to a fixed period)
    • unLock

    Implemented the interface and called the internal interface for locking operations

JDBC implementation

I like to run with you, the same rhythm, the same heartbeat, I feel very good, when you fall behind, I lead everyone to run together

  • JDBC class diagram



    Implement interface operation database


    Inherit the abstract class, passing the interface implementation class JdbcLockPersistence in the constructor.


    Declare a LockProvider Bean with @Configuration

  • JPA、、org.apache.servicecomb.pack.alpha.server.cluster.master.provider.jdbc.jpa.*

  • Create master_lock Table

  serviceName varchar(36) not NULL,
  instanceId  varchar(255) not NULL,
  PRIMARY KEY (serviceName)
  • serviceName Service name, this field takes the value $ {}
  • expireTime Lock expiration time, this field takes value lockedTime + 5s
  • lockedTime Last lock time
  • instanceId Service instance ID, this field takes the value $ {}]:$ {alpha.server.port}

Lock processing

The process of locking/updating a lock is a cyclically repeating action, as follows:

  • calls the lock method of interface once per second

  • The lock method in the abstract class will attempt to lock the iniLock method of, and the locked SQL implementation is defined in

    • If the table is empty then insert a record and return the lock successfully
    • Exception caught and lock failed if there are records with the same service name field in the table
       @Query(value = "INSERT INTO master_lock "
           + "(serviceName, expireTime, lockedTime, instanceId) "
           + "VALUES "
           + "(?1, ?2, ?3, ?4)", nativeQuery = true)
       int initLock(
           @Param("serviceName") String serviceName,
           @Param("expireTime") Date expireTime,
           @Param("lockedTime") Date lockedTime,
           @Param("instanceId") String instanceId);
  • If the initLock lock fails, try to update the lock by calling the updateLock method in

    • If the record instanceId of the same service name is the same as the instance instanceId, the update succeeds and the lock is successfully returned.
    • If the record expireTime of the same service name is less than the current lock time, the update is successful and the lock is successful.
       @Modifying(clearAutomatically = true)
       @Query("UPDATE org.apache.servicecomb.pack.alpha.server.cluster.master.provider.jdbc.jpa.MasterLock t "
           + "SET t.expireTime = :expireTime"
           + ",t.lockedTime = :lockedTime "
           + ",t.instanceId = :instanceId "
           + "WHERE t.serviceName = :serviceName AND (t.expireTime <= :lockedTime OR t.instanceId = :instanceId)")
       int updateLock(
           @Param("serviceName") String serviceName,
           @Param("lockedTime") Date lockedTime,
           @Param("expireTime") Date expireTime,
           @Param("instanceId") String instanceId);
  • UnLock

    Reserved interface for extending other ways of locking

Other lock implementations

Other types of locks can be implemented through the and interfaces, such as zookeeper or redis etc.


  • Need to synchronize time across multiple servers

  • Need to configure the correct time zone parameters when using MySQL database

Leave a Comment

Your email address will not be published. Required fields are marked *