How to syncing data between application instances? Easy!

4 min readApr 20, 2019

In the grim darkness of the far future, there is only data loss. Making simple data replication for you applications.

Syncing data with 12k requests/sec messages between instances

Replication. What is it?

Data replication is the process of syncing data between you services (databases, applications, etc). It can help you not to lose data in critical incidents and easily scale the increasing load on your services.

Replication description from the Wiki:
Replication in computing involves sharing information so as to ensure consistency between redundant resources, such as software or hardware components, to improve reliability, fault-tolerance, or accessibility.

For example, also you can save the limits of third-party api services, if only one instance of your application makes requests and transmit it to an other instances.

Me after a week recover lost data from the broken server

Replication will help you save data when some applications fail. At the right moment, it will save you from a lot of stress. Just trust.

Ok. Suppose I convinced you and you want to make your own replication in your application

But how?

First steps. Making the transaction model.

The basic transaction must have some data and an identifier for the transfer.

// In my opinion, protobufs is the best solution for the replication.
// Our identifier will be unix nano timestamp syntax = "proto3";

message Transaction {
    int64 timestamp = 1;
    map<string, bytes> data = 2;
}

After compiling it with protoc we will have Transaction model in the code.

Let’s create transaction pool as the best practice.

var transactionPool = sync.Pool{
   New: func() interface{} {
      return new(Transaction)
   },
}

func GetTransaction() *Transaction {
   return transactionPool.Get().(*Transaction)
}

func PutTransaction(transaction *Transaction) {
   transaction.Reset()
   transactionPool.Put(transaction)
}

Now we can create transactions:

func NewTransaction(data map[string][]byte) *Transaction {
   transaction := GetTransaction()
   transaction.Timestamp = time.Now().UnixNano()
   transaction.Data = data
   return transaction
}

Making the replication.

The simple replication must have the transmitter and receiver (for send and receive transactions), list of received transactions and list of any application hosts for sending transactions.

type Replication struct {
   Transmitter chan *Transaction
   Receiver    chan *Transaction

   Received map[string]int64

   Replicas map[string]string
}

Let’s create logic for the transmitting transactions:

func (r *Replication) Transmit() {
   for {
      transaction := <-r.Transmitter

      go func() {
         body, err := proto.Marshal(transaction)

         if err == nil {
            for addr, host := range r.Replicas {              //That's url what you services will listen for transactions
              _, err := http.Post(host + "/replication/receiver", "application/protobuf", bytes.NewReader(body))
               
               if err != nil {
                  log.Println("Master", addr, "is down:", err)
               }
            }
         } else {
            log.Println("Replication error:", err)
         }

         PutTransaction(transaction)
      }()
   }
}

And for receiving the transactions:

func (r *Replication) Receive() {
   for {
      transaction := <-r.Receiver

      go func() {
         for key, data := range transaction.Data {            if timestamp, ok := r.Received[key]; ok && timestamp <= transaction.Timestamp {
               continue
            }            // Put transaction data to you data store
            err := database.Client.Put(key, data)

            if err == nil {
               r.Received[key] = transaction.Timestamp
            } else {
               log.Println("Replication error:", err)
            }
         }

         PutTransaction(transaction)
      }()
   }
}

Sending message mechanism:

func (r *Replication) SendMessage(data map[string][]byte) {
   if len(r.Replicas) > 0 {
      r.Transmitter <- NewTransaction(data)
   }
}

Now we can init Replication in the app:

replication := &Replication{
   Transmitter: make(chan *Transaction, 2048),
   Receiver:    make(chan *Transaction, 2048),
   Received:    make(map[string]int64),
   Replicas:    hosts,
}

Replication usage

Just run receiver and transmitter in the new thread

go replication.Receive()
go replication.Transmit()

And create the simple method for receive new transactions for other instances:

http.HandleFunc("/replication/receiver", func(writer http.ResponseWriter, request *http.Request) {
   transaction := GetTransaction()
   body, _ := ioutil.ReadAll(request.Body)
   err := proto.Unmarshal(body, transaction)
   if err == nil {
      replication.Receiver <- transaction
      fmt.Fprintln(writer, "Received transaction")
   } else {
      fmt.Fprintln(writer, err)
   }
})