#StackBounty: #configuration #distributed-system #consul #key-value-store #consul-kv Using Consul for dynamic configuration management

Bounty: 50

I am working on designing a little project where I need to use Consul to manage application configuration in a dynamic way so that all my app machines can get the configuration at the same time without any inconsistency issue. We are using Consul already for service discovery purpose so I was reading more about it and it looks like they have a Key/Value store which I can use to manage my configurations.

All our configurations are json file so we make a zip file with all our json config files in it and store the reference from where you can download this zip file in a particular key in Consul Key/Value store. And all our app machines need to download this zip file from that reference (mentioned in a key in Consul) and store it on disk on each app machine. Now I need all app machines to switch to this new config at the same time approximately to avoid any inconsistency issue.

Let’s say I have 10 app machines and all these 10 machines needs to download zip file which has all my configs and then switch to new configs at the same time atomically to avoid any inconsistency (since they are taking traffic). Below are the steps I came up with but I am confuse on how loading new files in memory along with switch to new configs will work:

  1. All 10 machines are already up and running with default config files as of now which is also there on the disk.
  2. Some outside process will update the key in my consul key/value store with latest zip file reference.
  3. All the 10 machines have a watch on that key so once someone updates the value of the key, watch will be triggered and then all those 10 machines will download the zip file onto the disk and uncompress it to get all the config files.
  4. (..)
  5. (..)
  6. (..)

Now this is where I am confuse on how remaining steps should work.

  • How apps should load these config files in memory and then switch all at same time?
  • Do I need to use leadership election with consul or anything else to achieve any of these things?
  • What will be the logic around this since all 10 apps are already running with default configs in memory (which is also stored on disk). Do we need two separate directory one with default and other for new configs and then work with these two directories?

Let’s say if this is the node I have in Consul just a random design (could be wrong here) –

{"path":"path-to-new-config", "machines":"ip1:ip2:ip3:ip4:ip5:ip6:ip7:ip8:ip9:ip10", ...}

where path will have new zip file reference and machines could be a key here where I can have list of all machines so now I can put each machine ip address as soon as they have downloaded the file successfully in that key? And once machines key list has size of 10 then I can say we are ready to switch? If yes, then how can I atomically update machines key in that node? Maybe this logic is wrong here but I just wanted to throw out something.

Can someone outline the logic on how can I efficiently manage configuration on all my app machines dynamically and also avoid inconsistency issue at the same time?


Get this bounty!!!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.