Couchbase/Elasticsearch connector for multiple buckets - elasticsearch

Question

Is there a way to replicate 2 or many couchbase buckets to elasticsearch using a single configuration file?
I actually use this version of the couchbase elasticsearch connector:
https://docs.couchbase.com/elasticsearch-connector/4.0/index.html
I do replicate my data correctly, but need to run a command per bucket using a different configuration file (.toml) each time.
Could not by the way run the cbes command multiple times on the same server as the metrics port 31415 is already in use.
Is there any way to handle many connector groups in one time?

Answer 1

In version 4.0 a single connector process can replicate from only one bucket. This is because the indexing rules and all of the underlying network connections to Couchbase Server are scoped to the bucket level.
The current recommendation is to create multiple config files and run multiple connector processes. It's understood that this can be complicated to manage if you're replicating a large number of buckets.
If you're willing to get creative, you could use the same config file template for multiple buckets. The idea is that you'd write a config file with some placeholders in it, and then generate the actual config file by running a script that replaces the placeholders with the correct values for each connector.
The next update to the connector will add built-in support for environment variable substitution in the config file. This could make the templating approach easier.
Here are some options for avoiding the metrics port conflict:
Disable metrics reporting by setting the httpPort key in the [metrics] section to -1.
OR Use a random port by setting it to 0.
OR Use the templating idea described above, and plug a unique port number into each generated config file.
It's worth mentioning that a future version of the connector will support something we're calling "Autonomous Operations Mode". When the connector runs in this mode, the configuration will be stored in a central location (probably a Consul server). It will be possible to reconfigure a connector group on-the-fly, and add or remove workers to the group without having to stop all the workers and edit their config files. Hopefully this will simplify the management of large deployments.