I'm using Prometheus + Consul for service discovery - an awesome combo. Among the services (targets, in prometheus's lingo) discovered by Consul there are the Consul servers themselves. This is useful - I want to monitor the servers too, not just Consul clients.
In fact, I wish to scrape these servers twice - essentially read metrics from 2 different ports on each server.
I'm not sure how this can be achieved: I have no control over what Consul Servers expose to Prometheus about themselves.
Ideally, this could be performed with some regexp manipulation in Prometheus's configuration: I know how to drop a target from the list and I know how to mutate a target - but can a target be duplicated? is there some regex magic for that?
There's no way to create more targets. Instead have two scrape configs.
Related
I'm trying to use HAProxy as a dynamic proxy for backend hosts based on partial /path regex match. The use case is routing from an HTTPS frontend to a large number of nodes that come and go frequently, without maintaining an explicit mapping of /path to server hostnames.
Specifically in this case the nodes are members of an Amazon EMR cluster, and I'd like to reverse-proxy/rewrite HTTP requests like:
<haproxy>/emr/ip-99-88-77-66:4040 -> 99.88.77.66:4040
<haproxy>/emr/ip-55-44-33-22/ganglia -> 55.44.33.22/ganglia
<haproxy>/emr/ip-11-11-11-11:8088/cluster/nodes -> 11.11.11.11:8088/cluster/nodes
...etc
dynamically.
As-in, parse the path beginning at /emr and proxy requests to an IP captured by the regex:
emr\/ip-(\d{1,3}-\d{1,3}-\d{1,3}-\d{1,3})(.*)
Is this possible with HAProxy? I know it's probably not the right tool for the job, but if possible (even non-performant) I'd like to use the tooling we already have in place.
tl;dr basically nginx proxy_pass, but with HAProxy and plucking a backend IP from the url.
Thanks!
Yes its possible by using url filters in haproxy, see below link for more details.
https://fossies.org/linux/haproxy/doc/internals/filters.txt
Yes this can be done. I would recommend you use ACLs, as well as Roundrobin & checks, which will allow you to check to see if that instance is up before routing to it with a check. That way, the system will only route to service instances that are up and running, and will only have them preloaded for use if they are up.
In addition, this will also allow you to constantly cycle in and out instances, such as if your AWS instance costs change with any other providers you may have, and allows you to load balance with maximum cost savings in mind.
yes, this is possible.. check the official manual:
Using ACLs and fetching samples
We're considering using Consul's key-value store to enhance our configuration management tool (Ansible, at the moment). We're hoping to use it to solve two related problems:
Preventing scatter: Some items (namely: passwords, certificates etc) are scattered across our configuration files. Updating them requires manual search-and-replace which can be tiresome.
Ease of update: rather then edit-and-commit configuration changes into git, we could use Consul to store those items that change often.
We're looking for a set of recommendations on how to use/integrate Consul (or similar tools) for dynamic configurations. Naturally, there is no one answer, but a set of useful practices here. I'll provide a few approaches in my answer, but I'd like to hear additional ideas.
We've been tinkering with Consul as a key-value store for a while but I think the most interesting use comes with Consul Template and using that to update configuration on the fly.
I think the end state we're probably moving towards is going to be to use Ansible to configure a base image of things we know are slow changing plus configure Consul Template, then AMI this (these first 2 steps probably done via Packer) and then deploy into auto scaling groups in AWS using Terraform (which we already use for provisioning).
Then we will use Consul's key-value store to change properties that Consul Template will then propagate across a cluster of instances. We also intend to have instances register themselves in Consul which will also affect configuration on other instances such as load balancing members on Apache/NGINX configurations or lists of unicast addressable members for clustering.
On a slightly related note, and as mentioned by mahnve, Vault is a pretty nice add on to Consul for storing secrets. We're already using it for pretty static secrets but intend to start using some of the dynamic secret generation which allows you to request short lived API or SSH keys which can be tracked and revoked.
To mitigate #1 I'd suggest looking into Hashicorps Vault, https://www.vaultproject.io/, which is a tool to handle secrets, which can use Consul as a backend.
We've yet to do this, but are thinking about integrating consul into our Ansible plays. Ansible recently added a lookup option from consul:
https://github.com/ansible/ansible/blob/devel/test/integration/roles/test_consul_kv/tasks/main.yml#L70
- debug: msg='key contains {{item}}'
with_consul_kv:
- 'key/to/retrieve'
So we could directly populate our plays with values from Consul.
Another approach we're considering is to utilize consul's templating tool - and template entire configuration files after ansible plants them on our hosts.
I have an adapter (written in Spring Boot and Spring Integration) retrieving currency reates from two different sources (via REST and proprietary library). I filter unnecesary things, create instances of class known in my system and send rates to JMS cluster. I want this adapter to be replicated. Only one instance should be running at the same time. When one crashes (I know it from health endpoint) another one should start publishing rates. How can I achieve such effect? I know that available services can be registered using Eureka but how to turn one of them on automatically?
The solution to the problem is using spring-cloud-cluster. One can use either zookeeper or hazelcast to negotiate leadership. From few instances only one is given a leader role. If it crashes, another one takes its role (it is informed via event propagation). You can also use yieldLeadership method to manually relinquish leadership (if health indicator says something is wrong with the application).
Without knowing more details it is hard to give you a recommendation.
I'd personally say Eureka is not build for what you are trying to achieve. But it sounds more like you want to have a look into ZooKeeper. Also see Eureka FAQ for reference. ZooKeeper was exactly build for doing what you are trying to achieve: leader election.
On the other hand, if you can survive also with having the service down for a few seconds I'd suggest you use either your script that monitors the /health endpoint already to restart the service or use systems who already have this build in like Systemd or Docker, where you can define Restart policies.
I have the same application running on two WAS clusters. Each cluster has 3 application servers based in different datacenters. In front of each cluster are 3 IHS servers.
Can I specify a primary cluster, and a failover cluster within the plugin-cfg.xml? Currently I have both clusters defined within the plugin, but I'm only hitting 1 cluster for every request. The second cluster is completely ignored.
Thanks!
As noted already the WAS HTTP server plugin doesn't provide the function your're seeking as documented in the WAS KC http://www-01.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/rwsv_plugincfg.html?lang=en
assuming that by "failover cluster" what is actually meant is "BackupServers" in the plugin-cfg.xml
The ODR alternative mentioned previously likely isn't an option either, this because the ODR isn't supported for use in the DMZ (it's not been security hardened for DMZ deployment) http://www-01.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/twve_odoecreateodr.html?lang=en
From an effective HA/DR perspective what you're seeking to accomplish should handled at the network layer, using the global load balancer (global site selector, global traffic manager, etc) that is routing traffic into the data centers, this is usually accomplished by setting a "site cookie" using the load balancer
This is by design. IHS, at least at the 8.5.5 level, does not allow for what you are trying to do. You will have to implement such level of high availability in a higher level in your topology.
There are a few options.
If the environemnt is relatively static, you could post-process plugin-cfg.xml and combine them into a single ServerCluster with the "dc2" servers listed as <BackupServer>'s in the cluster. The "dc1" servers are probably already listed as <PrimaryServer>'s
BackupServers are only used when no PrimaryServers are reachable.
Another option is to use the Java On-Demand Router, which has first-class awareness of applications running in two cells. Rules can be written that dictate the behavior of applications residing in two clusters (load balancer, failover, etc.). I believe these are "ODR Routing Rules".
We have a two server load balanced ASP.NET MVC application that we're look to add search to - we like the look of Elasticsearch!
I'd like to build a bit of resiliency in so am considering installing ES on both servers and have them work like a cluster (which seems straightforward enough according the docs).
But I really want my MVC app to talk to "the cluster" not to each specific ES node, so if an ES node fails, or a server fails the application is unaffected:
Do I need to refer to the cluster in a special way from my application or setup some kind of internal domain for it?
I don't want to refer to "localhost:9200" or a server specific IP in my application I presume?
You should consider using the Elasticsearch .NET Client NEST. This client has built in support for failover connections when accessing an Elasticsearch Cluster.
If you want a failover client instead of passing a Uri pass an IConnectionPool see the Elasticsearch.Net documentation on cluster failover all of its implementations can also be used with NEST.