I have a mesos and chronos environment, I want to use chronos api to monitor jobs status, But ...
curl -L http://localhost:8080/leader
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /leader. Reason:
<pre> Not Found</pre></p>
<hr />Powered by Jetty:// 9.3.z-SNAPSHOT<hr/>
</body>
</html>
It return 404 to me,Can anybody has ideas?
The other features work fine.
The API URL is : chronos REST API DOC
You'll need to ensure you are connecting to the correct host on your cluster. It's likely chronos is not accessible via localhost or chronos-node given you are not receiving a response.
To determine where chronos is running, leverage mesos dns.
If your service is installed as chronos, look up the SRV record for all TCP endpoints as follows:
/# dig _chronos._tcp.marathon.mesos SRV
; <<>> DiG 9.12.1-P2 <<>> _chronos._tcp.marathon.mesos SRV
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39584
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; QUESTION SECTION:
;_chronos._tcp.marathon.mesos. IN SRV
;; ANSWER SECTION:
_chronos._tcp.marathon.mesos. 60 IN SRV 0 0 9669 chronos-jjmux-s11.marathon.mesos.
;; ADDITIONAL SECTION:
chronos-jjmux-s11.marathon.mesos. 60 IN A 10.0.2.15
;; Query time: 1 msec
;; SERVER: 198.51.100.1#53(198.51.100.1)
;; WHEN: Tue Oct 02 22:48:02 UTC 2018
;; MSG SIZE rcvd: 114
In this case above, I installed via the DC/OS universe, which automatically deploys with any available port via the host network driver, however I see there is a tcp listener on port 9669 via the DNS query. From there you can receive a response via curl -L -X GET http://chronos.marathon.mesos:9669/scheduler/jobs. (note:, the chronos documentation states to prefix with /v1, however in some deployments, that will not be the case).
Lastly, you'll likely not want to perform a lookup of an SRV record every time. If you installed via the DC/OS universe, chronos is deployed via marathon, however the universe package hides all the configuration options necessary to define a VIP via the DC/OS interface. This can be addressed by manually going to the original marathon interface, where you'll be able to change the configuration (or via the marathon api).
Related
The situation is, I wanna establish a QUIC connection based on quic-go from local to ECS server. The related tests using localhost are done both on local and remote device. That is:
#local: .$QUIC-GO-PATH/example/client/main -insecure -keylog ssl.log -qlog trial.log -v https://127.0.0.1:6121/demo/tile
#local: .$QUIC-GO-PATH/example/main -qlog -tcp -v
These tests are completed.
Now is the problem,when I start local-remote connection an error occurred:
#remote: .$QUIC-GO-PATH/example/main -qlog -tcp -v
#local: .$QUIC-GO-PATH/example/client/main -insecure -keylog ssl.log -qlog trial.log -v https://$REMOTE_IPADDR:6121/demo/tile
timeout: no recent network activity
When I go through a wireshark examination, it seems like the CRYPTO handshake never finishes:
Wireshark
Also client Qlog file atteched here:
Qlog file
Codes are all the same with https://github.com/lucas-clemente/quic-go
Help!
This problem has been solved.
Code $QUIC-GO-PATH/example/main.go has binded the port as a default onto 127.0.0.1:6121, which led to the problem that the server cannot get reached by client outside, just get this on server running:
-bind 0.0.0.0:6121
consul isn't publishing all the metrics defined in their document, from https://www.consul.io/docs/agent/telemetry.html#transaction-timing, it shows only raft metrics but not txn kvs, has anyone observed this problem?
Command to enable prometheus style metrics:
consul agent -dev -hcl 'telemetry{prometheus_retention_time="24h" disable_hostname=true}'
watch metrics:
watch -n 1 -d "curl -s localhost:8500/v1/agent/metrics?format=prometheus|grep -v ^# | grep -E 'kvs|txn|raft'"
Metrics will be exported only if they are available, i.e. if there are no transactions or KV store operations, then you will not see these metrics in the output.
I have managed to see kvs metrics in the example you have provided. While running Consul agent via command in the question, in browser open http://127.0.0.1:8500/ and click on Key/Value option in the top list (you should end up here http://127.0.0.1:8500/ui/dc1/kv). Click on Create to add new Key/Value pair. After clicking Save you should see something like this in the terminal running watch command:
consul_fsm_kvs{op="set",quantile="0.5"} 0.3572689890861511
consul_fsm_kvs{op="set",quantile="0.9"} 0.3572689890861511
consul_fsm_kvs{op="set",quantile="0.99"} 0.3572689890861511
consul_fsm_kvs_sum{op="set"} 0.3572689890861511
consul_fsm_kvs_count{op="set"} 1
consul_kvs_apply{quantile="0.5"} 2.6777150630950928
consul_kvs_apply{quantile="0.9"} 2.6777150630950928
consul_kvs_apply{quantile="0.99"} 2.6777150630950928
consul_kvs_apply_sum 2.6777150630950928
consul_kvs_apply_count 1
If there are no more transactions some of these values will be set to NaN value, depends on Prometheus metrics type.
Similarly, to see txn, you need to create Consul Transaction
Hope that helps you set up monitoring.
When I run an image in Kubernetes with kubectl run, environment variables are injected into the container.
My problem is that the values are wrong. I do not have anything running at 10.0.0.1. I believe the correct value there would be 10.1.0.1. This misconfiguration causes, as far as I know, among other things, the error from kube-dns reproduced below.
I would like to ask how are these variables injected into the container, preferably for a link into the code which takes care of this (I could not find anything). Also, some hints where the value 10.0.0.1 could be coming from.
pod variables:
$ kubectl run -i --image=busybox --restart=Never -t busybox
If you don't see a command prompt, try pressing enter.
/ # env
KUBERNETES_SERVICE_PORT=443
KUBERNETES_PORT=tcp://10.0.0.1:443
HOSTNAME=busybox
SHLVL=1
HOME=/root
TERM=xterm
KUBERNETES_PORT_443_TCP_ADDR=10.0.0.1
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP=tcp://10.0.0.1:443
KUBERNETES_SERVICE_HOST=10.0.0.1
PWD=/
kube-dns error:
$ kubectl --namespace kube-system logs kube-dns-2190035132-gxf80 kubedns
[...]
E0119 10:04:05.271499 55 reflector.go:199] k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: Get https://10.0.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.0.0.1:443: i/o timeout
I0119 10:04:05.771477 55 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
The closest thing to 10.0.0.1 that I have in my config is --service-cluster-ip-range=10.0.0.0/24 parameter I am giving to kube-apiserver.
I have the IP 10.0.0.1 in my etcd, in
# ETCDCTL_API=3 etcdctl get "" --from-key
[...]
/registry/services/specs/default/kubernetes
k8s
v1Service
kubernetes▒default"*$b198bc22-fcff-11e7-83a9-185e0fec8ce528B
Z
component apiserverZ
provider
kuberneteszC
▒
httpsTCP▒▒(10.0.0.1" ClusterIPClientIPBRZ`▒
▒"
/registry/services/specs/kube-system/kubernetes-dashboard
k8s
v1Service
kubernetes-dashboard▒
kube-system"*$b9f0daef-fcff-11e7-83a9-185e0fec8ce528B
ԾZ,
addonmanager.kubernetes.io/mode ReconcileZ
ppkubernetes-dashboardZ*
kubernetes.io/minikube-addons dashboardZ3
&kubernetes.io/minikube-addons-endpoint dashboardb
0kubectl.kubernetes.io/last-applied-configuration{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","app":"kubernetes-dashboard","kubernetes.io/minikube-addons":"dashboard","kubernetes.io/minikube-addons-endpoint":"dashboard"},"name":"kubernetes-dashboard","namespace":"kube-system"},"spec":{"ports":[{"nodePort":30000,"port":80,"targetPort":9090}],"selector":{"app":"kubernetes-dashboard"},"type":"NodePort"}}
z_
TCP▒PG▒(
ppkubernetes-dashboard▒ 10.0.0.82NodePort:NoneBRZCluster`▒
▒"
https://github.com/kubernetes/kubernetes/blob/v1.9.0/pkg/kubelet/envvars/envvars.go#L45-L48 which I found via git grep SERVICE_PORT
It's possible that if your kubernetes.default.svc.cluster.local is pointing to the wrong IP, then kubectl --namespace=kube-system edit svc kubernetes and changing the ClusterIP would sort that out; I don't have a cluster in front of me to test it, though
I've built a Heroku Application and added a custom domain for it.
I can verify this by running heroku domains on my project while in the terminal, or by using the heroku dashboard; a target DNS has been assigned.
In my GoDaddy Domain Manager, I've added the CNAME for "www" pointing to the heroku application URL. I've followed bunch of tutorials found on the web and yet cannot see my heroku application when typing my url (I've tried prepending http://, http://www, www)...
Is there anything I might be missing here? Please, I've been trying for long time and I have even called GoDaddy Support but in my country (México) they don't even know what Heroku is...
Any help is MUCH MUCH appreciated
Your DNS configuration is messed up, it seems that you have created a delegation (NS record) on the www.omnipagos.com domain to the heroku site, which should be removed. In fact, all records in that domain should be removed for you to be able to create and use a CNAME as per RFC1912
dig www.omnipagos.com any
; <<>> DiG 9.8.3-P1 <<>> www.omnipagos.com any
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13274
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 4
;; QUESTION SECTION:
;www.omnipagos.com. IN ANY
;; ANSWER SECTION:
www.omnipagos.com. 44 IN A 50.63.202.32
www.omnipagos.com. 60 IN NS www.omnipagos-alpha.herokuapp.com.
www.omnipagos.com. 60 IN NS ns25.domaincontrol.com.
www.omnipagos.com. 60 IN NS ns26.domaincontrol.com.
www.omnipagos.com. 60 IN SOA ns25.domaincontrol.com. dns.jomax.net. 2017083101 28800 7200 604800 600
;; ADDITIONAL SECTION:
ns25.domaincontrol.com. 72193 IN A 216.69.185.13
ns25.domaincontrol.com. 10595 IN AAAA 2607:f208:206::d
ns26.domaincontrol.com. 76947 IN A 208.109.255.13
ns26.domaincontrol.com. 10595 IN AAAA 2607:f208:302::d
I have a Go program that uses aws-sdk-go to talk to dynamodb. Dependencies are vendored. Go version 1.7.1. aws-sdk-go version 1.6.24. The program works as expected in all the following environments:
dev box from shell (Arch Linux)
docker container running on my dev box (Docker 1.13.1)
Ec2 instance from shell (Ubuntu 16.04)
When I run the docker container on kubernetes (same one I tested on my dev box), I get the following error:
2017/03/02 22:30:13 DEBUG ERROR: Request dynamodb/GetItem:
---[ REQUEST DUMP ERROR ]-----------------------------
net/http: invalid header field value "AWS4-HMAC-SHA256 Credential=hidden\n/20170302/us-east-1/dynamodb/aws4_request, SignedHeaders=accept-encoding;content-length;content-type;host;x-amz-date;x-amz-target, Signature=483f56dd0b17d8945d3c2f2044b7f97e531190602f132a4d5f828264b3a2cff2" for key Authorization
-----------------------------------------------------
2017/03/02 22:30:13 DEBUG: Response dynamodb/GetItem Details:
---[ RESPONSE ]--------------------------------------
HTTP/0.0 000 status code 0
Content-Length: 0
Based on:
https://golang.org/src/net/http/transport.go
https://godoc.org/golang.org/x/net/lex/httplex#ValidHeaderFieldValue
It looks like the problem is with the header value validation, yet I am at a loss to understand why it works everywhere except on my k8s cluster. The cluster is composed of Ec2 instances running the latest CoreOS stable ami (CoreOS stable 1235.8.0)
The docker image that works on my dev machine is scratch based. To troubleshoot I created an image based on Ubuntu latest with a separate go program that just does a simple get item from dynamodb. When this image is run on my k8s cluster and the program run from an interactive shell, I get the same errors. I have confirmed I can ping the dynamodb endpoints from this env.
I am having a hard time troubleshooting this issue: am I missing something stupid here? Can someone point me in the right direction or have an idea of what is going on?
remember the "-n" when you do this:
echo -n key | base64
The \n after hidden is certainly invalid. Not sure if it is actually there or somehow got inserted when you were cleansing for posting.
Consider:
package main
import (
"fmt"
"golang.org/x/net/lex/httplex"
)
func main() {
fmt.Println("Is valid (without new line)", httplex.ValidHeaderFieldValue("AWS4-HMAC-SHA256 Credential=hidden/20170302/us-east-1/dynamodb/aws4_request, SignedHeaders=accept-encoding;content-length;content-type;host;x-amz-date;x-amz-target, Signature=483f56dd0b17d8945d3c2f2044b7f97e531190602f132a4d5f828264b3a2cff2"))
fmt.Println("Is valid (with new line)", httplex.ValidHeaderFieldValue("AWS4-HMAC-SHA256 Credential=hidden\n/20170302/us-east-1/dynamodb/aws4_request, SignedHeaders=accept-encoding;content-length;content-type;host;x-amz-date;x-amz-target, Signature=483f56dd0b17d8945d3c2f2044b7f97e531190602f132a4d5f828264b3a2cff2"))
}
One guess would be wherever the real hidden value is getting pulled from (config file etc) mistakenly has the \n in there and it's happily getting pulled into your header, but only in this case.