I'm experiencing that the Kubernetes API server fails to start during cluster bootstrapping with the following error log, apparently due to being unable to initialize its "client CA configmap":
E1029 14:35:56.211083 5 client_ca_hook.go:78] Timeout: request did not complete within allowed duration
F1029 14:35:56.211121 5 hooks.go:126] PostStartHook “ca-registration” failed: unable to initialize client CA configmap: timed out waiting for the condition
It seems to happen here in the Kubernetes source code. What might cause this error?
See the full log here.
Update: It seems that my etcd cluster isn't accessible from master nodes, even though the same command works from etcd member machines:
$ sudo ETCDCTL_API=3 etcdctl --cacert=/opt/tectonic/tls/etcd-client-ca.crt \
--cert=/opt/tectonic/tls/etcd-client.crt --key=/opt/tectonic/tls/etcd-client.key \
--endpoints=https://coreos-testing-etcd-0.socialfoodie.club:2379 \
endpoint health
https://coreos-testing-etcd-0.socialfoodie.club:2379 is unhealthy: failed to connect: grpc: timed out when dialing
Error: unhealthy cluster
I found out that despite the cryptic error message in the API server, the cause is that it can't write to the etcd cluster. The reason was that the API server was configured with a different client certificate authority than what the etcd cluster was using, due to a timing issue wrt. copying certificates in my Terraform cluster setup. I figured out that the CA was the problem by using curl to contact the etcd cluster instead of etcdctl, as it gave a clear error message.
Thanks to #johnharris85 for suggesting etcd connectivity being an issue!
Related
I am trying to find the correct endpoint to use to connect to a minio bucket. I am running minio on a minikube cluster, and I am using argo workflows to launch pods. When I give the addresss I use to login to minio (http://127.0.0.1:29941/), I get:
Error (exit code 1): failed to create new S3 client: Endpoint url cannot have fully qualified paths.
Or when I use minio:9000 as endpoint i get:
Error (exit code 1): failed to put file: Get "http://minio:9000/my-bucket/?location=": dial tcp: lookup minio on 10.96.0.10:53: server misbehaving
Turned out to be the name of the service with the port. In my case for argo workflow it was:
argo-artifacts:9000
Getting this message when trying to install wget in RHEL EC2 instance. How to resolve this?
Error: Failed to download metadata for repo 'rhui-client-config-server-8': Cannot prepare internal mirrorlist: Curl error (28): Timeout was reached for https://rhui3.ap-south-1.aws.ce.redhat.com/pulp/mirror/protected/rhui-client-config/rhel/server/8/x86_64/os [Connection timed out after 30000 milliseconds]
If you set up SSH for the EC2 instance, try and SSH into the machine, and try
curl -I https://www.google.com
and if it times out, you perhaps can't connect to the internet.
You need to check if the relevant Security Group has an outbound rule allowing connections to the internet:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-security-groups.html?icmpid=docs_ec2_console#security-group-rules
There are a few useful hints in https://access.redhat.com/discussions/4656371 as well, if the issue is related to your subscription.
I tried to connect gcloud with gcloud init. I got Network connection error.
I tried to fix with Network connection problems in Google Cloud SDK while I have access to google or website like google app engine in my browser
gcloud config set proxy/type socks5
gcloud config set proxy/address 127.0.0.1
gcloud config set proxy/port 1086
But it still does not work for me.
My log is:
Welcome! This command will take you through the configuration of gcloud.
Settings from your current configuration [default] are:
core:
disable_usage_reporting: 'False'
proxy:
address: 127.0.0.1
port: '1080'
type: socks5
Pick configuration to use:
[1] Re-initialize this configuration [default] with new settings
[2] Create a new configuration
[3] Switch to and re-initialize existing configuration: [cindy]
Please enter your numeric choice: 1
Your current configuration has been set to: [default]
You can skip diagnostics next time by using the following flag:
gcloud init --skip-diagnostics
Network diagnostic detects and fixes local network connection issues.
Checking network connection...done.
ERROR: Reachability Check failed.
Cannot reach https://www.google.com (error)
Cannot reach https://accounts.google.com (error)
Cannot reach https://cloudresourcemanager.googleapis.com/v1beta1/projects (error)
Cannot reach https://www.googleapis.com/auth/cloud-platform (error)
Cannot reach https://dl.google.com/dl/cloudsdk/channels/rapid/components-2.json (error)
Network connection problems may be due to proxy or firewall settings.
Current effective Cloud SDK network proxy settings:
type = socks5
host = 127.0.0.1
port = 1080
username = None
password = None
What would you like to do?
[1] Change Cloud SDK network proxy properties
[2] Clear all gcloud proxy properties
[3] Exit
It is due to redirection from goolge account to localhost for acquiring token Try to login using firefox
Considering the error and mainly the following part of the error, it seems that you are facing issues with your proxy and firewall settings and configuration.
Network connection problems may be due to proxy or firewall settings.
Due to this error, you need to configure your SDK to be used with proxy and firewall. I would recommend you to take a look at the documentation Configuring Cloud SDK for use behind a proxy/firewall, to get more information and steps on how to achieve it and avoid the error that you are facing.
Besides that, I could find the below two posts from the Community, of users that are facing similar cases that you are facing.
ERROR: Reachability Check failed. #51
gcloud utility not working #25
Let me know if the information helped you!
For the people who still have this problem.
Try to use IPV4 first as IPV6 is still buggy for many systems.
For Linux, you can just uncomment or add the following line in /etc/gai.conf file:
precedence ::ffff:0:0/96 100
For other systems, you can google the set-up
A few minutes ago I clone search guard branch from here and I does everythink what README said.
After docker-compose up -d all services are working but elasticsearch_1 log one error every few secounds:
elasticsearch_1 | [2018-09-14T08:59:49,614][ERROR][c.f.s.a.BackendRegistry ] Not yet initialized (you may need to run sgadmin)
After that I run docker-compose exec -T elasticsearch bin/init_sg.sh, output:
Search Guard Admin v6
Will connect to localhost:9300 ... done
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/usr/share/elasticsearch/plugins/search-guard-6/netty-common-4.1.16.Final.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Elasticsearch Version: 6.3.2
Search Guard Version: 6.3.2-23.0
Connected as CN=kirk,OU=client,O=client,L=Test,C=DE
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
ERR: Timed out while waiting for a green or yellow cluster state.
* Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
* Make also sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
* If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
* Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
I guess that sgadmin can't connect to elasticsearch cluster but I does everythink what README said.
Any sugestions how to fix this?
Thanks for answers.
I already resolve this. Your product, works complitely fine. I had error with index win Kibana which make elasticsearch cluster RED status - never YELLOW.
If you want connect your sgadmin with elasticsearch cluster without waiting for YELLOW status use line: --accept-red-cluster in init_sg.sh script:
#!/bin/sh
plugins/search-guard-6/tools/sgadmin.sh \
-cd config/sg/ \
-ts config/sg/truststore.jks \
-ks config/sg/kirk-keystore.jks \
-nhnv \
-icl \
--accept-red-cluster
Then everything works fine and Kibana will show you why you have RED status - in my case kibana index problem.
Created new EC2 instance of neo4j via CloudFormation template found here (ubuntu host).
https://github.com/neo4j-contrib/ec2neo
Got the web interface to work fine, and DB is up and running.
Trying to connect with neo4j-shell from my local dev machine, and I am able to establish a connection to the remote EC2 server.
$ neo4j-shell -host ec2-xx-xx-xx-xx.compute-1.amazonaws.com
Welcome to the Neo4j Shell! Enter 'help' for a list of commands
NOTE: Remote Neo4j graph database service 'shell' at port 1337
neo4j-sh (?)$
netstat confirms that a connection has been ESTABLISHED
tcp6 0 0 xx.xx.xx.xx:1337 my.local.ip.add:13785 ESTABLISHED
At this point, I type help, or any neo4j command, and I get no response back from the server. The console just hangs. As soon as I stop the neo4j service on the server, I get the following exception on the client console.
java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is:
java.io.EOFException
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:229)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:194)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:148)
at com.sun.proxy.$Proxy1.interpretLine(Unknown Source)
at org.neo4j.shell.impl.AbstractClient.evaluate(AbstractClient.java:149)
at org.neo4j.shell.impl.AbstractClient.evaluate(AbstractClient.java:133)
at org.neo4j.shell.impl.AbstractClient.grabPrompt(AbstractClient.java:101)
at org.neo4j.shell.StartClient.grabPromptOrJustExecuteCommand(StartClient.java:383)
at org.neo4j.shell.StartClient.startRemote(StartClient.java:330)
at org.neo4j.shell.StartClient.start(StartClient.java:196)
at org.neo4j.shell.StartClient.main(StartClient.java:135)
Caused by: java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:267)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:215)
... 11 more
I have made the following change to neo4j-wrapper.conf :
wrapper.java.additional=-Djava.rmi.server.hostname=ec2-xx-xx-xx-xx.compute-1.amazonaws.com
All iptables are "disabled", to eliminate variables. I am able to run neo4j-shell on the server itself, to 127.0.0.1
What am I missing in my network config or neo4j server config?
Try to ssh into the instance and run it there. remote connections have been a pain for a long time because of the underlying Java RMI port handling.
You can also try out cycli which supports http and auth.