I am trying to connect to a Aerospike single node I set up using Vagrant on MacOSX. My AMC is running on localhost:2200. I am unable to connect to it successfully.
import com.aerospike.client.AerospikeClient;
public class AerospikeDriver {
public static void main(String[] args) {
AerospikeClient client = new AerospikeClient("127.0.0.1", 2200);
client.close();
}
}
I am getting this error in the first line itself. I tried changing the port to 3000 as well. Same error. Sometimes, I get SocketException as well.
Exception in thread "main" com.aerospike.client.AerospikeException$Connection: Error Code -8: Failed to connect to host(s):
127.0.0.1 2200 Error Code -1: java.io.EOFException
at com.aerospike.client.cluster.Cluster.seedNodes(Cluster.java:532)
at com.aerospike.client.cluster.Cluster.tend(Cluster.java:425)
at com.aerospike.client.cluster.Cluster.waitTillStabilized(Cluster.java:380)
at com.aerospike.client.cluster.Cluster.initTendThread(Cluster.java:286)
at com.aerospike.client.cluster.Cluster.<init>(Cluster.java:243)
at com.aerospike.client.AerospikeClient.<init>(AerospikeClient.java:234)
at com.aerospike.client.AerospikeClient.<init>(AerospikeClient.java:175)
at AerospikeDriver.main(AerospikeDriver.java:5)
My maven dependency for aerospike client is this:
<dependency>
<groupId>com.aerospike</groupId>
<artifactId>aerospike-client</artifactId>
<version>4.1.11</version>
</dependency>
This is my aerospike conf:
# Aerospike database configuration file.
# This stanza must come first.
service {
user root
group root
paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
pidfile /var/run/aerospike/asd.pid
# service-threads 4
# transaction-queues 4
# transaction-threads-per-queue 4
proto-fd-max 15000
node-id-interface eth1
}
logging {
# Log file must be an absolute path.
file /var/log/aerospike/aerospike.log {
context any info
}
file /var/log/aerospike/udf.log {
context udf info
context aggr info
}
}
network {
service {
address eth1
port 3000
# access-address <Published IP>
# access-address <NAT IP>
}
heartbeat {
mode multicast
multicast-group 239.1.99.222
address eth1
port 9918
protocol v3
# To use unicast-mesh heartbeats, comment out the 3 lines above and
# use the following 4 lines instead.
# mode mesh
# port 3002
# mesh-address 10.1.1.1
# mesh-port 3002
interval 150
timeout 10
}
fabric {
port 3001
address eth1
}
info {
port 3003
}
}
#namespace test {
# replication-factor 2
# memory-size 4G
# default-ttl 30d # 30 days, use 0 to never expire/evict.
#
# storage-engine memory
#}
namespace test {
replication-factor 2
memory-size 2G
default-ttl 5d # 5 days, use 0 to never expire/evict.
# To use file storage backing, comment out the line above and use the
# following lines instead.
storage-engine device {
file /opt/aerospike/data/test.dat
filesize 5G
data-in-memory true # Store data in memory in addition to file.
}
}
What am I doing wrong? Need some help here. I am very new to aerospike. I tried searching everywhere, but couldn't find anything.
UPDATE
I am now using IP address 172.28.128.4 (got it from ifconfig command) and port 3000 to connect to aerospike. I am now getting Socket Timeout Exception.
If you setup a single node on vagrant on Mac, and you running you are application in an ide on mac - say eclipse - locoalhost on vagrant is exposed to the mac as 172.28.128.3 typically. running ifconfig in your vagrant shell will confirm that. if your application is running inside vagrant itself, then 127.0.0.1 should work, in each case, your application should specify port 3000. thats where aerospike server is listening. amc is a webserver that talks to aerospike on port 3000 and serves the dashboard on port 8081 by default. so its a monitoring and management gateway to aerospike via a web browser. also, in your aerospike config, suggest you use mesh config instead of multicast though for a single node it does not matter - you are not making a cluster. If you are new, if you download CE, you get complimentary access to Aerospike Intro course in Aerospike Academy. Take advantage of that - few hours investment. Otherwise here are some intro videos on youtube. ( 02-Intro to Aerospike and 03-handson )
Related
I'm building an application on GKE (Google Kubernetes Engine) and a system using GCE instances of Redis.
When I try to connect from the application pod on GKE to Redis on GCE, I get connection refused.(dial tcp <REMOTE-IP>:6379: connect: connection refused)
The application is written in Go, and the redis library is go-redis(v8).
Why can't I connect?
The source code for the connection part and the part where the error occurs is as follows.
redisServerName = os.Getenv("REDIS_SERVER") // "sample.com:6379"
redisClient = redis.NewClient(&redis.Options{
Addr: redisServerName,
Password: "",
DB: 0,
})
p, err := redisClient.Ping(ctx).Result()
log.Println(p, err)
The hostname is resolved, so it is not a DNS problem, and the redis-cli commands are executable, so it does not seem to be a Firewall problem.
# redis-cli -h <REMOTE_IP> ping
PONG
Postscript
Here is the result of running the command from the Pod with the Go application running
/# redis-cli -h redis.sample.com
redis.sample.com:6379> // can connect
/# nc redis.sample.com 6379
// There is NO response.
I assert that every application in a container will have the same layer 4 (for redis, TCP) access to the network. Since Redis provides no significant access control, this means that if one app on your container has network access to redis server, all other apps on the same container will too. And if one can't contact redis, neither will the other.
On the same container. This is where testing gets tricky, because it isn't helpful or feasible to reproduce your k8s and gke config here.
ICMP Ping and tcp/6379 are different. Just because ping works, doesn't mean Redis can, and vise versa. And different containers will have different network access in k8s and gke.
Do this test on the app container, to take everything possible out of he equation.
apk add redis only pulls in a few packages, only 8MB and provides redis-cli when I tested, but you don't need any client app for redis; it's simple enough to be done with, say, netcat
. You don't have to issue a valid redis cmd, either - if you get an -ERR unknown command response, you know network works:
/ # echo "hi, redis!" |nc localhost 6379
-ERR unknown command `hi,`, with args beginning with: `redis!`,
If it works there and not in Go, it's probably because the environment variable REDIS_SERVER isn't set properly. So you might want to test that at the command line, as well.
nc $REDIS_SERVER 6379
I was trying to run consul in vagrant using the following comman
consul agent -dev -advertise 172.20.20.31
but it showing following error message
failed to get conn: dial tcp 127.0.0.1:0->172.20.20.31:8300: connect: invalid argument"
as a result when i connect form my local machine to this ip and run consul ui mode it showing
500 (The backend responded with an error)
my vagrant file is as follow:
Vagrant.configure("2") do |config|
config.vm.box = "centos/7"
config.vm.define "centos-consul" do |cs|
cs.vm.hostname = "centos-consul"
cs.vm.network "private_network",ip:"172.20.20.31"
end
end
and connect file is as
{
"ui": true,
"retry_join": ["172.20.20.31"],
"advertise_addr": "172.20.20.01",
"data_dir": "/tmp/consul/self"
}
I am using consul 1.7.2 in mac os and Vagrant 2.2.7 centos7
I am at a lost what i did wrong.
so any suggestion will be helpful.
According to the documentation, the error you are getting is the result of IP address 172.20.20.31 being not routable from your Vagrant box.
-advertise - The advertise address is used to change the address that we advertise to other nodes in the cluster. By default, the -bind
address is advertised. However, in some cases, there may be a routable
address that cannot be bound. This flag enables gossiping a different
address to support this. If this address is not routable, the node
will be in a constant flapping state as other nodes will treat the
non-routability as a failure. In Consul 1.0 and later this can be set
to a go-sockaddr template.
I'm experiencing intermittent failed to response when make an outbound connection such as RPC call, it is logged by my application (Java) like this :
org.apache.http.NoHttpResponseException: RPC_SERVER.com:443 failed to respond !
Outbound connection flow
Kubernetes Node -> ELB for internal NGINX -> internal NGINX ->[Upstream To]-> ELB RPC server -> RPC server instance
This problem is not occurred on usual EC2 (AWS).
I'm able to reproduce on my localhost by doing this
Run main application which act as client in port 9200
Run RPC server in port 9205
Client will make a connection to server using port 9202
Run $ socat TCP4-LISTEN:9202,reuseaddr TCP4:localhost:9205 that will listen on port 9202 and then forward it to 9205 (RPC Server)
Add rules on iptables using $ sudo iptables -A INPUT -p tcp --dport 9202 -j DROP
Trigger a RPC calling, and it will return the same error message as I desrcibe before
Hypothesis
Caused by NAT on kubernetes, as far as I know, NAT is using conntrack, conntrack and break the TCP connection if it was idle for some period of time, client will assume the connection is still established although it isn't. (CMIIW)
I also have tried scaling kube-dns into 10 replica, and the problem still occurred.
Node Specification
Use calico as network plugin
$ sysctl -a | grep conntrack
net.netfilter.nf_conntrack_acct = 0
net.netfilter.nf_conntrack_buckets = 65536
net.netfilter.nf_conntrack_checksum = 1
net.netfilter.nf_conntrack_count = 1585
net.netfilter.nf_conntrack_events = 1
net.netfilter.nf_conntrack_expect_max = 1024
net.netfilter.nf_conntrack_generic_timeout = 600
net.netfilter.nf_conntrack_helper = 1
net.netfilter.nf_conntrack_icmp_timeout = 30
net.netfilter.nf_conntrack_log_invalid = 0
net.netfilter.nf_conntrack_max = 262144
net.netfilter.nf_conntrack_tcp_be_liberal = 0
net.netfilter.nf_conntrack_tcp_loose = 1
net.netfilter.nf_conntrack_tcp_max_retrans = 3
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 3600
net.netfilter.nf_conntrack_tcp_timeout_established = 86400
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
net.netfilter.nf_conntrack_timestamp = 0
net.netfilter.nf_conntrack_udp_timeout = 30
net.netfilter.nf_conntrack_udp_timeout_stream = 180
net.nf_conntrack_max = 262144
Kubelet config
[Service]
Restart=always
Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
Environment="KUBELET_CLOUD_ARGS=--cloud-provider=aws"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_EXTRA_ARGS $KUBELET_CLOUD_ARGS
Kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean", BuildDate:"2017-08-31T09:14:02Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.7", GitCommit:"8e1552342355496b62754e61ad5f802a0f3f1fa7", GitTreeState:"clean", BuildDate:"2017-09-28T23:56:03Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Kube-proxy Log
W1004 05:34:17.400700 8 server.go:190] WARNING: all flags other than --config, --write-config-to, and --cleanup-iptables are deprecated. Please begin using a config file ASAP.
I1004 05:34:17.405871 8 server.go:478] Using iptables Proxier.
W1004 05:34:17.414111 8 server.go:787] Failed to retrieve node info: nodes "ip-172-30-1-20" not found
W1004 05:34:17.414174 8 proxier.go:483] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
I1004 05:34:17.414288 8 server.go:513] Tearing down userspace rules.
I1004 05:34:17.443472 8 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_max' to 262144
I1004 05:34:17.443518 8 conntrack.go:52] Setting nf_conntrack_max to 262144
I1004 05:34:17.443555 8 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I1004 05:34:17.443584 8 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I1004 05:34:17.443851 8 config.go:102] Starting endpoints config controller
I1004 05:34:17.443888 8 config.go:202] Starting service config controller
I1004 05:34:17.443890 8 controller_utils.go:994] Waiting for caches to sync for endpoints config controller
I1004 05:34:17.443916 8 controller_utils.go:994] Waiting for caches to sync for service config controller
I1004 05:34:17.544155 8 controller_utils.go:1001] Caches are synced for service config controller
I1004 05:34:17.544155 8 controller_utils.go:1001] Caches are synced for endpoints config controller
$ lsb_release -s -d
Ubuntu 16.04.3 LTS
Check the value of sysctl net.netfilter.nf_conntrack_tcp_timeout_close_wait inside the pod that contains your program. It is possible that the value on the node that you listed (3600) isn't the same as the value inside the pod.
If the value in the pod is too small (e.g. 60), and your Java client half-closes the TCP connection with a FIN when it finishes transmitting, but the response takes longer than the close_wait timeout to arrive, nf_conntrack will lose the connection state and your client program will not receive the response.
You may need to change the behavior of the client program to not use a TCP half-close, OR modify the value of net.netfilter.nf_conntrack_tcp_timeout_close_wait to be larger. See https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/.
This is my config file stored at /etc/logstash/conf
input
{
file{
path => ["PATH_OF_FILE"]
}
}
output
{
elasticsearch
{
host => "172.29.86.35"
index => "new"
}
}
and this is my elasticsearch.yaml file content for network and http
\# Set the bind address specifically (IPv4 or IPv6):
\#network.bind_host: 172.29.86.35
\# Set the address other nodes will use to communicate with this node. If not
\# set, it is automatically derived. It must point to an actual IP address.
\#network.publish_host: 192.168.0.1
\# Set both 'bind_host' and 'publish_host':
network.host: 172.29.86.35
\# Set a custom port for the node to node communication (9300 by default):
\#transport.tcp.port: 9300
\# Enable compression for all communication between nodes (disabled by default):
\#transport.tcp.compress: true
\# Set a custom port to listen for HTTP traffic:
\#http.port: 9200
I am running elasticsearch and logstash as service.The problem is when I start log stash as a service it does not send any data to elasticsearch. However if I use the same config in the logstash conf file and run logstash from the CLI it works perfectly fine. Even the logs do not show any error.
The version I am running is 1.4.3 for ES and 1.4.2 for LS.
The system env is RHEL 7
I also have encountered same issue...
When I exec command using -f option, it works normally, but when I start service, nothing happen and log file under /etc/log stash never updated.
What I did as the temporary counter measure is to exec the command below(with & option)
Logstash if conffile.conf &
With this, it work even if I logout from server.
I installed hadoop cluster using bdutil (instead of click to deploy). I am not able to access job tracker page at locahost:50030/jobtracker.jsp (https://cloud.google.com/hadoop/running-a-mapreduce-job)
I am checking it locally using lynx instead of from my client browser (so localhost instead of external ip)
My setting in my config file for bdutil is
MASTER_UI_PORTS=('8088' '50070' '50030')
but after deploying the hadoop cluster when I do firewall rules list I get following
NAME NETWORK SRC_RANGES RULES SRC_TAGS TARGET_TAGS
default-allow-http default 0.0.0.0/0 tcp:80,tcp:8080 http-server
default-allow-https default 0.0.0.0/0 tcp:443 https-server
default-allow-icmp default 0.0.0.0/0 icmp
default-allow-internal default 10.240.0.0/16 tcp:1-65535,udp:1-65535,icmp
default-allow-rdp default 0.0.0.0/0 tcp:3389
default-allow-ssh default 0.0.0.0/0 tcp:22
Now I dont see port 50030 in the list of rules. Why so?
so I run a command to add them (manually)
gcloud compute firewall-rules create allow-http --description "Incoming http allowed." --allow tcp:50030 --format json
Now it gets added and I can see in the output of firewall-rules list command.
But still when I do lynx locahost:50030/jobtracker.jsp I get unable to connect. Then, I run a hadoop job so that there is some output to view and then run lynx command but still see unable to connect.
Can someone tell me where I am going wrong in this complete process?
An ephemeral IP is an external IP. The difference between an ephemeral IP and a static IP is that a static IP can be reassigned to another virtual machine instance, while an ephemeral IP is released when the instance is destroyed. An ephemeral IP can be promoted to a static IP through the web UI or the gcloud command-line tool.
You can obtain the external IP of your host by querying the metadata API at http://169.254.169.254/0.1/meta-data/network. The response will be a JSON document that looks like this (pretty-printed for clarity):
{
"networkInterface" : [
{
"network" : "projects/852299914697/networks/rabbit",
"ip" : "10.129.14.59",
"accessConfiguration" : [
{
"externalIp" : "107.178.223.11",
"type" : "ONE_TO_ONE_NAT"
}
]
}
]
}
The firewall rule command seems reasonable, but you may want to choose a more descriptive name. If I saw a rule that said allow-http, I would assume it meant port 80. You may also want to restrict it to a target tag placed on your Hadoop dashboard instance; as written, your rule will allow access on that port to all instances in the current project.