can't run a couchbase cluster with more than one node - amazon-ec2

Question

I trying Couchbase. I wrote some code (with the help of couchbase documentation). Reading and writing work very fine when I have 1 server.
When I put a 2nd server, I can't write/or read... Connection is very long (it was blazing fast with 1 server)
What am I doing wrong ?
The code is clearly working. This is something I do when I add a node on EC2 ? All ports are open (to the entire world). When I joined the cluster, rebalancing was ok.
Example: my cluster is up and running (here with 5 nodes on EC2):
Here is my log :
Connection:
2014-07-04 17:25:20.600 INFO net.spy.memcached.auth.AuthThread: Authenticated to /54.247.158.182:11210
2014-07-04 17:25:20.811 INFO com.couchbase.client.vbucket.provider.BucketConfigurationProvider: Could bootstrap through carrier publication.
write:
2014-07-04 17:25:20.817 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=/10.57.35.15:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2014-07-04 17:25:20.818 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=/10.250.110.205:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2014-07-04 17:25:20.818 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=/10.250.153.230:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2014-07-04 17:25:20.819 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=/10.59.13.200:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2014-07-04 17:25:20.819 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=/10.65.51.132:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
read:
2014-07-04 17:25:30.820 INFO com.couchbase.client.vbucket.provider.BucketConfigurationProvider: Could bootstrap through carrier publication.
2014-07-04 17:25:45.847 INFO com.couchbase.client.CouchbaseClient: CouchbaseConnectionFactory{bucket='Users', nodes=[http://54.247.158.182:8091/pools], order=RANDOM, opTimeout=2500, opQueue=16384, opQueueBlockTime=10000, obsPollInt=10, obsPollMax=500, obsTimeout=5000, viewConns=10, viewTimeout=75000, viewWorkers=1, configCheck=10, reconnectInt=1100, failureMode=Redistribute, hashAlgo=NATIVE_HASH, authWaitTime=2500}
Connected to CouchbaseConnectionFactory{bucket='Users', nodes=[http://54.247.158.182:8091/pools], order=RANDOM, opTimeout=2500, opQueue=16384, opQueueBlockTime=10000, obsPollInt=10, obsPollMax=500, obsTimeout=5000, viewConns=10, viewTimeout=75000, viewWorkers=1, configCheck=10, reconnectInt=1100, failureMode=Redistribute, hashAlgo=NATIVE_HASH, authWaitTime=2500}
2014-07-04 17:26:10.908 INFO com.couchbase.client.CouchbaseClient: viewmode property isn't defined. Setting viewmode to production mode
2014-07-04 17:26:10.989 INFO com.couchbase.client.CouchbaseConnection: Node for key "user101" is not active (yet). Queueing up for retry and checking for stale configuration.
2014-07-04 17:26:11.362 INFO com.couchbase.client.CouchbaseConnection: Node for key "user101" is not active (yet). Queueing up for retry and checking for stale configuration.
2014-07-04 17:26:12.363 INFO com.couchbase.client.CouchbaseConnection: Node for key "user101" is not active (yet). Queueing up for retry and checking for stale configuration.
2014-07-04 17:26:13.365 INFO com.couchbase.client.CouchbaseConnection: Node for key "user101" is not active (yet). Queueing up for retry and checking for stale configuration.
2014-07-04 17:26:13.490 WARN net.spy.memcached.MemcachedConnection: handling node for operation is not set
java.lang.RuntimeException: Timed out waiting for operation
at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:142)
at com.t2s.injector.Injector.insertData(Injector.java:71)
at com.t2s.injector.Injector.main(Injector.java:27)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: <unknown>
at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:167)
at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:140)
... 7 more
2014-07-04 17:27:27.995 INFO com.couchbase.client.CouchbaseConnection: Reconnecting due to failure to connect to {QA sa=10.250.110.205/10.250.110.205:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
java.net.ConnectException: Connexion terminée par expiration du délai d'attente
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:677)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:436)
at com.couchbase.client.CouchbaseConnection.run(CouchbaseConnection.java:324)
java.lang.RuntimeException: Timed out waiting for operation
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:75)
at com.couchbase.client.CouchbaseClient.getView(CouchbaseClient.java:430)
at com.t2s.injector.Injector.queryView(Injector.java:85)
at com.t2s.injector.Injector.main(Injector.java:33)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for operation
at com.couchbase.client.internal.HttpFuture.waitForAndCheckOperation(HttpFuture.java:93)
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:82)
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:72)
... 8 more
2014-07-04 17:27:53.249 WARN com.couchbase.client.CouchbaseConnection: Closing, and reopening {QA sa=10.250.110.205/10.250.110.205:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}, attempt 1.
2014-07-04 17:27:53.249 INFO com.couchbase.client.CouchbaseConnection: Reconnecting due to failure to connect to {QA sa=10.250.153.230/10.250.153.230:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
java.net.ConnectException: Connexion terminée par expiration du délai d'attente
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:677)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:436)
at com.couchbase.client.CouchbaseConnection.run(CouchbaseConnection.java:324)
2014-07-04 17:28:08.534 INFO com.couchbase.client.ViewConnection: I/O reactor terminated
java code :
package com.t2s.injector;
import com.couchbase.client.CouchbaseClient;
import com.couchbase.client.protocol.views.Query;
import com.couchbase.client.protocol.views.View;
import com.couchbase.client.protocol.views.ViewResponse;
import com.couchbase.client.protocol.views.ViewRow;
import com.google.gson.Gson;
import com.t2s.injector.data.User;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
public class Injector {
public static void main(String[] args) throws Exception {
System.out.println("Starting...");
try {
CouchbaseClient client = createClient("Users", "54.247.158.182");//, "54.74.200.186");
try {
insertData(client);
} catch (Exception e) {
e.printStackTrace();
}
try {
queryView(client, "users", "by_firstname");
} catch (Exception e) {
e.printStackTrace();
}
// Shutting down properly
client.shutdown(5, TimeUnit.SECONDS);
System.exit(0);
} catch (Exception e) {
e.printStackTrace();
System.exit(1);
}
}
private static CouchbaseClient createClient(String bucket, String... ips) throws URISyntaxException, IOException {
// (Subset) of nodes in the cluster to establish a connection
Integer port = 8091;
List<URI> nodes = new ArrayList<>();
for (String ip : ips) {
nodes.add(new URI("http://" + ip + ":" + port + "/pools"));
}
// Password of the bucket (empty) string if none
String password = "";
// Connect to the Cluster
CouchbaseClient client = new CouchbaseClient(nodes, bucket, password);
System.out.println("Connected to " + client);
return client;
}
private static void insertData(CouchbaseClient client) throws InterruptedException, ExecutionException {
Gson gson = new Gson();
User user1 = new User("John", "Doe");
User user2 = new User("Matt", "Ingenthron");
User user3 = new User("Michael", "Nitschinger");
client.set("user101", gson.toJson(user1)).get();
client.set("user201", gson.toJson(user2)).get();
client.set("user301", gson.toJson(user3)).get();
}
private static void queryView(CouchbaseClient client, String designDoc, String viewName) {
/*
function (doc, meta) {
if(doc.firstname) {
emit(doc.firstname, doc.lastname);
}
}
*/
// 1: Load the View infos
View view = client.getView(designDoc, viewName);
// 2: Create a Query object to customize the Query
Query query = new Query();
query.setIncludeDocs(true); // Include the full document body
// 3: Actually Query the View and return the results
ViewResponse response = client.query(view, query);
// 4: Iterate over the Data and print out the full document
for (ViewRow row : response) {
System.out.println(row.getId() + ": " + row.getDocument());
}
}
}

Answer 1

I suspect that your clients cannot connect to the IP addresses which are returned by the cluster nodes. EC2 normally has a public and private address - can you check that your app server(s) can connect on ports 8091 & 11210 to each node?
See http://docs.couchbase.com/couchbase-manual-2.5/cb-admin/index.html#using-couchbase-in-the-cloud for details on how to configure - you'll generally have an easier time if you configure the Couchbase nodes to use hostnames and not IP addresses.

Answer 2

Dave is right, it appears you are connecting on the public IP, which will work for the initial connection, but the cluster is responding back with the private IP address space of the cluster and therefor you cannot connect to those perhaps.
Just to confirm, is your client local in AWS or remote?
Also, it is normally a bad practice to have your DB directly connected to the Internet. Open to the world just makes me nervous to even hear. You want to have your web/app servers on the public side and your db servers quite private with a web proxy to the Couchbase admin interface perhaps. Even better would be to put all of this in a VPC with Couchbase being in private subnets and your web/app and proxy in public subnets.
One other thing. It is best practice to have n+1 nodes in your connection string when connecting to a cluster in case one of the nodes is down when the app starts. I see that you have only one is all.