I need to connect to a redis instance in my Elasticache. As I understand from Amazon Elasticache Redis cluster - Can't get Endpoint, I can get the endpoint from this.
Now suppose I get the endpoint and I use this endpoint to create a JedisClient(Since I use java) then How do I provide the AWS IAM credentials?
I am going to secure ElastiCache using IAM policies. How do I ensure no other application connects to this redis?
static AWSCredentials credentials = null;
static {
try {
//credentials = new ProfileCredentialsProvider("default").getCredentials();
credentials = new SystemPropertiesCredentialsProvider().getCredentials();
} catch (Exception e) {
System.out.println("Got exception..........");
throw new AmazonClientException("Cannot load the credentials from the credential profiles file. "
+ "Please make sure that your credentials file is at the correct "
+ "location (/Users/USERNAME/.aws/credentials), and is in valid format.", e);
}
}
#Bean
public LettuceConnectionFactory redisConnectionFactory() {
AmazonElastiCache elasticacheClient = AmazonElastiCacheClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(credentials)).withRegion(Regions.US_EAST_1).build();
DescribeCacheClustersRequest dccRequest = new DescribeCacheClustersRequest();
dccRequest.setShowCacheNodeInfo(true);
DescribeCacheClustersResult clusterResult = elasticacheClient.describeCacheClusters(dccRequest);
List<CacheCluster> cacheClusters = clusterResult.getCacheClusters();
List<String> clusterNodes = new ArrayList <String> ();
try {
for (CacheCluster cacheCluster : cacheClusters) {
for (CacheNode cacheNode : cacheCluster.getCacheNodes()) {
String addr = cacheNode.getEndpoint().getAddress();
int port = cacheNode.getEndpoint().getPort();
String url = addr + ":" + port;
if(<ReplicationGroup Name>.equalsIgnoreCase(cacheCluster.getReplicationGroupId()))
clusterNodes.add(url);
}
}
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
LettuceConnectionFactory redisConnectionFactory = new LettuceConnectionFactory(new RedisClusterConfiguration(clusterNodes));
redisConnectionFactory.setUseSsl(true);
redisConnectionFactory.afterPropertiesSet();
return redisConnectionFactory;
}
Related
We have a spring-boot application which is deployed to lambda in AWS.
Code
public AbstractRedisClient getClient(String host, String port) {
LOG.info("redis-uri" + "redis://"+host+":"+port);
return RedisClient.create("redis://"+host+":"+port);
}
/**
* Returns the Redis connection using the Lettuce-Redis-Client
*
* #return RedisClient
*/
public RedisClient getConnection(String host, String port) {
LOG.info("redis-Host " + host);
LOG.info("redis-Port " + port);
RedisClient redisClient = (RedisClient) getClient(host, port);
redisClient.setDefaultTimeout(Duration.ofSeconds(10));
return redisClient;
}
private RedisCommands<String, String> getRedisCommands() {
StatefulRedisConnection<String, String> statefulConnection = openConnection();
if(statefulConnection != null)
return statefulConnection.sync();
else
return null;
}
public StatefulRedisConnection<String, String> openConnection() {
if(connection != null && connection.isOpen()) {
return connection;
}
String redisPort = "6379";
String redisHost = environment.getProperty("REDIS_HOST");
//String redisPort = environment.getProperty("REDIS_PORT");
LOG.info("Host: {}", redisHost);
LOG.info("Port: {}", redisPort);
UnifiedReservationRedisConfig lettuceRedisConfig = new UnifiedReservationRedisConfig();
String redisUri = "redis://"+redisHost+":"+redisPort;
redisClient = lettuceRedisConfig.getConnection(redisHost, redisPort);
ConnectionFuture<StatefulRedisConnection<String, String>> future = redisClient
.connectAsync(StringCodec.UTF8, RedisURI.create(redisUri));
try {
connection = future.get();
} catch(InterruptedException | ExecutionException exception) {
LOG.info(exception.getMessage());
closeConnectionsAsync();
connection = null;
Thread.currentThread().interrupt();
}
return connection;
}
private void closeConnectionsAsync() {
LOG.info("Close redis connection");
if(connection != null && connection.isOpen()) {
connection.closeAsync();
}
if(redisClient != null) {
redisClient.shutdownAsync();
}
}
The issue was happening all time, But frequently geting this issue like Caused by io.lettuce.core.rediscommandexecutionexception: moved 15596 XX.X.XXX.XX:6379, Any one can help to solve this issue
As per my knowledge, you are doing port-forwarding to your redis cluster/instance using port 15596 but the actual redis ports like 6379 are not accessible from your application's network.
When redis's java client get the access to redis then tries to connect to actual ports like 6379.
Try using RedisClusterClient, instead of RedisClient. The unhandled MOVED response indicates that you are trying to use the non-cluster-aware client with Redis deployed in cluster mode.
I am using Elastic search, and it works well, but not when I try to use it with a webservice with jetty and jersey.
Here is an example of a function that I want to use :
public boolean insertUser(RestHighLevelClient client, User user) throws IOException
{
java.util.Map<String, Object> jsonMap = new HashMap<String, Object>();
jsonMap.put("username", user.username);
jsonMap.put("password", user.password);
jsonMap.put("mail", user.mail);
jsonMap.put("friends", user.friends);
jsonMap.put("maps", user.maps);
System.out.println("insertUser");
IndexRequest indexRequest = new IndexRequest("users", "doc",user.username)
.source(jsonMap);
try {
IndexResponse indexResponse = client.index(indexRequest);
System.out.println("insertUser 222");
if (indexResponse.getResult() == DocWriteResponse.Result.CREATED) {
System.out.println("user "+user.username+" créé");
}
else if (indexResponse.getResult() == DocWriteResponse.Result.UPDATED) {
System.out.println("user "+user.username+" update dans insertUser (pas normal)");
}
} catch (IOException e) {
e.printStackTrace();
return false;
}
return true;
}
This function works well when I try it inside a test class. But If i start my server like this :
Server server = new Server();
// Add a connector
ServerConnector connector = new ServerConnector(server);
connector.setHost("0.0.0.0");
connector.setPort(8081);
connector.setIdleTimeout(30000);
server.addConnector(connector);
DAO.ClientConnection("0.0.0.0",8081);
// Configure Jersey
ResourceConfig rc = new ResourceConfig();
rc.packages(true, "com.example.jetty_jersey.ws");
rc.register(JacksonFeature.class);
// Add a servlet handler for web services (/ws/*)
ServletHolder servletHolder = new ServletHolder(new ServletContainer(rc));
ServletContextHandler handlerWebServices = new ServletContextHandler(ServletContextHandler.SESSIONS);
handlerWebServices.setContextPath("/ws");
handlerWebServices.addServlet(servletHolder, "/*");
// Add a handler for resources (/*)
ResourceHandler handlerPortal = new ResourceHandler();
handlerPortal.setResourceBase("src/main/webapp/temporary-work");
handlerPortal.setDirectoriesListed(false);
handlerPortal.setWelcomeFiles(new String[] { "homepage.html" });
ContextHandler handlerPortalCtx = new ContextHandler();
handlerPortalCtx.setContextPath("/");
handlerPortalCtx.setHandler(handlerPortal);
// Activate handlers
ContextHandlerCollection contexts = new ContextHandlerCollection();
contexts.setHandlers(new Handler[] { handlerWebServices, handlerPortalCtx });
server.setHandler(contexts);
// Start server
server.start();
And when I enter a form, then call this webservice :
#POST
#Path("/signup")
#Produces(MediaType.APPLICATION_JSON)
// #Consumes(MediaType.APPLICATION_FORM_URLENCODED)
public SimpleResponse signup(#Context HttpServletRequest httpRequest,
#FormParam("username") String username,
#FormParam("email") String email,
#FormParam("password") String password,
#FormParam("passwordConfirm") String passwordConfirm) {
System.out.println("k");
//if (httpRequest.getSession().getAttribute("user") != null) { //httpRequest.getUserPrincipal() == null) {
try {
if (password.equals(passwordConfirm)) {
User user = new User("jeanOknewmail#gmail.com", "abc");
user.username = "jeanok";
user.maps = new ArrayList<String>();
user.friends = new ArrayList<String>();
System.out.println(user);
System.out.println("avant insert");
DAO.getActionUser().createIndexUser();
//System.out.println(DAO.getActionUser().getOneUser(DAO.client, "joe"));
System.out.println("rdctfygbhunji,k");
DAO.getActionUser().insertUser(DAO.client, user);
System.out.println("après insert");
return new SimpleResponse(true);
}
} catch (IOException e) {
e.printStackTrace();
}
//}
return new SimpleResponse(false);
}
I get lots of errors :
avax.servlet.ServletException: ElasticsearchStatusException[Unable to parse response body]; nested: ResponseException[method [PUT], host [http://0.0.0.0:8081], URI [/users/doc/jeanok?timeout=1m], status line [HTTP/1.1 404 Not Found]|];
...
Caused by:
ElasticsearchStatusException[Unable to parse response body]; nested: ResponseException[method [PUT], host [http://0.0.0.0:8081], URI [/users/doc/jeanok?timeout=1m], status line [HTTP/1.1 404 Not Found]|];
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:598)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:501)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:474)
at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:335)
at DAO.UserDAO.insertUser(UserDAO.java:160)
Do you have any idea why the behaviour of my function isn't the same when I launch my server? And why this error? Thanks for your help
I wasn't connected to elastic search. My client was connected to the wrong port. Now it works
I was trying setup LDAP connection pooling using spring security and xml based configuration.
Below is my configuration,
<authentication-manager id="authenticationManager">
<ldap-authentication-provider server-ref="ldapServer"
user-dn-pattern="uid={0},ou=users"
group-search-filter="(&(objectClass=groupOfUniqueNames)(uniqueMember={0}))"
group-search-base="ou=groups"
group-role-attribute="cn"
role-prefix="ROLE_"
user-context-mapper-ref="ldapContextMapperImpl">
</ldap-authentication-provider>
</authentication-manager>
How do i provide all the connection pooling configuration?
I am intending to use PoolingContextSource class as it provides properties to configure pool size etc.
They explicitly removed pooling for ldap binds (or in Spring's case an authenticate):
https://github.com/spring-projects/spring-ldap/issues/216
ldapTemplate.authenticate searches for the user and calls contextSource.getContext to perform the ldap bind.
private AuthenticationStatus authenticate(Name base,
String filter,
String password,
SearchControls searchControls,
final AuthenticatedLdapEntryContextCallback callback,
final AuthenticationErrorCallback errorCallback) {
List<LdapEntryIdentification> result = search(base, filter, searchControls, new LdapEntryIdentificationContextMapper());
if (result.size() == 0) {
String msg = "No results found for search, base: '" + base + "'; filter: '" + filter + "'.";
LOG.info(msg);
return AuthenticationStatus.EMPTYRESULT;
} else if (result.size() > 1) {
String msg = "base: '" + base + "'; filter: '" + filter + "'.";
throw new IncorrectResultSizeDataAccessException(msg, 1, result.size());
}
final LdapEntryIdentification entryIdentification = result.get(0);
try {
DirContext ctx = contextSource.getContext(entryIdentification.getAbsoluteName().toString(), password);
executeWithContext(new ContextExecutor<Object>() {
public Object executeWithContext(DirContext ctx) throws javax.naming.NamingException {
callback.executeWithContext(ctx, entryIdentification);
return null;
}
}, ctx);
return AuthenticationStatus.SUCCESS;
}
catch (Exception e) {
LOG.debug("Authentication failed for entry with DN '" + entryIdentification.getAbsoluteName() + "'", e);
errorCallback.execute(e);
return AuthenticationStatus.UNDEFINED_FAILURE;
}
}
By default, context sources disable pooling. From AbstractContextSource.java (which is what LdapContextSource inherits from):
public abstract class AbstractContextSource implements BaseLdapPathContextSource, InitializingBean {
...
public DirContext getContext(String principal, String credentials) {
// This method is typically called for authentication purposes, which means that we
// should explicitly disable pooling in case passwords are changed (LDAP-183).
return doGetContext(principal, credentials, EXPLICITLY_DISABLE_POOLING);
}
private DirContext doGetContext(String principal, String credentials, boolean explicitlyDisablePooling) {
Hashtable<String, Object> env = getAuthenticatedEnv(principal, credentials);
if(explicitlyDisablePooling) {
env.remove(SUN_LDAP_POOLING_FLAG);
}
DirContext ctx = createContext(env);
try {
authenticationStrategy.processContextAfterCreation(ctx, principal, credentials);
return ctx;
}
catch (NamingException e) {
closeContext(ctx);
throw LdapUtils.convertLdapException(e);
}
}
...
}
And if you try to use the PoolingContextSource, then you will get an UnsupportedOperationException when authenticate tries to call getContext:
public class PoolingContextSource
extends DelegatingBaseLdapPathContextSourceSupport
implements ContextSource, DisposableBean {
...
#Override
public DirContext getContext(String principal, String credentials) {
throw new UnsupportedOperationException("Not supported for this implementation");
}
}
This code is from the spring-ldap-core 2.3.1.RELEASE maven artifact.
You can still do connection pooling for ldap searches using the PoolingContextSource, but connection pooling for authenticates won't work.
Pooled connections doesn't work with authentication, because the way LDAP authentication works is that the connection is authenticated on creation.
This is my LdapTemplate Class
public LdapTemplate getLdapTemplete(String ldapID)
{
if (ldapID.equalsIgnoreCase(Constants.LDAP1))
{
if (ldapTemplate1 == null)
{
try
{
PasswordCredential passwordCredential = j2cAliasUtility.getAliasDetails(ldapID);
String managerDN = passwordCredential.getUserName();
String managerPwd = new String(passwordCredential.getPassword());
log.info("managerDN :"+managerDN+":: password : "+managerPwd);
LdapContextSource lcs = new LdapContextSource();
lcs.setUrl(ldapUrl1);
lcs.setUserDn(managerDN);
lcs.setPassword(managerPwd);
lcs.setDirObjectFactory(DefaultDirObjectFactory.class);
lcs.afterPropertiesSet();
ldapTemplate1 = new LdapTemplate(lcs);
log.info("ldap1 configured");
return ldapTemplate1;
}
catch (Exception e)
{
log.error("ldapContextCreater / getLdapTemplete / ldap2");
log.error("Error in getting ldap context", e);
}
}
return ldapTemplate1;
}
This is my J2CAliasUtility Class--I dont know what is this method doing and what does it return ?
public PasswordCredential getAliasDetails(String aliasName) throws Exception
{
PasswordCredential result = null;
try
{
// ----------WAS 6 change -------------
Map map = new HashMap();
map.put(com.ibm.wsspi.security.auth.callback.Constants.MAPPING_ALIAS, aliasName); //{com.ibm.mapping.authDataAlias=ldap1}
CallbackHandler cbh = (WSMappingCallbackHandlerFactory.getInstance()).getCallbackHandler(map, null);
LoginContext lc = new LoginContext("DefaultPrincipalMapping", cbh);
lc.login();
javax.security.auth.Subject subject = lc.getSubject();
java.util.Set creds = subject.getPrivateCredentials();
result = (PasswordCredential) creds.toArray()[0];
}
catch (Exception e)
{
log.info("APPLICATION ERROR: cannot load credentials for j2calias = " + aliasName);
log.error(" "+e);
throw new RuntimeException("Unable to get credentials");
}
return result;
}
J2C alias is a feature that encrypts the password used by the adapter to access the database. The adapter can use it to connect to the database instead of using a user ID and password stored in an adapter property.
J2C alias eliminates the need to store the password in clear text in an adapter configuration property, where it might be visible to others.
It would seem that your class "J2CAliasUtility" retrieves a user name and password from an JAAS (Java Authentication and Authorization Service) authentication alias, in this case apparently looked-up from LDAP. An auth alias may be configured in WebSphere Application Server as described here and here. Your code uses WebSphere security APIs to retrieve the user id and password from the given alias. More details on the programmatic logins and JAAS made be found in this IBM KnowledgeCenter topic and it's related topics.
I am using Spring jmx to create jmx client which can interact with Cassandra cluster to get a mbean object attribute Livedicsspaceused.
So this Cassandra cluster had 3 node hence different serviceUrl (each having different ip address).
Now I realize that while creating MBeanServerConnectionFactoryBean bean I can specify only one service URl like below:
#Bean
MBeanServerConnectionFactoryBean getConnector() {
MBeanServerConnectionFactoryBean mBeanfactory = new MBeanServerConnectionFactoryBean();
try {
mBeanfactory.setServiceUrl("serviceUrl1");
} catch (MalformedURLException e) {
e.printStackTrace();
}
mBeanfactory.setConnectOnStartup(false);
return mBeanfactory;
}
Then in main I am accessing this as below:
objectName = newObjectName(QueueServicesConstant.MBEAN_OBJ_NAME_LIVE_DISC_USED);
long count = (Long)mBeanFactory.getObject().getAttribute(objectName, QueueServicesConstant.MBEAN_ATTR_NAME_COUNT);
How can i get this value in all three nodes?
You need 3 distinct connectors.
Or you can use something like a Jolokia Proxy to access multiple servers (using REST instead of JSR 160).
This is how I solved the problem ..Instead of using Spring-JMX, I am directly using javax.management apis..So my code below will get any one of the connector which will be sufficient to provide me correct attribute value however it will try to connect to ohther node if it fails to get connector from one server node.
#SuppressWarnings("restriction")
private Object getMbeanAttributeValue(String MbeanObectName,
String attributeName) throws IOException,
AttributeNotFoundException, InstanceNotFoundException,
MBeanException, ReflectionException, MalformedObjectNameException {
Object attributeValue = null;
JMXConnector jmxc = null;
try {
State state = metaTemplate.getSession().getState();
List<String> serviceUrlList = getJmxServiceUrlList(state
.getConnectedHosts());
jmxc = getJmxConnector(serviceUrlList);
ObjectName objectName = new ObjectName(MbeanObectName);
MBeanServerConnection mbsConnection = jmxc
.getMBeanServerConnection();
attributeValue = mbsConnection.getAttribute(objectName,
attributeName);
} finally {
if (jmxc != null)
jmxc.close();
}
return attributeValue;
}
// This will provide any one of the JMX Connector of cassandra cluster
#SuppressWarnings("restriction")
private JMXConnector getJmxConnector(List<String> serviceUrlList)
throws IOException {
JMXConnector jmxc = null;
for (String serviceUrl : serviceUrlList) {
JMXServiceURL url;
try {
url = new JMXServiceURL(serviceUrl);
jmxc = JMXConnectorFactory.connect(url, null);
return jmxc;
} catch (IOException e) {
log.error(
"getJmxConnector: Error while connecting to JMX sereice {} ",
serviceUrl, e.getMessage());
}
}
throw new IOException(
"Not able to connect to any of Cassandra JMX connector.");
}