Unable to connect to Locator via GFSH - spring-boot

I have started a GemFire Server and Locator via Spring Boot and when I try to connect to the Locator from GFSH, I getting the following issue:
gfsh> connect
Connecting to Locator at [host=localhost, port=10334] ..
Connection refused: connect
Below, is the Spring (Java) configuration:
#Configuration
#ComponentScan
#EnableGemfireRepositories(basePackages= "com.gemfire.demo")
#CacheServerApplication(locators = "localhost[10334]")
#EnableManager
public class GemfireConfiguration {
#Bean
Properties gemfireProperties() {
Properties gemfireProperties = new Properties();
gemfireProperties.setProperty("name", "SpringDataGemFireApplication");
gemfireProperties.setProperty("mcast-port", "0");
gemfireProperties.setProperty("log-level", "info");
return gemfireProperties;
}
#Bean
#Autowired
CacheFactoryBean gemfireCache() {
CacheFactoryBean gemfireCache = new CacheFactoryBean();
gemfireCache.setClose(true);
gemfireCache.setProperties(gemfireProperties());
return gemfireCache;
}
#Bean(name="employee")
#Autowired
LocalRegionFactoryBean<String, Employee> getEmployee(final GemFireCache cache) {
LocalRegionFactoryBean<String, Employee> employeeRegion = new LocalRegionFactoryBean<String, Employee>();
employeeRegion.setCache(cache);
employeeRegion.setClose(false);
employeeRegion.setName("employee");
employeeRegion.setPersistent(false);
employeeRegion.setDataPolicy(DataPolicy.PRELOADED);
return employeeRegion;
}
}
Ref:Spring Data Gemfire locator
as per John's advice, I have enabled the Manager though I am still unable to connect.

You are not able to connect to the Locator (using Gfsh) because you don't have a Locator (service, either standalone or embedded) running with just the Spring (Java) config shown above.
Note that the #CacheServerApplication(locators = "localhost[10334]") annotation, specifically with the locators attribute as you have configured above, does NOT start an embedded Locator. It simply allows this Spring Boot configured and bootstrapped Apache Geode or Pivotal GemFire peer Cache node to join an existing distributed system (cluster) with an "existing" Locator running on localhost, listening on port 10334.
For instance, you could have started a Locator using Gfsh (e.g. start locator --name=X ...), then started your Spring Boot application with the Spring (Java) config shown above and you would see the Spring Boot app as part of the cluster formed by the Gfsh started Locator.
It is simply a shortcut (convenience) to configure and start an "embedded" Locator, but to do so, you need to use the #EnableLocator annotation.
Therefore, to configure and start an (embedded) Locator service in the same Spring Boot application as the CacheServer (and Manager), you must also include the #EnableLocator annotation, like so:
#SpringBootApplicaton
#CacheServerApplication
#EnableLocator
#EnableManager(start = true)
public class GemFireServerApplication {
...
}
I have plenty of examples of this here, for instance here, and talk about this generally here, etc.
As a side note, your whole configuration (class) is confused and it is clear you don't quite understand what you are doing. For instance, declaring the gemfireProperties and gemfireCache beans in JavaConfig is redundant and unnecessary since you are using the #CacheServerApplication annotation. Your whole configuration could be simplified to:
#CacheServerApplication(
name = "SpringDataGemFireApplication",
locators = "localhost[10334]",
logLevel = "info"
)
#EnableLocator
#EnableManager(start = true)
#EnableGemfireRepositories(basePackages= "com.gemfire.demo")
#ComponentScan
public class GemfireConfiguration {
#Bean(name="employee")
LocalRegionFactoryBean<String, Employee> getEmployee(GemFireCache cache) {
LocalRegionFactoryBean<String, Employee> employeeRegion =
new LocalRegionFactoryBean<String, Employee>();
employeeRegion.setCache(cache);
employeeRegion.setClose(false);
employeeRegion.setName("employee");
employeeRegion.setPersistent(false);
employeeRegion.setDataPolicy(DataPolicy.PRELOADED);
return employeeRegion;
}
}
Two things:
1) First, I would be highly careful about using classpath component scanning (#ComponentScan). I am not a fan of this configuration approach, especially in production where things should be explicit.
2) I would encourage your to considering using the type-safe basePackageClasses attribute on the #EnableGemFireRepositorities annotation rather than the basePackages attribute. With basePackageClasses, you only need to refer to a single interface/class in the desired package (e.g. com.gemfire.demo) rather than every interface/class. The referenced interface/class serves as a pointer to identify the package to scan from, including all sub-packages. It is type-safe and when your interfaces/classes in that package are re-located, then your attribute is still valid after the refactoring.
Anyway...
Hope this helps.
-j

Related

Include newly added data sources into route Data Source object without restarting the application server

Implemented Spring's AbstractRoutingDatasource by dynamically determining the actual DataSource based on the current context.
Refered this article : https://www.baeldung.com/spring-abstract-routing-data-source.
Here on spring boot application start up . Created a map of contexts to datasource objects to configure our AbstractRoutingDataSource. All these client context details are fetched from a database table.
#Bean
#DependsOn("dataSource")
#Primary
public DataSource routeDataSource() {
RoutingDataSource routeDataSource = new RoutingDataSource();
DataSource defaultDataSource = (DataSource) applicationContext.getBean("dataSource");
List<EstCredentials> credentials = LocalDataSourcesDetailsLoader.getAllCredentails(defaultDataSource); // fetching from database table
localDataSourceRegistrationBean.registerDataSourceBeans(estCredentials);
routeDataSource.setDefaultTargetDataSource(defaultDataSource);
Map<Object, Object> targetDataSources = new HashMap<>();
for (Credentials credential : credentials) {
targetDataSources.put(credential.getEstCode().toString(),
(DataSource) applicationContext.getBean(credential.getEstCode().toString()));
}
routeDataSource.setTargetDataSources(targetDataSources);
return routeDataSource;
}
The problem is if i add a new client details, I cannot get that in routeDataSource. Obvious reason is that these values are set on start up.
How can I achieve to add new client context and I had to re intialize the routeDataSource object.
Planning to write a service to get all the client context newly added and reset the routeDataSource object, no need to restart the server each time any changes in the client details.
A simple solution to this situation is adding #RefreshScope to the bean definition:
#Bean
#Primary
#RefreshScope
public DataSource routeDataSource() {
RoutingDataSource routeDataSource = new RoutingDataSource();
DataSource defaultDataSource = (DataSource) applicationContext.getBean("dataSource");
List<EstCredentials> credentials = LocalDataSourcesDetailsLoader.getAllCredentails(defaultDataSource); // fetching from database table
localDataSourceRegistrationBean.registerDataSourceBeans(estCredentials);
routeDataSource.setDefaultTargetDataSource(defaultDataSource);
Map<Object, Object> targetDataSources = new HashMap<>();
for (Credentials credential : credentials) {
targetDataSources.put(credential.getEstCode().toString(),
(DataSource) applicationContext.getBean(credential.getEstCode().toString()));
}
routeDataSource.setTargetDataSources(targetDataSources);
return routeDataSource;
}
Add Spring Boot Actuator as a dependency:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
Then trigger the refresh endpoint POST to /actuator/refresh to update the DataSource (actually every refresh scoped bean).
So this will depend on how much you know about the datasources to be added, but you could set this up as a multi-tenant project. Another example of creating new datasources:
#Autowired private Map <String, Datasource> mars2DataSources;
public void addDataSourceAtRuntime() {
DataSourceBuilder dataSourcebuilder = DataSourcebuilder.create(
MultiTenantJPAConfiguration.class.getclassloader())
.driverclassName("org.postgresql.Driver")
.username("postgres")
.password("postgres")
.url("Jdbc: postgresql://localhost:5412/somedb");
mars2DataSources("tenantX", datasourcebuilder.build())
}
Given that you are using Oracle, you could also use its database change notification features.
Think of it as a listener in the JDBC driver that gets notified whenever something changes in your database table. So upon receiving a change, you could reinitialize/add datasources.
You can find a tutorial of how to do this here: https://docs.oracle.com/cd/E11882_01/java.112/e16548/dbchgnf.htm#JJDBC28820
Though, depending on your organization database notifications need some extra firewall settings for the communication to work.
Advantage: You do not need to manually call the REST Endpoint if something changes, (though Marcos Barberios answer is perfectly valid!)

Problem with connection to Neo4j test container using Spring boot 2 and JUnit5

Problem with connection to Neo4j test container using Spring boot 2 and JUnit5
int test context. Container started successfully but spring.data.neo4j.uri property has a wrong default port:7687, I guess this URI must be the same when I call neo4jContainer.getBoltUrl().
Everything works fine in this case:
#Testcontainers
public class ExampleTest {
#Container
private static Neo4jContainer neo4jContainer = new Neo4jContainer()
.withAdminPassword(null); // Disable password
#Test
void testSomethingUsingBolt() {
// Retrieve the Bolt URL from the container
String boltUrl = neo4jContainer.getBoltUrl();
try (
Driver driver = GraphDatabase.driver(boltUrl, AuthTokens.none());
Session session = driver.session()
) {
long one = session.run("RETURN 1",
Collections.emptyMap()).next().get(0).asLong();
assertThat(one, is(1L));
} catch (Exception e) {
fail(e.getMessage());
}
}
}
But SessionFactory is not created for the application using autoconfiguration following to these recommendations - https://www.testcontainers.org/modules/databases/neo4j/
When I try to create own primary bean - SessionFactory in test context I get the message like this - "URI cannot be returned before the container is not loaded"
But Application runs and works perfect using autoconfiguration and neo4j started in a container, the same cannot be told about the test context
You cannot rely 100% on Spring Boot's auto configuration (for production) in this case because it will read the application.properties or use the default values for the connection.
To achieve what you want to, the key part is to create a custom (Neo4j-OGM) Configuration bean. The #DataNeo4jTest annotation is provided by the spring-boot-test-autoconfigure module.
#Testcontainers
#DataNeo4jTest
public class TestClass {
#TestConfiguration
static class Config {
#Bean
public org.neo4j.ogm.config.Configuration configuration() {
return new Configuration.Builder()
.uri(databaseServer.getBoltUrl())
.credentials("neo4j", databaseServer.getAdminPassword())
.build();
}
}
// your tests
}
For a broader explanation have a look at this blog post. Esp. the section Using with Neo4j-OGM and SDN.

Set heartbeatintervalseconds using spring xml

I am using spring-data-Cassandra v1.3.2 in my project.
Is it possible to set heartbeatintervalseconds using spring configuration XML file.
Getting 4 lines of hearbeat DEBUG logs every 30 seconds in my application logs and i am not sure how to avoid them.
Unfortunately, no.
After reviewing the SD Cassandra CassandraCqlClusterParser class, it is apparent that you can specify both "local" and "remote" connection pooling options, however, neither handler handles all the Cassandra Java driver "pooling options" appropriately (such as heartbeatIntervalSeconds).
It appears several other options are missing as well: idleTimeoutSeconds, initializationExecutor, poolTimeoutMillis, and protocolVersion.
Equally unfortunate is it appears the SD Cassandra PoolOptionsFactoryBean does not support these "pooling options" either.
However, not all is lost.
While your SD Cassandra application may resolve it's configuration primarily from XML, it does not preclude you from using a combination of Java config and XML.
For instance, you could use a Spring Java config class to configure your cluster and express your PoolingOptions in Java config...
#Configuration
#ImportResource("/class/path/to/cassandra/config.xml")
class CassandraConfig {
#Bean
PoolingOptions poolingOptions() {
PoolingOptions poolingOptions = new PoolingOptions();
poolingOptions.setHeartbeatIntervalSeconds(30);
poolingOptions.setIdleTimeoutSeconds(300);
poolingOptions.setMaxConnectionsPerHost(50);
poolingOptions.set...
return poolingOptions;
}
#Bean
CassandraClusterFactoryBean cluster() {
CassandraClusterFactoryBean cluster = new CassandraClusterFactoryBean()
cluster.setContactPoints("..");
cluster.setPort(1234);
cluster.setPoolingOptions(poolingOptions());
cluster.set...
return cluster;
}
}
Hope this helps.
As an FYI, you may want to upgrade to the "current" Spring Data Cassandra version, 1.4.1.RELEASE.
Sadly, but the answer is no. It's not possible to configure the heartbeat interval using XML configuration. Only the following local/remote properties can be configured in PoolingOptions:
min-simultaneous-requests
max-simultaneous-requests
core-connections
max-connections
If you switch to Java-based configuration, then you're able to configure PoolingOptions by extending AbstractClusterConfiguration:
#Configuration
public class MyConfig extends AbstractClusterConfiguration {
#Override
protected PoolingOptions getPoolingOptions() {
PoolingOptions poolingOptions = new PoolingOptions();
poolingOptions.setHeartbeatIntervalSeconds(10);
return poolingOptions
}
}

How to set custom connection properties on DataSource in Spring Boot 1.3.x with default Tomcat connection pool

I need to set some specific Oracle JDBC connection properties in order to speed up batch INSERTs (defaultBatchValue) and mass SELECTs (defaultRowPrefetch).
I got suggestions how to achieve this with DBCP (Thanks to M. Deinum) but I would like to:
keep the default Tomcat jdbc connection pool
keep application.yml for configuration
I was thinking about a feature request to support spring.datasource.custom_connection_properties or similar in the future and because of this tried to pretent this was already possible. I did this by passing the relevant information while creating the DataSource and manipulated the creation of the DataSource like this:
#Bean
public DataSource dataSource() {
DataSource ds = null;
try {
Field props = DataSourceBuilder.class.getDeclaredField("properties");
props.setAccessible(true);
DataSourceBuilder builder = DataSourceBuilder.create();
Map<String, String> properties = (Map<String, String>) props.get(builder);
properties.put("defaultRowPrefetch", "1000");
properties.put("defaultBatchValue", "1000");
ds = builder.url( "jdbc:oracle:thin:#xyz:1521:abc" ).username( "ihave" ).password( "wonttell" ).build();
properties = (Map<String, String>) props.get(builder);
log.debug("properties after: {}", properties);
} ... leaving out the catches ...
}
log.debug("We are using this datasource: {}", ds);
return ds;
}
In the logs I can see that I am creating the correct DataSource:
2016-01-18 14:40:32.924 DEBUG 31204 --- [ main] d.a.e.a.c.config.DatabaseConfiguration : We are using this datasource: org.apache.tomcat.jdbc.pool.DataSource#19f040ba{ConnectionPool[defaultAutoCommit=null; ...
2016-01-18 14:40:32.919 DEBUG 31204 --- [ main] d.a.e.a.c.config.DatabaseConfiguration : properties after: {password=wonttell, driverClassName=oracle.jdbc.OracleDriver, defaultRowPrefetch=1000, defaultBatchValue=1000, url=jdbc:oracle:thin:#xyz:1521:abc, username=ihave}
The actuator shows me that my code replaced the datasource:
But the settings are not activated, which I can see while profiling the application. The defaultRowPrefetch is still at 10 which causes my SELECTs to be much slower than they would be if 1000 was activated.
Setting the pools connectionProperties should work. Those will be passed to the JDBC driver. Add this to application.properties:
spring.datasource.connectionProperties: defaultRowPrefetch=1000;defaultBatchValue=1000
Edit (some background information):
Note also that you can configure any of the DataSource implementation
specific properties via spring.datasource.*: refer to the
documentation of the connection pool implementation you are using for
more details.
source: spring-boot documentation
As Spring Boot is EOL for a long time I switched to Spring Boot 2.1 with its new default connection pool Hikari. Here the solution is even more simply and can be done in the application.properties or (like shown here) application.yml:
spring:
datasource:
hikari:
data-source-properties:
defaultRowPrefetch: 1000
(In a real-life config there would be several other configuration items but as they are not of interest for the question asked I simply left them out in my example)
Some additional information to complement the answer by #Cyril. If you want to upvote use his answer, not mine.
I was a little bit puzzled how easy it is to set additional connection properties that in the end get used while creating the database connection. So I did a little bit of research.
spring.datasource.connectionProperties is not mentioned in the reference. I created an issue because of this.
If I had used the Spring Boot YML editor, I would have seen which properties are supported. Here is what STS suggests when you create an application.yml and hit Ctrl+Space:
The dash does not matter because of relaxed binding but if you interpret it literally the propertys name is spring.datasource.connection-properties.
The correct setup in application.yml looks like this:
spring:
datasource:
connection-properties: defaultBatchValue=1000;defaultRowPrefetch=1000
...
This gets honored which is proven by my perf4j measurements of mass SELECTs.
Before:
2016-01-19 08:58:32.604 INFO 15108 --- [ main]
org.perf4j.TimingLogger : start[1453190311227]
time[1377] tag[get elements]
After:
2016-01-19 08:09:18.214 INFO 9152 --- [ main]
org.perf4j.TimingLogger : start[1453187358066]
time[147] tag[get elements]
The time taken to complete the SQL statement drops from 1377ms to 147, which is an enormous gain in performance.
After digging around in the Tomcat code for a bit, I found that the dataSource.getPoolProperties().getDbProperties() is the Properties object that will actually get used to generate connections for the pool.
If you use the BeanPostProcessor approach mentioned by #m-deinum, but instead use it to populate the dbProperties like so, you should be able to add the properties in a way that makes them stick and get passed to the Oracle driver.
import java.util.Properties;
import org.apache.tomcat.jdbc.pool.DataSource;
import org.apache.tomcat.jdbc.pool.PoolConfiguration;
#Component
public class OracleConfigurer implements BeanPostProcessor {
#Override
public Object postProcessBeforeInitialization(Object bean, String name) throws BeansException {
if (bean instanceof DataSource) {
DataSource dataSource = (DataSource)bean;
PoolConfiguration configuration = dataSource.getPoolProperties();
Properties properties = configuration.getDbProperties();
if (null == properties) properties = new Properties();
properties.put("defaultRowPrefetch", 1000);
properties.put("defaultBatchValue", 1000);
configuration.setDbProperties(properties);
}
return bean;
}
#Override
public Object postProcessAfterInitialization(Object bean, String name) throws BeansException {
return bean;
}
}

Camel: use datasource configured by spring-boot

I have a project and in it I'm using spring-boot-jdbc-starter and it automatically configures a DataSource for me.
Now I added camel-spring-boot to project and I was able to successfully create routes from Beans of type RouteBuilder.
But when I'm using sql component of camel it can not find datasource. Is there any simple way to add Spring configured datasource to CamelContext? In samples of camel project they use spring xml for datasource configuration but I'm looking for a way with java config. This is what I tried:
#Configuration
public class SqlRouteBuilder extends RouteBuilder {
#Bean
public SqlComponent sqlComponent(DataSource dataSource) {
SqlComponent sqlComponent = new SqlComponent();
sqlComponent.setDataSource(dataSource);
return sqlComponent;
}
#Override
public void configure() throws Exception {
from("sql:SELECT * FROM tasks WHERE STATUS NOT LIKE 'completed'")
.to("mock:sql");
}
}
I have to publish it because although the answer is in the commentary, you may not notice it, and in my case such a configuration was necessary to run the process.
The use of the SQL component should look like this:
from("timer://dbQueryTimer?period=10s")
.routeId("DATABASE_QUERY_TIMER_ROUTE")
.to("sql:SELECT * FROM event_queue?dataSource=#dataSource")
.process(xchg -> {
List<Map<String, Object>> row = xchg.getIn().getBody(List.class);
row.stream()
.map((x) -> {
EventQueue eventQueue = new EventQueue();
eventQueue.setId((Long)x.get("id"));
eventQueue.setData((String)x.get("data"));
return eventQueue;
}).collect(Collectors.toList());
})
.log(LoggingLevel.INFO,"******Database query executed - body:${body}******");
Note the use of ?dataSource=#dataSource. The dataSource name points to the DataSource object configured by Spring, it can be changed to another one and thus use different DataSource in different routes.
Here is the sample/example code (Java DSL). For this I used
Spring boot
H2 embedded Database
Camel
on startup spring-boot, creates table and loads data. Then camel route, runs "select" to pull the data.
Here is the code:
public void configure() throws Exception {
from("timer://timer1?period=1000")
.setBody(constant("select * from Employee"))
.to("jdbc:dataSource")
.split().simple("${body}")
.log("process row ${body}")
full example in github

Resources