I try to manage network segmentation in a Ignite cluster that is embedded in a tomcat webapp. I create a custom segmentationResolver that ping database server.
So I configure Ignite with spring context
<beans:bean id="segmentationResolver" class="my.cluster.JdbcPingSegmentationResolver"/>
<beans:bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<beans:property name="gridLogger">
<beans:bean class="org.apache.ignite.logger.slf4j.Slf4jLogger"/>
</beans:property>
<beans:property name="workDirectory" value="#{systemProperties[igniteTmpDir]}"/>
<beans:property name="segmentationResolvers" ref="segmentationResolver" />
<beans:property name="segmentCheckFrequency" value="10000" />
<beans:property name="segmentationPolicy" value="NOOP" />
<beans:property name="waitForSegmentOnStart" value="true" />
<beans:property name="communicationSpi">....
Constructor of my JdbcPingSegmentationResolver is call by spring but its method isValidSegment() is never calls. Even if I start / stop another Ignite node (it has to be called when cluster topology change).
#Override
public boolean isValidSegment() throws IgniteCheckedException {
var reachable = false;
try {
Process exec = Runtime.getRuntime().exec("ping " + hostname);
//0 - normal termination
reachable = exec.waitFor(1, TimeUnit.SECONDS);
} catch (IOException | InterruptedException e) {
LOGGER.error("segmentation network : {}", e.getMessage(), e);
}
LOGGER.debug("check segmentation network status : {}", reachable);
return reachable;
}
Segmentation resolvers are called by a GridSegmentationProcessor. The default processor is org.apache.ignite.internal.processors.segmentation.os.GridOsSegmentationProcessor which is noop in fact. GridGain Enterprise Edition (it's a vendor-supported delivery of Apache Ignite) specifies a processor that handles resolvers properly.
I believe that the machinery that's being called on topology changes (node left, join etc) is topology validator.
Related
Spring MVC + Java 8 + Tomcat 8 stack
I am maintaining my configuration in yaml and flattening the properties using Spring's PropertyPlaceholderConfigurer and maintaining the configuration in a bean.
Today, it has a inherent problem as I am required to restart the server whenever there is a change to the YML files.
I believe there are ways to refresh the bean without restart, but my main concern is how to do in fail safe manner.
Lets assume, there was a request and that time the config was A, and then we refresh the configuration so now its B, but if any subsequent user request was dependent on the configuration, then it will blow up.
Add this configuration to your servlet-context.xml to catch property changes on the fly:
<context:property-placeholder
location="file:${A_CONFIG_LOCATION}/configuration.properties" />
<beans:bean id="propertiesLoader"
class="org.springframework.context.support.ReloadableResourceBundleMessageSource">
<beans:property name="cacheSeconds" value="1" />
<beans:property name="basenames">
<beans:list>
<beans:value>file:${A_CONFIG_LOCATION}/configuration
</beans:value>
</beans:list>
</beans:property>
</beans:bean>
And then you can read property values like this:
#Component
public class PropertiesReader {
private String value = "some_default_value";
#Autowired
MessageSource propertiesLoader;
public String getValue() {
value = propertiesLoader.getMessage("configuration.value", null, null);
return value;
}
}
I am trying this use case:
Poll for a message in a queue->transform the message->Call a method with the transformed message.
Here is my code
<jms:message-driven-channel-adapter id="jmsIn"
destination-name="test"
channel="jmsInChannel"/>
<channel id="jmsInChannel"/>
<channel id="consoleOut"/>
<int:transformer input-channel="jmsInChannel" ref="xmlMsgToVORPojoTransformer" output-channel="consoleOut">
</int:transformer>
<beans:bean id="xmlMsgToVORPojoTransformer" class="com.order.jmspublisher.ValidateOrderMessageTransformer">
<beans:property name="unmarshaller" ref="marshaller" />
</beans:bean>
<beans:bean id="marshaller" class="org.springframework.oxm.jaxb.Jaxb2Marshaller">
<beans:property name="classesToBeBound">
<beans:list>
<beans:value>com.order.jmspublisher.ValidateOrderResponseType</beans:value>
</beans:list>
</beans:property>
</beans:bean>
<logging-channel-adapter id="consoleOutloggerChannel" channel="consoleOut" log-full-message="true" level="DEBUG"/>
<int:service-activator id="sa" input-channel="consoleOut" output-channel="someChannel" method="handleVOR">
<beans:bean id="vorActivator" class="com.order.jmspublisher.VORServiceActivator"/>
</int:service-activator>
My Transformer code is as follows:
#SuppressWarnings("rawtypes")
public Message transform(String message)
{
try {
XMLInputFactory xif = XMLInputFactory.newFactory();
XMLStreamReader xsr = xif.createXMLStreamReader(new StreamSource(new StringReader(message)));
xsr.nextTag(); // Advance to Envelope tag
while(xsr.hasNext() ) {
int next = xsr.next();
if(next == XMLStreamReader.START_ELEMENT)
if(xsr.getLocalName().equals("ValidateOrderResponse"))
{
break;
}
}
ValidateOrderResponseType vor = (ValidateOrderResponseType)marshaller.unmarshal(new StAXSource(xsr));
return MessageBuilder.withPayload(vor).build();
} catch (XmlMappingException e) {
return MessageBuilder.withPayload(e).build();
} catch(Exception e){
return MessageBuilder.withPayload(e).build();
}
}
========================================================================
My service activator method code is as follows:
public class VORServiceActivator {
private static final Logger logger = LoggerFactory.getLogger(VORServiceActivator.class);
#ServiceActivator
public String handleVOR(ValidateOrderResponseType vor)
{
logger.info("vor information received and invoke some services here....\r\n"+vor);
return vor.toString();
}
}
=========================================================================
My service activator method is not getting called. But my transform is getting called and I can see the same in logs.
Please help me to know where I am going wrong.
Thanks in advance.
<channel id="consoleOut"/> is a DirectChannel; you have 2 consumers subscribed to the channel (logging adapter and service activator).
The default dispatcher will distribute messages to these competing consumers in round-robin fashion; so each will get alternate messages.
You need to change consoleOut to a <publish-subscribe-channel /> if you want both consumers to get all messages.
You can read about channel types here.
For Spring security setup in Spring Boot. The LDAP Authentication provider is configured by default to use BindAuthenticator class.
This Class contains method
/**
* Allows subclasses to inspect the exception thrown by an attempt to bind with a
* particular DN. The default implementation just reports the failure to the debug
* logger.
*/
protected void handleBindException(String userDn, String username, Throwable cause) {
if (logger.isDebugEnabled()) {
logger.debug("Failed to bind as " + userDn + ": " + cause);
}
}
This Method is to handle the authentication related Exceptions like invalid credentials.
I want to over-ride this method so i can handle this issue and return proper error message on the basis of error codes returned by LDAP. like invalid password or the account is locked.
Current LDAP implementation always returns "Bad Credentials" that does not give the right picture that why my credentials are invalid. i want to cover the cases
where the account is Locked
password is expired so i can redirect to change password
account locked due to number of invalid password retries
Please help
The issue i fixed by defining the LDAP context instead of using the Spring Boot LDAPAuthenticationProviderConfigurer.
Then created the FilterBasedLdapUserSearch and Over-written the BindAuthentication with my ConnectBindAuthenticator.
i created a separate LDAPConfiguration class for spring boot configuration and registered all these custom objects as Beans.
From the above Objects i created LDAPAuthenticationProvider by passing my Custom Objects to constructor
The Config is as below
#Bean
public DefaultSpringSecurityContextSource contextSource() {
DefaultSpringSecurityContextSource contextSource = new DefaultSpringSecurityContextSource(env.getProperty("ldap.url"));
contextSource.setBase(env.getProperty("ldap.base"));
contextSource.setUserDn(env.getProperty("ldap.managerDn"));
contextSource.setPassword(env.getProperty("ldap.managerPassword"));
return contextSource;
}
#Bean
public ConnectBindAuthenticator bindAuthenticator() {
ConnectBindAuthenticator connectBindAuthenticator = new ConnectBindAuthenticator(contextSource());
connectBindAuthenticator.setUserSearch(ldapUserSearch());
connectBindAuthenticator.setUserDnPatterns(new String[]{env.getProperty("ldap.managerDn")});
return connectBindAuthenticator;
}
#Bean
public LdapUserSearch ldapUserSearch() {
return new FilterBasedLdapUserSearch("", env.getProperty("ldap.userSearchFilter"), contextSource());
}
You have to change your spring security configuration to add your extension of BindAuthenticator:
CustomBindAuthenticator.java
public class CustomBindAuthenticator extends BindAuthenticator {
public CustomBindAuthenticator(BaseLdapPathContextSource contextSource) {
super(contextSource);
}
#Override
protected void handleBindException(String userDn, String username, Throwable cause) {
// TODO: Include here the logic of your custom BindAuthenticator
if (somethingHappens()) {
throw new MyCustomException("Custom error message");
}
super.handleBindException(userDn, username, cause);
}
}
spring-security.xml
<beans:bean id="contextSource"
class="org.springframework.security.ldap.DefaultSpringSecurityContextSource">
<beans:constructor-arg value="LDAP_URL" />
<beans:property name="userDn" value="USER_DN" />
<beans:property name="password" value="PASSWORD" />
</beans:bean>
<beans:bean id="userSearch"
class="org.springframework.security.ldap.search.FilterBasedLdapUserSearch">
<beans:constructor-arg index="0" value="USER_SEARCH_BASE" />
<beans:constructor-arg index="1" value="USER_SEARCH_FILTER" />
<beans:constructor-arg index="2" ref="contextSource" />
</beans:bean>
<beans:bean id="ldapAuthProvider"
class="org.springframework.security.ldap.authentication.LdapAuthenticationProvider">
<beans:constructor-arg>
<beans:bean class="com.your.project.CustomBindAuthenticator">
<beans:constructor-arg ref="contextSource" />
<beans:property name="userSearch" ref="userSearch" />
</beans:bean>
</beans:constructor-arg>
</beans:bean>
<security:authentication-manager alias="authenticationManager">
<security:authentication-provider ref="ldapAuthProvider" />
</security:authentication-manager>
Hope it's helpful.
We are using spring integration adapters for file ftp in our project, the problem we are facing is, the adapters are not closing the open socket connections.
As a result, other modules which are in the same managed server are failing with "Too many open files" socket connection exception. Is there a way to close the unused open socket connections from the channel adapters Or Can we get the underlying jsch connections and close the sockets from sftp channel adapters.
We have tried caching session factory and it did not close the open sockets. The file handles kept on piling up. Thanks in advance for the inputs.
We have two xmls one with outboundAdapter and the other with InboundAdapter. These two are in different xmls as they are different jobs that are run using spring batch. We are expected to send files to a location.
We are using spring batch 2.2.0 and spring integration 2.1.6 and spring integration 2.1.6.
Here is the configuration:
We have one session factory and it is wrapped by cachingSession factory:
<beans:bean id="sftpSessionFactory"
class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<beans:property name="host" value="hostname"/>
<beans:property name="privateKey" value="somepath"/>
<beans:property name="port" value="22"/>
</beans:bean>
<bean id="cachingSessionFactory"
class="org.springframework.integration.file.remote.session.CachingSessionFactory">
<constructor-arg ref="sftpSessionFactory"/>
<constructor-arg value="10"/>
<property name="sessionWaitTimeout" value="1000"/>
</bean>
**and then we have a channel**
<int:channel id="ftpChannel" />
**and then we have the following outbound Channel adapter**
<int-sftp:outbound-channel-adapter id="sftpOutboundAdapter"
session-factory="cachingSessionFactory"
channel="inputChannel"
charset="UTF-8"
use-temporary-filename="false"/>
**With the above configuration we are using the ftpChannel to send the files by constructing a payload like this:**
message = MessageBuilder.withPayLoad(f).build() // MessageBuilder is //org.springframework.integration.support.MessageBuilder and f is the file
ftpChannel.send(message)
**In another inbound job, the following is the configuration of adapters:
Session factory:**
<beans:bean id="sftpSessionFactory2"
class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<beans:property name="host" value="hostname"/>
<beans:property name="privateKey" value="somepath"/>
<beans:property name="port" value="22"/>
</beans:bean>
**Caching session factory:**
<bean id="cachingSessionFactory2"
class="org.springframework.integration.file.remote.session.CachingSessionFactory">
<constructor-arg ref="sftpSessionFactory2"/>
<constructor-arg value="10"/>
<property name="sessionWaitTimeout" value="1000"/>
</bean>
**and another channel:**
<int:channel id="ftpChannel2" />
**Now we have the following adapter in this xml:**
<int-sftp:outbound-channel-adapter id="sftpInboundAdapter"
session-factory="cachingSessionFactory2"
channel="inputChannel"
charset="UTF-8"
use-temporary-filename="false"/>
With this configuration in the above xml we are trying to get session from the cachingSessionFactory configured in the first xml, getting a session out of it, getting a list of files and then sending some files with ftpChannel2.send() and doing session.close() in finally block. When I do session.isOpen() in after session.close(), I see true being returned.
With these two jobs, I could see a lot of open file handles, which are socket connections and I am absolutely clueless as to how I can close those opened sockets.
The session will be closed when the operation is complete as long as you don't use the caching session factory - that is intended to keep the session open for the next use.
If you turn on DEBUG logging, you should get some insight into what it wrong.
EDIT
Just ran this with no problems:
#Test
public void test() throws Exception {
DefaultFtpSessionFactory sf = new DefaultFtpSessionFactory();
sf.setHost("10.0.0.3");
sf.setUsername("ftptest");
sf.setPassword("ftptest");
FtpSession session = sf.getSession();
Thread.sleep(10000);
session.close();
assertFalse(session.isOpen());
System.out.println("closed");
Thread.sleep(10000);
}
During the first sleep netstat -ntp shows the socket open; socket is gone after the close.
The session is the socket...
public void disconnect() throws IOException
{
closeQuietly(_socket_);
...
}
EDIT2
I had forgotten that with 2.1.x there was the cache-sessions attribute (2.1.x is very old).
I just tested with this (and 2.1.6) ...
<bean id="sftpSessionFactory"
class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<property name="host" value="10.0.0.3" />
<property name="privateKey" value="file:/somPathTo/.ssh/id_rsa" />
<property name="port" value="22" />
<property name="user" value="ftptest" />
</bean>
<int:channel id="inputChannel" />
<int-sftp:outbound-channel-adapter id="sftpOutboundAdapter"
session-factory="sftpSessionFactory"
channel="inputChannel"
charset="UTF-8"
cache-sessions="false"
use-temporary-file-name="false"
remote-directory="." />
public class Main {
public static void main(String[] args) throws Exception {
ConfigurableApplicationContext context = new ClassPathXmlApplicationContext("context.xml");
File f = new File("foo.txt");
FileOutputStream fos = new FileOutputStream(f);
fos.write("bar".getBytes());
fos.close();
context.getBean("inputChannel", MessageChannel.class).send(MessageBuilder.withPayload(f).build());
System.out.println("Sleeping - check socket");
Thread.sleep(60000); // check socket
context.close();
System.exit(0);
}
}
With no problems (the socket is closed); if I set the cache-sessions to true, the socket remains open as expected.
I do notice you don't have a remote-directory attribute - that's illegal:
exactly one of 'remote-directory' or 'remote-directory-expression' is required on a remote file outbound adapter
I am trying to use BasicDataSource to pool connections with JDBCTemplate in a spring application. From everything I've read, this should be really simple: Just configure the BasicDataSource in XML, inject the data source into a bean, and in the setter method, create a new JDBCTemplate.
When I did this, I noticed my performance was terrible. So then I switched to Spring's SingleConnectionDataSource, just to see what would happen, and my performance got much better. I started investigating with a profiler tool, and I noticed that when using BasicDataSource, a new connection was being created for every query.
Investigating further, I can see where the connection is being closed after the query is finished. Specifically in Spring's DataSourceUtil class:
public static void doReleaseConnection(Connection con, DataSource dataSource) throws SQLException {
if (con == null) {
return;
}
if (dataSource != null) {
ConnectionHolder conHolder = (ConnectionHolder) TransactionSynchronizationManager.getResource(dataSource);
if (conHolder != null && connectionEquals(conHolder, con)) {
// It's the transactional Connection: Don't close it.
conHolder.released();
return;
}
}
// Leave the Connection open only if the DataSource is our
// special SmartDataSoruce and it wants the Connection left open.
if (!(dataSource instanceof SmartDataSource) || ((SmartDataSource) dataSource).shouldClose(con)) {
logger.debug("Returning JDBC Connection to DataSource");
con.close();
}
}
The thing that I notice is there is some special logic for 'SmartDataSource' which leaves the connection open. This partially explains the behavior I was seeing: Since SingleConnectionDataSource implements SmartDataSource, the connection is not closed. However, I thought by using BasicDataSource, the close() method on the connection would just return the connection to the pool. However, when I look at what is happening in my profiler, the close method is actually being called on my sybase connection: not any kind of 'Pooled Connection Wrapper' like I would expect to see.
One last thing (this is what I'm going to investigate now): I use TransactionTemplate for some of my queries (involving commits to the database), but simple queries are not inside a transactionTemplate. I don't know if that has anything to do with the problem or not.
EDIT 1:
Ok finally got some more time to investigate after getting pulled off the project a bit and, here is a very simple test that shows the problem
public class DBConnectionPoolTest {
#Autowired
#Qualifier("myDataSource")
private DataSource dataSource;
#Test
public void test() throws Exception{
JdbcTemplate template = new JdbcTemplate(dataSource);
StopWatch sw = new StopWatch();
sw.start();
for(int i=0; i<1000; i++){
template.queryForInt("select count(*) from mytable");
}
sw.stop();
System.out.println("TIME: " + sw.getTotalTimeSeconds() + " seconds");
}}
Here are my two datasource configurations:
<bean id="myDataSource" class="org.springframework.jdbc.datasource.SingleConnectionDataSource">
<property name="driverClassName" value="${db.driver}" />
<property name="url" value="${db.url}" />
<property name="username" value="${db.username}" />
<property name="password" value="${db.password}" />
</bean>
<bean id="myDataSource" class="org.apache.commons.dbcp.BasicDataSource">
<property name="driverClassName" value="${db.driver}" />
<property name="url" value="${db.url}" />
<property name="username" value="${db.username}" />
<property name="password" value="${db.password}" />
</bean>
When I run the test with the first configuration, it takes about 2.1 seconds. When I run it with the second configuration, it takes about 4.5 seconds. I have tried various parameters on BasicDataSource, such as setting maxActive=1 and testOnBorrow=false, but nothing makes a difference.
I think the problem in my case was that my jdbc libraries for sybase were outdated.