I am using Spring integration sftp to put files on a remote server and below is configuration.
<spring-integration.version>5.2.5.RELEASE</spring-integration.version>
I have configurated #MessagingGateway.
#MessagingGateway
public interface SftpMessagingGateway {
#Gateway(requestChannel = "sftpOutputChannel")
void sendToFTP(Message<?> message);
}
I have configured the MessageHandler as below,
#Bean
#ServiceActivator(inputChannel = "sftpOutputChannel" )
public MessageHandler genericOutboundhandler() {
SftpMessageHandler handler = new SftpMessageHandler(outboundTemplate(), FileExistsMode.APPEND);
handler.setRemoteDirectoryExpressionString("headers['remote_directory']");
handler.setFileNameGenerator((Message<?> message) -> ((String) message.getHeaders().get(Constant.FILE_NAME_KEY)));
handler.setUseTemporaryFileName(false);
return handler;
}
I have configured SftpRemoteFileTemplate as below
private SftpRemoteFileTemplate outboundTemplate;
public SftpRemoteFileTemplate outboundTemplate(){
if (outboundTemplate == null) {
outboundTemplate = new SftpRemoteFileTemplate(sftpSessionFactory());
}
return outboundTemplate;
}
This is the configuration for SessionFactory
public SessionFactory<LsEntry> sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory();
factory.setHost(host);
factory.setPort(port);
factory.setUser(username);
factory.setPassword(password);
factory.setAllowUnknownKeys(true);
factory.setKnownHosts(host);
factory.setSessionConfig(configPropertiesOutbound());
CachingSessionFactory<LsEntry> csf = new CachingSessionFactory<LsEntry>(factory);
csf.setSessionWaitTimeout(1000);
csf.setPoolSize(10);
csf.setTestSession(true);
return csf;
}
I have configured all this in one of the service.
Now the problem is,
Sometimes the entire operation takes more than 15 min~ specially if the service is ideal for few hours and I am not sure what is causing this issue.
It looks like it is spending time on getting the active session from CachedSessionFactory the after operations are pretty fast below is the snap from one of the tool where I have managed to capture the time.
It usually takes few miliseconds to transfer files.
I have recently made below changes but before that as well I was getting the same issue,
I have set isShareSession to false earlier it was DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
There was no pool size I have set it to 10
I think I have configured something incorrectly and that's why I end up piling connection ? Or there is something else ?
Observation :
The time taking to complete the operation is somewhat similar all the time when issue occurs i.e 938000 milliseconds +
If I restart the application daily it works perfectly fine.
Related
Background: I am calling a REST API to download a text file from a spring boot app. The problem is that the server takes time to generate the file and then make it ready for download.
So, I am using RetryTemplate to wait for 1 minute and attempt 60 times (60 attempts in 1 hour). When the download file is not ready, the server response will be empty, but when the download file is ready, the server will respond with the file URL.
Problem: Now the problem is, the way I've configured RetryTemplate, it'll keep calling the API even if there is an exception from the server.
Desired behavior: RetryTemplate should retry calling the API only when the server response is EMPTY and should NOT retry if the server response contains the download file URL or on a server-side exception.
RetryTemplate bean configuration:
#Bean
public RetryTemplate retryTemplate() {
final SimpleRetryPolicy retryPolicy = new SimpleRetryPolicy();
retryPolicy.setMaxAttempts(60);
final FixedBackOffPolicy backOffPolicy = new FixedBackOffPolicy();
backOffPolicy.setBackOffPeriod(60000L);
final RetryTemplate retryTemplate = new RetryTemplate();
retryTemplate.setRetryPolicy(retryPolicy);
retryTemplate.setBackOffPolicy(backOffPolicy);
return retryTemplate;
}
Service class method where it's used:
#Override
public DownloadResult getDownloadFile(final int id) {
// retry for 1 hour to download the file. Call to EBL will be made every minute for 60 minutes until the file is ready
final DownloadResult result = retryTemplate.execute(downloadFile -> dataAccess.getFile(id));
final InputStream inputStream = Try.of(result::getFileUrl)
.map(this::download)
.onFailure(throwable -> log.error("MalformedURLException: {}", result.getFileUrl()))
.get();
return result.toBuilder().downloadFileInputStream(inputStream).build();
}
The server response when download file is ready for download looks like this:
{
fileUrl:"www.download.com/fileId"
}
Empty server response if download file is not ready for download:
{
}
Thanks in advance for your help!
spring-retry is entirely based on exceptions. You need to check the result within the lambda and throw an exception...
retryTemplate.execute(downloadFile -> {
DownloadResult result = dataAccess.getFile(id));
if (resultIsEmpty(result)) {
throw new SomeException();
}
return result;
}
My use case is that I have got a spring boot application with a JdbcPollingChannelAdapter to fetch data from a postgresql database, updating the fetched rows and moving foreward with message flow (using IntegrationFlowBuilder) to process some transform to the ResultSet and publish the results to RabbitMQ.
JdbcPollingChannelAdapter is configured to fetch data each 60 seonds with a select for update query followed by an update query to flag the status form NEW to PUBLISH status:
The sql query :select * from table where status= 'NEW' order by tms_creation limit 100 for update;
The update query : update table set cod_etat = 'PUBLISH', tms_modification = now() where id in (:id)
Also, there is no Max Row per Poll to fetch data, which means that the jdbc poller will execute the sql request as many time as data (with status NEW) is present.
First issue: I stop my RabbitMQ and let my microservice running, the JdbcPollingChannelAdapter fetch the first ResultSet pass them through the Message flow and process the update. The message flow process the resultSet to send them through a channel to rabbitMQ(using spring cloud stream). The send fail and no Rollback has occured which means that the resultSet has been flagged as published.
I Have been loking around in documentation to figure out what I have missed. So any help would be appreciate.
Second issue: I run 3 instances of my application on PCF, and handle the concurrent access to the rows in the datable. My transaction and the select for update query in The JdbcPollingChannelAdapter suppose to get Row-level Lock Modes for the current transaction as per sql query (select for update). But what is happening is that more than one instance could get the same rows which is supposed to be managed by the current lock. Thus, it leads to multiple instances handling the same data and publishing them multiple times.
My code is as
#EnableConfigurationProperties(ProprietesSourceJdbc.class)
#Component
public class KafkaGuy {
private static final Logger LOG = LoggerFactory.getLogger(KafkaGuy.class);
private ProprietesSourceJdbc proprietesSourceJdbc;
private DataSource sourceDeDonnees;
private DemandeSource demandeSource;
private ObjectMapper objectMapper;
private JdbcTemplate jdbcTemplate;
public KafkaGuy(ProprietesSourceJdbc proprietesSourceJdbc, DemandeSource demandeSource, DataSource dataSource, JdbcTemplate jdbcTemplate, ObjectMapper objectMapper) {
this.proprietesSourceJdbc = proprietesSourceJdbc;
this.demandeSource = demandeSource;
this.sourceDeDonnees = dataSource;
this.objectMapper = objectMapper;
this.jdbcTemplate = jdbcTemplate;
}
#Bean
public MessageSource<Object> jdbcSourceMessage() {
JdbcPollingChannelAdapter jdbcSource = new JdbcPollingChannelAdapter(this.sourceDeDonnees, this.proprietesSourceJdbc.getQuery());
jdbcSource.setUpdateSql(this.proprietesSourceJdbc.getUpdate());
return jdbcSource;
}
#Bean
public IntegrationFlow fluxDeDonnees() {
IntegrationFlowBuilder flowBuilder = IntegrationFlows.from(jdbcSourceMessage());
flowBuilder
.split()
.log(LoggingHandler.Level.INFO, message ->
message.getHeaders().get("sequenceNumber")
+ " événements publiés sur le bus de message sur "
+ message.getHeaders().get("sequenceSize")
+ " événements lus (lot)")
.transform(Transformers.toJson())
.enrichHeaders(h -> h.headerExpression("type", "payload.typ_evenement"))
.publishSubscribeChannel(publishSubscribeSpec -> publishSubscribeSpec
.subscribe(flow -> flow
.transform(Transformers.toJson())
.transform(kafkaGuyTransformer())
.channel(this.demandeSource.demandePreinscriptionOuput()))
);
return flowBuilder.get();
}
#Bean
public KafkaGuyTransformer kafkaGuyTransformer() {
return new KafkaGuyTransformer();
}
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata defaultPoller() {
PollerMetadata pollerMetadata = new PollerMetadata();
PeriodicTrigger trigger = new PeriodicTrigger(this.proprietesSourceJdbc.getTriggerDelay(), TimeUnit.SECONDS);
pollerMetadata.setTrigger(trigger);
pollerMetadata.setMaxMessagesPerPoll(proprietesSourceJdbc.getMaxRowsPerPoll());
return pollerMetadata;
}
public class KafkaGuyTransformer implements GenericTransformer<Message, Message> {
#Override
public Message transform(Message message) {
Message<String> msg = null;
try {
DemandeRecueDTO dto = objectMapper.readValue(message.getPayload().toString(), DemandeRecueDTO.class);
msg = MessageBuilder.withPayload(dto.getTxtDonnee())
.copyHeaders(message.getHeaders())
.build();
} catch (Exception e) {
LOG.error(e.getMessage(), e);
}
return msg;
}
}
}
I am new In spring integration and sorry if is not well explained. Any help is appreciate.
Everything looks good and should be as you have described. Only the problem I see that there is no transaction configured for the IntegrationFlows.from(jdbcSourceMessage()).
Consider to PollerMetadata.setAdviceChain() with a TransactionInterceptor.
Another way is to use a PollerSpec with its transactional() option.
This way you won't use local data base transactions which are committed exactly after return from the ResultSet processing. With transaction on the application level there is not going to be a commit until you exit a thread.
I have a CXF client configured in my Spring Boot app like so:
#Bean
public ConsumerSupportService consumerSupportService() {
JaxWsProxyFactoryBean jaxWsProxyFactoryBean = new JaxWsProxyFactoryBean();
jaxWsProxyFactoryBean.setServiceClass(ConsumerSupportService.class);
jaxWsProxyFactoryBean.setAddress("https://www.someservice.com/service?wsdl");
jaxWsProxyFactoryBean.setBindingId(SOAPBinding.SOAP12HTTP_BINDING);
WSAddressingFeature wsAddressingFeature = new WSAddressingFeature();
wsAddressingFeature.setAddressingRequired(true);
jaxWsProxyFactoryBean.getFeatures().add(wsAddressingFeature);
ConsumerSupportService service = (ConsumerSupportService) jaxWsProxyFactoryBean.create();
Client client = ClientProxy.getClient(service);
AddressingProperties addressingProperties = new AddressingProperties();
AttributedURIType to = new AttributedURIType();
to.setValue(applicationProperties.getWex().getServices().getConsumersupport().getTo());
addressingProperties.setTo(to);
AttributedURIType action = new AttributedURIType();
action.setValue("http://serviceaction/SearchConsumer");
addressingProperties.setAction(action);
client.getRequestContext().put("javax.xml.ws.addressing.context", addressingProperties);
setClientTimeout(client);
return service;
}
private void setClientTimeout(Client client) {
HTTPConduit conduit = (HTTPConduit) client.getConduit();
HTTPClientPolicy policy = new HTTPClientPolicy();
policy.setConnectionTimeout(applicationProperties.getWex().getServices().getClient().getConnectionTimeout());
policy.setReceiveTimeout(applicationProperties.getWex().getServices().getClient().getReceiveTimeout());
conduit.setClient(policy);
}
This same service bean is accessed by two different threads in the same application sequence. If I execute this particular sequence 10 times in a row, I will get a connection timeout from the service call at least 3 times. What I'm seeing is:
Caused by: java.io.IOException: Timed out waiting for response to operation {http://theservice.com}SearchConsumer.
at org.apache.cxf.endpoint.ClientImpl.waitResponse(ClientImpl.java:685) ~[cxf-core-3.2.0.jar:3.2.0]
at org.apache.cxf.endpoint.ClientImpl.processResult(ClientImpl.java:608) ~[cxf-core-3.2.0.jar:3.2.0]
If I change the sequence such that one of the threads does not call this service, then the error goes away. So, it seems like there's some sort of a race condition happening here. If I look at the logs in our proxy manager for this service, I can see that both of the service calls do return a response very quickly, but the second service call seems to get stuck somewhere in the code and never actually lets go of the connection until the timeout value is reached. I've been trying to track down the cause of this for quite a while, but have been unsuccessful.
I've read some mixed opinions as to whether or not CXF client proxies are thread-safe, but I was under the impression that they were. If this actually not the case, and I should be creating a new client proxy for each invocation, or use a pool of proxies?
Turns out that it is an issue with the proxy not being thread-safe. What I wound up doing was leveraging a solution kind of like one posted at the bottom of this post: Is this JAX-WS client call thread safe? - I created a pool for the proxies and I use that to access proxies from multiple threads in a thread-safe manner. This seems to work out pretty well.
public class JaxWSServiceProxyPool<T> extends GenericObjectPool<T> {
JaxWSServiceProxyPool(Supplier<T> factory, GenericObjectPoolConfig poolConfig) {
super(new BasePooledObjectFactory<T>() {
#Override
public T create() throws Exception {
return factory.get();
}
#Override
public PooledObject<T> wrap(T t) {
return new DefaultPooledObject<>(t);
}
}, poolConfig != null ? poolConfig : new GenericObjectPoolConfig());
}
}
I then created a simple "registry" class to keep references to various pools.
#Component
public class JaxWSServiceProxyPoolRegistry {
private static final Map<Class, JaxWSServiceProxyPool> registry = new HashMap<>();
public synchronized <T> void register(Class<T> serviceTypeClass, Supplier<T> factory, GenericObjectPoolConfig poolConfig) {
Assert.notNull(serviceTypeClass);
Assert.notNull(factory);
if (!registry.containsKey(serviceTypeClass)) {
registry.put(serviceTypeClass, new JaxWSServiceProxyPool<>(factory, poolConfig));
}
}
public <T> void register(Class<T> serviceTypeClass, Supplier<T> factory) {
register(serviceTypeClass, factory, null);
}
#SuppressWarnings("unchecked")
public <T> JaxWSServiceProxyPool<T> getServiceProxyPool(Class<T> serviceTypeClass) {
Assert.notNull(serviceTypeClass);
return registry.get(serviceTypeClass);
}
}
To use it, I did:
JaxWSServiceProxyPoolRegistry jaxWSServiceProxyPoolRegistry = new JaxWSServiceProxyPoolRegistry();
jaxWSServiceProxyPoolRegistry.register(ConsumerSupportService.class,
this::buildConsumerSupportServiceClient,
getConsumerSupportServicePoolConfig());
Where buildConsumerSupportServiceClient uses a JaxWsProxyFactoryBean to build up the client.
To retrieve an instance from the pool I inject my registry class and then do:
JaxWSServiceProxyPool<ConsumerSupportService> consumerSupportServiceJaxWSServiceProxyPool = jaxWSServiceProxyPoolRegistry.getServiceProxyPool(ConsumerSupportService.class);
And then borrow/return the object from/to the pool as necessary.
This seems to work well so far. I've executed some fairly heavy load tests against it and it's held up.
I'm new to Spring Framework and, indeed, I'm learning and using Spring Boot. Recently, in the app I'm developing, I made Quartz Scheduler work, and now I want to make Spring Integration work there: FTP connection to a server to write and read files from.
What I want is really simple (as I've been able to do so in a previous java application). I've got two Quartz Jobs scheduled to fired in different times daily: one of them reads a file from a FTP server and another one writes a file to a FTP server.
I'll detail what I've developed so far.
#SpringBootApplication
#ImportResource("classpath:ws-config.xml")
#EnableIntegration
#EnableScheduling
public class MyApp extends SpringBootServletInitializer {
#Autowired
private Configuration configuration;
//...
#Bean
public DefaultFtpsSessionFactory myFtpsSessionFactory(){
DefaultFtpsSessionFactory sess = new DefaultFtpsSessionFactory();
Ftp ftp = configuration.getFtp();
sess.setHost(ftp.getServer());
sess.setPort(ftp.getPort());
sess.setUsername(ftp.getUsername());
sess.setPassword(ftp.getPassword());
return sess;
}
}
The following class I've named it as a FtpGateway, as follows:
#Component
public class FtpGateway {
#Autowired
private DefaultFtpsSessionFactory sess;
public void sendFile(){
// todo
}
public void readFile(){
// todo
}
}
I'm reading this documentation to learn to do so. Spring Integration's FTP seems to be event driven, so I don't know how can I execute either of the sendFile() and readFile() from by Jobs when the trigger is fired at an exact time.
The documentation tells me something about using Inbound Channel Adapter (to read files from a FTP?), Outbound Channel Adapter (to write files to a FTP?) and Outbound Gateway (to do what?):
Spring Integration supports sending and receiving files over FTP/FTPS by providing three client side endpoints: Inbound Channel Adapter, Outbound Channel Adapter, and Outbound Gateway. It also provides convenient namespace-based configuration options for defining these client components.
So, I haven't got it clear as how to follow.
Please, could anybody give me a hint?
Thank you!
EDIT:
Thank you #M. Deinum. First, I'll try a simple task: read a file from the FTP, the poller will run every 5 seconds. This is what I've added:
#Bean
public FtpInboundFileSynchronizer ftpInboundFileSynchronizer() {
FtpInboundFileSynchronizer fileSynchronizer = new FtpInboundFileSynchronizer(myFtpsSessionFactory());
fileSynchronizer.setDeleteRemoteFiles(false);
fileSynchronizer.setPreserveTimestamp(true);
fileSynchronizer.setRemoteDirectory("/Entrada");
fileSynchronizer.setFilter(new FtpSimplePatternFileListFilter("*.csv"));
return fileSynchronizer;
}
#Bean
#InboundChannelAdapter(channel = "ftpChannel", poller = #Poller(fixedDelay = "5000"))
public MessageSource<File> ftpMessageSource() {
FtpInboundFileSynchronizingMessageSource source = new FtpInboundFileSynchronizingMessageSource(inbound);
source.setLocalDirectory(new File(configuracion.getDirFicherosDescargados()));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<File>());
return source;
}
#Bean
#ServiceActivator(inputChannel = "ftpChannel")
public MessageHandler handler() {
return new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
Object payload = message.getPayload();
if(payload instanceof File){
File f = (File) payload;
System.out.println(f.getName());
}else{
System.out.println(message.getPayload());
}
}
};
}
Then, when the app is running, I put a new csv file intro "Entrada" remote folder, but the handler() method isn't run after 5 seconds... I'm doing something wrong?
Please add #Scheduled(fixedDelay = 5000) over your poller method.
You should use SPRING BATCH with tasklet. It is far easier to configure bean, crone time, input source with existing interfaces provided by Spring.
https://www.baeldung.com/introduction-to-spring-batch
Above example is annotation and xml based both, you can use either.
Other benefit Take use of listeners and parallel steps. This framework can be used in Reader - Processor - Writer manner as well.
We have a use case to download files from FTP, and there is a strange behavior that the ftp inbound adapter stops to work after network resets, here is the steps to reproduce the problem:
start application
application starts to download files from ftp server to local
there are filename.writing file appearing in defined local directory
pull out the network cable (to simulate a network reset situation)
application stops to download file (obviously no network connection)
plug in the network cable.
download is not restarted or reset, application stays still..
there is no LOG at all to identify this problem.
Thanks in advance!
UPDATE
This problem should be fixed by adding timeout defSession.setConnectTimeout(Integer.valueOf(env.getProperty("ftp.timeout.connect")));
AND The code below is a WORKING EXAMPLE on FTP reading client.
Here are the code snippet:
#Bean
public DefaultFtpSessionFactory ftpSessionFactory() {
DefaultFtpSessionFactory defSession = new DefaultFtpSessionFactory();
defSession.setUsername(env.getProperty("ftp.username"));
defSession.setPassword(env.getProperty("ftp.password"));
defSession.setPort(21);
defSession.setHost(env.getProperty("ftp.host"));
defSession.setClientMode(FTPClient.PASSIVE_LOCAL_DATA_CONNECTION_MODE);
defSession.setControlEncoding("UTF-8");
return defSession;
}
#Bean
PollableChannel ftpChannel() {
return new QueueChannel(Integer.valueOf(env.getProperty("ftp.channel.size")));
}
#Bean
public FtpInboundFileSynchronizer ftpInboundFileSynchronizer() {
FtpInboundFileSynchronizer ftpInboundFileSynchronizer = new FtpInboundFileSynchronizer(ftpSessionFactory());
ftpInboundFileSynchronizer.setDeleteRemoteFiles(Boolean.valueOf(env.getProperty("ftp.directory.delete")));
FtpRegexPatternFileListFilter ftpRegexPatternFileListFilter = new FtpRegexPatternFileListFilter(pattern);
ftpInboundFileSynchronizer.setFilter(ftpRegexPatternFileListFilter);
ftpInboundFileSynchronizer.setRemoteDirectory(env.getProperty("ftp.directory.remote"));
return ftpInboundFileSynchronizer;
}
#Bean
#InboundChannelAdapter(value = "ftpChannel")
public FtpInboundFileSynchronizingMessageSource ftpInboundFileSynchronizingMessageSource() {
FtpInboundFileSynchronizingMessageSource ftpInboundFileSynchronizingMessageSource = new FtpInboundFileSynchronizingMessageSource(ftpInboundFileSynchronizer());
ftpInboundFileSynchronizingMessageSource.setLoggingEnabled(true);
ftpInboundFileSynchronizingMessageSource.setCountsEnabled(true);
ftpInboundFileSynchronizingMessageSource.setAutoCreateLocalDirectory(true);
ftpInboundFileSynchronizingMessageSource.setLocalDirectory(new File(env.getProperty("ftp.directory.local")));
return ftpInboundFileSynchronizingMessageSource;
}
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata defaultPoller() {
PollerMetadata pollerMetadata = new PollerMetadata();
pollerMetadata.setErrorHandler(t -> log.error("Failed to retrieve data from FTP: {}", t.getMessage(), t));
pollerMetadata.setTrigger(new PeriodicTrigger(60, TimeUnit.SECONDS));
return pollerMetadata;
}
Most likely the thread is still hanging on the read - if you pull the cable from the actual adapter on the computer, the network stack should notify the java process that the socket is gone, but if you pull the cable from a downstream router, there may be no signal. jstack will show what the thread is doing.
You need to set timeouts on the session factory - see the documentation and the FtpClient javadocs - the dataTimeout is used for reads.