Source files are not deleting from s3 bucket once after successfully transfer it to target directory.
steps 1. Using Inbound streaming channel adapter to stream source files from S3 to local directory.(working fine)
step 2 : Want to delete source files once successfully transferred (not working)
configuration code are below
<int-aws:s3-inbound-streaming-channel-adapter id="s3FilesInbound"
channel="s3FilesChannel"
session-factory="s3SessionFactory"
filename-regex="^.*\\.(txt|csv)$"
remote-directory-expression="bucket_name"
auto-startup="true" >
<integration:poller id="s3FilesChannelPoller"
fixed-delay="1000"
max-messages-per-poll="1">
</integration:poller>
</int-aws:s3-inbound-streaming-channel-adapter>
<integration:stream-transformer id="streamTransformer" input-channel="s3FilesChannel" output-channel="s3FilesChannelOut"/>
<integration:chain id="filesS3ChannelChain"
input-channel="s3FilesChannelOut">
<file:outbound-gateway
id="fileInS3ArchiveChannel"
directory="local_directory"
filename-generator-expression="headers.file_remoteFile">
<file:request-handler-advice-chain>
<ref bean="retryAdvice" />
</file:request-handler-advice-chain>
</file:outbound-gateway>
<integration:gateway request-channel="nullChannel"
error-channel="errorChannel" />
</integration:chain>
Regards,
Since you use there a <integration:stream-transformer>, I don't see reason to rely on the <int-aws:s3-inbound-streaming-channel-adapter>. With the first one you just eliminate the streaming purpose of the last one.
I'd suggest you take a look into the regular <int-aws:s3-inbound-channel-adapter>, which already has a delete-remote-files="true" option.
On the other hand you still can do that with what you have so far, but you need to something like <integration:outbound-channel-adapter expression="#s3SessionFactory.getSession().remove(headers[file_remoteDirectory] + '/' + headers[file_remoteFile])">.
Those headers are populated by the AbstractRemoteFileStreamingMessageSource.
Related
In spring integration, I want to poll files from different source directories (each interface configured have different source directories) which is configured in as sourcePath in yml file (dynamically) like below. N number of interfaces can be added by user.
interfaces:
-
sourceType: NFS
sourcePath: /Interface-1/Inbound/text
target: Interface-1
targetType: S3
targetPath: test-bucket-1
-
sourceType: NFS
sourcePath: /Interface-2/Inbound/text
target: Interface-2
targetType: S3
targetPath: test-bucket-2
Is it possible to poll the files from different source folders using single inbound adapter (using atomic reference) or need more than one inbound adapter?
Currently application polls files from base directory.
<file:inbound-channel-adapter id="filesInboundChannel"
directory="file:${base.path}" auto-startup="false" scanner="scanner" auto-create-directory="true">
<integration:poller id="poller" max-messages-per-poll="${max.messages.per.poll}" fixed-rate="${message.read.frequency}" task-executor="pollingExecutor">
<integration:transactional transaction-manager="transactionManager" />
</integration:poller>
</file:inbound-channel-adapter>
Can someone give an advice on this or is there any other way can also achieve the same goal
Yes, you can use a single <file:inbound-channel-adapter> for this task. To make it rotate over a list of directories for scanning you need to configure an AbstractMessageSourceAdvice implementation for the <poller> of that adapter to change a directory when afterReceive(boolean messageReceived, MessageSource<?> source) gets a false for the receive operation. So, this way the next poll will get already a new directory for scanning.
As a sample you can take a look into the recently introduced a RotatingServerAdvice: https://github.com/spring-projects/spring-integration/blob/master/spring-integration-file/src/main/java/org/springframework/integration/file/remote/aop/RotatingServerAdvice.java
https://docs.spring.io/spring-integration/docs/current/reference/html/messaging-channels-section.html#conditional-pollers
new to Spring and could use some help on figuring out how to correctly chain together an sftp outbound gateway with a file outbound gateway. I want to confirm a file has sftp'ed, and then move it to an archive location.
Essentially, I have a directory where files are sent to be sftp'd somewhere else. The file is then supposed to be moved to an Archive directory, after the file has been transferred.
Each code piece works independently, but fails when I attempt to connect the two. I am unable to use the reply channel that I would normally because the reply channel confirms where the file has been saved to remotely, and that .msg is moved to the archive directory.
I suspect that order does not do what I think it does.
Currently, the file moves 90% of the time to the archive directory, without sftping the file.
Is this possible, or am I just barking up the wrong tree? Is there a way to configure the sftp:outbound-gateway downstream, or should I try using a different method?
<!-- START: SFTP files-->
<int-file:inbound-channel-adapter
directory="file:${sftp.repo}"
channel="SFTPchannel"
prevent-duplicates="false"
ignore-hidden="true" />
<int-sftp:outbound-gateway
session-factory="SFTPFactory"
request-channel="SFTPchannel"
order="1"
command="mput"
command-options="-1"
expression="payload"
mode="REPLACE"
use-temporary-file-name="false"
remote-filename-generator="filenameGenerator"
auto-create-directory="false"
remote-directory="${sftp.remote.destination}"/>
<int-file:outbound-gateway
request-channel="SFTPchannel"
order="2"
directory-expression="'${repository.directory}/'+new java.text.SimpleDateFormat('yyyyMMdd').format(new java.util.Date())"
mode="REPLACE"
auto-create-directory="true"
filename-generator="filenameGenerator"
delete-source-files="true"
reply-channel="nullChannel" />
<!-- END: SFTP files-->
See the retry-and-more sample, and particularly the Expression Evaluating Advice Demo therein. It covers your exact use case.
When my FTP mput transfer is successfully complete, I want to rename the file in the local directory. For this, I need to use the local dir path from PropertyPlaceholderConfigurer. But this doesn't seem to be working. Please can you suggest the syntaxt to expand the value of the property? ${local.request.dir} represents a directory path like /home/jainr/REQUEST.
<int-ftp:request-handler-advice-chain>
<bean id="requestFileRename" class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="trapException" value="true" />
<property name="onSuccessExpression" value="T(org.apache.commons.io.FileUtils).moveFile(new java.io.File(#{${local.request.dir}} + '/' + headers['RequestFileName']), new java.io.File(#{${local.request.dir}} + '/' + headers['RequestFileName'] + '.processed'))" />
</bean>
</int-ftp:request-handler-advice-chain>
You need to define the resolved placeholder as a literal, from a SpEL perspective...
new java.io.File('${local.request.dir}/' + headers...
Notice I also removed the #{...} - that is initialization time SpEL - this is runtime SpEL.
In future, instead of statements like...
But this doesn't seem to be working.
...provide the error message and/or stack trace.
Issue: Using spring batch, i need to read a file which has todays date. E.g test_02032015.txt.This file will be in a directory /test/example. Its an unix environment that i need to fetch file from.
question is how to configure spring batch xml so that above mentioned file is read
Any pointers to relevant website or solution would be of great help.
You have a few ways to address a requirement like this:
If you don't need to worry about the other files in a directory, you can just use a wild card in the file name like this:
<property name="resource" value="data/iosample/input/*.xml" />
Another alternative would be to pass the value into the job as a parameter and reference it like this:
<property name="resource" value="#{jobParameters['input.file']}" />
Finally you could use SpEL to build the file name (Sorry I don't have an example of that handy).
In WiX DirectorySearch can be used to determine if a specific directory exists on the target computer. But I don't understand if there's a consistent way to determine that a directory does not exist.
For example:
<Property Id="INSTALLDIR" Secure="yes">
<RegistrySearch Id='InstallDir' Type='directory'
Root='HKLM' Key='Software\Company\Product\Install' Name='InstallPath'/>
</Property>
<Property Id='IS_INSTALLED' Secure='yes'>
<DirectorySearch Id='IsInstalled' Path='[INSTALLDIR]' />
</Property>
When both the registry key and the directory exist, the IS_INSTALLED property is set to the path returned by DirectorySearch.
When the directory does not exist, IS_INSTALLED appears to be set to "C:\".
Is a condition like:
<Condition>NOT (IS_INSTALLED = "C:\")</Condition>
a reliable way to detect that the directory was found? Is there a better way?
Answer: Here is WiX code based on mrnxs answer that I accepted
<Property Id="PRODUCT_IS_INSTALLED" Secure="yes">
<RegistrySearch Id='RegistrySearch1' Type='directory'
Root='HKLM' Key='Software\Company\Product\Version\Install' Name='Path'>
<DirectorySearch Id='DirectorySearch1' Path='[PRODUCT_IS_INSTALLED]'/>
</RegistrySearch>
</Property>
<CustomAction Id='SET_INSTALLDIR'
Property='INSTALLDIR'
Value='[PRODUCT_IS_INSTALLED]'/>
<InstallExecuteSequence>
<Custom Action='SET_INSTALLDIR' After='AppSearch'></Custom>
</InstallExecuteSequence>
Usually this happens when the property is used as a property-based folder. In this case the CostFinalize action automatically sets the property to a valid path (for example "C:\") so the folder can be used by Windows Installer.
Since this path is generated automatically, you cannot be sure that it will be "C:\" on all your client machines, so you shouldn't use this value in your condition. Instead, you can try this:
use a custom property for your folder
use a type 51 custom action (property set with formatted text) to set this property to a valid default path (for example "[ProgramFilesFolder]MyCompany\MyProduct")
use another property for the search
use another type 51 custom action to set the folder property to your search property
For example, if your search is IS_INSTALLED your folder can use IS_INSTALLED_PATH. IS_INSTALLED_PATH can be set to a default path and after AppSearch action you can set it to IS_INSTALLED if the search found something.
This way you can use for conditioning:
IS_INSTALLED
or
NOT IS_INSTALLED
Understaing AppSearch's RegLocator and DrLocator patterns can be a little tricky. I reccomend ignoring the condition for a moment and logging the install to verify that AppSearch is properly setting the properties you want. Fix the problems you find on that end first. When it works the property will be set to the value of the registry or the path to the directory.
Then you should be able to use:
<Condition>IS_INSTALLED/> <!-- it's not important what the value is, just that it exists -->
<Condition>Not IS_INSTALLED/>
Btw, I would avoid using the property INSTALLDIR. In my installers ( InstallShield ) that has special meaning as the main focal point of the installation.
Another approach could be this, in this you can continue the Install Sequence if you want to set the InstallDir to anywhere else, if the SystemDir and RegisteryDir is not same
<Property Id="RegisteryDir" Secure="yes">
<RegistrySearch Id='InstallDir' Type='directory'
Root='HKLM' Key='Software\Company\Product\Install' Name='InstallPath'/>
</Property>
<Property Id='SystemDir' Secure='yes'>
<DirectorySearch Id='IsInstalled' Path='[RegisteryDir]' />
</Property>
<CustomAction Id="SET_INSTALL_DIR" Property="INSTALLDIR" Value="[SystemDir] />
<InstallExecuteSequence>
<Custom Action='SET_INSTALLDIR' After='AppSearch'>
SystemDir AND SystemDir=RegisteryDir
</Custom>
</InstallExecuteSequence>