Below is code for copy files(any type of file) from remote machine or any other machine which is in our LAN. How to automatically copy files Using Apche Camel.
I want to copy images, so any method to copy only images.
I implement this in spring.
final String sourceFolder = "ftp:172.30.83.119\\D:MyCLone\\WSIMS\\images?username=shailesh.bhad&password=Password2";
final String destinationFolder = "file:D:\\outbox";
final CamelContext camelContext = new DefaultCamelContext();
camelContext.addRoutes(new RouteBuilder() {
#Override
public void configure() {
from(sourceFolder).to(destinationFolder);
}
});
camelContext.start();
Thread.sleep(30000);
camelContext.stop();
}
See the FTP example
http://camel.apache.org/ftp-example.html
And then insted of sending to another FTP server, you can change the url to use the file component so it can store the files in your LAN
And see also this FAQ how to keep a Java JVM running
http://camel.apache.org/running-camel-standalone-and-have-it-keep-running.html
Related
I just deployed my first microservice. My microservice is working fine. All routes are working. But the service class inside the microservice is not working properly. The service class is not reading data from the CSV file.
Below is the code I am using to read data from CSV file.
public class ReadCsvUtil {
private static final Logger LOGGER = LoggerFactory.getLogger(ReadCsvUtil.class);
public List<String[]> readData() throws IOException {
String file = ".\\src\\main\\resources\\pensioners.csv";
List<String[]> content = new ArrayList<>();
try(BufferedReader br = new BufferedReader(new FileReader(file))) {
String line = "";
while ((line = br.readLine()) != null) {
content.add(line.split(","));
}
} catch (FileNotFoundException e) {
LOGGER.debug(e.getMessage());
}
return content;
}
}
The service class invokes the above function to get details of all the people.
The above code is working fine on my desktop and I am able to get details but code is not working on AWS. Also, I tried to remove the CSV and manually enter the values and it's working in AWS. So I am 99% sure there is some problem in reading the CSV files.
Is there anyway I can fix this?
If the path to the file one directory up, instead of
String file = ".\\src\\main\\resources\\pensioners.csv";
try,
String file = "..\src\main\resources\pensioners.csv";
I presume the AWS sever is ubuntu and your local is windows OS.
My current project is based on Spring Integration. I am developing this project by using spring Boot.
My goal is to use Spring Integration to complete the below task.
Connect to SFTP
check if directory is created in the local at a specific folder
check the eligible file extension of the file specific to (CSV & XLSX)
Down load all the content from SFTP remote directory to local directory & need to track the file transfer start time and file transfer end time.
Read the file from local directory line by line and extract the certain column info.
Can you give me some suggestions ?
And how can I get the transfer start time ?
Note : This requirement i have to develop as an rest api. Please provide some guidance how i can achieve this by using spring integration?
Thanks. :)
public class SftpConfig {
#Value("${sftp.host}")
private String sftpHost;
#Value("${sftp.port:22}")
private int sftpPort;
#Value("${sftp.user}")
private String sftpUser;
#Value("${sftp.password:#{null}}")
private String sftpPasword;
#Value("${sftp.remote.directory:/}")
private String sftpRemoteDirectory;
#Value("${sftp.privateKey:#{null}}")
private Resource sftpPrivateKey;
#Value("${sftp.privateKeyPassPhrase:}")
private String privateKeyPassPhrase;
#Value("${sftp.remote.directory.download.filter:*.*}")
private String sftpRemoteDirectoryDownloadFilter;
#Value("${sftp.remote.directory.download:/}")
private String sftpRemoteDirectoryDownload;
#Value("${sftp.local.directory.download:${java.io.tmpdir}/localDownload}")
private String sftpLocalDirectoryDownload;
/*
* The SftpSessionFactory creates the sftp sessions. This is where you define
* the host , user and key information for your sftp server.
*/
// Creating session for Remote Destination SFTP server Folder
#Bean
public SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost(sftpHost);
factory.setPort(sftpPort);
factory.setUser(sftpUser);
if (sftpPrivateKey != null) {
factory.setPrivateKey(sftpPrivateKey);
factory.setPrivateKeyPassphrase(privateKeyPassPhrase);
} else {
factory.setPassword("sftpPassword");
}
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<ChannelSftp.LsEntry>(factory);
}
/*
* The SftpInboundFileSynchronizer uses the session factory that we defined above.
* Here we set information about the remote directory to fetch files from.
* We could also set filters here to control which files get downloaded
*/
#Bean
public SftpInboundFileSynchronizer SftpInboundFileSynchronizer () {
SftpInboundFileSynchronizer synchronizer = new SftpInboundFileSynchronizer();
return null;
}
If you take a look into the SftpStreamingMessageSource instead: https://docs.spring.io/spring-integration/docs/current/reference/html/sftp.html#sftp-streaming, plus use a FileSplitter to read that file line by line (which supports a "first like as header", too), you won't need to worry about transferring the file to local dir. You will do everything with the remote content on demand.
On the other hand, since you talk about a REST API, you probably are going to have some #RestController or Spring Integration HTTP Inbound Gateway: https://docs.spring.io/spring-integration/docs/current/reference/html/http.html#http-inbound, then you need to think about using an SftpOutboundGateway with an MGET command: https://docs.spring.io/spring-integration/docs/current/reference/html/sftp.html#sftp-outbound-gateway.
If you still need to track a download time for every single file, you need to consider to use that gateway twice: with a LIST command and NAME_ONLY, and the second time for GET command. At this point you can add ChannelInterceptor for input and output channels of the second gateway, so you will have a file name info to correlate and catch start and stop time before and after this gateway.
I have a folder structure /data/reports on a file system, which contains all reports.
How can I configure a SpringBoot application to serve the contents of this file sytem.
Currently I have tried few options, but none working
#Configuration
#EnableWebMvc
public class AppConfig implements WebMvcConfigurer {
#Value(value = "${spring.resources.static-locations:#{null}}")
private String fileSystem;
#Override
public void addResourceHandlers(ResourceHandlerRegistry registry) {
registry
.addResourceHandler("/data/reports/**")
.addResourceLocations(fileSystem)
.setCachePeriod(3600)
.resourceChain(true)
.addResolver(new PathResourceResolver());
}
}
and in application.properties I have defined
spring.resources.static-locations=file:///data/reports
server.servlet.jsp.init-parameters.listings=true
But in both cases, when I try
http://host:port/application/data/reports
I'm getting 404
What am I missing ?
Based on the suggestions given, I realized that one mistake I'm doing is to access the reports via
http://host:port/application/data/reports
instead of
http://host:port/data/reports
if I use application in the request, those calls will go through RequestDispatcher and will try to find for a matching RequestMapping, which does not exist. I think I'm convinced so far.
But the problem I'm seeing now is, I'm getting SocketTimeoutException while trying to read from the resource listed in the URL. I had put some breakpoints in Spring source "ResourceHttpMessageConverter.java"
protected void writeContent(Resource resource, HttpOutputMessage outputMessage)
throws IOException, HttpMessageNotWritableException {
try {
InputStream in = resource.getInputStream(); //It is timing out here
try {
StreamUtils.copy(in, outputMessage.getBody());
}
catch (NullPointerException ex) {
// ignore, see SPR-13620
}
The resource is a small text file with 1 line "Hello World". Yet it is timing out.
The resource in the above class is a FileUrlResource opened on file:///c:/data/reports/sample.txt
On the other hand, I tried to read that resource as
File file = new File("c:/data/reports/sample.txt");
System.out.println(file.exists());
URL url = file.toURI().toURL();
URLConnection con = url.openConnection();
InputStream is = con.getInputStream(); //This works
Thanks
I'm using Spring Integration to poll files from remote FTP server and process them.
Is there a way to configure FtpInboundFileSynchronizer (or other component) to fetch and process remote files in specific order. Say i have file1 and file2 in remote directory, is it possible to fetch and process file1 before file2.
Thanks in advance
There are (at least) 3 techniques to achieve this:
Add a custom FileListFilter<FTPFile> (that sorts the FTPFile objects into the order you desire) to the synchronizer.
Use two FTP outbound gateways, one the list (ls) the files, and one to get each file as needed.
Use the FtpRemoteFileTemplate from within your own code to list and fetch files.
EDIT
Actually, for #1, you would also need a custom FileListFilter<File> in the local filter to sort the File objects. Since the local files are emitted as message payloads after the synchronization is complete.
EDIT2 Remote file template example
This just copies the first file in the list, but it should give you what you need...
#SpringBootApplication
public class So49462148Application {
public static void main(String[] args) {
SpringApplication.run(So49462148Application.class, args);
}
#Bean
public ApplicationRunner runner(FtpRemoteFileTemplate template) {
return args -> {
FTPFile[] files = template.list("*.txt");
System.out.println(Arrays.toString(files));
template.get(files[0].getName(), is -> {
File file = new File("/tmp/" + files[0].getName());
FileOutputStream os = new FileOutputStream(file);
FileCopyUtils.copy(is, os);
System.out.println("Copied: " + file.getAbsolutePath());
});
};
}
#Bean
public DefaultFtpSessionFactory sf() {
DefaultFtpSessionFactory sf = new DefaultFtpSessionFactory();
sf.setHost("...");
sf.setUsername("...");
sf.setPassword("...");
return sf;
}
#Bean
public FtpRemoteFileTemplate template(DefaultFtpSessionFactory sf) {
FtpRemoteFileTemplate template = new FtpRemoteFileTemplate(sf);
template.setRemoteDirectoryExpression(new LiteralExpression("foo"));
return template;
}
}
I use spring integration sftp to download and upload files.In the document ,I found
Spring Integration supports sending and receiving files over SFTP by providing three client side endpoints: Inbound Channel Adapter, Outbound Channel Adapter, and Outbound Gateway
When I want to download files I must assign the local directory and when I want to upload files I must assign the remote directory.But if I can't assign the directory when I write the code such as my directory is association with date.How can I assign the directory at runtime?
Here is my code:
#Bean
public SessionFactory<LsEntry> sftpSessionFactory(){
DefaultSftpSessionFactory defaultSftpSessionFactory = new DefaultSftpSessionFactory();
defaultSftpSessionFactory.setHost(host);
defaultSftpSessionFactory.setPort(Integer.parseInt(port));
defaultSftpSessionFactory.setUser(username);
defaultSftpSessionFactory.setPassword(password);
defaultSftpSessionFactory.setAllowUnknownKeys(true);
return new CachingSessionFactory<LsEntry>(defaultSftpSessionFactory);
}
#Bean
public SftpRemoteFileTemplate sftpRemoteFileTemplate(){
SftpRemoteFileTemplate sftpRemoteFileTemplate = new SftpRemoteFileTemplate(sftpSessionFactory());
return sftpRemoteFileTemplate;
}
#Bean
#ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler handlerGet() {
SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(sftpSessionFactory(), "mget", "payload");
sftpOutboundGateway.setLocalDirectory(new File(localDirectory));
sftpOutboundGateway.setFilter(new SftpSimplePatternFileListFilter("*.txt"));
sftpOutboundGateway.setSendTimeout(1000);
return sftpOutboundGateway;
}
In the messageHandler,I must assign the localDirectory in the outboundGateway. And when I want change my localDirectory by days.I must download the file to the localDirectory and move to the target directory. How can I assign the localDirectory at runtime .such as today I download to 20170606/ and tomorrow I download to 20170607 ?
edit
this is my option and test
public interface OutboundGatewayOption {
#Gateway(requestChannel = "sftpChannel")
public List<File> getFiles(String dir);
}
#Test
public void test2(){
outboundGatewayOption.getFiles("upload/20160920/");
}
sftpOutboundGateway.setLocalDirectoryExpression(
new SpelExpressionParser().parseExpression("headers['whereToPutTheFiles']");
or parseExpression("#someBean.getDirectoryName(payload)")
etc.
The expression must evaluate to a String representing the directory absolute path.
While evaluating the expression, the remote directory is available as a variable #remoteDirectory.