I was following a tutorial on how to listen to a folder with spring integration and SseEmitter. I have this code now:
#Bean
IntegrationFlow inboundFlow ( #Value("${input-dir:file:C:\\Users\\kader\\Desktop\\Scaned\\}") File in){
return IntegrationFlows.from(Files.inboundAdapter(in).autoCreateDirectory(true),
poller -> poller.poller(spec -> spec.fixedRate(1000L)))
.transform(File.class, File::getAbsolutePath)
.handle(String.class, (path, map) -> {
sses.forEach((sse) -> {
try {
String p = path;
sse.send(SseEmitter.event().name("spring").data(p));
}
catch (IOException e) {
throw new RuntimeException(e);
}
});
return null ;
})
.get();
}
and it works but it sends all the files in the specified directory including the files that already exist, is there any way to make it ignore them and send the new files only???
Well, actually since you don't configure any filters on the Files.inboundAdapter(), there is a logic like this:
// no filters are provided
else if (Boolean.FALSE.equals(this.preventDuplicates)) {
filtersNeeded.add(new AcceptAllFileListFilter<File>());
}
else { // preventDuplicates is either TRUE or NULL
filtersNeeded.add(new AcceptOnceFileListFilter<File>());
}
Therefore an AcceptOnceFileListFilter is applied and no any already polled files are not going to be picked up on the subsequent poll tasks.
However you really may talk about something like "after application restart", so yes, in this case all the files are going to be pulled.
I believe you need to study what is the FileListFilter and use an appropriate for your use-case: https://docs.spring.io/spring-integration/docs/current/reference/html/files.html#file-reading
Related
Somebody asked for this and there is a pull-request which contains code that somehow was rewritten before it got merged and somebody managed to code a solution based on the pull-request. However, there is no example for the final version in that library.
Therefore, that doesn't really help me with my limited understanding of ssh and all. Basically there are two scenarios I want to solve:
common SSH-session via some jump-hosts:
user1#jump1.com
user2#jump2.com
user3#jump3.com
admin#server.com
ending in an ssh-session where the connecting user is free to work around in that ssh-shell at server.com, i.e. what a normal ssh admin#server.com-command would do in the shell on jump3.com.
like the above but ending in a port forwarding to server.com:80
That is possible with ssh's ProxyCommand, but I want to code this with SSHJ. And that's where I fail to figure out how to do this.
What I have now is
SSHClient hop1 = new SSHClient();
try {
Path knownHosts = rootConfig.getKnownHosts();
if (knownHosts != null) {
hop1.loadKnownHosts(knownHosts.toFile());
} else {
hop1.loadKnownHosts();
}
Path authenticationFile = hop1Config.getAuthenticationFile();
if (authenticationFile != null) {
KeyProvider keyProvider = hop1.loadKeys(authenticationFile.toString(), (String) null);
hop1.authPublickey(hop1Config.getUser(), keyProvider);
} else {
hop1.authPassword(hop1Config.getUser(), hop1Config.getPassword());
}
// I found these methods:
hop1.getConnection();
hop1.getSocket();
// and now what?
} catch (IOException e) {
logger.error("Failed to open ssh-connection to {}", hop1Config, e);
}
I noticed class LocalPortForwarder.DirectTCPIPChannel, but I don't know with what values I should instantiate it or how to use it with the rest afterwards.
I am working with spring boot and spring content. I want to store all my pictures and videos in one directory but my code continues to create different dir every time I rerun the application
I have such bean and when I run the app again it shows null pointer because the dir already exists but I want it to create it just once and every file is stored there
every time i run this tries to create the dir again
#Bean
File filesystemRoot() {
try {
return Files.createDirectory(Paths.get("/tmp/photo_video_myram")).toFile();
} catch (IOException io) {}
return null;
}
#Bean
FileSystemResourceLoader fileSystemResourceLoader() {
return new FileSystemResourceLoader(filesystemRoot().getAbsolutePath());
}
One solution, would be to check if the directory exists:
#Bean
File filesystemRoot() {
File tmpDir = new File("tmp/photo_video_myram");
if (!tmpDir.isDirectory()) {
try {
return Files.createDirectory(tmpDir.toPath()).toFile();
} catch (IOException e) {
e.printStackTrace();
}
}
return tmpDir;
}
You can use isDirectory() method first to check if the directory already exists. In case it does not exist, then create a new one.
Meanwhile there is another way to achieve this, when you use Spring Boot and accordingly spring-content-fs-boot-starter.
According to the documentation at https://paulcwarren.github.io/spring-content/refs/release/fs-index.html#_spring_boot_configuration it should be sufficient to add
spring.content.fs.filesystemRoot=/tmp/photo_video_myram
to your application.properties file.
I have problem with correctly saving a file after its download is complete in Project Reactor.
class HttpImageClientDownloader implements ImageClientDownloader {
private final ExchangeFunction exchangeFunction;
HttpImageClientDownloader() {
this.exchangeFunction = ExchangeFunctions.create(new ReactorClientHttpConnector());
}
#Override
public Mono<File> downloadImage(String url, Path destination) {
ClientRequest clientRequest = ClientRequest.create(HttpMethod.GET, URI.create(url)).build();
return exchangeFunction.exchange(clientRequest)
.map(clientResponse -> clientResponse.body(BodyExtractors.toDataBuffers()))
//.flatMapMany(clientResponse -> clientResponse.body(BodyExtractors.toDataBuffers()))
.flatMap(dataBuffer -> {
AsynchronousFileChannel fileChannel = createFile(destination);
return DataBufferUtils
.write(dataBuffer, fileChannel, 0)
.publishOn(Schedulers.elastic())
.doOnNext(DataBufferUtils::release)
.then(Mono.just(destination.toFile()));
});
}
private AsynchronousFileChannel createFile(Path path) {
try {
return AsynchronousFileChannel.open(path, StandardOpenOption.CREATE);
} catch (Exception e) {
throw new ImageDownloadException("Error while creating file: " + path, e);
}
}
}
So my question is:
Is DataBufferUtils.write(dataBuffer, fileChannel, 0) blocking?
What about when the disk is slow?
And second question about what happens when ImageDownloadException occurs ,
In doOnNext I want to release the given data buffer, is that a good place for this kind operation?
I think also this line:
.map(clientResponse -> clientResponse.body(BodyExtractors.toDataBuffers()))
could be blocking...
Here's another (shorter) way to achieve that:
Flux<DataBuffer> data = this.webClient.get()
.uri("/greeting")
.retrieve()
.bodyToFlux(DataBuffer.class);
Path file = Files.createTempFile("spring", null);
WritableByteChannel channel = Files.newByteChannel(file, StandardOpenOption.WRITE);
Mono<File> result = DataBufferUtils.write(data, channel)
.map(DataBufferUtils::release)
.then(Mono.just(file));
Now DataBufferUtils::write operations are not blocking because they use non-blocking IO with channels. Writing to such channels means it'll write whatever it can to the output buffer (i.e. may write all the DataBuffer or just part of it).
Using Flux::map or Flux::doOnNext is the right place to do that. But you're right, if an error occurs, you're still responsible for releasing the current buffer (and all the remaining ones). There might be something we can improve here in Spring Framework, please keep an eye on SPR-16782.
I don't see how your last sample shows anything blocking: all methods return reactive types and none are doing blocking I/O.
We are using Spring Integration to process a JSON payload passed into a RESTful endpoint. As part of this flow we are using a filter to validate the JSON:
.filter(schemaValidationFilter, s -> s
.discardFlow(f -> f
.handle(message -> {
throw new SchemaValidationException(message);
}))
)
This works great. However, if the validation fails we want to capture the parsing error and return that to the user so they can act on the error. Here is the overridden accept method in the SchemaValidationFilter class:
#Override
public boolean accept(Message<?> message) {
Assert.notNull(message);
Assert.isTrue(message.getHeaders().containsKey(TYPE_NAME));
String historyType = (String)message.getHeaders().get(TYPE_NAME);
JSONObject payload = (JSONObject) message.getPayload();
String jsonString = payload.toJSONString();
try {
ProcessingReport report = schemaValidator.validate(historyType, payload);
return report.isSuccess();
} catch (IOException | ProcessingException e) {
throw new MessagingException(message, e);
}
}
What we have done is in the catch block we throw a MessageException which seems to solve the problem. However this seems to break what a filter should do (simply return a true or false).
Is there a best practice for passing the error details from the filter to the client? Is the filter the right solution for this use case?
Thanks for your help!
John
I'd say you go correct way. Please, refer to the XmlValidatingMessageSelector, so your JsonValidatingMessageSelector should be similar and must follow the same design.
Since we have a throwExceptionOnRejection option we always can be sure that throwing Exception instead of just true/false is correct behavior.
What Gary says is good, too, but according to the existing logic in that MessageSelector impl we can go ahead with the same and continue to use .filter(), but, of course, already without .discardFlow(), because we won't send invalid message to the discardChannel.
When your JsonValidatingMessageSelector is ready, feel free to contribute it back to the Framework!
It's probably more correct to do the validation in a <service-activator/>...
public Message<?> validate(Message<?> message) {
...
try {
ProcessingReport report = schemaValidator.validate(historyType, payload);
return message;
}
catch (IOException | ProcessingException e) {
throw new MessagingException(message, e);
}
}
...since you're never really filtering.
dirPath contains 200k files. I want to read them one by one and do some processing. The following snippet causes java.nio.file.FileSystemException: dirPath/file-N Too many open files. Isn't the terminal operation forEach() supposed to close the open stream (i.e. the open file) before moving to the next one? In other words, do I have to add try-with-resources for the streamed files?
Files.list(dirPath)
.forEach(filePath -> {
Files.lines(filePath).forEach() { ... }
});
No forEach does not close the stream (created by Files.list or Files.lines). It is documented in the javadoc, for example for Files.list:
The returned stream encapsulates a Reader. If timely disposal of file system resources is required, the try-with-resources construct should be used to ensure that the stream's close method is invoked after the stream operations are completed.
A nested forEach is the wrong tool, in most cases.
The code
Files.list(dirPath).forEach(filePath -> Files.lines(filePath).forEach(line -> { ... });
can and should be replaced by
Files.list(dirPath).flatMap(filePath -> Files.lines(filePath)).forEach(line -> { ... });
or well, since it’s not that easy in this case:
Files.list(dirPath).flatMap(filePath -> {
try { return Files.lines(filePath);}
catch(IOException ex) { throw new UncheckedIOException(ex); }
}).forEach(line -> { });
as a side-effect, you get the following for free:
Stream.flatMap(…):
Each mapped stream is closed after its contents have been placed into this stream.
So that’s the preferred solution. Or well, to make it entirely correct:
try(Stream<Path> dirStream = Files.list(dirPath)) {
dirStream.flatMap(filePath -> {
try { return Files.lines(filePath);}
catch(IOException ex) { throw new UncheckedIOException(ex); }
}).forEach(line -> { });
}