I probably hate writing noob questions as much as other people hate answering them, but here goes.
I need to split a message retrieved from a JdbcPollingChannelAdapter into multiple messages based on the operation requested in each row of the resultset in the payload.
The split operation is simple enough. What is proving to be a challenge is conditionally routing the message to one flow or the other.
After much trial and error, I believe that this flow represents my intention
/- insertUpdateAdapter -\
Poll Table -> decorate headers -> split -> router -< >- aggregator -> cleanup
\---- deleteAdapter ----/
TO that end I have constructed this Java DSL:
final JdbcOutboundGateway inboundAdapter = createInboundAdapter();;
final JdbcOutboundGateway deleteAdapter = createDeleteAdapter();
final JdbcOutboundGateway insertUpdateAdapter = createInsertUpdateAdapter();
return IntegrationFlows
.from(setupAdapter,
c -> c.poller(Pollers.fixedRate(1000L, TimeUnit.MILLISECONDS).maxMessagesPerPoll(1)))
.enrichHeaders(h -> h.headerExpression("start", "payload[0].get(\"start\")")
.headerExpression("end", "payload[0].get(\"end\")"))
.handle(inboundAdapter)
.split(insertDeleteSplitter)
.enrichHeaders(h -> h.headerExpression("operation", "payload[0].get(\"operation\")"))
.channel(c -> c.executor("stepTaskExecutor"))
.routeToRecipients (r -> r
.recipientFlow("'I' == headers.operation or 'U' == headers.operation",
f -> f.handle(insertUpdateAdapter))
// This element is complaining "Syntax error on token ")", ElidedSemicolonAndRightBrace expected"
// Attempted to follow patterns from https://github.com/spring-projects/spring-integration-java-dsl/wiki/Spring-Integration-Java-DSL-Reference#routers
.recipientFlow("'D' == headers.operation",
f -> f.handle(deleteAdapter))
.defaultOutputToParentFlow())
)
.aggregate()
.handle(cleanupAdapter)
.get();
Assumptions I have made, based on prior work include:
The necessary channels are auto-created as Direct Channels
Route To Recipients is the appropriate tool for this function (I have also considered expression router, but the examples of how to add sub-flows were less clear than the Route To Recipients)
Insert an ExecutorChannel somewhere between the splitter and router if you want to run the splits in parallel. You can limit the pool size of the executor to control the concurrency.
There is an extra parenthesis after .defaultOutputToParentFlow())
The corrected code is:
return IntegrationFlows
.from(setupAdapter,
c -> c.poller(Pollers.fixedRate(1000L, TimeUnit.MILLISECONDS).maxMessagesPerPoll(1)))
.enrichHeaders(h -> h.headerExpression("ALC_startTime", "payload[0].get(\"ALC_startTime\")")
.headerExpression("ALC_endTime", "payload[0].get(\"ALC_endTime\")"))
.handle(inboundAdapter)
.split(insertDeleteSplitter)
.enrichHeaders(h -> h.headerExpression("ALC_operation", "payload[0].get(\"ALC_operation\")"))
.channel(c -> c.executor(stepTaskExecutor))
.routeToRecipients (r -> r
.recipientFlow("'I' == headers.ALC_operation or 'U' == headers.ALC_operation",
f -> f.handle(insertUpdateAdapter))
// This element is complaining "Syntax error on token ")", ElidedSemicolonAndRightBrace expected"
// Attempted to follow patterns from https://github.com/spring-projects/spring-integration-java-dsl/wiki/Spring-Integration-Java-DSL-Reference#routers
.recipientFlow("'D' == headers.ALC_operation",
f -> f.handle(deleteAdapter))
.defaultOutputToParentFlow())
.aggregate()
.handle(cleanupAdapter)
.get();
Related
Mono<Void> enrich(List<String> idList) {
return Flux.fromIterable(idList)
.flatMap(idList -> aEnricher.getAById(idList))
.map(aEnrichResponse -> aEnrichResponse.getLookupId())
.buffer(10)
.flatMap(lookupIdList -> bEnricher.getBByLookupId(lookupIdList)) //WRONG: idList is not available, but response of previous call is available
.map(bResponse -> save(bResponse))
.flatMap(lookupIdList-> cEnricher.getCByLookupId(lookupIdList)); //WRONG: same as above
.map(cResponse -> save(cResponse))
}
How can I call multiple calls (bEnricher and cEnricher) simultaneously with same arguments. I cannot use merge or zip since return types and return size are different
Suppose I want to run two sql queries in a transaction I have code like the below:
jdbi.useHandle(handle -> handle.useTransaction(h -> {
var id = handle.createUpdate("some query")
.executeAndReturnGeneratedKeys()
.mapTo(Long.class).findOne().orElseThrow(() -> new IllegalStateException("No id"));
handle.createUpdate("INSERT INTO SOMETABLE (id) " +
"VALUES (:id , xxx);")
.bind("id")
.execute();
}
));
Now as the complexity grows I want to extract each update in into it's own method:
jdbi.useHandle(handle -> handle.useTransaction(h -> {
var id = someQuery1(h);
someQuery2(id, h);
}
));
...with someQuery1 looking like:
private Long someQuery1(Handle handle) {
return handle.createUpdate("some query")
.executeAndReturnGeneratedKeys()
.mapTo(Long.class).findOne().orElseThrow(() -> new IllegalStateException("No id"));
}
Now when I refactor to the latter I get a SonarQube blocker bug on the someQuery1 handle.createUpdate stating:
Resources should be closed
Connections, streams, files, and other
classes that implement the Closeable interface or its super-interface,
AutoCloseable, needs to be closed after use....*
I was under the impression, that because I'm using jdbi.useHandle (and passing the same handle to the called methods) that a callback would be used and immediately release the handle upon return. As per the jdbi docs:
Both withHandle and useHandle open a temporary handle, call your
callback, and immediately release the handle when your callback
returns.
Any help / suggestions appreciated.
TIA
SonarQube doesn't know any specifics regarding JDBI implementation and just triggers by AutoCloseable/Closable not being closed. Just suppress sonar issue and/or file a feature-request to SonarQube team to improve this behavior.
First, I'm listing out the files in a directory which works fine. After listing, I want to move the files to a different directory which also works. However, what doesn't work is to retain the original file name. Here is my code;
#Bean(value = "moveFile")
public IntegrationFlow moveFile() {
IntegrationFlow flow = IntegrationFlows
.fromSupplier(() -> "/", e -> e.poller(Pollers.fixedDelay(Duration.ofSeconds(5))))
.handle(Ftp.outboundGateway(sf(), AbstractRemoteFileOutboundGateway.Command.LS, null)
.regexFileNameFilter("(dir1|dir2|.*[0-9]|.*.txt)")
.options(AbstractRemoteFileOutboundGateway.Option.NAME_ONLY,
AbstractRemoteFileOutboundGateway.Option.RECURSIVE))
.split()
.handle(Ftp.outboundGateway(sf(), AbstractRemoteFileOutboundGateway.Command.MV, null)
.renameExpression("'/newdirectory/' + #remoteFileName"))
.channel("nullChannel")
.get();
return flow;
}
#remoteFileName doesn't work here but works when I use it in an mget command, and also without going through another outbound. headers['file_relativePath'] and headers['file_remoteFile'] does not work either, which are some of the results I searched up.
Another bonus question if you don't mind, (dir1|dir2|.[0-9]|..txt) regex works for me, but that is because .*[0-9] works for all my folder in that directory based on how it named. What I actually want is expression for any folder there.
Thanks in advance.
UPDATE
I manage to solve it. The SpEL variable was 'payload' which is the string of original file directory and file name.
IntegrationFlow flow = IntegrationFlows
.fromSupplier(() -> "/", e -> e.poller(Pollers.fixedDelay(Duration.ofSeconds(5))))
.handle(Ftp.outboundGateway(sf(), AbstractRemoteFileOutboundGateway.Command.LS, null)
.regexFileNameFilter("(dir1|dir2|.*[0-9]|.*.txt)")
.options(AbstractRemoteFileOutboundGateway.Option.RECURSIVE,AbstractRemoteFileOutboundGateway.Option.NAME_ONLY))
.split()
.handle(Ftp.outboundGateway(sf(), AbstractRemoteFileOutboundGateway.Command.MV, null)
.renameExpression("'/newdirectory/' + payload"))
.channel("nullChannel")
.get();
return flow;
The #remoteFileName is a SpEL variable for the remote file in the context of the local file generation with MGET command.
The MV command does not have a context of remote file. It works only with one remote file. You can provide all the necessary info in the request message: payload and its headers. Then you can use their payload and headers properties in that renameExpression.
See more info in docs: https://docs.spring.io/spring-integration/docs/current/reference/html/ftp.html#using-the-mv-command
I suggest you to raise a new SO question about your regexp concern. That's not how SO works: https://stackoverflow.com/help/how-to-ask
I need to implement the following architecture:
I have data that must be sent to systems (Some external application ) using JMS.
Depending on the data you need to send only to the necessary systems (For example, if the number of systems is 4, then you can send from 1 to 4 )
It is necessary to wait for a response from the systems to which the messages were sent, after receiving all the answers, it is required to process the received data (or to process at least one timeout)
The correlation id is contained in the header of both outgoing and incoming JMS messages
Each new such process can be started asynchronously and in parallel
Now I have it implemented only with the help of Spring JMS. I synchronize the threads manually, also manually I manage the thread pools.
The correlation ids and information about the systems in which messages were sent are stored as a state and update it after receiving new messages, etc.
But I want to simplify the logic and use Spring-integration Java DSL, Scatter gather pattern (Which is just my case) and other useful Spring features.
Can you help me show an example of how such an architecture can be implemented with the help of Spring-integration/IntregrationFlow?
Here is some sample from our test-cases:
#Bean
public IntegrationFlow scatterGatherFlow() {
return f -> f
.scatterGather(scatterer -> scatterer
.applySequence(true)
.recipientFlow(m -> true, sf -> sf.handle((p, h) -> Math.random() * 10))
.recipientFlow(m -> true, sf -> sf.handle((p, h) -> Math.random() * 10))
.recipientFlow(m -> true, sf -> sf.handle((p, h) -> Math.random() * 10)),
gatherer -> gatherer
.releaseStrategy(group ->
group.size() == 3 ||
group.getMessages()
.stream()
.anyMatch(m -> (Double) m.getPayload() > 5)),
scatterGather -> scatterGather
.gatherTimeout(10_000));
}
So, there is the parts:
scatterer - to send messages to recipients. In your case all those JMS services. That can be a scatterChannel though. Typically PublishSubscribeChannel, so Scatter-Gather might not know subscrbibers in adavance.
gatherer - well, it is just an aggregator with all its possible options.
scatterGather - is just for convenience for the direct properties of the ScatterGatherHandler and common endpoint options.
I am trying to make parallel calls to getPrice method, for each product in products. I have this piece of code and verified that getPrice is running in separate threads, but they are running sequentially, not in parallel. Can anyone please point me to what am I missing here?
Thanks a lot for your help.
ExecutorService service = Executors.newFixedThreadPool(THREAD_POOL_SIZE);
Set<Product> decoratedProductSet = products.stream()
.map(product -> CompletableFuture
.supplyAsync(() -> getPrice(product.getId(), date, context), service))
.map(t -> t.exceptionally(throwable -> null))
.map(t -> t.join())
.collect(Collectors.<Product>toSet());
You are streaming your products, sending each of to a CompletableFuture but then wait for it with join, before the stream processes the next one.
Why not use:
products.parallelStream()
.map(p -> getPrice(p.getId(), date, context))
.collect(Collectors.<Product>toSet());