I've integrated my application with spring XD through the REST API.
I have a job def set up and deployed for each datasource (my module is jdbc related) and then launch specific jobs using the appropriate job def as a template with jobParameters provided in the POST request, containing criteria unique to that job run.
I then expected my REST client to be able to poll for metrics related to the job, it just initialized, but for such calls jobExecutionId is needed and the client doesn't have this value, only the jobName which is of course not unique.
I would have expected POST to /jobs/executions to respond with an executionId, the fact that it doesn't makes me question the way I'm using xd.
Can anyone shed some light on this?
I agree with you; this seems to be an oversight...
/**
* Send the request to launch Job. Job has to be deployed first.
*
* #param name the name of the job
* #param jobParameters the job parameters in JSON string
*/
#RequestMapping(value = "", method = RequestMethod.POST, params = "jobname")
#ResponseStatus(HttpStatus.CREATED)
public void launchJob(#RequestParam("jobname") String name, #RequestParam(required = false) String jobParameters) {
...
}
I can see the problem; we'd need to somehow feed the job execution back to the controller method (right now the there's no result from the launch request).
There is a work-around...
Use a job execution tap to dump the job execution to some sink; see the job event tap documentation.
xd:>stream create --name jobex --definition "tap:job:myJob.job > file --name=exec --dir=/tmp" --deploy
This will write jobExecution information to /tmp/exec.out which you can monitor (instead of polling the REST API).
If you are in distributed mode, you could dump the data to a Redis (or similar) sink.
But I agree, it would be better if the launch returned the id.
Please open a JIRA Issue.
Related
I'm using SpringBatch for my app. In one of the batch jobs, I need to process multiple data. Each data requires several database updates. And I need to make one transaction for one data. Meaning, if when processing one data an exception is thrown, database updates are rolled back for that data, then keep processing the next data.
I've put all database updates in one method in service layer. In my springbatch tasklet, I call that method for each data, like this;
for (RequestViewForBatch request : requestList) {
orderService.processEachRequest(request);
}
In the service class the method is like this;
Transactional(propagation = Propagation.NESTED, timeout = 100, rollbackFor = Exception.class)
public void processEachRequest(RequestViewForBatch request) {
//update database
}
When executing the task, it gives me this error message
org.springframework.transaction.NestedTransactionNotSupportedException: Transaction manager does not allow nested transactions by default - specify 'nestedTransactionAllowed' property with value 'true'
but i don't know how to solve this error.
Any suggestion would be appreciated. Thanks in advance.
The tasklet step will be executed in a transaction driven by Spring Batch. You need to remove the #Transactional on your processEachRequest method.
You would need a fault-tolerant chunk-oriented step configured with a skip policy. In this case, only faulty items will be skipped. Please refer to the Configuring Skip Logic section of the documentation. You can find an example here.
Intro:
We're currently using the spring mail integration to receive and send emails which works without flaws if there's no exception such as a connection error to the exchange server or the database.
These mails come in as Messages and are passed to a handler method which will parse the MimeMessage to a custom mail data object. JPA saves those entities as the last step to our database.
Question/Problem:
There's a problem if the database is down or the mail can't be processed for any other reason, as the IntegrationFlow will still mark it as /SEEN once the message gets passed to the handler.
Setting this flag to false won't fix our problem, because we want Spring to set the /SEEN flag if the mail is processed and saved correctly
shouldMarkMessagesAsRead(false)
Searching for:
Would there be a possibility to set flags AFTER successfully saving the mail to the database?
We'd like to process the failed email again after the cause for the responsible error is fixed, which won't work as long Spring marks them as /SEEN no matter the result.
Reference:
The messages comes in and gets passed to the handler which will parse the mail and execute the CRUD-Repository save(mailDAO) method. The handleMimeMessage() is more or less just a mapper.
#Bean
fun imapIdleFlow(imapProperties: ImapProperties): IntegrationFlow {
imapProperties.username.let { URLEncoder.encode(it, charset) }
return IntegrationFlows
.from(
Mail.imapIdleAdapter(
ImapMailReceiver("imap://${imapProperties.username}:${imapProperties.password}#${imapProperties.host}/Inbox")
.apply {
setSimpleContent(true)
setJavaMailProperties(imapProperties.properties.toProperties())
})
.autoStartup(true)
.shouldReconnectAutomatically(true)
)
.handle(this::handleMimeMessage)
.get()
}
Is it even possible to mark the messages in the same flow afterward as you need to access the exchange a second time or would I need a second flow to get and flag the same mail?
I think it is possible with something like transaction synchronization: https://docs.spring.io/spring-integration/reference/html/mail.html#mail-tx-sync
So, you set transactional(TransactionManager transactionManager) on that Mail.imapIdleAdapter to the JpaTransactionManager to start transaction from this IMAP Idle channel adapter and propagate it to your handleMimeMessage() where you do those JPA saves.
Plus you add:
/**
* Configure a {#link TransactionSynchronizationFactory}. Usually used to synchronize
* message deletion with some external transaction manager.
* #param transactionSynchronizationFactory the transactionSynchronizationFactory.
* #return the spec.
*/
public ImapIdleChannelAdapterSpec transactionSynchronizationFactory(
TransactionSynchronizationFactory transactionSynchronizationFactory) {
To react for commit and rollback of the mentioned transaction.
The DefaultTransactionSynchronizationFactory with some TransactionSynchronizationProcessor impl can give you a desired behavior, where you take a Message and its payload from the provided IntegrationResourceHolder and perform something like message.setFlag(Flag.SEEN, true); on the MimeMessage.
You may consider to use the mentioned in docs an ExpressionEvaluatingTransactionSynchronizationProcessor.
To avoid folder reopening, you may consider to use a public ImapIdleChannelAdapterSpec autoCloseFolder(boolean autoCloseFolder) { with a false option. You need to consider to close it in that TX sync impl or some other way.
I am working on a Spring Webflux project,
I want to do something like, When client make API call, I want to send success message to client and perform large file operation in background.
So client does not have to wait till my entire file is process.
For try out I made sample code as below
REST controller
#GetMapping(value = "/{jobId}/process")
#ApiOperation("Start import job")
public Mono<Integer> process(#PathVariable("jobId") long jobId) {
return service.process(jobId);
}
File processing Service
public Mono<Integer> process(Integer jobId) {
return repository
.findById(jobId)
.map(
job -> {
File file = new File("read.csv");
return processFile(file);
});
}
Following is my stack
Spring Webflux 2.2.2.RELEASE
I try to make this call using WebClient, but till entire file is not processed I am not getting response.
As one of the options, you can run processing in a different thread.
For example:
Create an Event Listener Link
Enable #Async and #EnableAsync Link
Or use deferent types of Executors from Java concurrency package
Or manually run the thread
Also for Kotlin you can use Coroutines
You can use the subscribe method and start a job with its own scope in background.
Mono.delay(Duration.ofSeconds(10)).subscribeOn(Schedulers.newElastic("myBackgroundTask")).subscribe(System.out::println);
As long as you do not tie this to your response publisher using one of the zip/merge or similar operators your job will be run on background on its own scheduler pool.
subscribe() method returns a Disposable instance which can later be used cancel the background job by calling dispose() method.
This questions is related to Return immediately in spring web flux but I don't think it's the same (at least the answer there is not satisfactory for me).
I have a function returning a Mono that when invoked starts a long-running job. This function is invoked when a call is made to a Spring Webflux HTTP API. Here's an example:
#PutMapping("/{jobId}")
fun startNewJob(#PathVariable("jobId") jobId: String,
request: ServerHttpRequest): Mono<ResponseEntity<Unit>> {
val longRunningJob : Mono<Job> = startNewJob(jobId)
longRunningJob.map { job ->
val jobUri = generateJobUri(request, job.id)
ResponseEntity.created(jobURI).build<Unit>()
}
}
The problem with the code above is that "201 Created" is created after the long running job is completed. I want to kick-off the longRunningJob in the background and return "201 Created" immediately.
I could perhaps do something like this:
#PutMapping("/{jobId}")
fun startNewJob(#PathVariable("jobId") jobId: String,
request: ServerHttpRequest): Mono<ResponseEntity<Unit>> {
startNewJob(jobId)
.subscribeOn(Schedulers.newSingle("thread"))
.subscribe()
val jobUri = generateJobUri(request, job.id)
val response = ResponseEntity.created(jobURI).build<Unit>()
Mono.just(response)
}
But it doesn't seem very idiomatic to me to have to call subscribe() manually (e.g. intellij is complaining that I call subscribe() in non-blocking scope). Isn't there a better way to compose the two "streams" without using an explicit subscribe? If so how do I modify the startNewJob function above to achieve this?
AFAIK, using one of the subscribe methods is the only way to really start a job in the background with its own lifecycle (not tied to the returned publisher).
If you were to use one of the operators to combine the job publisher and the response publisher (e.g. zip or merge), then the lifecycle of the job publisher would be tied to the response publisher, which is not what you want for a background job.
One thing you might want to consider is kicking off the background job within the response publisher stream, rather than directly in the method body. e.g. via doOnSubscibe or from an operator upstream of the response.
This would tie the start of the background job to the onSubscribe events of the response publisher, but still allow it to complete in the background.
Also note, that if you want to be able to cancel the background job (e.g. maybe during application shutdown), you'll need to save the Disposable returned from subscribe so you can later call dispose on it. This might be better done from some type of BackgroundJobManager that could keep track of all the jobs running.
private static final Scheduler backgroundTaskScheduler = Schedulers.newParallel("backgroundTaskScheduler", 2);
backgroundTaskScheduler.schedule(() -> doBackgroundJob());
I have an Enterprise Service Bus (ESB) that posts Data to Microservices (MCS) via Rest. I use Spring to do this. The main Problem is that i have 6 Microservices, that run one after one. So it looks like this: MCS1 -> ESB -> MCS2 -> ESB -> ... -> MCS6
So my Problem looks like this: (ESB)
#RequestMapping(value = "/rawdataservice/container", method = RequestMethod.POST)
#Produces(MediaType.APPLICATION_JSON)
public void rawContainer(#RequestBody Container c)
{
// Here i want to do something to directly send a response and afterwards execute the
// heavy code
// In the heavy code is a postForObject to the next Microservice
}
And the Service does something like this:
#RequestMapping(value = "/container", method = RequestMethod.POST)
public void addDomain(#RequestBody Container container)
{
heavyCode();
RestTemplate rt = new RestTemplate();
rt.postForObject("http://134.61.64.201:8080/rest/rawdataservice/container",container, Container.class);
}
But i dont know how to do this. I looked up the post for Location method, but i dont think it would solve the Problem.
EDIT:
I have a chain of Microservices. The first Microservice waits for a Response of the ESB. In the response the ESB posts to another Microservice and waits for a response and the next one does the same as the first one. So the Problem is that the first Microservice is blocked as long as the complete Microservice Route is completed.
ESB Route
Maybe a picture could help. 1.rawdataService 2.metadataservice 3.syntaxservice 4.semantik
// Here i want to do something to directly send a response and afterwards execute the
// heavy code
The usual spelling of that is to use the data from the http request to create a Runnable that knows how to do the work, and dispatch that runnable to an executor service for later processing. Much the same, you copy the data you need into a queue, which is polled by other threads ready to complete the work.
The http request handler then returns as soon as the executor service/queue has accepted the pending work. The most common implementation is to return a "202 Accepted" response, including in the Location header the url for a resource that will allow the client to monitor the work in progress, if desired.
In Spring, it might be ResponseEntity that manages the codes for you. For instance
ResponseEntity.accepted()....
See also:
How to respond with HTTP 400 error in a Spring MVC #ResponseBody method returning String?
REST - Returning Created Object with Spring MVC
From the caller's point of view, it would invoke RestTemplate.postForLocation, receive a URI, and throw away that URI because the microservice only needs to know that the work as been accepted
Side note: in the long term, you are probably going to want to be able to correlate the activities of the different micro services, especially when you are troubleshooting. So make sure you understand what Gregor Hohpe has to say about correlation identifiers.