I am trying to trigger a spring batch job and expect run.id as a job param to be passed after adding RunIdIncrementer while defining the job bean . But from the logs it looks like the param is not getting passed.
Job Bean
public Job firstPartyStatsIngestionJob() {
return jobBuilderFactory.get(TradingCenterJobName.firstPartyStatsIngestionJob.name())
.incrementer(new RunIdIncrementer())
.start(getFirstPartyIngestionStep())
.build();
}
Here I expect a job param run.id to be passed when the job is triggered but in the logs, all I see is launched with the following parameters: [{}]
[INFO ] 2022-12-16 09:04:07 [main ] [/] [] JobLauncherApplicationRunner:154 - Running default command line with: []
[INFO ] 2022-12-16 09:04:07 [main ] [/] [] SimpleJobLauncher:146 - Job: [SimpleJob: [name=firstPartyStatsIngestionJob]] launched with the following parameters: [{}]
key dependencies and versions
spring-boot: 2.5.12
spring-batch-core : 4.3.5
Try launching the job using the JobOperator.startNextInstance method...
https://docs.spring.io/spring-batch/docs/current/api/org/springframework/batch/core/launch/JobOperator.html#startNextInstance(java.lang.String)
If running from the command line, you can specify the -next argument...
https://docs.spring.io/spring-batch/docs/current/api/org/springframework/batch/core/launch/support/CommandLineJobRunner.html
Related
DebeziumEngine looking for kafka topic eventhough i have not specified KafkaOffsetBackingStore for offset.storage
Reference : DebeziumEngine Config
Config
Configuration config = Configuration.create()
.with("name", "oracle_debezium_connector")
.with("connector.class", "io.debezium.connector.oracle.OracleConnector")
.with("offset.storage", "org.apache.kafka.connect.storage.FileOffsetBackingStore")
.with("offset.storage.file.filename", "/Users/dk/Documents/work/ACET/offset.dat")
.with("offset.flush.interval.ms", 2000)
.with("database.hostname", "localhost")
.with("database.port", "1521")
.with("database.user", "pravin")
.with("database.password", "*****")
.with("database.sid", "ORCLCDB")
.with("database.server.name", "mServer")
.with("database.out.server.name", "dbzxout")
.with("database.history", "io.debezium.relational.history.FileDatabaseHistory")
.with("database.history.file.filename", "/Users/dk/Documents/work/ACET/dbhistory.dat")
.with("topic.prefix","cycowner")
.with("database.dbname", "ORCLCDB")
.build();
DebeziumEngine
DebeziumEngine<ChangeEvent<String, String>> engine = DebeziumEngine.create(Json.class)
.using(config.asProperties())
.using(connectorCallback)
.using(completionCallback)
.notifying(record -> {
System.out.println(record);
})
.build();
Error :
2022-10-29T16:06:16,457 ERROR [pool-2-thread-1] i.d.c.Configuration: The 'schema.history.internal.kafka.topic' value is invalid: A value is required
2022-10-29T16:06:16,457 ERROR [pool-2-thread-1] i.d.c.Configuration: The 'schema.history.internal.kafka.bootstrap.servers' value is invalid: A value is required**
2022-10-29T16:06:16,458 INFO [pool-2-thread-1] i.d.c.c.BaseSourceTask: Stopping down connector
2022-10-29T16:06:16,463 INFO [pool-3-thread-1] i.d.j.JdbcConnection: Connection gracefully closed
2022-10-29T16:06:16,465 INFO [pool-2-thread-1] o.a.k.c.s.FileOffsetBackingStore: Stopped FileOffsetBackingStore
connector stopped successfully
---------------------------------------------------
success status: false, message : Unable to initialize and start connector's task class 'io.debezium.connector.oracle.OracleConnectorTask' with config: {connector.class=io.debezium.connector.oracle.OracleConnector, database.history.file.filename=/Users/dkuma416/Documents/work/ACET/dbhistory.dat, database.user=pravin, database.dbname=ORCLCDB, offset.storage=org.apache.kafka.connect.storage.FileOffsetBackingStore, database.server.name=mServer, offset.flush.timeout.ms=5000, errors.retry.delay.max.ms=10000, database.port=1521, database.sid=ORCLCDB, offset.flush.interval.ms=2000, topic.prefix=cycowner, offset.storage.file.filename=/Users/dkuma416/Documents/work/ACET/offset.dat, errors.max.retries=-1, database.hostname=localhost, database.password=********, name=oracle_debezium_connector, database.out.server.name=dbzxout, errors.retry.delay.initial.ms=300, value.converter=org.apache.kafka.connect.json.JsonConverter, key.converter=org.apache.kafka.connect.json.JsonConverter, database.history=io.debezium.relational.history.MemoryDatabaseHistory}, **Error: Error configuring an instance of KafkaSchemaHistory; check the logs for details**
We were using fabric8 kubernetes client 5.3.x for watcher and it worked fine. Recently when we moved to 5.11.2 there were many changes observed and eventually the JUnit Tests started failing.
We use io.fabric8.kubernetes.client.server.mock.KubernetesServer
Earlier we were using ContainerStatus .withNewReady which now seem to be removed.
And then we added the following annotation
#Rule
public KubernetesServer myMockServer = new KubernetesServer(false, true);
After this, we are getting the following logs stating unsupported label requirement while the application code is sending this label.
[2022-02-03T07:30:22.733Z] 07:30:17.812 [pool-1-thread-1] DEBUG io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager - Watching http://localhost:40033/api/v1/namespaces/test/pods?labelSelector=app.kubernetes.io%2Fname%20in%20%28apps%29&timeoutSeconds=0&allowWatchBookmarks=true&watch=true...
[2022-02-03T07:30:22.733Z] 07:30:17.814 [MockWebServer /127.0.0.1:49882] WARN io.fabric8.kubernetes.client.server.mock.KubernetesAttributesExtractor - Ignoring unsupported label requirement: app.kubernetes.io/name in (apps)
[2022-02-03T07:30:22.733Z] 07:30:17.815 [MockWebServer /127.0.0.1:49882] DEBUG io.fabric8.kubernetes.client.server.mock.KubernetesAttributesExtractor - fromPath /api/v1/namespaces/test/pods?labelSelector=app.kubernetes.io%2Fname%20in%20%28apps%29&timeoutSeconds=0&allowWatchBookmarks=true&watch=true : {attributes: {namespace={key:namespace, value:test}, version={key:version, value:v1}, plural={key:plural, value:pods}}}
[2022-02-03T07:30:22.733Z] 07:30:17.815 [OkHttp http://localhost:40033/...] DEBUG io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener - WebSocket successfully opened
[2022-02-03T07:30:22.733Z] 07:30:20.818 [OkHttp http://localhost:40033/...] DEBUG io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager - Scheduling reconnect task
Is there something more should be done? Tests are in ERROR state and not Failed.
Sample JUnit program
public void testAddNewPodWatchEvent()
{
//given
doReturn(myClientMock).when(myTestObj).getClient();
doReturn(myWatcherSpy).when(myTestObj).createEventHandler();
String PATH =
"/api/v1/namespaces/test/pods?labelSelector=app.kubernetes.io%2Fname%20in%20%28apps%29&timeoutSeconds=0&watch=true";
Map<String, String> mockLabelMap = new HashMap<>();
mockLabelMap.put("foo", "testlabel");
mockLabelMap.put("app.kubernetes.io/name", "apps");
Pod accPod = createAppsPod(mockLabelMap, true);
myMockServer.expect()
.get()
.withPath(PATH)
.andUpgradeToWebSocket()
.open()
.waitFor(100)
.andEmit(new WatchEvent(accPod, "ADDED"))
.done()
.once();
//when
myTestObj.activate(mockProps);
sleepForWatchToBeInvoked();
//then
verify(myWatcherSpy, atLeastOnce()).eventReceived(Action.ADDED, accPod);
}
I am trying to trigger a job A(this is configured as trigger remote) remotely from another job B, and job B needs to hold until results come back to show success or failure, I initially tried using rest API using curl command, it perfectly works.here's the curl code:
curl -v -X POST 'https://xxx.xxx/xxx-xxx/job/xxx/job/master/buildWithParameters?config_files=./jenkins/unit-tests.json' --user xxxx:110f4dfa33ba8f8ef5d8d299beb6aa1543
I choose parameterized plugin code which installed on Jenkins server because it handles the polling mechanism internally and also has handler friendly methods. please see below code for remoteJob, but it fails with 405 error, that means method not allowed in HTTP language, looks like plugin is using GET method instead of post. I added an option for logging , but it does not seems to be showing more log.
def handle = triggerRemoteJob(
remoteJenkinsName: 'remote-master',
job: 'https://xxx.xxx.com/xxx-xxx/job/xxx/job/master/buildWithParameters',
remoteJenkinsUrl: 'https://xxx.xxx.xxx/xxx-xxx/job/xxx/job/master/buildWithParameters',
auth: TokenAuth(apiToken: hudson.util.Secret.fromString('110f4dfa33ba8f8ef5d8d299beb6aa1543'), userName: 'xxxx'),
parameters: 'config_files=./jenkins/unit-tests')
I am getting following error -
[Pipeline] triggerRemoteJob
##########################################################################
Parameterized Remote Trigger Configuration:
- job: https://xxx.xxx.xxx/xxx-xxx/job/xxx/job/master/buildWithParameters
- remoteJenkinsUrl: https://xxx.xxx.xxx/xxx-xxx/job/ius/job/master/buildWithParameters
- auth: 'Token Authentication' as user 'sseri'
- parameters: [config_files=./jenkins/unit-tests]
- blockBuildUntilComplete: true
- connectionRetryLimit: 5
- trustAllCertificates: false
##########################################################################
Connection to remote server failed [405], waiting to retry - 10 seconds until next attempt. URL: https://xxx.xxx.xxx/xxx-xxx/job/xxx/job/master/buildWithParameters/api/json, parameters:
Retry attempt #1 out of 5
Please help me in this regard!
I am not sure about the plugins you are using, but it's quite simple to implement this scenario "call a downstream job from upstream and fail upstream if the downstream fail" without any plugins.
Take a look at my example below.
let say if you have 2 jobs called jobA and jobB and your goal is to call jobB from jobA and fail the jobA if jobB fail.
**Scripted Pipeline for jobA **
node() {
try {
def jobB = build(job: jobName,parameters: [string(name:"parameterName",value: "parameterValue")])
def jobBStatus = jobB.getResult()
if(jobBStatus == "failed") {
throw new RuntimeException("Downstream job-b failed with reason ...");
}
...
}catch(Exception e) {
throw e
}
}
Declarative Pipeline for jobA
pipeline {
agent any;
stages {
stage('call jobB') {
steps {
script {
def jobB = build(job: jobName,parameters: [
string(name:"parameterName",value: "parameterValue")
])
def jobBStatus = jobB.getResult()
if(jobBStatus == "failed") {
error("Downstream job-b failed with reason ...")
}
}
}
}
}
}
Try using this Parameterized-Remote-Trigger-Plugin. It should give you what you want. I'm having some problems configuring it using authentication tokens and users using Jenkinsfile but if you are using the GUI im sure you will get the job done.
I have couple of routes (route 1 and route 2) in my Spring Boot application. I have been researching how to gracefully shutdown the application after processing both the routes. I have referred the documentation (https://camel.apache.org/manual/latest/graceful-shutdown.html) but couldn't successfully achieve what I needed. Maybe my understanding is wrong.
Below are my two routes
Route 1
from("timer://runOnce?repeatCount=1")
.to("{{sql.selectAll}}")
......... SOME PROCESSING
.to("direct:checkStatus")
Route 2
from("direct:checkStatus")
.delay(5000)
.loopDoWhile(CONDITION)
.process(DO_SOMETHING)
.end()
.to("jpa:com.pqr.MyClass)
.stop();
I have tried all these options
1. Automatic shutdown after 60 seconds
camel.springboot.duration-max-seconds = 60
It does GRACEFULLY shutdown the 2 routes but then WARNs about FORCEFUL shutdown ExecutorsService and also it doesn't stop the main thread to stop the application.
2020-03-01 18:28:25.507 WARN 30279 --- [otTerminateTask] o.a.c.i.e.BaseExecutorServiceManager : Forcing shutdown of ExecutorService: org.apache.camel.util.concurrent.SizedScheduledExecutorService#17fbfb02[CamelSpringBootTerminateTask] due first await termination elapsed.
2020-03-01 18:28:25.507 WARN 30279 --- [otTerminateTask] o.a.c.i.e.BaseExecutorServiceManager : Forcing shutdown of ExecutorService: org.apache.camel.util.concurrent.SizedScheduledExecutorService#17fbfb02[CamelSpringBootTerminateTask] due interrupted.
2020-03-01 18:28:25.508 INFO 30279 --- [otTerminateTask] o.a.c.i.e.BaseExecutorServiceManager : Shutdown of ExecutorService: org.apache.camel.util.concurrent.SizedScheduledExecutorService#17fbfb02[CamelSpringBootTerminateTask] is shutdown: true and terminated: false took: 10.004 seconds.
2020-03-01 18:28:25.508 WARN 30279 --- [otTerminateTask] o.a.c.i.e.BaseExecutorServiceManager : Forced shutdown of 1 ExecutorService's which has not been shutdown properly (acting as fail-safe)
2020-03-01 18:28:25.508 WARN 30279 --- [otTerminateTask] o.a.c.i.e.BaseExecutorServiceManager : forced -> org.apache.camel.util.concurrent.SizedScheduledExecutorService#17fbfb02[CamelSpringBootTerminateTask]
2. Initiate shutdown from the Route2
from("direct:checkStatus")
.delay(5000)
.loopDoWhile(CONDITION)
.process(DO_SOMETHING)
.end()
.to("jpa:com.pqr.MyClass)
.process(exchange -> {
exchange.getContext().getRouteController().stopRoute("route1");
exchange.getContext().getRouteController().stopRoute("route2");
System.out.println("Route1 -->"+exchange.getContext().getRouteController().getRouteStatus("route1"));
System.out.println("Route2 -->"+exchange.getContext().getRouteController().getRouteStatus("route2"));
exchange.getContext().shutdown();
});
"route1" is gracefully stopped but "route2" fails to be gracefully stopped with below message and waits for default timeout (300s).
2020-03-01 18:35:29.113 INFO 30504 --- [read #4 - Delay] o.a.c.i.engine.DefaultShutdownStrategy : Starting to graceful shutdown 1 routes (timeout 300 seconds)
2020-03-01 18:35:29.116 INFO 30504 --- [ - ShutdownTask] o.a.c.i.engine.DefaultShutdownStrategy : Route: route1 shutdown complete, was consuming from: timer://runOnce?repeatCount=1
2020-03-01 18:35:29.116 INFO 30504 --- [read #4 - Delay] o.a.c.i.engine.DefaultShutdownStrategy : Graceful shutdown of 1 routes completed in 0 seconds
2020-03-01 18:35:29.117 INFO 30504 --- [read #4 - Delay] o.a.c.s.boot.SpringBootCamelContext : Route: route1 is stopped, was consuming from: timer://runOnce?repeatCount=1
2020-03-01 18:35:29.117 INFO 30504 --- [read #4 - Delay] o.a.c.i.engine.DefaultShutdownStrategy : Starting to graceful shutdown 1 routes (timeout 300 seconds)
2020-03-01 18:35:29.118 INFO 30504 --- [ - ShutdownTask] o.a.c.i.engine.DefaultShutdownStrategy : Waiting as there are still 1 inflight and pending exchanges to complete, timeout in 300 seconds. Inflights per route: [route2 = 1]
It looks like there is a pending exchange message to be consumed. Do I need to manually clear/consume the exchange message in order to clear and facilitate a graceful shutdown?
Either option doesn't stop the main application. Do I have to write a custom Shutdown strategy instead of DefaultShutdownStrategy to achieve this? Can someone kindly point to an example to shut down the Spring Boot application after completion of the routes? Thanks in advance!!!
Did you try to use exchange.getContext().stop() to stop main application?
To force stop route without waiting for default timeout you can use exchange.getContext().stopRoute(routeId, 1L, TimeUnit.SECONDS); or set your timeout in seconds context.getShutdownStrategy().setTimeout(30);
You have to stop the currently running route from a new thread. The onCompletion() DSL is to make sure every message has been processed.
The attached code is in Kotlin, but it should be easy to transfer it to Java:
fromF(route).id(routeId)
.process(someProcessor)
.to("jdbc:dataSource")
.onCompletion()
.choice().`when`(exchangeProperty("CamelBatchComplete"))
.process(object : Processor {
override fun process(exchange: Exchange) {
Thread {
try {
exchange.context.routeController.stopRoute(routeId)
exchange.context.stop()
} catch (e: Exception) {
throw RuntimeException(e)
}
}.start()
}
}
)
// must use end to denote the end of the onCompletion route
.end()
If you want to stop the entire application, you can use this class and add a call of shutdownManager.initiateShutdown() after the exchange.context.stop().
#Component
class ShutdownManager {
companion object {
val logger = LoggerFactory.getLogger(ShutdownManager::class.java)
}
#Autowired
private val appContext: ApplicationContext? = null
fun initiateShutdown(returnCode: Int) {
logger.info("Shutting down with a Shutdown manager")
SpringApplication.exit(appContext, ExitCodeGenerator { returnCode })
System.exit(returnCode)
}
}
I use the following logic to restart the Spring Batch uncompleted(for example after application abnormal termination) jobs:
public void restartUncompletedJobs() {
LOGGER.info("Restarting uncompleted jobs");
try {
jobRegistry.register(new ReferenceJobFactory(documetPipelineJob));
List<String> jobs = jobExplorer.getJobNames();
for (String job : jobs) {
Set<JobExecution> runningJobs = jobExplorer.findRunningJobExecutions(job);
for (JobExecution runningJob : runningJobs) {
runningJob.setStatus(BatchStatus.FAILED);
runningJob.setEndTime(new Date());
jobRepository.update(runningJob);
jobOperator.restart(runningJob.getId());
LOGGER.info("Job restarted: " + runningJob);
}
}
} catch (Exception e) {
LOGGER.error(e.getMessage(), e);
}
}
This works fine but with one side effect - it doesn't restart the failed job execution but creates a new execution instance. How to change this logic in order to restart the failed execution from the failed step and do not create a new execution ?
UPDATED
When I try the following code:
public void restartUncompletedJobs() {
try {
jobRegistry.register(new ReferenceJobFactory(documetPipelineJob));
List<String> jobs = jobExplorer.getJobNames();
for (String job : jobs) {
Set<JobExecution> jobExecutions = jobExplorer.findRunningJobExecutions(job);
for (JobExecution jobExecution : jobExecutions) {
jobOperator.restart(jobExecution.getId());
}
}
} catch (Exception e) {
LOGGER.error(e.getMessage(), e);
}
}
it fails with the following exception:
2018-07-30 06:50:47.090 ERROR 1588 --- [ main] c.v.p.d.service.batch.BatchServiceImpl : Illegal state (only happens on a race condition): job execution already running with name=documetPipelineJob and parameters={ID=826407fa-d3bc-481a-8acb-b9643b849035, inputDir=/home/public/images, STORAGE_TYPE=LOCAL}
org.springframework.batch.core.UnexpectedJobExecutionException: Illegal state (only happens on a race condition): job execution already running with name=documetPipelineJob and parameters={ID=826407fa-d3bc-481a-8acb-b9643b849035, inputDir=/home/public/images, STORAGE_TYPE=LOCAL}
at org.springframework.batch.core.launch.support.SimpleJobOperator.restart(SimpleJobOperator.java:283) ~[spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobOperator$$FastClassBySpringCGLIB$$44ee6049.invoke(<generated>) ~[spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [spring-core-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:684) [spring-aop-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobOperator$$EnhancerBySpringCGLIB$$7659d4c.restart(<generated>) ~[spring-batch-core-4.0.1.RELEASE.jar!/:4.0.1.RELEASE]
at com.example.pipeline.domain.service.batch.BatchServiceImpl.restartUncompletedJobs(BatchServiceImpl.java:143) ~[domain-0.0.1.jar!/:0.0.1]
The following code creates new executions in jobstore database:
public void restartUncompletedJobs() {
try {
jobRegistry.register(new ReferenceJobFactory(documetPipelineJob));
List<String> jobs = jobExplorer.getJobNames();
for (String job : jobs) {
Set<JobExecution> jobExecutions = jobExplorer.findRunningJobExecutions(job);
for (JobExecution jobExecution : jobExecutions) {
jobExecution.setStatus(BatchStatus.STOPPED);
jobExecution.setEndTime(new Date());
jobRepository.update(jobExecution);
Long jobExecutionId = jobExecution.getId();
jobOperator.restart(jobExecutionId);
}
}
} catch (Exception e) {
LOGGER.error(e.getMessage(), e);
}
}
The question is - how to continue to run the old uncompleted executions without creating new ones after application restart?
TL;DR: Spring Batch will always create new Job Execution and will not reuse a previous failed job execution to continue its execution.
Longer answer: First you need to understand three similar but different concept in Spring Batch: Job, Job Instance, Job Execution
I always use this example:
Job : End-Of-Day Batch
Job Instance : End-Of-Day Batch for 2018-01-01
Job Execution: End-Of-Day Batch for 2018-01-01, execution #1
In high-level, that's how Spring Batch's recovery works:
Assuming your first execution failed in the step 3. You can submit the same Job (End-of-Day Batch) with same Parameters (2018-01-01). Spring Batch will try to look up last Job Execution (End-Of-Day Batch for 2018-01-01, execution #1) of the submitted Job Instance (End-of-Day Batch for 2018-01-01), and found that it has previously failed in step 3. Spring Batch will then create a NEW execution, [End-Of-Day Batch for 2018-01-01, execution #2], and start the execution from step 3.
So by design, what Spring trying to recover is a previously failed Job Instance (instead of Job Execution). Spring batch will not reuse execution when you are re-running a previous-failed execution.