Here's my use case:
I have 1 DB and 1 set of tables for quartz
I have a spring boot application X which have N nodes (VMs) and job class A, B and C
I have another spring boot application Y which have M nodes and job class D, E and F
How should I define application.properties for each application to make sure the quartz schedule the job to the right clusters? (job A,B,C goes to N and job D,E,F goes to M)
I guess this should be required for both applications:
spring.quartz.properties.org.quartz.jobStore.isClustered=true
But I'm not sure which of below property make the call, or I need define both for each application:
spring.quartz.properties.org.quartz.scheduler.instanceName=???
spring.quartz.properties.org.quartz.scheduler.instanceId=???
Thanks in advance!
Related
Setting 'common' properties for child tasks is not working
The SCDF version I'm using is 2.9.6.
I want to make CTR A-B-C, each of tasks does follows:
A : sql select on some source DB
B : process DB data that A got
C : sql insert on some target DB
Simplest way to make this work seems to define shared work directory folder Path "some_work_directory", and pass it as application properties to A, B, C. Under {some_work_directory}, I just store each of task result as file, like select.result, process.result, insert.result, and access them consequently. If there is no precedent data, I could assume something went wrong, and make tasks exit with 1.
================
I tried with a composed task instance QWER, with two task from same application "global" named as A, B. This simple application prints out test.value application property to console, which is "test" in default when no other properties given.
If I tried to set test.value in global tab on SCDF launch builder, it is interpreted as app.*.test.value in composed task's log. However, SCDF logs on child task A, B does not catch this configuration from parent. Both of them fail to resolve input given at launch time.
If I tried to set test.value as row in launch builder, and pass any value to A, B like I did when task is not composed one, this even fails. I know this is not 'global' that I need, it seems that CTR is not working correctly with SCDF launch builder.
The only workaround I found is manually setting app.QWER.A.test.value=AAAAA and app.QWER.B.test.value=BBBBB in launch freetext. This way, input is converted to app.QWER-A.app.global4.test.value=AAAAA, app.QWER-B.app.global4.test.value=BBBBB, and print well.
I understand that, by this way, I could set detailed configurations for each of child task at launch time. However, If I just want to set some 'global' that tasks in one CTR instance would share, there seems to be no feasible way.
Am I missing something? Thanks for any information in advance.
CTR will orchestrate the execution of a collection of tasks. There is no implicit data transfer between tasks. If you want the data from A to be the input to B and then output of B becomes the input of C you can create one Task / Batch application that have readers and writers connected by a processor OR you can create a stream application for B and use JDBC source and sink for A and C.
I am facing a hard time finding a solution for a problem I am facing which is:
Let's say I have 4 microservices (A,B,C,D) that interact with each other (rest APIs)
A calls B and B calls D. So that path for a single request is A/B/D
Below is the logging pattern.
logging.pattern.console=%d{yyyy-MM-dd HH:mm:ss.SSS} ${LOG_LEVEL_PATTERN:-%5p} ${PID:- } [%15.15t] %logger{10}:%L | %m%n
I want to add path in it.
So lets say Request is initiated from A to B, in the logs of B I want it to display Path:A/
now B calls D, the Path should be in the logs of D: A/B
Please suggest how do I manage this.
I am sorry for naïve question since I am new to SLF4J
as I understand you want to know the path your request took!!! for that you can use Spring Cloud Sleuth provides Spring Boot auto-configuration for distributed tracing
https://developer.okta.com/blog/2021/07/26/spring-cloud-sleuth
I want to write a spring boot batch application where I have a database table full of events. What I want to do is have a multi threaded spring boot batch application which will work in this way:
I want to have 5 threads running each thread will keep an offset to track what event it has read so that no other thread reads the same event again. How am I thinking to do it:
Thread 1 will pick up the event if eventId % 5 == 1
Thread 2 will pick up the event if eventId % 5 == 2
Thread 2 will pick up the event if eventId % 5 == 3
Thread 2 will pick up the event if eventId % 5 == 4
Thread 2 will pick up the event if eventId % 5 == 0
so I want to be able to keep offset in a database table for each thread. Is there a way to make Spring boot environment to work in this way?
You can partition your table according to what you described (I guess you will have 5 partitions) and use a partitioned step configured with a TaskExecutorPartitionHandler of 5 threads. With this setup you will achieve what you are looking for.
Hope this helps.
Seeking for an advise on the below faced COA correlation issue.
Background: there is an application A which is feeding data to an application B via MQ (nothing special - remote queue def pointing to the local q def on remote QM). Where the sending app A is requesting COAs. That is a stable setup working for years:
App A -> QM.A[Q1] -channel-> QM.B[Q2] -> App B
Here:
Q1 is a remote q def pointing to the Q2.
Problem: there is an application C which requires exactly the same data feed which A is sending to B via MQ. => it is required to duplicate data feed considering the following constraint.
Constraint: neither code, nor app config of applications A and B could be changed - duplication of the data feed from A to B should be transparent for applications A and B - A puts messages to the same queue Q1 on QM.A; B gets messages from the same queue Q2 on the QM.B
Proposed solution: duplicate the feed on the MQ layer by creation of the Topic/subscirbers configuration on the QM of the app B:
App A -> QM.A[Q1] -channel-> QM.B[QA->T->{S2,S3}->{Q2,Q3}] -> {App B, QM.C[Q4] -> App C}
Here:
Q1 - has the rname property updated to point to the QA for Topic
instead of Q2
QA - Queue Alias for Topic T
T - Topic
S2, S3 - subscribers publishing data to the Q2 and Q3
Q2 - unchanged, the same local queue definition where App B consumes from
Q3 - remote queue definition pointing to the Q4
Q4 - local queue definition on the QM.C, the queue with copy of messages sent from A to B
With this set up duplication of the messages from the app A to the app B and C works fine.
But ... there is an issue.
Issue: application A is not able to correlate COAs and that is the problem.
I'm not sure if app A is not able to correlate COAs at all, or (what is more likely guess) it is not able to correlate additional COAs e.g. from the QM.C
Any idea or advise is very much appreciated.
I have two kinds of tasks in spark : A and B
In spark.scheduler.pool, I have two pools: APool and BPool.
I want task A to be executed aways in APool while B is in BPool.
The resources in APool is preserved to A.
Because task B may take too much resources to execute. Every time when B is executing, A needs to wait. I want no matter when the task is submitted, there will always be some resource for A to execute.
I am using spark with java in standalone mode. I submit the job like javaRDD.map(..).reduce... The javaRDD is a sub-clesse extended form JavaRDD. Task A and B have different RDD class like ARDD and BRDD. They run in the same spark application.
The procedure is like: The app start up -> spark application created, but no job runs -> I click "run A" on the app ui, then ARDD will run. -> I click "run B" on the app ui, then BRDD will run in the same spark application as A.