I have a kettle job with two transformations.
Trans1: runs and the result is logged into kettle transformation & step log tables.
Trans2: checks the step log table for result of Trans1 and success if Trans1 run without error.
the job works fine if i call it manually, but when i schedule it the transformation and step logs got delayed and Trans2 fails.
I am not sure what causes the log delay.
Related
I have a batch that is scheduled to run every day at 00:05.
On some days, a job in the batch fails immediately with the ModelNotFoundException.
However, the model that was not found does exist.
There were no change to any of the concerned models in the database. (Field, Category, Condition)
There is also no code in the application that allows to delete the category.
Retrying the job manually from the horizon dashboard make the job pass.
The dba said there are no error in the logs and no scheduled maintenance at that time.
What can possibly cause this?
My spring batch application is running on PCF platform which is connected to MySQL database (single instance), it's running fine when only an instance is up & running but when it comes to more than one application instance, I'm getting exception org.springframework.dao.DuplicateKeyException. This might be happening because similar batch job is firing at the same time & trying to update batch instance table with same job ID. Is there any way to restrict this kind of failure or in another way, I wanted a solution where only one batch job will run at a time even there are multiple instances running.
For me , it is a good sign that DuplicateKeyException is thrown. Because it exactly achieves what you want to do is that spring-batch already makes sure that the same job execution will not executed in parallel. (i.e. Only one server instance execute the job successfully while other fail to execute)
So I see no harms in your case. If you don't like this exception , you can catch it and re-throw it as your application level exception saying something like "The job is executing by other sever instances , so skip to execute it."
If you really want that only one server instance will try to trigger to execute a job and other servers will not try to trigger in the meantime , it is not the problem of spring-batch but is a problem about how you ensure that only one server node will fires the request in the distributed environment. If the batch job is fired as a scheduled task using #Scheduled , you can consider to use a distributed lock such as ShedLock to make sure that it is executed at most once at the same time on one node only.
I am new for Talend ETL. So which is the proper way to store talend logs when its running in automate.
1. Job running time
2. Error in-case if Job return error
3. Number of rows
In Talend Open Studio tStatCatcher components listens to components that have the tStatCatcher Statistics option set to true, and writes statistics information to the defined output. tStatCatcher also listens for the start and end of a Job's execution.
End of the job's execution, pass the logs from tStatCatcher to database
I'm using infosphere DataStage of IBM for loading data in ETL processes.
I have some problem with one of my jobs.
This job scheduler twice a month, and when it run automatic by the tool - it get an oracle error:
ORA-00813 : object no longer exists
But when we run it manualy after it failed - there is no error at all and it's finished fine.
I tried to run the query in oracle directly and it just fine.
That problem happend twice, and always after the failure - it run good with manual execution.
any idea?
Thanks.
I have created a dbms scheduler job which should write a short string to the alert log of each instance of the 9 databases on our 2-node 11g Oracle RAC cluster, once every 24 hours.
The job action is:
'dbms_system.ksdwrt(2, to_char(sysdate+1,''rrrr-mm-dd'') || ''_First'');'
which should write a line like:
2014-08-27_First
The job runs succesfully according to its log, and it does write what it's supposed to, but not always. It's only been scheduled for a few days, so I can't be certain, but it looks as if it will only write to one instance's alert log. Logs on both sides seem to be getting written to, but if it's on one side it's not on the other, so it appears. There is however no indication whatever of any failure in the job itself.
Can anyone shed any light on this behaviour? Thanks.