Expression transformation is failing to load data - etl

I have written a simple expression transformation where the mapping concatenates the first and last name and based on salary of the customer it stores the string high,low or medium into the target table.
The workflow is validated and there are no errors. But when the workflow is started it reads the rows from the database and then applies the transformation and then tries to load the target table. Here while loading the data into the target table it doesn't move forward and gets struck and after long time when Integration service gets disconnected it throws a repository error.
The interesting thing is when I run some other workflows it doesn't give any error. Below are the session logs:
<pre><code>
DIRECTOR> TM_6014 Initializing session [s_mp_person_exp_person] at [Thu Feb
18 22:21:21 2016].
DIRECTOR> TM_6683 Repository Name: [infa_training]
DIRECTOR> TM_6684 Server Name: [infa_training_IS]
DIRECTOR> TM_6686 Folder: [training]
DIRECTOR> TM_6685 Workflow: [wf_person_exp_person] Run Instance Name: [] Run Id: [38]
DIRECTOR> TM_6101 Mapping name: mp_person_exp_person.
DIRECTOR> TM_6964 Date format for the Session is [MM/DD/YYYY HH24:MI:SS.US]
DIRECTOR> TM_6708 Using configuration property [EnableDataEncryption,no]
DIRECTOR> TM_6708 Using configuration property [StoreHAPersistenceInDB,no]
DIRECTOR> TM_6703 Session [s_mp_person_exp_person] is run by 64-bit Integration Service [node01_Gizmo], version [9.6.1], build [0606].
MANAGER> PETL_24058 Running Partition Group [1].
MANAGER> PETL_24000 Parallel Pipeline Engine initializing.
MANAGER> PETL_24001 Parallel Pipeline Engine running.
MANAGER> PETL_24003 Initializing session run.
MAPPING> CMN_1569 Server Mode: [ASCII]
MAPPING> CMN_1570 Server Code page: [MS Windows Latin 1 (ANSI), superset of Latin1]
MAPPING> TM_6151 The session sort order is [Binary].
MAPPING> TM_6156 Using low precision processing.
MAPPING> TM_6180 Deadlock retry logic will not be implemented.
MAPPING> TM_6187 Session target-based commit interval is [10000].
MAPPING> TM_6307 DTM error log disabled.
MAPPING> TE_7022 TShmWriter: Initialized
MAPPING> TM_6007 DTM initialized successfully for session [s_mp_person_exp_person]
DIRECTOR> PETL_24033 All DTM Connection Info: [<NONE>].
MANAGER> PETL_24004 Starting pre-session tasks. : (Thu Feb 18 22:21:21 2016)
MANAGER> PETL_24027 Pre-session task completed successfully. : (Thu Feb 18 22:21:21 2016)
DIRECTOR> PETL_24006 Starting data movement.
MAPPING> TM_6660 Total Buffer Pool size is 1219648 bytes and Block size is 65536 bytes.
READER_1_1_1> DBG_21438 Reader: Source is [orcl], user [kakarrot]
READER_1_1_1> BLKR_16003 Initialization completed successfully.
WRITER_1_*_1> WRT_8147 Writer: Target is database [orcl], user [target], bulk mode [OFF]
WRITER_1_*_1> WRT_8124 Target Table PERSON :SQL INSERT statement:
INSERT INTO PERSON(ID,FULLNAME,TOTALSAL,ANNUALSAL,CITY,STATE,MOB,SALSTATUS) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?)
WRITER_1_*_1> WRT_8124 Target Table PERSON :SQL UPDATE statement:
UPDATE PERSON SET FULLNAME = ?, TOTALSAL = ?, ANNUALSAL = ?, CITY = ?, STATE = ?, MOB = ?, SALSTATUS = ? WHERE ID = ?
WRITER_1_*_1> WRT_8124 Target Table PERSON :SQL DELETE statement:
DELETE FROM PERSON WHERE ID = ?
WRITER_1_*_1> WRT_8270 Target connection group #1 consists of target(s) [PERSON]
WRITER_1_*_1> WRT_8003 Writer initialization complete.
READER_1_1_1> BLKR_16007 Reader run started.
WRITER_1_*_1> WRT_8005 Writer run started.
WRITER_1_*_1> WRT_8158
*****START LOAD SESSION*****
Load Start Time: Thu Feb 18 22:21:21 2016
Target tables:
PERSON
READER_1_1_1> RR_4010 SQ instance [SQ_PERSON] SQL Query [SELECT kakarrot.Person.ID, kakarrot.Person.FNAME, kakarrot.Person.LNAME, kakarrot.Person.SALPERMONTH, kakarrot.Person.COMMISION, kakarrot.Person.CITY, kakarrot.Person.STATE, kakarrot.Person.MOB FROM kakarrot.Person]
READER_1_1_1> RR_4049 SQL Query issued to database : (Thu Feb 18 22:21:21 2016)
READER_1_1_1> RR_4050 First row returned from database to reader : (Thu Feb 18 22:21:21 2016)
READER_1_1_1> BLKR_16019 Read [6] rows, read [0] error rows for source table [Person] instance name [PERSON]
READER_1_1_1> BLKR_16008 Reader run completed.
TRANSF_1_1_1> DBG_21216 Finished transformations for Source Qualifier [SQ_PERSON]. Total errors [0]
WRITER_1_*_1> WRT_8167 Start loading table [PERSON] at: Thu Feb 18 22:21:21 2016
CMN_1761 Timestamp Event: [Fri Feb 19 08:01:22 2016]
REP_12014 An error occurred while accessing the repository
CMN_1761 Timestamp Event: [Fri Feb 19 08:01:23 2016]
REP_12400 Repository Error (Connection to the Repository Service [infa_training] is broken.)
CMN_1761 Timestamp Event: [Fri Feb 19 08:01:23 2016]
REP_12400 Repository Error ([REP_55101] Connection to the Repository Service [infa_training] is broken.)
CMN_1761 Timestamp Event: [Fri Feb 19 08:01:23 2016]
REP_12400 Repository Error ([REP_55114] Reconnecting to the Repository Service [infa_training]. The resilience time is 180 seconds.)
CMN_1761 Timestamp Event: [Fri Feb 19 08:01:50 2016]
REP_12400 Repository Error ([REP_51490] Communication with client application on host [169.254.164.2] and port [6013] has failed because of network errors. [System Error (errno = 10054): Unknown error].)
CMN_1761 Timestamp Event: [Fri Feb 19 10:00:14 2016]
REP_12400 Repository Error ([REP_55112] Unable to connect to the Repository Service [infa_training] since the resilience time is up.)
CMN_1761 Timestamp Event: [Fri Feb 19 10:00:14 2016]
REP_12400 Repository Error (Failed to connect to repository service [infa_training].)
CMN_1761 Timestamp Event: [Fri Feb 19 10:00:14 2016]
REP_12014 An error occurred while accessing the repository
CMN_1761 Timestamp Event: [Fri Feb 19 10:00:14 2016]
REP_12400 Repository Error (Failed to connect to repository service [infa_training].)
CMN_1761 Timestamp Event: [Fri Feb 19 10:00:14 2016]
REP_12400 Repository Error ([REP_55102] Failed to connect to repository service [infa_training].)
CMN_1761 Timestamp Event: [Fri Feb 19 10:00:15 2016]
PETL_24074 Failed to send updates to the master service process. Session run
`will be terminated.`
</pre></code>
Hope you guys can shed light on this problem.

The issue definitely appears to be with the database. According to your log, the session hung for 10 hours and then seems to have lost its connection to the repository database as well. You should try changing the target connection to a flat file and/or any other database you have access to in order to test the session itself. Also, if you don't have the ability to do it yourself, you should ask a DBA to look at the session as it is running to see why the oracle connection is waiting/hung up.

Related

Analysis of Redis error logs "LOADING Redis is loading the dataset in memory" and more

I am frequently seeing these messages in the redis logs
1#
602854:M 23 Dec 2022 09:48:54.028 * 10 changes in 300 seconds. Saving...
602854:M 23 Dec 2022 09:48:54.035 * Background saving started by pid 3266364
3266364:C 23 Dec 2022 09:48:55.844 * DB saved on disk
3266364:C 23 Dec 2022 09:48:55.852 * RDB: 12 MB of memory used by copy-on-write
602854:M 23 Dec 2022 09:48:55.938 * Background saving terminated with success
2#
LOADING Redis is loading the dataset in memory
3#
7678:signal-handler (1671738516) Received SIGTERM scheduling shutdown...
7678:M 22 Dec 2022 23:48:36.300 # User requested shutdown...
7678:M 22 Dec 2022 23:48:36.300 # systemd supervision requested, but NOTIFY_SOCKET not found
7678:M 22 Dec 2022 23:48:36.300 * Saving the final RDB snapshot before exiting.
7678:M 22 Dec 2022 23:48:36.300 # systemd supervision requested, but NOTIFY_SOCKET not found
7678:M 22 Dec 2022 23:48:36.720 * DB saved on disk
7678:M 22 Dec 2022 23:48:36.720 * Removing the pid file.
7678:M 22 Dec 2022 23:48:36.720 # Redis is now ready to exit, bye bye...
7901:C 22 Dec 2022 23:48:37.071 # WARNING supervised by systemd - you MUST set appropriate values for TimeoutStartSec and TimeoutStopSec in your service unit.
7901:C 22 Dec 2022 23:48:37.071 # systemd supervision requested, but NOTIFY_SOCKET not found
7914:C 22 Dec 2022 23:48:37.071 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
7914:C 22 Dec 2022 23:48:37.071 # Redis version=6.0.9, bits=64, commit=00000000, modified=0, pid=7914, just started
7914:C 22 Dec 2022 23:48:37.071 # Configuration loaded
Are these messages concerning?
Let me know if there's any optimization to be carried out in terms of settings.
The first set of informational messages is related to Redis persistence: it appears your Redis node is configured to save the database to disk if 300 seconds elapsed and it surpassed 10 write operations against it. You can change that according to your needs through the Redis configuration file.
The message LOADING Redis is loading the dataset in memory, on the other side, is an error message received while attempting to connect to a Redis instance which is loading its dataset in memory: that occurs during the startup for standalone servers and master nodes or when replicas reconnect and fully resynchronize with master. If you are seeing this error too often and not right after a system restart, I would suggest to check your system log files and learn why your Redis instance is restarting or resynchronizing (depending on your topology).

PostgresSQL: could not stat directory "base/<db_oid>": Unknown error

I am getting error in starting PostgreSQL server after a machine restart on windows server 2019.
This server has a multiple databases and in logs it is showing error for one of the database may be corrupted.
I tried restart server few times, but it it is getting shut down after some time. How can I make server up again with other databases running?
2022-04-20 07:32:49.843 PDT [3720] LOG: starting PostgreSQL 13.3, compiled by Visual C++ build 1914, 64-bit
2022-04-20 07:32:49.844 PDT [3720] LOG: listening on IPv6 address "::", port 5432
2022-04-20 07:32:49.844 PDT [3720] LOG: listening on IPv4 address "0.0.0.0", port 5432
2022-04-20 07:32:57.949 PDT [6600] LOG: database system was interrupted while in recovery at 2022-04-20 06:29:00 PDT
2022-04-20 07:32:57.949 PDT [6600] HINT: This probably means that some data is corrupted and you will have to use the last backup for recovery.
2022-04-20 07:34:29.599 PDT [852] FATAL: the database system is starting up
2022-04-20 07:44:04.349 PDT [2480] FATAL: the database system is starting up
2022-04-20 07:52:28.107 PDT [4800] FATAL: the database system is starting up
2022-04-20 08:09:30.847 PDT [1872] FATAL: the database system is starting up
2022-04-20 08:21:41.884 PDT [6600] LOG: could not stat file "./base/22504705": Unknown error
2022-04-20 08:21:47.121 PDT [6600] LOG: database system was not properly shut down; automatic recovery in progress
2022-04-20 08:22:01.957 PDT [6600] LOG: redo starts at 22E/5419D678 2022-04-20 08:22:01.994 PDT [6600] FATAL: could not stat directory "base/22504705": Unknown error
2022-04-20 08:22:01.994 PDT [6600] CONTEXT: WAL redo at 22E/5422EDD8 for Heap/LOCK: off 3: xid 26378859: flags 0x00 LOCK_ONLY KEYSHR_LOCK
2022-04-20 08:22:02.000 PDT [3720] LOG: startup process (PID 6600) exited with exit code 1
2022-04-20 08:22:02.000 PDT [3720] LOG: aborting startup due to startup process failure
2022-04-20 08:22:02.085 PDT [3720] LOG: database system is shut down
logs

Java 8 Time API - DB2/400 / JT400

I'm currently upgrading an application that uses DB2/400 database, and I'm migrating to the new java time api (LocalDate, LocalTime, ...), using the latest version of jt400.jar.
However, when executing statement setObject as follows
stmt.setObject(++columns, LocalDate.now());
I'm getting this exception
Thread[main,5,main] Ter jan 18 10:46:56:852 GMT 2022 as400: PreparedStatementImpl STMT0001 (1588970020) : setObject().
Thread[main,5,main] Ter jan 18 10:46:56:852 GMT 2022 as400: PreparedStatementImpl STMT0001 (1588970020) : parameter index: 2 type: java.time.LocalDate toString():2022-01-18.
Thread[main,5,main] Ter jan 18 10:46:56:853 GMT 2022 as400: Unknown com.ibm.as400.access.SQLDate#3c679bde (1013423070) : Unable to assign object(2022-01-18) of class(class java.time.LocalDate).
Thread[main,5,main] Ter jan 18 10:46:56:853 GMT 2022 as400: Unknown com.ibm.as400.access.SQLDate#3c679bde (1013423070) : Throwing exception, sqlState: 07006 reason: Data type mismatch. vendor code -99999.java.sql.SQLException: Data type mismatch.
at com.ibm.as400.access.JDError.createSQLExceptionSubClass(JDError.java:891)
at com.ibm.as400.access.JDError.throwSQLException(JDError.java:436)
at com.ibm.as400.access.SQLDate.set(SQLDate.java:452)
at com.ibm.as400.access.AS400JDBCPreparedStatementImpl.setValue(AS400JDBCPreparedStatementImpl.java:3655)
at com.ibm.as400.access.AS400JDBCPreparedStatementImpl.setObject(AS400JDBCPreparedStatementImpl.java:3124)
at mypackage.jdbc.LocalDateTester.main(LocalDateTester.java:26)
Also, when trying to read from the DB using
rs.getObject(2, LocalDate.class)
I'm also getting the exception
Exception in thread "main" java.sql.SQLException: Data type not valid.
at com.ibm.as400.access.JDError.createSQLExceptionSubClass(JDError.java:985)
at com.ibm.as400.access.JDError.throwSQLException(JDError.java:436)
at com.ibm.as400.access.JDError.throwSQLException(JDError.java:413)
at com.ibm.as400.access.AS400JDBCResultSet.getObject(AS400JDBCResultSet.java:7457)
at mypackage.jdbc.LocalDateTester.main(LocalDateTester.java:47)
The DB column type on iSeries is Date.
I'm I doing something wrong, or is the new java.time api not supported?
If so, does anyone know if there are any plans for Java Time API support?
Thanks in advance
You must use java.sql.date with JDBC. Convert with :
java.sql.Date.valueOf(LocalDate.now());

ODI to BI EE, LKM can't load data

I Try load data from ODI 12c to OBIEE 12c, I use reverse engineer and Knowledge Modules. I create connect from ODI to OBIEE, but after run mapping which should create new table and insert data I get
WARNING: SQLCommand execution failure: java.io.IOException: prepare query failed[nQSError: 43113] Message returned from OBIS.
[nQSError: 27002] Near <table>: Syntax error [nQSError: 26012] .
Failed query text:
drop table "Sample Targets Lite"...C_0DEFAULT
Feb 04, 2019 11:39:08 PM oracle.odi.runtime.agent.execution.sql.SQLCommand
WARNING: SQLCommand execution failure: java.io.IOException: prepare query failed[nQSError: 43113] Message returned from OBIS.
[nQSError: 27002] Near <table>: Syntax error [nQSError: 26012] .
Failed query text:
create table "Sample Targets Lite"...C_0DEFAULT
(DEPARTMENT_NAME VARCHAR(30)
);
Feb 04, 2019 11:39:09 PM oracle.odi.agent
WARNING: Agent Internal encountered an error: ODI-1226: Step Physical_STEP fails after 1 attempt(s).
Feb 04, 2019 11:39:09 PM oracle.odi.agent
SEVERE: Session ggfgfggf_Physical_SESS (59) fails with return code ODI-1298.
ODI-1217: Session ggfgfggf_Physical_SESS (59) fails with return code ODI-1298.
ODI-1226: Step Physical_STEP fails after 1 attempt(s).
ODI-1227: Task SERIAL-MAP_MAIN- fails on the source connection <Empty Value>.
ODI-1298: Serial task "SERIAL-MAP_MAIN- (10)" failed because child task "SERIAL-EU-LS_HR_CLASS_PROJECT_1_UNIT (20)" is in error.
ODI-1298: Serial task "SERIAL-EU-LS_HR_CLASS_PROJECT_1_UNIT (20)" failed because child task "Create work table-LKM SQL to SQL (JYTHON)- (40)" is in error.
ODI-1228: Task Create work table-LKM SQL to SQL (JYTHON)- fails on the target connection BI_connect.
Caused By: java.sql.SQLException: java.io.IOException: prepare query failed[nQSError: 43113] Message returned from OBIS.
[nQSError: 27002] Near <table>: Syntax error [nQSError: 26012] .
at oracle.bi.jdbc.AnaJdbcStatementImpl.executeQuery(AnaJdbcStatementImpl.java:387)
...
Feb 04, 2019 11:39:09 PM oracle.odi.agent
SEVERE: Agent encountered an error: ODI-1217: Session ggfgfggf_Physical_SESS (59) fails with return code ODI-1298. Caused by: ODI-1226: Step Physical_STEP fails after 1 attempt(s).
Feb 04, 2019 11:58:51 PM null
INFO: Disconnecting from Source.
Feb 04, 2019 11:58:51 PM null
INFO: Disconnecting from Repository
Feb 04, 2019 11:58:51 PM null
INFO: Reverse End.
I use all LKM but all throw same error or can't run.

aerospike on openvz 2core 4Gb ram doesn't start and doesn't give errors

after installation without any trouble I've started aerospike on a openvz vps with 2cores and 4gb ram.
this is the result:
root#outland:~# /etc/init.d/aerospike start
* Start aerospike: asd [OK]
then check for running asd:
root#outland:~# /etc/init.d/aerospike status
* Halt aerospike: asd [fail]
what is going wrong?
adding logs:
Mar 03 2015 15:17:57 GMT: INFO (config): (cfg.c::3033) system file descriptor limit: 100000, proto-fd-max: 15000
Mar 03 2015 15:17:57 GMT: WARNING (cf:misc): (id.c::249) Tried eth,bond,wlan and list of all available interfaces on device.Failed to retrieve physical address with errno 19 No such device
Mar 03 2015 15:17:57 GMT: CRITICAL (config): (cfg.c:3363) could not get unique id and/or ip address
Mar 03 2015 15:17:57 GMT: WARNING (as): (signal.c::120) SIGINT received, shutting down
Mar 03 2015 15:17:57 GMT: WARNING (as): (signal.c::123) startup was not complete, exiting immediately
This is your config problem
Mar 03 2015 15:17:57 GMT: WARNING (cf:misc): (id.c::249) Tried eth,bond,wlan and list of all available interfaces on device.Failed to retrieve physical address with errno 19 No such device
Mar 03 2015 15:17:57 GMT: CRITICAL (config): (cfg.c:3363) could not get unique id and/or ip address
Basically the vps has a non standard interface name.
The solution is to add your interface name as network-interface-name to the config.
http://www.aerospike.com/docs/operations/troubleshoot/startup/#problem-with-network-interface
Which OS are your using btw?

Resources