Passing shell variable in hive script - shell

I am running hive script using shell job, with passing some parameters.
CREATE EXTERNAL TABLE IF NOT EXISTS db_${hivevar:myEnv}_travail.tble91_formation_eligible(
ID_FORM_ELIG BIGINT,
LIBELLE_COURT_FORM VARCHAR(50),
DEBUT_VALID_FORM TIMESTAMP,
FIN_VALID_FORM TIMESTAMP,
CODE_BRANCHE_FORM VARCHAR(4),
CODE_REGION_FORM VARCHAR(3),
CODE_PUBLIC_FORM VARCHAR(1),
ETAT_FORM VARCHAR(3),
DATE_CREAT_FORM TIMESTAMP,
ORIGINE_CREAT_FORM VARCHAR(20),
DATE_MAJ_FORM TIMESTAMP,
ORIGINE_MAJ_FORM VARCHAR(20),
ID_LISTE_ELIG BIGINT,
LIBELLE_FORMATION VARCHAR(255),
CODE_DIPLOME INT,
NUMERO_FORMATION VARCHAR(50),
CODE_REFERENTIEL VARCHAR(50) COMMENT 'Code associé au type de référentiel (ex: RNCP, RS, ELU, CPF, ...)',
CODE_RECONNAISSANCE VARCHAR(50)COMMENT 'Code de la reconnaissance (ex: RNCP1234, ELU134444, ...)',
NIVEAU_CONTROLE_HABILITATION VARCHAR(50) COMMENT 'Niveau de contrôle d\'habilitation d\'une reconnaissance éligible CPF (informatif, bloquant, absent)'
)
PARTITIONED BY TO_DATE('${hivevar:date_archive}')
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\001'
LINES TERMINATED BY '\n'
STORED AS TEXTFILE
LOCATION '/${hivevar:myEnv}/ep/traitement/le5/app_db/db_${hivevar:myEnv}_le5_travail.db/tble91_formation_eligible/';
The problem is about passing the date_archive as a partition column, it'is sourced from shell script as String, but I wanna convert it as a date column, so that I used TO_DATE function.
For information, date_archive has many date values like : 2022-02-03
But, this make a problem
Error: Error while compiling statement: FAILED: ParseException line
22:15 extraneous input 'TO_DATE' expecting ( near '(' in create table
partition specification (state=42000,code=40000)
I tried :
BY (TO_DATE('${hivevar:date_archive}'))
BY (TO_DATE("${hivevar:date_archive}"))
BY (date_archive=TO_DATE("${hivevar:date_archive}"))
But always, I have errors, Any help, please ? Thank you

Related

Not able to set default value in sqlloader

I am trying to load default value using sqlloader from a unix box, while my file only contains one column.
Below is my table structure:
CREATE TABLE DATA_LOAD(
SEQUENCE_NUMBER NUMBER(15)
CONSTRAINT SEQUENCE_NUMBER NOT NULL,
Account NUMBER(12)
CONSTRAINT Account NOT NULL,
SYS_CREATION_DATE DATE NOT NULL,
SYS_UPDATE_DATE DATE ,
OPERATOR_ID NUMBER(9) ,
APP_ID CHAR(6),
SERVICE_CODE CHAR(5) NOT NULL,
UPDATE_STAMP NUMBER(4),
ACCOUNT_TYPE CHAR(1),
ACCOUNT_SUB_TYPE CHAR(1),
STATUS VARCHAR2(2),
OPER_IND CHAR(1),
ERROR_REASON VARCHAR2(240)
)
File will contain only:
123476810
and I am trying to load with
LOAD DATA
INFILE '$INPUT_FILE'
INTO TABLE DATA_LOAD
FIELDS TERMINATED BY '$'
TRAILING NULLCOLS
(
Account,
SEQUENCE_NUMBER default 2,
STATUS default R,
OPER_IND default N,
SERVICE_CODE default abcde,
SYS_CREATION_DATE default to_date(sysdate,'DD-MON-YY')
)
getting this error:
SQL*Loader-350: Syntax error at line 9.
Expecting "," or ")", found "default".
SEQUENCE_NUMBER default 2,
Your syntax seems to be wrong. You need to be using CONSTANT
OPER_IND CONSTANT "N"

error while trying to select from external table

I am getting the below error while trying to select * from ext_poc
ORA-29913: error in executing ODCIEXTTABLEOPEN callout
ORA-29400: data cartridge error
KUP-00554: error encountered while parsing access parameters
KUP-01005: syntax error: found "identifier": expecting one of: "binary_double, binary_float, comma, char, date, defaultif, decimal, double, float, integer, (, no, nullif, oracle_date, oracle_number, position, raw, recnum, ), unsigned, varrawc, varchar, varraw, varcharc, zoned"
KUP-01008: the bad identifier was: varchar2
KUP-01007: at line 4 column 10
29913. 00000 - "error in executing %s callout"
*Cause: The execution of the specified callout caused an error.
*Action: Examine the error messages take appropriate action.
Below is the ddl for the table:
CREATE TABLE "JDASTG"."EXT_POC"
( "ID" varchar2(100),
"NAME" varchar2(100),
"DOB" varchar2(100)
)
ORGANIZATION EXTERNAL
( TYPE ORACLE_LOADER
DEFAULT DIRECTORY "SCPO_EXT_DATA"
ACCESS PARAMETERS
( RECORDS DELIMITED BY NEWLINE
FIELDS TERMINATED BY ','
MISSING FIELD VALUES ARE NULL
(id varchar2(100),
name varchar2(100),
dob varchar2(100)
)
)
LOCATION
( 'xyz_aldrin.csv'
)
);
PS:
This error however doesn't occur if the varchar2(100) is changed to char(100):
MISSING FIELD VALUES ARE NULL
(id char(100),
name char(100),
dob char(100)
)
It is important to distinguish between oracle internal data types specified on the table, and the external data types specified in the access parameters for the external table (control file for sqlldr). Read the manual again with that in mind. It should become clear. CHAR is allowed in SQL loader whereas VARCHAR2 is not

Identifier is too long while loading from SQL*Loader

I have a table structure like this
CREATE TABLE acn_scr_upload_header
(
FILE_RECORD_DESCRIPTOR varchar2(5) NOT NULL,
schedule_no Number(10) NOT NULL,
upld_time_stamp Date NOT NULL,
seq_no number NOT NULL,
filename varchar2(100) ,
schedule_date_time Date
);
When I try to load my file via SQL*Loader I'm getting an error on this value in the column filename: Stock_Count_Request_01122014010101.csv. The error is:
Error on table ACN_SCR_UPLOAD_HEADER, column FILENAME.
ORA-00972: identifier is too long".
If I try to insert the same value into the table using an INSERT statement it works fine.
My data file Stock_Count_Request_01122014010101.csv looks like
FHEAD,1,12345,20141103
FDETL,7,100,W,20141231,SC100,B,N,1,5
FTAIL,8,6
and control file
LOAD DATA
INFILE '$IN_DIR/$FILENAME'
APPEND
INTO TABLE ACN_SCR_UPLOAD_HEADER
WHEN FILE_RECORD_DESCRIPTOR = 'FHEAD'
FIELDS TERMINATED BY ","
TRAILING NULLCOLS
(
FILE_RECORD_DESCRIPTOR position(1),
LINE_NO FILLER,
schedule_no ,
schedule_date_time,
upld_time_stamp sysdate,
seq_no "TJX_STOCK_COUNT_REQ_UPLD_SEQ.NEXTVAL",
FILENAME constant ""
)

Why did SQL*Loader load 808594481 when using the INTEGER data-type?

I was loading data using SQL*Loader and when making the control file I used the table definition and accidentally left the INTEGER data type on the "version" line.
And in the "version" field (data type integer) it inserted the value 808594481.
I'm having a hard time understanding how it processed this value -- I'm assuming it took it as a literal ... but is that the sum of the ASCII representations of each letter?
NOPE!
SELECT ASCII('I')+ascii('N')+ASCII('T')+ASCII('E')+ASCII('G')+ASCII('E')+ASCII('G')+ASCII('E')+ASCII('R')
FROM SYS.DUAL
returns 666 (which, btw is hilarious).
concatenate ascii values?
SELECT ASCII('I')||ascii('N')||ASCII('T')||ASCII('E')||ASCII('G')||ASCII('E')||ASCII('G')||ASCII('E')||ASCII('R')
FROM SYS.DUAL
returns 737884697169716982
I'm hoping someone out there knows the answer.
This is the actual control file:
OPTIONS (SKIP=1)
LOAD DATA
APPEND into table THETABLE
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
(id ,
parent_id ,
record_id ,
version INTEGER,
created_at ,
updated_at ,
created_by ,
updated_by ,
species_and_cohort ,
species_and_cohort_count)
Table DDL:
create table THETABLE
(
id VARCHAR2(36),
parent_id VARCHAR2(36),
record_id VARCHAR2(36),
version INTEGER,
created_at VARCHAR2(25),
updated_at VARCHAR2(25),
created_by VARCHAR2(50),
updated_by VARCHAR2(50),
species_and_cohort VARCHAR2(150),
species_and_cohort_other VARCHAR2(150),
species_and_cohort_count NUMBER
)
Data:
id,parent_id,record_id,version,created_at,updated_at,created_by,updated_by,species_and_cohort,species_and_cohort_other,species_and_cohort_count
60D90F54-C5F2-47AF-951B-27A424EAE8E3,f9fe8a3b-3470-4caf-b0ba-3682a1c79731,f9fe8a3b-3470-4caf-b0ba-3682a1c79731,1,2014-09-23 21:02:54 UTC,2014-09-23 21:02:54 UTC,x#gmail.com,x#gmail.com,"PRCA Cherrylaurel,Sapling","",5
FC6A2120-AA0B-4238-A2F6-A6AEDD9B8202,f9fe8a3b-3470-4caf-b0ba-3682a1c79731,f9fe8a3b-3470-4caf-b0ba-3682a1c79731,1,2014-09-23 21:03:02 UTC,2014-09-23 21:03:02 UTC,x7#gmail.com,x7#gmail.com,"JUVI Eastern Redcedar,Sapling","",45
If you split 808594481 into bytes as it would be encoded in a 32 bit twos complement encoding, and treat each byte as an ascii-encoded character, you get "02,1" or "1,20" depending on byte order. You probably inserted a string that starts or ends with one of those, and some layer between your code and the database silently converted it to an integer.

running sqlldr control files in UNIX

I have a question. I have this control files that works fine when I run it from a windows client. However when I run it directly in Linux, it shows load complete but when I look at my oracle data, there is NO DATA and there are even bad records. Below is my control file that works well in windows but fails in Linux.
NOTE: The control file works if I remove the string or date converted fields
Control file
load data
infile 'HOME/INPUT/FILEA.dat'
badfile 'HOME/BAD/FILEA.bad'
discardfile 'HOME/DIS/FILEA.dsc'
truncate
into table TEST
fields terminated by '|'
trailing nullcols
( ABCcode CHAR(11),
ABCID CHAR(6),
ABC_SEQNO "to_number(:ABC_SEQNO,'999999')",
PSNO "to_number(:PSNO,'99999999999.999')",
ABDF CHAR(1),
ABCFI CHAR(1),
ABC_DATE NULLIF ABC_DATE="00000000" "to_date(:ABC_DATE, 'YYYYMMDD')",
XZY_date NULLIF XZY_date="00000000" "to_date(:XZY_date, 'YYYYMMDD')",
DESC CHAR(1))
Any help or ideas to get this code to run in Linux will be appreciated
Notes about the logfile: The logfile had the following
ORA-00604: error occurred at recursive SQL level 1
ORA-12899: value too large for column "ABCschema"."TEST"."ABC_DATE" (actual: 9, maximum: 8)
Also, the date conversion had the following
NULL if ABC_DATE = 0X3030303030303030(character '00000000')
SQL string for column : "to_date(:ABC_DATE, 'YYYYMMDD')"
Your TEST table has the ABC_DATE column defined as VARCHAR2(8), not as a DATE.
If I create a table as:
create table test (
ABCcode VARCHAR2(11),
ABCID VARCHAR2(6),
ABC_SEQNO NUMBER,
PSNO NUMBER,
ABDF VARCHAR2(1),
ABCFI VARCHAR2(1),
ABC_DATE DATE,
XZY_date DATE,
"DESC" VARCHAR2(1)
);
and have a data file with:
A|B|1|2.3|C|D|20140217|20140218|E
then it loads fine. If I recreate the table as:
create table test (
ABCcode VARCHAR2(11),
ABCID VARCHAR2(6),
ABC_SEQNO NUMBER,
PSNO NUMBER,
ABDF VARCHAR2(1),
ABCFI VARCHAR2(1),
ABC_DATE VARCHAR2(8),
XZY_date DATE,
"DESC" VARCHAR2(1)
);
... then the same control file and data file now give me:
Record 1: Rejected - Error on table TEST, column ABC_DATE.
ORA-12899: value too large for column "<schema>"."TEST"."ABC_DATE" (actual: 9, maximum: 8)
You are converting the string value to a date, but then you're doing an implicit conversion back to a string when it actually inserts the data into the VARCHAR2 column. When it does that it's using your NLS_DATE_FORMAT settings, and the error I got was from having that set to DD-MON-RR.
You have three options really. Either modify your table to have actual DATE columns; or change the control file so it just inserts the plain text value and doesn't do the date conversion at all; or massage your environment so the conversion back to a string gets the format you want the string to be.
Only the first one is really sensible - if it's a date value, always store it as a DATE, never as a string.
The 0X30... thing isn't a problem, that's just showing the internal representation it's using.

Resources