Suppose i have the following file.csv
DATE Name Email
26-Sep-19 Name1 Name1#email.com
26-Sep-19 Name2 Name2#email.com
26-Sep-19 Name3 Name3#email.com
I am trying to insert values from file.csv into a table
import cx_Oracle
import csv
import os
from datetime import datetime
con = cx_Oracle.connect(uname, pwd, hostname + ': ' + port + '/' + service)
cursor = con.cursor()
with open('file.csv', 'r') as csv_file:
csv_reader = csv.reader(csv_file, delimiter=',')
for lines in csv_reader:
cursor.execute( "INSERT INTO table1 ( DATE,Name,Email) values (:1, :2, :3)",
(lines[0],lines[1],lines[2])
cursor.close()
con.commit()
con.close()
I get this error:
(lines[0],lines[1],lines[2]) cx_Oracle.DatabaseError: ORA-01858: a
non-numeric character was found where a numeric was expected
After some debugging, i was able to nail it down to the issue being with the date, so instead of lines[0], i replaced with hard-coded date, and it worked!
cursor.execute( "INSERT INTO table1 ( DATE,Name,Email) values (:1, :2, :3)",
('26-Sep-19',lines[1],lines[2])
why is it not working with lines[0] variable but with a hardcoded value its working just fine?
The string format of your date must match what Oracle is expecting. You can set the NLS_DATE_FORMAT parameter for your session, or you can simply modify your code to do something like the following instead:
cursor.execute("""
insert into table1 (date, name, email)
values (to_date(:1, 'dd-Mon-YY'), :2, :3)""",
(lines[0], lines[1], lines[2]))
Now you've solved the date format issue, consider using executemany() for performance. Something like:
With file.csv containing:
26-Sep-19,Name1,Name1#email.com
26-Sep-19,Name2,Name2#email.com
26-Sep-19,Name3,Name3#email.com
and the table created like:
create table table1 ("DATE" date, name varchar2(20), email varchar2(20));
Then this file using the date conversion that #anthony mentioned works:
# Set cursor sizes to match the table definition or known max data sizes
# create table table1 ("DATE" date, name varchar2(20), email varchar2(20));
cursor.setinputsizes(None, 20, 20)
# Adjust the batch size to meet your memory and performance requirements
batchSize = 1000
with open('file.csv', 'r') as csv_file:
csv_reader = csv.reader(csv_file, delimiter=',')
i = 0
data = []
for line in csv_reader:
data.append((line[0],line[1],line[2]))
i = i + 1
if (i % batchSize == 0):
print('batch')
cursor.executemany("""insert into table1 ("DATE",name, email) values (to_date(:1, 'DD-Mon-YY'), :2, :3)""", data)
data = []
i = 0
if (i % batchSize != 0):
print('final')
cursor.executemany("""insert into table1 ("DATE",name, email) values (to_date(:1, 'DD-Mon-YY'), :2, :3)""", data)
con.commit()
To load some rows to a DB on the far side of the world took 4 seconds with a similar script (mostly connection time costs) vs 36 seconds with execute()
Related
I have a program that has been working for years. Today, we upgraded from SAS 9.4M3 to 9.4M7.
proc setinit
Current version: 9.04.01M7P080520
Since then, I am not able to get the same results as before the upgrade.
Please note that I am querying on Oracle databases directly.
Trying to replicate the issue with a minimal, reproducible SAS table example, I found that the issue disappear when querying on a SAS table instead of on Oracle databases.
Let's say I have the following dataset:
data have;
infile datalines delimiter="|";
input name :$8. id $1. value :$8. t1 :$10.;
datalines;
Joe|A|TLO
Joe|B|IKSK
Joe|C|Yes
;
Using the temporary table:
proc sql;
create table want as
select name,
min(case when id = "A" then value else "" end) as A length 8
from have
group by name;
quit;
Results:
name A
Joe TLO
However, when running the very same query on the oracle database directly I get a missing value instead:
proc sql;
create table want as
select name,
min(case when id = "A" then value else "" end) as A length 8
from have_oracle
group by name;
quit;
name A
Joe
As per documentation, the min() function is behaving properly when used on the SAS table
The MIN function returns a missing value (.) only if all arguments are missing.
I believe this happens when Oracle don't understand the function that SAS is passing it - the min functions in SAS and Oracle are very different and the equivalent in SAS would be LEAST().
So my guess is that the upgrade messed up how is translates the SAS min function to Oracle, but it remains a guess. Does anyone ran into this type of behavior?
EDIT: #Richard's comment
options sastrace=',,,d' sastraceloc=saslog nostsuffix;
proc sql;
create table want as
select t1.name,
min(case when id = 'A' then value else "" end) as A length 8
from oracle_db.names t1 inner join oracle_db.ids t2 on (t1.tid = t2.tid)
group by t1.name;
ORACLE_26: Prepared: on connection 0
SELECT * FROM NAMES
ORACLE_27: Prepared: on connection 1
SELECT UI.INDEX_NAME, UIC.COLUMN_NAME FROM USER_INDEXES UI,USER_IND_COLUMNS UIC WHERE UI.TABLE_NAME='NAMES' AND
UIC.TABLE_NAME='NAMES' AND UI.INDEX_NAME=UIC.INDEX_NAME
ORACLE_28: Executed: on connection 1
SELECT statement ORACLE_27
ORACLE_29: Prepared: on connection 0
SELECT * FROM IDS
ORACLE_30: Prepared: on connection 1
SELECT UI.INDEX_NAME, UIC.COLUMN_NAME FROM USER_INDEXES UI,USER_IND_COLUMNS UIC WHERE UI.TABLE_NAME='IDS' AND
UIC.TABLE_NAME='IDS' AND UI.INDEX_NAME=UIC.INDEX_NAME
ORACLE_31: Executed: on connection 1
SELECT statement ORACLE_30
ORACLE_32: Prepared: on connection 0
select t1."NAME", MIN(case when t2."ID" = 'A' then t1."VALUE" else ' ' end) as A from
NAMES t1 inner join IDS t2 on t1."TID" = t2."TID" group by t1."NAME"
ORACLE_33: Executed: on connection 0
SELECT statement ORACLE_32
ACCESS ENGINE: SQL statement was passed to the DBMS for fetching data.
NOTE: Table WORK.SELECTED_ATTR created, with 1 row and 2 columns.
! quit;
NOTE: PROCEDURE SQL used (Total process time):
real time 0.34 seconds
cpu time 0.09 seconds
Use the SASTRACE= system option to log SQL statements sent to the DBMS.
options SASTRACE=',,,d';
will provide the most detailed logging.
From the prepared statement you can see why you are getting a blank from the Oracle query.
select
t1."NAME"
, MIN ( case
when t2."ID" = 'A' then t1."VALUE"
else ' '
end
) as A
from
NAMES t1 inner join IDS t2 on t1."TID" = t2."TID"
group by
t1."NAME"
The SQL MIN () aggregate function will exclude null values from consideration.
In SAS SQL, a blank value is also interpreted as null.
In SAS your SQL query returns the min non-null value TLO
In Oracle transformed query, the SAS blank '' is transformed to ' ' a single blank character, which is not-null, and thus ' ' < 'TLO' and you get the blank result.
The actual MIN you want to force in Oracle is min(case when id = "A" then value else null end) which #Tom has shown is possible by omitting the else clause.
The only way to see the actual difference is to run the query with trace in the prior SAS version, or if lucky, see the explanation in the (ignored by many) "What's New" documents.
Why are you using ' ' or '' as the ELSE value? Perhaps Oracle is treating a string with blanks in it differently than a null string.
Why not use null in the ELSE clause?
or just leave off the ELSE clause and let it default to null?
libname mylib oracle .... ;
proc sql;
create table want as
select name
, min(case when id = "A" then value else null end) as A length 8
from mylib.have_oracle
group by name
;
quit;
Also try running the Oracle code yourself, instead of using implicit pass thru.
proc sql;
connect to oracle ..... ;
create table want as
select * from connection to oracle
(
select name,
min(case when id = "A" then value else null end) as A length 8
from have_oracle
group by name
)
;
quit;
When I try to reproduce this in Oracle I get the result you are looking for so I suspect it has something to do with SAS (which I'm not familiar with).
with t as (
select 'Joe' name, 'A' id, 'TLO' value from dual union all
select 'Joe' name, 'B' id, 'IKSK' value from dual union all
select 'Joe' name, 'C' id, 'Yes' value from dual
)
select name
, min(case when id = 'A' then value else '' end) as a
from t
group by name;
NAME A
---- ----
Joe TLO
Unrelated, if you are only interested in id = 'A' then a better query would be:
select name
, min(value) as a
from t
where id = 'A'
group by name;
I try to get part of JSON column in each result row this select
SELECT TRIM(a.symbol),
TRIM(a.ex_name),
to_char(a.date_rw, 'dd-MON-yyyy'),
a.pwr,
a.last,
JSON_QUERY(b.mval, '$."-9"') as value
FROM adviser_log a
INNER JOIN profit_model_d b
ON a.date_rw = b.date_rw
WHERE a.date_rw = '08-OCT-2021'
select result:
VERY NAS 08-OCT-2021 -9 8.9443 {"sl":-3.6,"tp":5,"avg":1.368,"max":5,"min":-3.6,"count":1}
As a json path I put "-9" literal but I wanna put as path a.pwr is it possible
I tried put CONCAT('$.', a.pwr) without result
Is it any way to create dynamical json path into JSON_QUERy
I want to match part json which key compared with a.pwr to each row in select
Thx
You can use a function to dynamically get the JSON value:
WITH FUNCTION get_value(
value IN CLOB,
path IN VARCHAR2
) RETURN VARCHAR2
IS
BEGIN
RETURN JSON_OBJECT_T( value ).get_object( path ).to_string();
END;
SELECT TRIM(a.symbol) AS symbol,
TRIM(a.ex_name) AS ex_name,
to_char(a.date_rw, 'dd-MON-yyyy') AS date_rw,
a.pwr,
a.last,
get_value(b.mval, a.pwr) AS value
FROM adviser_log a
INNER JOIN profit_model_d b
ON a.date_rw = b.date_rw
WHERE a.date_rw = DATE '2021-10-08'
Which, for your sample data:
CREATE TABLE adviser_log (symbol, ex_name, date_rw, pwr, last) AS
SELECT 'VERY', 'NAS', DATE '2021-10-08', -9, 8.9443 FROM DUAL;
CREATE TABLE profit_model_d (date_rw DATE, mval CLOB CHECK (mval IS JSON));
INSERT INTO profit_model_d (
date_rw,
mval
) VALUES (
DATE '2021-10-08',
'{"-9":{"sl":-3.6,"tp":5,"avg":1.368,"max":5,"min":-3.6,"count":1}}'
);
Outputs:
SYMBOL
EX_NAME
DATE_RW
PWR
LAST
VALUE
VERY
NAS
08-OCT-2021
-9
8.9443
{"sl":-3.6,"tp":5,"avg":1.368,"max":5,"min":-3.6,"count":1}
db<>fiddle here
I have table
TableA
ID PRICE DATE
123 200 01-SEP-2015
456 500 01-AUG-2015
In my query i want to show date when (date + X months)< sysdate else show null.
I have issue using variable in that query.
Forexample in record 1
(01-SEP-2015 + 3 months) = 01-DEC-2015 is not less than sysdate so, I don't want to show that date.
but on the other hand for second record
(01-AUG-2015 + 3 months) = 01-NOV-2015 is less than sysdate so, I want to show that date.
declare
x number(5);
set x:=3;
select
a.ID,
a.PRICE,
case when( ADD_MONTHS(a.DATE, x) < sysdate() )
then a.DATE
else
NULL
end as NEW_DATE
from TableA a;
You need to put an ampersand (&) before the variable name to tell SQL*Plus to substitute in the value of the variable:
declare
x number(5);
set x:=3;
select a.ID,
a.PRICE,
case
when ADD_MONTHS(a.DATE, &x) < sysdate then a.DATE
else NULL
end as NEW_DATE
from TableA a;
Also, DATE is a data type in Oracle - don't use it as a column name. You can do that but it's going to cause problems down the line somewhere.
Best of luck.
I am recieving information from a csv file from one department to compare with the same inforation in a different department to check for discrepencies (About 3/4 of a million rows of data with 44 columns in each row). After I have the data in a table, I have a program that will take the data and send reports based on a HQ. I feel like the way I am going about this is not the most efficient. I am using oracle for this comparison.
Here is what I have:
I have a vb.net program that parses the data and inserts it into an extract table
I run a procedure to do a full outer join on the two tables into a new table with the fields in one department prefixed with '_c'
I run another procedure to compare the old/new data and update 2 different tables with detail and summary information. Here is code from inside the procedure:
DECLARE
CURSOR Cur_Comp IS SELECT * FROM T.AEC_CIS_COMP;
BEGIN
FOR compRow in Cur_Comp LOOP
--If service pipe exists in CIS but not in FM and the service pipe has status of retired in CIS, ignore the variance
If(compRow.pipe_num = '' AND cis_status_c = 'R')
continue
END IF
--If there is not a summary record for this HQ in the table for this run, create one
INSERT INTO t.AEC_CIS_SUM (HQ, RUN_DATE)
SELECT compRow.HQ, to_date(sysdate, 'DD/MM/YYYY') from dual WHERE NOT EXISTS
(SELECT null FROM t.AEC_CIS_SUM WHERE HQ = compRow.HQ AND RUN_DATE = to_date(sysdate, 'DD/MM/YYYY'))
-- Check fields and update the tables accordingly
If (compRow.cis_loop <> compRow.cis_loop_c) Then
--Insert information into the details table
INSERT INTO T.AEC_CIS_DET( Fac_id, Pipe_Num, Hq, Address, AutoUpdatedFl,
DateTime, Changed_Field, CIS_Value, FM_Value)
VALUES(compRow.Fac_ID, compRow.Pipe_Num, compRow.Hq, compRow.Street_Num || ' ' || compRow.Street_Name,
'Y', sysdate, 'Cis_Loop', compRow.cis_loop, compRow.cis_loop_c);
-- Update information into the summary table
UPDATE AEC_CIS_SUM
SET cis_loop = cis_loop + 1
WHERE Hq = compRow.Hq
AND Run_Date = to_date(sysdate, 'DD/MM/YYYY')
End If;
END LOOP;
END;
Any suggestions of an easier way of doing this rather than an if statement for all 44 columns of the table? (This is run once a week if it matters)
Update: Just to clarify, there are 88 columns of data (44 of duplicates to compare with one suffixed with _c). One table lists each field in a row that is different so one row can mean 30+ records written in that table. The other table keeps tally of the number of discrepencies for each week.
First of all I believe that your task can be implemented (and should be actually) with staight SQL. No fancy cursors, no loops, just selects, inserts and updates. I would start with unpivotting your source data (it is not clear if you have primary key to join two sets, I guess you do):
Col0_PK Col1 Col2 Col3 Col4
----------------------------------------
Row1_val A B C D
Row2_val E F G H
Above is your source data. Using UNPIVOT clause we convert it to:
Col0_PK Col_Name Col_Value
------------------------------
Row1_val Col1 A
Row1_val Col2 B
Row1_val Col3 C
Row1_val Col4 D
Row2_val Col1 E
Row2_val Col2 F
Row2_val Col3 G
Row2_val Col4 H
I think you get the idea. Say we have table1 with one set of data and the same structured table2 with the second set of data. It is good idea to use index-organized tables.
Next step is comparing rows to each other and storing difference details. Something like:
insert into diff_details(some_service_info_columns_here)
select some_service_info_columns_here_along_with_data_difference
from table1 t1 inner join table2 t2
on t1.Col0_PK = t2.Col0_PK
and t1.Col_name = t2.Col_name
and nvl(t1.Col_value, 'Dummy1') <> nvl(t2.Col_value, 'Dummy2');
And on the last step we update difference summary table:
insert into diff_summary(summary_columns_here)
select diff_row_id, count(*) as diff_count
from diff_details
group by diff_row_id;
It's just rough draft to show my approach, I'm sure there is much more details should be taken into account. To summarize I suggest two things:
UNPIVOT data
Use SQL statements instead of cursors
You have several issues in your code:
If(compRow.pipe_num = '' AND cis_status_c = 'R')
continue
END IF
"cis_status_c" is not declared. Is it a variable or a column in AEC_CIS_COMP?
In case it is a column, just put the condition into the cursor, i.e. SELECT * FROM T.AEC_CIS_COMP WHERE not (compRow.pipe_num = '' AND cis_status_c = 'R')
to_date(sysdate, 'DD/MM/YYYY')
That's nonsense, you convert a date into a date, simply use TRUNC(SYSDATE)
Anyway, I think you can use three single statements instead of a cursor:
INSERT INTO t.AEC_CIS_SUM (HQ, RUN_DATE)
SELECT comp.HQ, trunc(sysdate)
from AEC_CIS_COMP comp
WHERE NOT EXISTS
(SELECT null FROM t.AEC_CIS_SUM WHERE HQ = comp.HQ AND RUN_DATE = trunc(sysdate));
INSERT INTO T.AEC_CIS_DET( Fac_id, Pipe_Num, Hq, Address, AutoUpdatedFl, DateTime, Changed_Field, CIS_Value, FM_Value)
select comp.Fac_ID, comp.Pipe_Num, comp.Hq, comp.Street_Num || ' ' || comp.Street_Name, 'Y', sysdate, 'Cis_Loop', comp.cis_loop, comp.cis_loop_c
from T.AEC_CIS_COMP comp
where comp.cis_loop <> comp.cis_loop_c;
UPDATE AEC_CIS_SUM
SET cis_loop = cis_loop + 1
WHERE Hq IN (Select Hq from T.AEC_CIS_COMP)
AND trunc(Run_Date) = trunc(sysdate);
They are not tested but they should give you a hint how to do it.
I have two rows that have a varchar column that are different according to a Java .equals(). I can't easily change or debug the Java code that's running against this particular database but I do have access to do queries directly against the database using SQLDeveloper. The fields look the same to me (they are street addresses with two lines separated by some new line or carriage feed/new line combo).
Is there a way to see all of the hidden characters as the result of a query?I'd like to avoid having to use the ascii() function with substr() on each of the rows to figure out which hidden character is different.
I'd also accept some query that shows me which character is the first difference between the two fields.
Try
select dump(column_name) from table
More information is in the documentation.
As for finding the position where the character differs, this might give you an idea:
create table tq84_compare (
id number,
col varchar2(20)
);
insert into tq84_compare values (1, 'hello world');
insert into tq84_compare values (2, 'hello' || chr(9) || 'world');
with c as (
select
(select col from tq84_compare where id = 1) col1,
(select col from tq84_compare where id = 2) col2
from
dual
),
l as (
select
level l from dual
start with 1=1
connect by level < (select length(c.col1) from c)
)
select
max(l.l) + 1position
from c,l
where substr(c.col1,1,l.l) = substr(c.col2,1,l.l);
SELECT DUMP('€ÁÑ', 1016)
FROM DUAL
... will print something like:
Typ=96 Len=3 CharacterSet=WE8MSWIN1252: 80,c1,d1