I am using Oracle 19c.
I have a table,which I need to change the data-type of one of its columns:
from number to number (24,8)
The column contains data, nearly 300.000 records and I am required to keep the data.
If I do this without truncating/deleting data:
Does it harm data integrity?
Does it effect the datatype of existing data?
The reason for this operation is that the column should have had 7 or 8 decimals but it has been limited to 4 decimal somehow, even though data type is number. Either my etl tool (informatica ) or oracle db has limited, I do not know.
thanks in advance
Your problem doesn't appear to be with Oracle.
CREATE TABLE T1 (
num NUMBER
);
INSERT INTO T1 (num) VALUES (123.12345678);
SELECT * FROM T1;
NUM
123.12345678
Related
I need to insert values with a precision of 5 decimal places into an Oracle interface table via OPENQUERY because the values are originally stored in an SQL database. The data type of the Oracle table column is NUMBER (with no scale/precision specified). Using OPENQUERY to insert a value of 1.4, results in a value of 1.3999999999999999 stored in the Oracle table. I cannot change the data type of the Oracle table to NUMBER(38,5) because it is a standard Oracle table (GL_DAILY_RATES_INTERFACE).
According to Oracle https://docs.oracle.com/cd/B28359_01/server.111/b28318/datatype.htm#CNCPT1832
"If a precision is not specified, the column stores values as given."
Which means that if I insert 1.4, it should be stored in a NUMBER column as is. But it doesn't. So does that mean that when inserting through OPENQUERY to a linked Oracle server, the Oracle Provider for OLE DB does some addition conversion that results in a floating point error?
How do I insert values to a precision of 5 decimal places into an Oracle table NUMBER column that does not have precision or scale specified?
Update:
My insert statement does round the values when inserting. But it doesn't solve the issue.
For example:
INSERT INTO OPENQUERY(LINKEDSERVER, "SELECT CONVERSION_RATE FROM GL_DAILY_RATES_INTERFACE") VALUES(ROUND(1.4,5))
Since inserting values through OPENQUERY to a linked Oracle server causes some floating point error, I tried using EXEC('') AT LINKEDSERVER and it worked. Because the statement is executed directly on the ORACLE server, there is no longer any issue of the Oracle Provider for OLE DB doing any unexpected conversion.
My overall solution was to first insert values from the SQL table to the Oracle table using OPENQUERY, then use EXEC() to update and round the values in the Oracle table again.
I have a table that stores the changes to a transaction. All the changes are captured into a table. One of the column that comes as part of the transaction can have many comma separated values. The number of occurrences cannot be predicted. Also this field is not mandatory and can have null values as well.
The total number of transactions that i have in the table is around 100M. Out of those the number of records for which the value is populated is 1M. Out of the 1M transactions the number of records for which the length of the record exceeds 4000 is ~37K.
I mentioned the length as 4000 since in my oracle table the column which would save this has been defined as varchar2(4000).
I check at places and found that if I have to save something of unknown length then i should define the table column datatype as clob. But clob is expensive for me since only a very small amount of data has length > 4000. If I snowflake my star schema and create another table to store the values then even though, I have transactions for which the length is much smaller than 4000 would be saved as part of the clob column. This would be expensive both in terms of storage and performance.
Can someone suggest me an approach to solve this problem.
Thanks
S
You could create a master - detail table to store the comma separated values, then you could have rows rather than save all comma separated values in a single column. This could be managed with a foregn key using a pseudo key between master and detail table.
Here's one option.
Create two columns, e.g.
create table storage
(id number primary key,
long_text_1 varchar2(4000),
long_text_2 varchar2(4000)
);
Store values like
insert into storage (id, long_text_1, long_text_2)
values (seq.nextval,
substr(input_value, 1, 4000),
substr(input_value, 4001, 4000)
);
When retrieving them from the table, concatenate them:
select id,
long_text_1 || long_text_2 as long_text
from storage
where ...
You might benefit from using inline SecurFile CLOBs. With inline CLOBs, up to about 4000 bytes of data can be stored in rows like a regular VARCHAR2 and only the larger values will be stored in a separate CLOB segment. With SecureFiles, Oracle can significantly improve CLOB performance. (For example, import and export of SecureFiles is much faster than the old-fashioned BasicFile LOB format.)
Depending on your version, parameters, and table DDL, your database may already store CLOBs as inline SecureFiles. Ensure that your COMPATIBLE setting is 11.2 or higher, and that DB_SECUREFILE is one of "permitted", "always", or "preferred":
select name, value from v$parameter where name in ('compatible', 'db_securefile') order by 1;
Use a query like this to ensure that your tables were setup correctly, and nobody overrode the system settings:
select dbms_metadata.get_ddl('TABLE', 'YOUR_TABLE_NAME') from dual;
You should see something like this in the results:
... LOB ("CLOB_NAME") STORE AS SECUREFILE (... ENABLE STORAGE IN ROW ...) ...
One of the main problems with CLOBs is that they are stored in a separate segment, and a LOB index must be traversed to map each row in the table to a value in another segment. The below demo creates two tables to show that LOB segments do not need to be used when the the data is small and stored inline.
--drop table clob_test_inline;
--drop table clob_test_not_in;
create table clob_test_inline(a number, b clob) lob(b) store as securefile (enable storage in row);
create table clob_test_not_in(a number, b clob) lob(b) store as (disable storage in row);
insert into clob_test_inline select level, lpad('A', 900, 'A') from dual connect by level <= 10000;
insert into clob_test_not_in select level, lpad('A', 900, 'A') from dual connect by level <= 10000;
commit;
The inline table segment is large, because it holds all the data. The out of line table segment is small, because all of its data is held elsewhere.
select segment_name, bytes/1024/1024 mb_inline
from dba_segments
where segment_name like 'CLOB_TEST%'
order by 1;
SEGMENT_NAME MB_INLINE
---------------- ---------
CLOB_TEST_INLINE 27
CLOB_TEST_NOT_IN 0.625
Looking at the LOB segments, the sizes are reversed. The inline table doesn't store anything in the LOB segment.
select table_name, bytes/1024/1024 mb_out_of_line
from dba_segments
join dba_lobs
on dba_segments.owner = dba_lobs.owner
and dba_segments.segment_name = dba_lobs.segment_name
where dba_lobs.table_name like 'CLOB_TEST%'
order by 1;
TABLE_NAME MB_OUT_OF_LINE
------------ --------------
CLOB_TEST_INLINE 0.125
CLOB_TEST_NOT_IN 88.1875
Despite the above, I can't promise that CLOBs will still work for you. All I can say is that it's worth testing the data using CLOBs. You'll still need to look out for a few things. CLOBs store text slightly differently (UCS2 instead of UTF8), which may take up more space depending on your character sets. So check the segment sizes. But also beware that segment sizes can lie when they are small - there's a lot of auto-allocated overhead for sample data, so you'll want to use realistic sizes when testing.
Finally, as Raul pointed out, storing non-atomic values in a field is usually a terrible mistake. That said, there are rare times when data warehouses need to break the rules for performance, and data needs to be stored as compactly as possible. Before you store the data this way, ensure that you will never need to join based on those values, or query for individual values. If you think dealing with 100M rows is tough, just wait until you try to split 100M values and then join them to another table.
I am new to Oracle sql, but I have some experience with MSSQL. I was sent a script to create some tables, but because of the BLOB columns, I am getting a couple errors when I try to limit the size.
I tried asking my co-workers about this, but they aren't sure either. This was basically grabbed from somewhere else, so they're not sure how to fix this.
Essentially, the table looks something like this (table and column names were changed):
CREATE TABLE table1
(
ID CHAR(32),
NAME CHAR(50),
KEY $(BLOB)(64),
BUFFER $(BLOB)(20),
SORTNO NUMERIC(8) CONSTRAINT UK_WIU UNIQUE,
CONSTRAINT PK_ID PRIMARY KEY (ID)
)
;
When running this, I get the error "invalid character" because of the dollar sign ($). But if I change data type to BLOB(64), I get an error saying "missing right parenthesis." If I just do "BLOB," it runs fine. Is there any way to define the length for BLOB?
Thank you
BLOB column's size can't be restricted. It allows you to store up to [(4 gigabytes - 1) * (database block size)] data, and that's it. If you need less than that, fine, no problem.
More about BLOB (and other Oracle 18c datatypes) here: https://docs.oracle.com/en/database/oracle/oracle-database/18/sqlrf/Data-Types.html#GUID-1A71C635-188E-4EC9-B821-1DBEC2B45451
Table you meant to create would in Oracle look like this:
SQL> create table table1
2 (id varchar2(32) constraint pk_id primary key,
3 name varchar2(50),
4 sortno number(8) constraint uk_wiu unique,
5 key blob,
6 buffer blob
7 );
Table created.
SQL>
Yet another objection regarding your code: don't use CHAR but VARCHAR2 (unless you have to); CHAR will right-pad values with spaces, up to the maximum length of the column.
No, columns are just defined as a CLOB or BLOB.
There is no size associated with the column. You can put in something as big or small as you like, with a few restrictions.
"Maximum size: (4 GB - 1) * DB_BLOCK_SIZE initialization parameter (8 TB to 128 TB)"..as of 11g of Oracle.
To limit the size, you could, do so at the application layer - prevent folks from uploading files larger than X, although someone with DB access could do this.
You could write a stored procedure for handling INSERT/UPDATES, and have logic in there to prevent DML if the size is 'too large.'
You might be able to do a check constraint using a function call to get the length of the lob...an example here
A database in oracle with column datatype as NUMBER on retrieval using pyspark gets converted to float.
eg. ID column - datatype NUMBER has value 111
On retrieval the value is shown as 111.000000
The column names are dynamic and i dont want to CAST it by hard coding. Can the data in oracle be retrieved the way it appears in the database.
I don't want the zeroes to be appended.
select CAST(111.00000 AS NUMBER(38)) from dual
Is there any major performance issues if I use virtual columns in an Oracle table?
We have a scenario where the db has fields stored as strings. Since other production apps run off those fields we can't easily convert them.
I am tasked with generating reports from the same db. Since I need to be able to filter by dates (which are stored as strings) it was brought to my attention that we could create a virtual date field so that I can query against that.
Has anyone ran into any roadblocks with this approach?
A virtual column is defined using an expression that is evaluated when you select from the table. There is no performance hit on inserts/updates on the table.
For example:
create table t1 (
datestr varchar2(100),
datedt date generated always as (to_date(datestr,'YYYYMMDD'))
);
Table created.
SQL> insert into t1 (datestr) values ('20160815');
1 row created.
SQL> insert into t1 (datestr) values ('xxx');
1 row created.
SQL> commit;
Commit complete.
Note that I was able to insert an invalid date value into datestr. Now we can try to select the data:
SQL> select * from t1 where datedt = date '2016-08-15';
ERROR:
ORA-01841: (full) year must be between -4713 and +9999, and not be 0
This could be a problem for you if you can't guarantee all the strings hold valid dates.
As for performance, when you run the above query what you are really running is:
select * from t1 where to_date(datestr,'YYYYMMDD') = date '2016-08-15';
So the query will not be able to use an index on the datestr column (probably), and you may want to add an index on the virtual column. Again, this won't work if any of the strings don't contain valid dates.
Another consideration is potential impact on existing code. Hopefully you won't have any code like insert into t1 values (...); i.e. not specifying the column list. If you do you will get the error:
ORA-54013: INSERT operation disallowed on virtual columns