Is it possible to add a metadata field at column-level (in the Oracle Data Dictionary)?
The purpose would be to hold a flag identifying where individual data items in a table have been anonymised.
I'm an analyst (not a DBA) and I'm using Oracle SQL Developer which surfaces (and enables querying of) the COLUMN_NAME, DATA_TYPE, NULLABLE, DATA_DEFAULT, COLUMN_ID, and COMMENTS metadata fields of our Oracle DB (see pic).
I'd be looking to add another metadata field at this level (essentially, to add a second 'COMMENTS' field) to hold the 'Anonymisation' flag, to support easy querying of our flagged-anonymised data.
If it's possible (and advisable / supportable), I'd be grateful for any advice for describing the steps required to enable this, which I can then discuss with our Developer and DBA.
Short answer: NO.
But where could you keep that information?
In your data model.
Oracle provides a free data modeling solution, Oracle SQL Developer Data Modeler. It provides the ability to mark table/view columns as sensitive or PII.
Those same models can be stored back in your database so they can be accessed via SQL.
Once you've marked up all of your sensitive attributes/columns, and store it back into the database, you can query it back out.
Disclaimer: I work for Oracle, I'm the product manager for Data Modeler.
[TL;DR] Don't do it. Find another way.
If it's advisable
NO
Never modify the data dictionary; (unless Oracle support tells you to) you are likely to invalidate your support contract with Oracle and may break the database and make it unusable.
If it's possible
Don't do this.
If you really want to try it then still don't.
If you really, really want to try it then find a database you don't care about (the don't care about bit is important!) and log on as a SYSDBA user and:
ALTER TABLE whichever_data_dictionary_table ADD anonymisation_flag VARCHAR2(10);
Then you can test whether the database breaks (and it may not break immediately but at some point later), but if it does then you almost certainly will not get any support from Oracle in fixing it.
Did we say, "Don't do it"... we mean it.
As you already know, you shouldn't do that.
But, nothing prevents you from creating your own table which will contain such an info.
For example:
SQL> CREATE TABLE my_comments
2 (
3 table_name VARCHAR2 (30),
4 column_name VARCHAR2 (30),
5 anonymisation VARCHAR2 (10)
6 );
Table created.
Populate it with some data:
SQL> insert into my_comments (table_name, column_name)
2 select table_name, column_name
3 from user_tab_columns
4 where table_name = 'DEPT';
3 rows created.
Set the anonymisation flag:
SQL> update my_comments set anonymisation = 'F' where column_name = 'DEPTNO';
1 row updated.
When you want to get such an info (along with some more data from user_tab_columns, use (outer) join:
SQL> select u.table_name, u.column_name, u.data_type, u.nullable, m.anonymisation
2 from user_tab_columns u left join my_comments m on m.table_name = u.table_name
3 and m.column_name = u.column_name
4 where u.column_name = 'DEPTNO';
TABLE_NAME COLUMN_NAME DATA_TYPE N ANONYMISATION
---------- --------------- ------------ - ---------------
DEPT DEPTNO NUMBER N F
DSV DEPTNO NUMBER N
DSMV DEPTNO NUMBER Y
EMP DEPTNO NUMBER Y
SQL>
Advantages: you won't break the database & you'll have your additional info.
Drawbacks: you'll have to maintain the table manually.
Related
Is there a way to disable/restrict/alert-when-using some column in Oracle in a where clauses?
The reason that I'm asking this is because I have a very complex system (~30 services span cross millions of lines of code with thousends of sqls in it, in a sensitive production environment) working with an Oracle DB I need to migrate from using one column that is part of a key (and have a very not uniqu name) to another column.
Simple search is impossible....
The steps I'm having are:
populate new column
Add indexes on with the second column whenever there's an index with the first one.
Migrate all uses in where caluses from old to new column
Stop reading from the first column
Stop writing to the first column
Delete the column
I'm currently done step 3 and want to verify I've found all of the cases.
So, you're replacing one column with another. Which benefit do you expect once you're done? How will that improve overall experience with that application? I hope it is worth the effort.
As of your question: query user_source (or expand it to all_source or even dba_source, but you'll need additional privileges to do that) and see where's that very not unique name used. Something like this:
SQL> select * from user_source where lower(text) like '%empno%';
NAME TYPE LINE TEXT
--------------- ------------ ----- --------------------------------------------------------------------------------
P_RAISE PROCEDURE 22 WHERE empno = par_empno;
P_RAISE PROCEDURE 14 WHERE empno = par_empno;
P_RAISE PROCEDURE 1 PROCEDURE p_raise (par_empno IN emp.empno%TYPE)
GET_LIST FUNCTION 7 'select empno, ename, job, sal from emp where deptno = 10 order by '
SQL>
I was working on a trigger, now I want to make some changes and see the code, is it possible to see, if so then how? the trigger is working fine, is there any command in plsql to where I can check out the code? I am using sql command line
user_triggers (or all_triggers) have the trigger source code. Similarly user_source (or all_source) have other source code.
General rule though that I follow is to always always always have a file structure where I keep the source. 100% rule that no one in the team is allowed to violate ever. Then I view the act of creating trigger equivalent to "compiling" in traditional programming. In that model, you would have a directory tree for example
<project>/src
<project>/src/plsql
<project>/src/plsql/<folder for each package>/<files>
<project>/src/plsql/<folder for each table>/<triggers>
And then "modifying" is simply changing them here and "compiling" again (compiling will imply running these via sqlplus - or still better creating a shell script.
In this model, you can easily incorporate several available version control tools as well.
GUI would display it prettier, but SQL*Plus can do it as well. Here's an example:
Creating a sample trigger:
SQL> create or replace trigger trg_update_percentage
2 after update or insert on item
3 for each row
4 begin
5 insert into students_percentage (sid, total_per)
6 select sid, total from student_report
7 where sid = :new.sid;
8 end;
9 /
Trigger created.
Fetch its description from USER_TRIGGERS; as the body is stored into the LONG datatype column, set long should be used (otherwise you won't see the complete code).
SQL> set long 4000
SQL> select trigger_name, trigger_type, triggering_event, table_name, trigger_body
2 from user_Triggers where trigger_name = upper('trg_update_percentage');
TRIGGER_NAME TRIGGER_TYPE TRIGGERING_EVENT TABLE_NAME
------------------------- -------------------- -------------------- ----------
TRIGGER_BODY
--------------------------------------------------------------------------------
TRG_UPDATE_PERCENTAGE AFTER EACH ROW INSERT OR UPDATE ITEM
begin
insert into students_percentage (sid, total_per)
select sid, total from student_report
where sid = :new.sid;
end;
SQL>
I been working with a complex view written by some other company in 2005. I am trying to understand what it is doing for reasons beyond this post. By the highly complex nature of this view (over 500 lines of code) I take it that the writers new what they where doing.
I keep finding things like TO_NUMBER(null), TO_DATE(null) in various places.
Seems to me like totally unnecessary use of a function.
Is there any technical reasons or advantages that justify why this was design like this?
By default NULL does not have a data type:
SQL> select dump(null) from dual;
DUMP
----
NULL
SQL>
However, if we force Oracle into making a decision it will default to making it a string:
SQL> create or replace view v1 as
select 1 as id
, null as dt
from dual
/
2 3 4 5
View created.
SQL> desc v1
Name Null? Type
-------------- -------- ----------------------------
ID NUMBER
DT VARCHAR2
SQL>
But this not always desirable. We might need to use NULL in a view for a number of reasons (defining an API, filling out a jagged UNION, etc) and so we cast the NULL to another datatype to get the projection we need.
SQL> create or replace view v1 as
select 1 as id
, to_date(null) as dt
from dual
/
2 3 4 5
View created.
SQL> desc v1
Name Null? Type
-------------- -------- ----------------------------
ID NUMBER
DT DATE
SQL>
Later versions have got smarter with regards to handling UNION. On my 11gR2 database, even though I use the null in first declared query (and that usually drives things) I still get the correct datatype:
SQL> create or replace view v1 as
select 1 as id
, null as dt
from dual
union all
select 2 as id
, sysdate as something_else
from dual
/
2 3 4 5 6 7 8 9
View created.
SQL>
SQL> desc v1
Name Null? Type
-------------- -------- ----------------------------
ID NUMBER
DT DATE
SQL>
Explicitly casting NULL may be left over from 8i, or to workaround a bug, or as ammoQ said, "superstitious".
In some old and rare cases the implicit conversion of NULL in set operations caused errors like ORA-01790: expression must have same datatype as corresponding expression.
I can't find any great references for this old behavior, but Google returns a few results that claim a query like this would fail in 8i:
select 'a' a from dual
union
select null a from dual;
And there is at least one similar bug, "Bug 9456979 Wrong result from push of NVL / DECODE into UNION view with NULL select list item - superceded".
But don't let 16 year-old software and some rare bug dictate how to program. And don't think there's a positive correlation between code size programming skill. There's a negative correlation: good programmers will create smaller, more readable code, and won't leave as many mysteries for future coders.
Well, I would prefer CAST(NULL AS DATE) or CAST(NULL AS NUMBER) instead of TO_DATA(NULL), looks more logical in my eyes.
I know two scenarios where such an expression is required.
One it the case of UNION, as already stated in the other aswers.
Another scenario is the case of overloaded procedures/functions, for example:
CREATE OR REPLACE PROCEDURE MY_PROC(val IN DATE) AS
BEGIN
DELETE FROM EMP WHERE HIRE_DATE = val;
END;
/
CREATE OR REPLACE PROCEDURE MY_PROC(val IN NUMBER) AS
BEGIN
DELETE FROM EMP WHERE EMP_ID = val;
END;
/
Calling the procedure like MY_PROC(NULL); does not work, Oracle does not know which procedure to execute. You must call it like MY_PROC(CAST(NULL AS DATE)); for example.
As a oracle PL/SQL programmer, I really don't find any logical reason for doing the things you have specified. The only logical approach to deal with null in oracle is to use nvl(), I really don't find any reason to use TO_NUMBER(null), TO_DATE(null) in a complex view.
We have three databases: dev, staging, and production. We do all our coding in the dev environment. We then push all our code and database changes to staging so the client can see how it works in a live environment. After they sign off, we do the final deployment to the production environment.
Now, about these CLOB columns: When using desc and/or querying the all_tab_columns view for the dev database, CLOBs show a data length of 4,000. However, in the staging and production databases, data lengths for dev-equivalent CLOB columns are odd numbers like 86. I've searched for every possible solution as to how this could have come about. I've even tried adding a new CLOB(86) column thinking it would work like it does for VARCHAR2, but Oracle just spits out an error.
Could the DBAs have botched something up? Is this even something to worry about? Nothing has ever seemed to break as a result of this, but I just like the metadata to be the same across all environments.
First of all, I - as a dba - feel sorry to see the lack of cooperation between you and the dbas. We all need to cooperate to be successful. Clob data lengths can be less than 4000 bytes.
create table z ( a number, b clob);
Table created.
insert into z values (1, 'boe');
1 row created.
exec dbms_stats.gather_table_stats (ownname => 'ronr', tabname => 'z');
PL/SQL procedure successfully completed.
select owner, avg_row_len from dba_tables where table_name = 'Z'
SQL> /
OWNER AVG_ROW_LEN
------------------------------ -----------
RONR 109
select length(b) from z;
LENGTH(B)
----------
3
Where do you find that a clob length can not be less than 4000?
DATA_LENGTH stores the maximun # of bytes that will be taken up within the row for a column. If the CLOB can be stored in row, then the maximum is 4000. LOBS will never take up more than 4000 bytes. If in row storage is disabled, then the LOB will only store the pointer information it needs to find the LOB data, which is much less than 4000 bytes.
SQL> create table t (clob_in_table clob
2 , clob_out_of_table clob
3 ) lob (clob_out_of_table) store as (disable storage in row)
4 , lob (clob_in_table) store as (enable storage in row)
5 /
Table created.
SQL> select table_name, column_name, data_length
2 from user_tab_columns
3 where table_name = 'T'
4 /
TABLE_NAME COLUMN_NAME DATA_LENGTH
------------------------------ ------------------------------ -----------
T CLOB_IN_TABLE 4000
T CLOB_OUT_OF_TABLE 86
EDIT, adding info on *_LOBS view
Use the [DBA|ALL|USER]_LOBS view to look at the defined in row out of row storage settings:
SQL> select table_name
2 , cast(substr(column_name, 1, 30) as varchar2(30))
3 , in_row
4 from user_lobs
5 where table_name = 'T'
6 /
TABLE_NAME CAST(SUBSTR(COLUMN_NAME,1,30)A IN_
------------------------------ ------------------------------ ---
T CLOB_IN_TABLE YES
T CLOB_OUT_OF_TABLE NO
EDIT 2, some references
See LOB Storage in Oracle Database Application Developer's Guide - Large Objects for more information on defining LOB storage, especially the third note that talks about what can be changed:
Note:
Only some storage parameters can be modified. For example, you
can use the ALTER TABLE ... MODIFY LOB statement to change RETENTION,
PCTVERSION, CACHE or NO CACHE LOGGING or NO LOGGING, and the STORAGE
clause.
You can also change the TABLESPACE using the ALTER TABLE ...
MOVE statement.
However, once the table has been created, you cannot change the CHUNK
size, or the ENABLE or DISABLE STORAGE IN ROW settings.
Also, LOBs in Index Organized Tables says:
By default, all LOBs in an index organized table created without an overflow segment will be stored out of line. In other words, if an index organized table is created without an overflow segment, then the LOBs in this table have their default storage attributes as DISABLE STORAGE IN ROW. If you forcibly try to specify an ENABLE STORAGE IN ROW clause for such LOBs, then SQL will raise an error.
This explains why jonearles did not see 4,000 in the data_length column when he created the LOB in an index organized table.
CLOBs don't have a specified length. When you query ALL_TAB_COLUMNS, e.g.:
select table_name, column_name, data_length
from all_tab_columns
where data_type = 'CLOB';
You'll notice that data_length is always 4000, but this should be ignored.
The minimum size of a CLOB is zero (0), and the maximum is anything from 8 TB to 128 TB depending on the database block size.
As ik_zelf and Jeffrey Kemp pointed out, CLOBs can store less than 4000 bytes.
But why are CLOB data_lengths not always 4000? The number doesn't actually limit the CLOB, but you're probably right to worry about the metadata being
different on your servers. You might want to run DBMS_METADATA.GET_DDL on the objects on all servers and compare the results.
I was able to create a low data_length by adding a CLOB to an index organized table.
create table test
(
column1 number,
column2 clob,
constraint test_pk primary key (column1)
)
organization index;
select data_length from user_tab_cols
where table_name = 'TEST' and column_name = 'COLUMN2';
On 10.2.0.1.0, the result is 116.
On 11.2.0.1.0, the result is 476.
Those numbers don't make any sense to me and I'd guess it's a bug. But I don't have a good understanding of the different storage options, maybe I'm just missing something.
Does anybody know what's really going on here?
Is there a option to see if existing table/record from a Oracle database is updated?
From a monitoring perspective (not intended to find previous changes), you have several options including but not limited to triggers, streams, and a column with a default value of sysdate. A trigger will allow you to execute a bit of programming logic (stored directly in the trigger or in an external database object) whenever the record changes (insert, update, delete). Streams can be used to track changes by monitoring the redo logs. One of the easiest may be to add a date column with a default value of sysdate.
Are you talking about within a transaction or outside of it?
Within our program we can use things like SQL%ROWCOUNT to see whether our DML succeeded...
SQL> set serveroutput on size unlimited
SQL> begin
2 update emp
3 set job = 'SALESMAN', COMM=10
4 where empno = 8083;
5 dbms_output.put_line('Number of records updated = '||sql%rowcount);
6 end;
7 /
Number of records updated = 1
PL/SQL procedure successfully completed.
SQL>
Alternatively we might test for SQL%FOUND (or SQL%NOTFOUND).
From outside the transaction we can monitor ORA_ROWSCN to see whether a record has changed.
SQL> select ora_rowscn from emp
2 where empno = 8083
3 /
ORA_ROWSCN
----------
83828715
SQL> update emp
2 set comm = 25
3 where empno = 8083
4 /
1 row updated.
SQL> commit
2 /
Commit complete.
SQL> select ora_rowscn from emp
2 where empno = 8083
3 /
ORA_ROWSCN
----------
83828780
SQL>
By default ORA_ROWSCN is set at the block level. If you want to track it at the lower level your need to create the table with the ROWDEPENCIES keyword.
These are ad hoc solutions. If you want to proactive monitoring then you need to implementing some form of logging. Using triggers to write log records is a common solution. If you have Enterprise Edition you should consider using Fine Grained Auditing: Dan Morgan's library has a useful demo of how to use FGA to track changes.
You can see if a table definition has change by querying the last_ddl_time from the user_objects view.
Without using triggers or materialized logs (which would be a total hack) there is no way I know of to see when any particular row in a table has been updated.