Is there any encryption or obfuscation Oracle Function (standard or third party) that can limit the output size?
I need to obfuscate a column smaller than 20 char by limiting the output to 20 char .
I'm using the following function that generates an output of 32 Characters , the constraint comes from a System external to mine , I couldn't negotiate a larger size of the output (their input).
create or replace function md5 (valor varchar) return varchar2 is
v_input varchar2(2000) := valor;
hexkey varchar2(32) := null;
begin
hexkey := rawtohex(dbms_obfuscation_toolkit.md5(input => utl_raw.cast_to_raw(v_input)));
return nvl(hexkey,'');
end;
I need something like obfuscate a USA Social Service in a 20 char length output.
I need also some security because us applied to private medical information.
Related
I have a table containing nvarchar datatype columns (contains text in different languages). I want to encrypt data before inserting into table and decrypt the same while fetching records.
Please suggest how i can achieve this.
Encryption and decryption should be done through a private key.
Hoping, my question is clear. Please confirm if i need to provide more information.
Note that it is probably wiser to crypt and decrypt your data directly in your application rather than in the database.
You can use Oracle's DBMS_CRYPTO package. There is an example in the middle of the documentation page.
First you need to make a package to access the cipher type from SQL expression. Let's say you want AES256 in CBC mode with padding:
CREATE PACKAGE pkg_so_42979606
AS
FUNCTION cipher_type RETURN PLS_INTEGER;
END pkg_so_42979606;
/
CREATE PACKAGE BODY pkg_so_42979606
AS
ctype CONSTANT PLS_INTEGER := DBMS_CRYPTO.ENCRYPT_AES256
+ DBMS_CRYPTO.CHAIN_CBC
+ DBMS_CRYPTO.PAD_PKCS5;
FUNCTION cipher_type RETURN PLS_INTEGER
IS
BEGIN
RETURN ctype;
END;
END pkg_so_42979606;
/
Then you will need a key. You can ask Oracle to generate one. To easily handle it I'll move it in Base64. Let's draw one:
DECLARE
key_bytes_raw RAW(32);
key_char NVARCHAR2(64);
BEGIN
key_bytes_raw := DBMS_CRYPTO.RANDOMBYTES(32);
key_char := UTL_I18N.RAW_TO_CHAR(UTL_ENCODE.BASE64_ENCODE(key_bytes_raw), 'AL32UTF8');
DBMS_OUTPUT.PUT_LINE('Key: ' || key_char);
END;
/
Key: pMV3D4xhyfNxp3YyfLWzAErGcKkIjK3X6uc/WIeVTls=
Thus the cipher key I'll use is pMV3D4xhyfNxp3YyfLWzAErGcKkIjK3X6uc/WIeVTls=.
Now I'll use a test table
CREATE TABLE so_42979606 (
id NUMBER PRIMARY KEY,
data NVARCHAR2(2000));
You can insert encrypted data:
INSERT INTO so_42979606
VALUES (1,
DBMS_CRYPTO.ENCRYPT(UTL_I18N.STRING_TO_RAW('My clear data', 'AL32UTF8'),
pkg_so_42979606.cipher_type(),
UTL_ENCODE.BASE64_DECODE(UTL_I18N.STRING_TO_RAW('pMV3D4xhyfNxp3YyfLWzAErGcKkIjK3X6uc/WIeVTls=', 'AL32UTF8'))));
And retrieve the encrypted data in clear.
SELECT id, UTL_I18N.RAW_TO_NCHAR(DBMS_CRYPTO.DECRYPT(data,
pkg_so_42979606.cipher_type(),
UTL_ENCODE.BASE64_DECODE(UTL_I18N.STRING_TO_RAW('pMV3D4xhyfNxp3YyfLWzAErGcKkIjK3X6uc/WIeVTls=', 'AL32UTF8'))),
'AL32UTF8') data
FROM so_42979606;
ID DATA
-- ----------------------
1 My clear data
Can I convert an oracle BLOB to Base64 CLOB in One go?
like:
CREATE TABLE test
(
image BLOB,
imageBase64 CLOB
);
INSERT INTO test(image)
VALUES (LOAD_FILE('/full/path/to/new/image.jpg'));
UPDATE test SET imageBase64 = UTL_ENCODE.base64_encode(image);
commit;
I know I can add functions/Stored proc to do the work. Performance aspect is very important,so I am asking if there is a way to overcome the 32K limitation by directly pushing the data into a CLOB.
This function got from here should do the job.
CREATE OR REPLACE FUNCTION base64encode(p_blob IN BLOB)
RETURN CLOB
-- -----------------------------------------------------------------------------------
-- File Name : http://oracle-base.com/dba/miscellaneous/base64encode.sql
-- Author : Tim Hall
-- Description : Encodes a BLOB into a Base64 CLOB.
-- Last Modified: 09/11/2011
-- -----------------------------------------------------------------------------------
IS
l_clob CLOB;
l_step PLS_INTEGER := 12000; -- make sure you set a multiple of 3 not higher than 24573
BEGIN
FOR i IN 0 .. TRUNC((DBMS_LOB.getlength(p_blob) - 1 )/l_step) LOOP
l_clob := l_clob || UTL_RAW.cast_to_varchar2(UTL_ENCODE.base64_encode(DBMS_LOB.substr(p_blob, l_step, i * l_step + 1)));
END LOOP;
RETURN l_clob;
END;
/
Then the update can look like
UPDATE test SET imageBase64 = base64encode(image);
Note that maybe the function should be optimized with the function DBMS_LOB.APPEND instead of that concatenation operator. Try that if you have performance problems.
Provided that stored procs would despite be a viable alternative for you, here's one possible solution to your problem ...
First, let's make that nice base64encode() function of Tim Hall's into a procedure ...
create or replace procedure base64encode
( i_blob in blob
, io_clob in out nocopy clob )
is
l_step pls_integer := 22500; -- make sure you set a multiple of 3 not higher than 24573
l_converted varchar2(32767);
l_buffer_size_approx pls_integer := 1048576;
l_buffer clob;
begin
dbms_lob.createtemporary(l_buffer, true, dbms_lob.call);
for i in 0 .. trunc((dbms_lob.getlength(i_blob) - 1 )/l_step) loop
l_converted := utl_raw.cast_to_varchar2(utl_encode.base64_encode(dbms_lob.substr(i_blob, l_step, i * l_step + 1)));
dbms_lob.writeappend(l_buffer, length(l_converted), l_converted);
if dbms_lob.getlength(l_buffer) >= l_buffer_size_approx then
dbms_lob.append(io_clob, l_buffer);
dbms_lob.trim(l_buffer, 0);
end if;
end loop;
dbms_lob.append(io_clob, l_buffer);
dbms_lob.freetemporary(l_buffer);
end;
The "trick" here is to directly use the persistent LOB locators in calls to procedures/functions. Why "persistent"? Because if you create a function that returns a LOB, then there's a temporary LOB created in background and this means some TEMP disk/memory usage and LOB content copying involved. For large LOBs this may imply a performance hit. In order to satisfy your requirement of making this the most performing possible, you should avoid this TEMP space usage. Hence, for this approach, a stored procedure instead of a function must be used.
Then, of course, the procedure must be fed with persistent LOB locators. You have to do that, again, with a stored procedure, where you e.g. insert an empty LOB (effectively creating a new LOB locator) to a table first, and then supplying that newly created LOB locator to the base64 encoding routine ...
create or replace procedure load_and_encode_image
( i_file_name in varchar2 )
is
l_input_bfile bfile := bfilename('DIR_ANYTHING', i_file_name);
l_image_base64_lob test.imageBase64%type;
l_image_raw test.image%type;
begin
insert into test(image, imageBase64)
values (empty_blob(), empty_clob())
returning image, imageBase64
into l_image_raw, l_image_base64_lob;
begin
dbms_lob.fileopen(l_input_bfile);
dbms_lob.loadfromfile(
dest_lob => l_image_raw,
src_lob => l_input_bfile,
amount => dbms_lob.getlength(l_input_bfile)
);
dbms_lob.fileclose(l_input_bfile);
exception
when others then
if dbms_lob.fileisopen(l_input_bfile) = 1 then
dbms_lob.fileclose(l_input_bfile);
end if;
raise;
end;
base64encode(
i_blob => l_image_raw,
io_clob => l_image_base64_lob
);
end;
Note: Of course, if you base64-encode only small files (the actual size depends on your PGA settings, I guess; a question for a DBA, this is), then the function-based approach may be equally performing than this procedure-based one. Base64-encoding a 200MB file on my laptop took 55 seconds with the function+update approach, 14 seconds with the procedure approach. Not exactly a speed demon, so choose what suits your needs.
Note: I believe this procedure-based approach may be further speeded up by reading the file to inmemory chunks in loop, base64-encoding the chunks to another inmemory chunks and appending them both to the target persistent LOBs. That way you should make the workload even easier by avoiding re-reading the full test.image LOB contents by the base64encode() procedure.
I solved this same problem at work by using a Java stored procedure. There is no chunking/contatenation of VARCHAR2s involved in such an approach, since the ability to encode/decode base64 is natively built into Java, simply writing an Oracle function that thinly wraps the Java method works well and is high-performance since as soon as you've executed it a few times, the HotSpot JVM compiles the Java proc into low-level code (high performance just like a C stored function). I'll edit this answer later and add the details about that Java code.
But to step back just one step, question why are you storing this data as both a BLOB and base64 encoded (CLOB)? Is it because you have clients that want to consume the data in the latter format? I'd really prefer to only store the BLOB format. One reason why is that the base64 encoded version can be double the size of the original binary BLOB, so storing them both means possibly 3x the storage.
One solution, the one I implemented at work, to create your own Java stored function base64_encode() that encodes binary --> base64 and then use that function to encode base64 on the fly at query time (it's not expensive). From the application/client side, you would query something like SELECT base64_encode(image) FROM test WHERE ...
If the application code can't be touched (ie COTS application) or if your developers aren't thrilled about using a function, you could abstract this for them (since you are using 11g+) by using a VIRTUAL (computed) column on the table which contains the computed base64_encode(image). It would function like a view, in that it wouldn't physically store the encoded CLOBs, but would generate them at query time. To any client, they would not be able to tell they are not reading a physical column. The other benefit is that if you ever update the jpg (BLOB), the virtual CLOB is immediately and automatically updated. If you ever have to insert/update/delete a huge batch of BLOBs, you'd save 66% of the redo/archivelog volume from not having to process all the CLOBs.
Lastly, for performance, make very sure you are using SecureFile LOBs (both for BLOBs and CLOBs). They really are much faster and better in just about every way.
UPDATE - I found my code, at least the version that uses a Java Stored Procedure to do the opposite (converting a base64 encoded CLOB to its binary BLOB version). It would not be that difficult to write the inverse.
--DROP FUNCTION base64_decode ;
--DROP java source base64;
-- This is a PLSQL java wrapper function
create or replace
FUNCTION base64_decode (
myclob clob)
RETURN blob
AS LANGUAGE JAVA
NAME 'Base64.decode (
oracle.sql.CLOB)
return oracle.sql.BLOB';
/
-- The Java code that base64 decodes a clob and returns a blob.
create or replace and compile java source named base64 as
import java.sql.*;
import java.io.*;
import oracle.sql.*;
import sun.misc.BASE64Decoder;
import oracle.jdbc.driver.*;
public class Base64 {
public static oracle.sql.BLOB decode(oracle.sql.CLOB myBase64EncodedClob)
{
BASE64Decoder base64 = new BASE64Decoder();
OutputStream outstrm = null;
oracle.sql.BLOB myBlob = null;
ByteArrayInputStream instrm = null;
try
{
if (!myBase64EncodedClob.equals("Null"))
{
Connection conn = new OracleDriver().defaultConnection();
myBlob = oracle.sql.BLOB.createTemporary(conn, false,oracle.sql.BLOB.DURATION_CALL);
outstrm = myBlob.getBinaryOutputStream();
ByteArrayOutputStream byteOutStream = new ByteArrayOutputStream();
InputStream in = myBase64EncodedClob.getAsciiStream();
int c;
while ((c = in.read()) != -1)
{
byteOutStream.write((char) c);
}
instrm = new ByteArrayInputStream(byteOutStream.toByteArray());
try // Input stream to output Stream
{
base64.decodeBuffer(instrm, outstrm);
}
catch (Exception e)
{
e.printStackTrace();
}
outstrm.close();
instrm.close();
byteOutStream.close();
in.close();
conn.close();
}
}
catch (Exception e)
{
e.printStackTrace();
}
return myBlob;
} // Public decode
} // Class Base64
;
/
The easiest way that I found, that works with special characters (in your case you don't have this problem), is using dbms_lob.converttoclob.
Create encapsulated Procedure:
CREATE OR REPLACE FUNCTION blob2clob(blob_i IN BLOB) RETURN CLOB IS
l_clob CLOB;
l_dest_offset NUMBER := 1;
l_src_offset NUMBER := 1;
l_amount INTEGER := dbms_lob.lobmaxsize;
l_clob_csid NUMBER := nls_charset_id('WE8ISO8859P15'); --dbms_lob.default_csid;
l_lang_context INTEGER := dbms_lob.default_lang_ctx;
l_warning INTEGER;
BEGIN
---------------------------
-- Create Temporary BLOB --
---------------------------
dbms_lob.createtemporary(lob_loc => l_clob,
cache => TRUE);
--------------------------
-- Convert CLOB to BLOB --
--------------------------
dbms_lob.converttoclob(dest_lob => l_clob,
src_blob => blob_i,
amount => l_amount,
dest_offset => l_dest_offset,
src_offset => l_src_offset,
blob_csid => l_clob_csid,
lang_context => l_lang_context,
warning => l_warning);
--
RETURN l_clob;
END blob2clob;
Then you can use:
blob2clob(utl_encode.base64_encode(image))
How to do for encrypting a number
column in Oracle 11 (and also be able to decrypt them back) so that the same column stores crypted numbers along with (unencrypted ones (based on a boolean saved elsewhere)?
I actually did a function like this that takes a number to produce a crypted number and it worked fine in Oracle 10 but now it doesn't work anymore in ORACLE 11:
function crypt (key varchar2, n number) return number
raw_input RAW(128);
encrypted_raw RAW(2048);
raw_key_ RAW(128) := UTL_RAW.CAST_TO_RAW(CONVERT(key,'AL32UTF8','US7ASCII'));
begin
raw_input_ := UTL_RAW.cast_from_number (n);
encrypted_raw := dbms_crypto.Encrypt(src => raw_input_, typ =>DBMS_CRYPTO.DES3_CBC_PKCS5, KEY=>raw_key_);
return UTL_RAW.cast_to_number(encrypted_raw);
end;
it throws this error with no explanations:
ORA-06502: PL/SQL: numeric or value error
ORA-06512: at "SYS.UTL_RAW", line 388
...
Thank you
Looks like you're trying to cast the encrypted raw to a number. If I understand what you want this function to do, it seems you want to test that the input number = returned number (that you can encrypt and decrypt successfully). If you wanted to just encrypt, you would return a raw, not a number.
Anyway, you should be casting the decrypted raw value to a number to test that the input number = returned number. Something like:
create or replace function test_crypt(k varchar2, n number)
return number as
raw_input_ RAW(128);
encrypted_raw RAW(2000);
decrypted_raw RAW (2000);
raw_key_ RAW(128) := UTL_RAW.CAST_TO_RAW(k);
begin
raw_input_ := UTL_RAW.cast_from_number (n);
encrypted_raw := dbms_crypto.Encrypt(src => raw_input_, typ =>DBMS_CRYPTO.DES3_CBC_PKCS5, KEY=>raw_key_);
--return UTL_RAW.cast_to_number(encrypted_raw);
-- use dbms_crypto to decrypt and return (hopefully same) number
decrypted_raw := DBMS_CRYPTO.DECRYPT
(
src => encrypted_raw,
typ => DBMS_CRYPTO.DES3_CBC_PKCS5,
key => raw_key_
);
return UTL_RAW.cast_to_number(decrypted_raw);
end;
On my system anyway, the input number = returned number (key is from dbms_crypto.randombytes(32) ). For example:
select test_crypt('50610FB89D98C7D906CB0A9917413221E4FE6FA62A9604302EE2C8F63E6BAD91', 234.21) from dual;
Output:
234.21
I'm using Oracle 11g and I'm trying to find out the length of a text. I normally use select length(myvar) from table, but I can't do that.
The table which I want to query has a BLOB column that saves characters or photos. I want to know the number of characters that my BLOB column has.
I tried to convert my BLOB into a char using UTL_RAW.CAST_TO_VARCHAR2(myblob) from table, but this functions isn't working correctly or maybe I'm making a mistake.
For example:
My BLOB have the word Section, when I see this in the databse in the hexadecimal form I see S.e.c.t.i.o.n.. I don't know why it have those points in between each letter.
Then I used the this query:
select UTL_RAW.CAST_TO_VARCHAR2(myblob)
from table
The result of this query is 'S' so it's not the complete word that my BLOB has, and when I make this query:
select length(UTL_RAW.CAST_TO_VARCHAR2(myblob))
from table
the result is 18, but the word Sections doesn't have 18 characters.
I was trying to convert the BLOB into a VARCHAR, although I think my best choise would be a CLOB because the length of the text that it can save is more than the limit that VARCHAR has. I tried to do that by making this query (I'm not sure if this is correct but is what I found in the internet):
select UTL_RAW.CAST_TO_VARCHAR2(DBMS_LOB.SUBSTR(myblob, 32767, 1))
from table
This query also returns 'S'
For anyone coming to this thread and wants to know how to convert a blob to a clob. Here is an example.
create function clobfromblob(p_blob blob) return clob is
l_clob clob;
l_dest_offsset integer := 1;
l_src_offsset integer := 1;
l_lang_context integer := dbms_lob.default_lang_ctx;
l_warning integer;
begin
if p_blob is null then
return null;
end if;
dbms_lob.createTemporary(lob_loc => l_clob
,cache => false);
dbms_lob.converttoclob(dest_lob => l_clob
,src_blob => p_blob
,amount => dbms_lob.lobmaxsize
,dest_offset => l_dest_offsset
,src_offset => l_src_offsset
,blob_csid => dbms_lob.default_csid
,lang_context => l_lang_context
,warning => l_warning);
return l_clob;
end;
To convert blob to clob, try this:
SELECT TO_CLOB(UTL_RAW.CAST_TO_VARCHAR2(DBMS_LOB.SUBSTR(MYBLOB,2000)))
FROM MYTABLE;
SELECT DBMS_LOB.GetLength( myblob ) length_in_bytes
FROM table
will return the length of the BLOB in bytes. It sounds like the character data in your BLOB is probably encoded using the UTF-16 character set so the number of bytes is probably twice the number of characters (depending on the version of Unicode that is being used and the specific data being stored, some characters might require 4 bytes of storage but it is relatively unlikely that you're dealing with any of those characters).
You can use the DBMS_LOB.ConvertToClob procedure to convert a BLOB to a CLOB (though since this is a procedure, you'll need to call it in a PL/SQL block). As part of that conversion, you'll almost certainly need to specify the character set that the data is encoded in-- my assumption is that your application is using the UTF-16 character set but that's just an assumption.
Only converting to CLOB:
select TO_CLOB(UTL_RAW.CAST_TO_VARCHAR2(YOURCLOB)) from DUAL;
Inspired by Craig without limitation on size.
In Oracle, the full text search syntax of Contains operator is:
CONTAINS(
[schema.]column,
text_query VARCHAR2
[,label NUMBER]) RETURN NUMBER;
which means the text_query can not be more than 4000 characters long or an error will occur. I repeatedly have text_query longer than 4000 characters long in many cases. How would you, as an Oracle expert, suggest to get around such limitation if possible?
To further clarify the situation in which 4000 is easily reached is that if you combine many Contains Query Operators to construct your text_query, it is quite possible to exceed such 4000 characters limitation.
The 4000 character limit is not some arbitrary boundary: it is the maximum amount of VARCHAR2 characters that Oracle SQL can handle.
4000 characters is a lot of text. In English it's around 600 words, or an A4 page and a bit in a reasonable point font. There are not many applications I can think of which require searching for such large chunks of verbiage. Even colleges checking students' essays for plagiarism would operate at no more than the paragraph level.
However, if you really have a situation in which matching on a scant 4000 characters generates false positives all you can do is split the query string into chunks and search on them. This means you have to use PL/SQL:
create or replace function big_search (p_search_text in varchar2)
return sys_refcursor
is
return_value sys_refcursor;
p_srch1 varchar2(4000);
p_srch2 varchar2(4000);
begin
dbms_output.put_line('search_length='||to_char(length(p_search_text)));
p_srch1 := substr(p_search_text, 1, 4000);
p_srch2 := substr(p_search_text, 4001, 4000);
open return_value for
select docname
, (score(1) + score(2))/2 as score
from t23
where contains ( text_column, p_srch1 , 1) != 0
and contains ( text_column, p_srch2 , 2) != 0;
return return_value;
end;
/
If you don't know the size of the search text beforehand, then you'll need to use dynamic SQL to assemble this. Note that passing null search terms to CONTAINS() will hurl DRG-50901: text query parser syntax error.
The current version supports now a CLOB parameter
CONTAINS(
[schema.]column,
text_query [VARCHAR2|CLOB]
[,label NUMBER])
RETURN NUMBER;
http://docs.oracle.com/cd/B28359_01/text.111/b28304/csql.htm#i997503