I have definitely searched FAR and WIDE for an answer to this, but I can't find anything! I am using a UTL_FILE script to pull down some file BLOBS from an Oracle table and save them to a file directory. It's working for a lot of the files, but I have narrowed it down by process of elimination that it's having issues for files that have an "unconventional" file name, albeit still a valid one, the files are getting corrupted in the transfer. They may have only been 30kb originally, but export as 5kb and cannot be opened. So I know it's not a large file size issue. The files open just fine through the application, have a valid MIME type encoding, and would otherwise open fine on a file system, but UTL_FILE doesn't seem to like them. They are files that have an extra "." in them ie: john.smith.doc, or a pound sign ie: Smith #12345.doc or parentheses, etc. I cannot change the source file names in the Oracle table, but I have been concatenating an ID number on to them when saving them out so I can reference it as a key for an ETL load into SQL file table later. Maybe I also need to write a complicated REGEXP to rename the files on the fly and strip out the bad characters, but I'm not sure that will work because I don't know at what point UTL_FILE is choking on them. If it's at the source, then that won't help. Has anyone else encountered this problem? Here is my script:
DECLARE
CURSOR C1 IS Select FILE_ID || '---' || substr(DOCUMENTLOCATION,1,instr
(DOCUMENTLOCATION,'.')-1)||'.doc' as FILE_NAME, FILE_BLOB, FILE_ID
From DOCUMENTS d inner join CASEJOURNAL c on d.FILE_ID = c.JOURNALENTRYID
where (JOURNAL_ENTRY_TYPE = 117 or JOURNAL_ENTRY_TYPE = 3) AND
c.DOCUMENTLOCATION Is Not Null AND d.MIME_TYPE = 'application/msword'
AND FILE_ID BETWEEN 785 AND 3380;
l_file UTL_FILE.FILE_TYPE;
l_buffer RAW(32000);
l_amount INTEGER := 32000;
l_pos INTEGER := 1;
l_blob BLOB;
l_blob_len INTEGER;
l_filename varchar2(255);
BEGIN
--Select BLOB file into variables
FOR I in C1
LOOP
Select FILE_ID || '---' || substr(DOCUMENTLOCATION,1,instr
(DOCUMENTLOCATION,'.')-1) ||'.doc' as FILE_NAME, FILE_BLOB INTO l_filename,
l_blob From DOCUMENTS d inner join CASEJOURNAL c on d.FILE_ID =
c.JOURNALENTRYID where (JOURNAL_ENTRY_TYPE = 117 or JOURNAL_ENTRY_TYPE =
3) AND c.DOCUMENTLOCATION Is Not Null AND d.MIME_TYPE
= 'application/msword' and d.FILE_ID = I.FILE_ID;
-- Define the output directory
l_file := UTL_FILE.FOPEN('\\myfiledirectory',l_filename,'wb',32000);
l_pos := 1;
l_amount := 32000;
--Get length of BLOB file and save to variable.
l_blob_len := DBMS_LOB.getlength(l_blob);
-- Write the data to the file
--If small enough for single write:
IF l_blob_len < 32000 THEN
UTL_FILE.PUT_RAW (l_file, l_blob);
UTL_FILE.FFLUSH(l_file);
--Write in pieces if larger than 32k
ELSE
l_pos := 1;
WHILE l_pos < l_blob_len AND l_amount > 0
LOOP
DBMS_LOB.read(l_blob, l_amount, l_pos, l_buffer);
UTL_FILE.PUT_RAW(l_file, l_buffer);
UTL_FILE.FFLUSH(l_file);
--Set start position for next write
l_pos := l_pos + l_amount;
--Set end position if less than 32k.
l_blob_len := l_blob_len - l_amount;
IF l_blob_len < 32000 THEN
l_amount := l_blob_len;
END IF;
END LOOP;
END IF;
UTL_FILE.FCLOSE(l_file);
END LOOP;
END;
The file name isn't going to affect how the bytes are written out once the file has been opened. You seem to be truncating the file if it's more than 32k. Your loop does this:
WHILE l_pos < l_blob_len AND l_amount > 0
LOOP
... but then you change both l_pos and l_blob_len within the loop; once the adjusted l_pos falls below the remaining l_blob_len you exit the loop, too early. You don't need to adjust l_blob_len, or even adjust l_amount - that is the maximum number of bytes to read, it doesn't matter if it's higher than what's left.
So change the loop to:
WHILE l_pos < l_blob_len AND l_amount > 0
LOOP
DBMS_LOB.read(l_blob, l_amount, l_pos, l_buffer);
UTL_FILE.PUT_RAW(l_file, l_buffer);
UTL_FILE.FFLUSH(l_file);
--Set start position for next write
l_pos := l_pos + l_amount;
END LOOP;
Not really related to your problem, but you don't need to reselect the data inside your cursor loop. You've already got the values you need in your i cursor variable, so you can do:
FOR I in C1
LOOP
l_filename := i.file_name;
l_blob := i.file_blob;
-- Define the output directory
...
Or don't bother with the l_filename and l_blob local variables at all; since you only refer to them inside the cursor loop anyway, use i.file_name and i.file_blob directly everywhere, e.g.
l_file := UTL_FILE.FOPEN('\\myfiledirectory',i.file_name,'wb',32000);
l_blob_len := DBMS_LOB.getlength(i.file_blob);
etc.
Related
Currently I am doing this to upload cwallet.sso (ie a "normal" file, not an export, etc.) to an Oracle Autonomous Database...
BEGIN
DBMS_CLOUD.GET_OBJECT(
object_uri => 'https://objectstorage.us-ashburn-1.oraclecloud.com/p/Uasdfadsfasdf7Icmer6HMkv/n/sadf/b/paul/o/cwallet.sso',
directory_name => 'DATA_PUMP_DIR');
END;
/
I would prefer to not rely on object store as I have the cwallet.sso locally and so it would seem an unnecessary additional step. Is there a straightforward PL/SQL command to just upload the file from local location to DATA_PUMP_DIR (or any dir really)? I couldn't quite tell from doc.
Autonomous Database does offer access to "directories" and "files." Under the covers, these are implemented as a virtual filesystem with the storage coming from your database, so it is charged to you as database quota.
It's a little awkward, but you can get files into this filesystem with a PL/SQL procedure if you're able to load your input into a BLOB:
PROCEDURE write_file(
directory_name IN VARCHAR2,
file_name IN VARCHAR2,
contents IN BLOB
)
IS
l_file UTL_FILE.file_type;
l_data_len INTEGER;
l_buffer RAW(32000);
l_pos INTEGER := 1;
l_amount INTEGER := 32000;
BEGIN
-- Get the data length to write
l_data_len := DBMS_LOB.getlength(contents);
-- Write the contents to local file
l_file := UTL_FILE.FOPEN(directory_name, file_name, 'wb', l_amount);
WHILE l_pos < l_data_len
LOOP
DBMS_LOB.read(contents, l_amount, l_pos, l_buffer);
UTL_FILE.PUT_RAW(l_file, l_buffer, TRUE);
l_pos := l_pos + l_amount;
END LOOP;
UTL_FILE.FCLOSE(l_file);
EXCEPTION
WHEN OTHERS THEN
UTL_FILE.FCLOSE(l_file);
RAISE;
END write_file;
How you get your data into a BLOB depends on your client.
Is there any way to return a clob as text (or file) without splitting into smaller pieces first?
I tried creating a GET handler with PL/SQL source that would just do this:
declare
txt clob;
begin
...-- set clob
htp.p(txt);
end;
But then I was getting the error ORA-06502: PL/SQL: numeric or value error.
I then tried cutting the clob in smaller segments and calling htp.p multiple times, which worked, but I was wondering if there was a way to send the whole thing in one go.
The doc states that htp.p and htp.prn take only VARCHAR2 so you're limited by the max size of a varchar2 and if the clob length exceeds that it will throw an error. This is what you can do:
Loop through the clob in 4k chunks and output using htp.prn. Avoid using htp.p in a loop because that generates a newline characters which could mess up the output, for example if you're generating json. It's also good practice to let the browser know what he's getting by setting the mime header.
DECLARE
l_clob CLOB;
l_amt INTEGER := 4000;
l_pos INTEGER := 1;
l_buf VARCHAR2(4000);
BEGIN
owa_util.mime_header ('text/html', true);
l_clob := '....';
LOOP
BEGIN
dbms_lob.read(
l_clob,
l_amt,
l_pos,
l_buf
);
l_pos := l_pos + l_amt;
-- need htp.prn since htp.p generates a newline char at the end.
htp.prn(l_buf);
EXCEPTION
WHEN no_data_found THEN
EXIT;
END;
END LOOP;
END;
I have an xlsx file sample.xlsx stored in a remote directory with around 1,699 KB in Size.
I have tried two popular PL/SQL packages (UTL_COMPRESS and AS_ZIP) that compresses them into gzip and zip, rescpectively.
With the code below using AS_ZIP, I have compressed the file to 1,619 KB:
declare
g_zipped_blob blob;
l_file_name varchar2(100) := 'sample.xlsx';
l_directory varchar2(100) := 'EXT_TAB_DATA';
begin
as_zip.add1file( g_zipped_blob, l_file_name, as_zip.file2blob(l_directory, l_file_name));
as_zip.finish_zip( g_zipped_blob );
as_zip.save_zip( g_zipped_blob, l_directory, 'my2.zip' );
dbms_lob.freetemporary( g_zipped_blob );
end;
With the code below (taken from the original post) using UTL_COMPRESS, I have compressed the file to 1,618 KB:
DECLARE
in_filename VARCHAR2(100) := 'sample.xlsx';
l_directory varchar2(100) := 'EXT_TAB_DATA';
src_file BFILE;
v_content BLOB;
v_blob_len INTEGER;
v_file utl_file.file_type;
v_buffer RAW(32767);
v_amount BINARY_INTEGER := 32767;
v_pos INTEGER := 1;
BEGIN
src_file := bfilename(l_directory, in_filename);
dbms_lob.fileopen(src_file, dbms_lob.file_readonly);
v_content := utl_compress.lz_compress(src_file, 9);
v_blob_len := dbms_lob.getlength(v_content);
v_file := utl_file.fopen(l_directory,
in_filename || '.gz',
'wb');
WHILE v_pos < v_blob_len LOOP
dbms_lob.READ(v_content, v_amount, v_pos, v_buffer);
utl_file.put_raw(v_file, v_buffer, TRUE);
v_pos := v_pos + v_amount;
END LOOP;
utl_file.fclose(v_file);
EXCEPTION
WHEN OTHERS THEN
IF utl_file.is_open(v_file) THEN
utl_file.fclose(v_file);
END IF;
RAISE;
END;
Although minimal, it seems that UTL_COMPRESS has better compression in terms of file size.
I was wondering if there was some unseen advantage using the custom AS_ZIP over the Oracle-supplied UTL_COMPRESS?
Thank you.
Anton Scheffer explains why he wrote the AS_ZIP package in this blog post . It should answer your question. Basically it's to support additional zip formats.
Also it has a link to a more recent version of the package than the link in your post.
As for which one to use, my standard line is always to use the Oracle built-in functionality unless we really need the something extra from a third-party offering.
Using Oracle's standard functionality means:
Oracle Support covers us
we don't have to maintain the code
our code base is that much simpler for new joiners
I think this one will do the job of writing all the files to a directory all at once if I can get past the "invalid operation error"
Your help, as always, is greatly appreciated.
create or replace
PROCEDURE GetbFile
IS
l_output utl_file.file_type;
vstart NUMBER := 1;
bytelen NUMBER := 32000;
x NUMBER;
my_vr RAW(32000);
v_name VARCHAR2(32760);
BEGIN
FOR recFiles IN (SELECT dbms_lob.getlength(BLOB_VALUE) as len,
FILE_NAME,
BLOB_VALUE from Gfile)
LOOP
l_output := utl_file.fopen('THE_DIR', 'file_name'||'.dot', 'w', 32760);
IF recFiles.len < 32760 THEN
utl_file.put_raw(l_output, recFiles.BLOB_VALUE);
utl_file.fflush(l_output);
ELSE -- write in pieces
vstart := 1;
WHILE vstart < recFiles.len
LOOP
dbms_lob.read(recFiles.BLOB_VALUE, bytelen, vstart, my_vr);
utl_file.put_raw(l_output, my_vr);
utl_file.fflush(l_output);
-- set the start position for the next cut
vstart := vstart + bytelen;
-- set the end position if less than 32000 bytes
x := x - bytelen;
IF x < 32000 THEN
bytelen := x;
END IF;
END LOOP;
END IF;
End Loop;
dbms_output.put_line('End');
utl_file.fclose(l_output);
END GetFile;
Why did you replace the prior version of the code with this version? If you already have a version of the code that is working to write a single BLOB to the file system, it's very easy to just call that code in a loop. It's also a better way of designing modular code.
When you get an error, please post the error stack. That will include the Oracle error number, the error message, and the line number of the error. Tell us what line that corresponds to in your code (particularly if there are formatting differences between what you post here and what code you actually run).
You cannot in a single thread copy every LOB to a file at the same time. A single thread can do one thing at a time so it can copy one file at a time. You can loop so that you copy each file sequentially. I'm still not clear whether that is what you want to do or if you really want to spawn 800 threads each of which writes a single LOB to the file system.
You need to close the file inside the loop since you open the file in the loop (note that keeping the old code would make it much easier to avoid this sort of error). And assuming that you want to use the file name from the table, you'd want to use recFiles.file_name in your call to fopen, not a hard-coded string 'file_name' which would try to write every LOB to the same physical file.
Given that, my guess is that you want something like this (note that it would still be better form to modularize this code but since you're trying to avoid that, I'll assume you have a good reason for that)
create or replace
PROCEDURE GetbFile
IS
l_output utl_file.file_type;
vstart NUMBER := 1;
bytelen NUMBER := 32000;
x NUMBER;
my_vr RAW(32000);
v_name VARCHAR2(32760);
BEGIN
FOR recFiles IN (SELECT dbms_lob.getlength(BLOB_VALUE) as len,
FILE_NAME,
BLOB_VALUE from Gfile)
LOOP
l_output := utl_file.fopen('THE_DIR', recFiles.file_name||'.dot', 'w', 32760);
IF recFiles.len < 32760 THEN
utl_file.put_raw(l_output, recFiles.BLOB_VALUE);
utl_file.fflush(l_output);
ELSE -- write in pieces
vstart := 1;
WHILE vstart < recFiles.len
LOOP
dbms_lob.read(recFiles.BLOB_VALUE, bytelen, vstart, my_vr);
utl_file.put_raw(l_output, my_vr);
utl_file.fflush(l_output);
-- set the start position for the next cut
vstart := vstart + bytelen;
-- set the end position if less than 32000 bytes
x := x - bytelen;
IF x < 32000 THEN
bytelen := x;
END IF;
END LOOP;
END IF;
utl_file.fclose(l_output);
End Loop;
dbms_output.put_line('End');
END GetFile;
The following stored procedure is intended to grab all the BLOB values from an Oracle database and save them into a folder called OraFolder.
It compiles fine but I have 2 questions.
1, there are 2 parameters, pname and display_name. I must admit that I don't know what they are there for because I just googled the code which seems to fit into our need.
My question is do I really need the 2 params given that we are trying to extract ALL BLOB values into a folder?
2, If your answer is yes, I do need them, how I do I use them?
Finally, there is an Entry_Id, I just kept getting an error that it is not declared. I had to remove it. What is it used for?
Sorr, I am not an Oracle guy, just trying to figure out a wa to fix a problem that is dropped on my laps.
Thanks in advance
Here is the complete stored proc.
create or replace PROCEDURE blob2file(pfname VARCHAR2, display_name in varchar2) IS
vblob BLOB;
vstart NUMBER := 1;
bytelen NUMBER := 32000;
len NUMBER;
my_vr RAW(32000);
x NUMBER;
v_name varchar2(100);
lv_str_len NUMBER;
l_output utl_file.file_type;
BEGIN
-- define output directory
lv_str_len := length(pfname);
--v_name := display_name||upper(substr(pfname,lv_str_len-3,lv_str_len));
v_name := display_name;
l_output := utl_file.fopen('MY_FOLDER', v_name, 'w', 32760);
-- get length of blob
SELECT dbms_lob.getlength(blob_content)
INTO len
FROM portal.WWDOC_DOCUMENT$
WHERE FILENAME = pfname;
-- dbms_output.put_line('Length: '||len);
-- save blob length
x := len;
-- select blob into variable
SELECT blob_content
INTO vblob
FROM portal.WWDOC_DOCUMENT$
WHERE FILENAME = pfname;
-- if small enough for a single write
IF len < 32760 THEN
-- dbms_output.put_line('Single write ');
utl_file.put_raw(l_output,vblob);
utl_file.fflush(l_output);
ELSE -- write in pieces
-- dbms_output.put_line('multi write '||vstart);
vstart := 1;
WHILE vstart < len
LOOP
dbms_lob.read(vblob,bytelen,vstart,my_vr);
utl_file.put_raw(l_output,my_vr);
utl_file.fflush(l_output);
-- set the start position for the next cut
vstart := vstart + bytelen;
-- set the end position if less than 32000 bytes
x := x - bytelen;
IF x < 32000 THEN
bytelen := x;
END IF;
END LOOP;
END IF;
dbms_output.put_line('End');
utl_file.fclose(l_output);
EXCEPTION
when others then dbms_output.put_line('ERROR:'||entry_id);
END blob2file;
The pfname param defines which file to grab, so you don't need that since you want them all.
The display_name param defines the output directory, so if you want to hard code the directory, you could.
Since you want all files, you'll need to loop through all records in the table and output them one at a time:
CREATE OR REPLACE PROCEDURE blob2file
IS
l_output utl_file.file_type;
vstart NUMBER := 1;
bytelen NUMBER := 32000;
x NUMBER;
my_vr RAW(32000);
BEGIN
FOR recFiles IN (SELECT dbms_lob.getlength(BLOB_CONTENT) as len,
FILENAME,
BLOB_CONTENT
FROM PORTAL.WWDOC_DOCUMENT$)
LOOP
l_output := utl_file.fopen('MY_FOLDER', '/hard code the path here/', 'w', 32760);
IF recFiles.len < 32760 THEN
utl_file.put_raw(l_output, recFiles.BLOB_CONTENT);
utl_file.fflush(l_output);
ELSE -- write in pieces
vstart := 1;
WHILE vstart < refFiles.len
LOOP
dbms_lob.read(recFiles.BLOB_CONTENT, bytelen, vstart, my_vr);
utl_file.put_raw(l_output, my_vr);
utl_file.fflush(l_output);
-- set the start position for the next cut
vstart := vstart + bytelen;
-- set the end position if less than 32000 bytes
x := x - bytelen;
IF x < 32000 THEN
bytelen := x;
END IF;
END LOOP;
END IF;
utl_file.fclose(l_output);
dbms_output.put_line('End');
EXCEPTION
WHEN OTHERS THEN
dbms_output.put_line('ERROR: ' || SQLERRM);
END blob2file;
Looks like vname is the name of the file which the blob is written to. And pname is the key of the blob in the table. So if you're dumping all blobs in the table then you don't need either of these, but you will need to come up with a unique filename for each blob.