I can't process to deleting my table. I get this error:
"ERROR: File SASUSER.MCO.DATA is not a SAS data set."
I've tried many ways to delete but neither works.
Thanks for felp!
tested with (proc delete / proc sql drop / %deltable )
I have used the bellow codes:
proc sql; drop table sasuser.MCO; quit;
%deltable (tables=sasuser.MCO)
proc datasets nolist lib=sasuser; delete MCO ; quit;
log and result of proc datasets lib=sasuser; run;
this is the log:
here
and the result is here
From your error message it seems that the file is NOT an actual SAS dataset. I have never seen a SAS dataset on Unix that as only one thousand bytes long, even an empty dataset is normally more like 14K, depending on the default block size that SAS uses to create the files.
So just use the operating system to delete the file. The name of the file should be mco.sas7bdat and it should be in the directory that the SASUSER libref is pointing to. So if you have XCMD option active you could just use code like this:
x "rm %sysfunc(pathname(sasuser))/mco.sas7bdat";
If XCMD is not active then you will need to use the FDELETE() function instead.
Related
I want to use an external table to load a csv file as it's very convenient, but the problem is how do i make sure i don't load the same file twice in a row? i can't validate the data loaded because it can be the same information as before; i need to find a way to make sure the user doesnt load the same file as 2h ago for example.
I thought about uploading the file with a different name each time and issuing an alter table command to change the name of the file in the definition of the external table, but it sounds kinda risky.
I also thought about marking each row in the file with a sequence to help differentiate files, but i doubt the client would accept it as they would need to manually do this (the file is exported from somewhere).
Is there any better way to make sure i don't load the same file in the external table except changing the file's name and executing an alter on the table?
Thank you
when you bring the data from external table to your database you can use MERGE command instead of insert. it let you don't worry about duplicate data
see the blog about The Oracle Merge Command
What's more, we can wrap up the whole transformation process into this
one Oracle MERGE command, referencing the external table and the table
function in the one command as the source for the MERGED Oracle data.
alter session enable parallel dml;
merge /*+ parallel(contract_dim,10) append */
into contract_dim d
using TABLE(trx.go(
CURSOR(select /*+ parallel(contracts_file,10) full (contracts_file) */ *
from contracts_file ))) f
on d.contract_id = f.contract_id
when matched then
update set desc = f.desc,
init_val_loc_curr = f.init_val_loc_curr,
init_val_adj_amt = f.init_val_adj_amt
when not matched then
insert values ( f.contract_id,
f.desc,
f.init_val_loc_curr,
f.init_val_adj_amt);
So there we have it - our complex ETL function all contained within a
single Oracle MERGE statement. No separate SQL*Loader phase, no
staging tables, and all piped through and loaded in parallel
I can only think of a solution somewhat like this:
Have a timestamp encoded in the datafile name (like: YYYYMMDDHHMISS-file.csv), where YYYYMMDDHHMISS is the timestamp.
Create a table with the fields timestamp (as above).
Create a shell scripts that:
extracts the timestamp from the datafilename.
calls an sqlscript with the timestamp as the parameter, and return 0 if that timestamp does not exist, <>0 if the timestamp already exist, and in that case exit the script with the error: File: YYYYMMDDHHMISS-file.csv already loaded.
copy the YYYYMMDDDHHMISS-file.csv to input-file.csv.
run the sql loader script that loads the input-file.csv file
when succes: run a second sql script with parameter timestamp that inserts the record in the database to indicate that the file is loaded and move the original file to a backup folder.
when failure: report the failure of the load script.
I have a dozen web servers each writing data to a log file. At the beginning of each hour, the data from the previous hour is loaded to hive using a cron script running the command:
hive -e "LOAD DATA LOCAL INPATH 'myfile.log' INTO TABLE my_table PARTITION(dt='2015-08-17-05')"
In some cases, the command fails and exits with a code other than 0, in which case our script awaits and tries again. The problem is, in some cases of failure, the data loading does not fail, even though it shows a failure message. How can I know for sure whether or not the data has been loaded?
Example for such a "failure" where the data is loaded:
Loading data to table default.my_table partition (dt=2015-08-17-05)
Failed with exception
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter
partition. FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
Edit:
Alternatively, is there a way to query hive for the filenames loaded into it? I can use DESCRIBE to see the number of files. Can I know their names?
About "which files have been loaded in a partition":
if you had used an EXTERNAL TABLE and just uploaded your raw data
file in the HDFS directory mapped to LOCATION, then you could
(a) just run a hdfs dfs -ls on that directory from command line (or use the equivalent Java API call)
(b) run a Hive query such as select distinct INPUT__FILE__NAME from (...)
but in your case, you copy the data into a "managed" table, so there
is no way to retrieve the data lineage (i.e. which log file was used
to create each managed datafile)
...unless you add explicitly the original file name inside the log file, of
course (either on "special" header record, or at the beginning of each record - which can be done with good old sed)
About "how to automagically avoid duplication on INSERT": there is a way, but it would require quite a bit of re-engineering, and would cost you in terms of processing time /(extra Map step plus MapJoin)/...
map your log file to an EXTERNAL TABLE so that you can run an
INSERT-SELECT query
upload the original file name into your managed table using INPUT__FILE__NAME pseudo-column as source
add a WHERE NOT EXISTS clause w/ correlated sub-query, so that if the source file name is already present in target then you load nothing more
INSERT INTO TABLE Target
SELECT ColA, ColB, ColC, INPUT__FILE__NAME AS SrcFileName
FROM Source src
WHERE NOT EXISTS
(SELECT DISTINCT 1
FROM Target trg
WHERE trg.SrcFileName =src.INPUT__FILE__NAME
)
Note the silly DISTINCT that is actually required to avoid blowing away the RAM in your Mappers; it would be useless with a mature DBMS like Oracle, but the Hive optimizer is still rather crude...
I don't believe you can simply do this is in Hadoop/Hive. So here are the basics of an implementation in python:
import subprocess
x=subprocess.check_output([hive -e "select count(*) from my_table where dt='2015-08-17-05'"])
print type(x)
print x
But you have to spend some time working with backslashes to get hive -e to work using python. It can be very difficult. It may be easier to write a file with that simple query in it first, and then use hive -f filename. Then, print the output of subprocess.check_output in order to see how the output is stored. You may need to do some regex or type conversions, but I think it should just come back as a string. Then simply use an if statement:
if x > 0:
pass
else:
hive -e "LOAD DATA LOCAL INPATH 'myfile.log' INTO TABLE my_table PARTITION(dt='2015-08-17-05')"
I want to upload some data from UAT DB to DEV DB. When I try to do this from Export function in SQL Developer, I got an error File C:\Users\xxx\export.sql was not opened because it exceeds the maximum automatic open size
How can I copy the UAT data to DEV ?
ORACLE Version 12C
SQL Developer Version 4.0.0.13
found the below answer from a SQL Developer forum :
It appears that the "maximum automatic open size" is hard-coded to a value of 500000 (bytes, I believe) with no way to override it. By
limiting this, we nip in the bud any potential complaints of Java
OutOfMemory upon trying to open a huge file.
To view the file from within SQL Developer despite this limitation,
just use the File|Open menu. For those huge files, please use an
external editor. And if you don't want to open files automatically in
order to suppress the warning dialog, use
Tools|Preferences|Database|Export/View DDL Options and un-check the
"Open Sql File When Exported" box.
Are you certain the export file does not contain all the insert rows?
That would be a bug unless you hit an OutOfMemory or disk full
condition. I just tried your scenario on at 55000 row table that
produced an export.sql of about 20MB. All rows were included.
Regards,
Gary Graham
SQL Developer Team
and as the summary, it suggested that the SQL developer is not the best tool to open a large size of data file.
hope Gary's answer will guide you to some extent.
If you need to get an idea of some tools that you can open large files, check this LINK
Solution 1:
Set these values to some higher value!
Solution 2:
change "save to" to worksheet!
I was having this error when exporting database in insert format, selecting loader format on the 1st Export wizard screen fixed the issue.
This is probably because insert format creates a single SQL script with DDL and data as insert statements. So all the database is dumped in a single script file.
loader option produces multiple files: control file, data file, and sql files. And there are separate files for each table. As a result the export will consist of hundreds of files and no one file will reach the size limit.
This may not however work with single tables with very large amounts of data as that table's data file would hit the limit.
You can try different options like below.
On SQL developer, when right click on Table and click export, export wizard will be launched you can select either "Save As" - "separate files" that will export data in same SQL file.
OR you can change the format type on the same wizard to CSV that will export data in CSV format.
If you want to transfer large amounts of data (or small amounts, too) from one database to another, you should consider the tools that were specifically designed for such tasks.
First and foremost, look into data pump. It has a bit of a learning curve, though.
exp and imp (also by Oracle) are a bit easier to handle, but they're older and not nearly as powerful as data pump.
You might also want to look into the SQL*Plus copy command.
There is a trick to copy large chunk of data (from SQL developer) into excel sheet.
steps to be followed : Right click ---> export data ----> select format type as 'Text' ---> select type as "Clipboard" ----> open an excel sheet and try to paste keeping the below in mind :)
Then paste the data
NOTE : **Do Not paste the data on the first cell of the excel. Ctrl+v in any of the columns **
This will work.
Thanks
You can use spool the query and save the results as CSV or XLSX files for larger results. For example:
spool "D:\Temp\Report.csv"
SELECT /*csv*/ select id,name,age from EMP;
spool off;
1-You can create a database link (db link) on DEV DB pointing to UAT DB, to INSERT rows in DEV DB.
2-Or you can build in PL/SQL a procedure in UAT DB to export data to a file in CSV format and in DEV DB use oracle external tables to SELECT from that files.
Be carefull about DATE acolumns, write down using TO_CHAR.
3-Use Datapump to export data from UAT DB and then import into DEV DB; it's a bit tricky.
Oracle database commands can run both in SqlCl by Oracle and in SQL developer, so this is easy:
set feedback only -- for Oracle 12.2+, turn off terminal output
set sqlformat insert -- data in "insert into ..." format
-- set sqlformat csv -- data in csv format
spool /path/to/your/file.sql
select * from t; -- lines to export
spool off
set feedback off -- restore terminal output
Simplest way to this is to modify the "Save As" below in the screenshot to save to multiple files instead of single file while exporting-
I have a table "Images" with two fields:
Name VARCHAR2
Data BLOB
I would like to export that table to a .sql file which I could import on another system. I tried to do so using the "Database unload" assistant of Oracle SQL Developer. However the generated file does just have the content for the names in it but not the data. Thus after importing I would end up with all the names but the data field would be null everywhere.
I'd really prefer it just to be one file (I saw some examples that included dumping the data to one file per field on the fs...)
Is it possible to generate such a script with SQL Developer? or is there any other way/tool to do so?
I don't think this is possible with SQL Developer (but then I don't use it very often).
The SQL client I am using - SQL Workbench/J - can do this.
There are several ways to export this data.
Generate a proprietary script
It can create a SQL script that uses a special (tool specific) notation to reference an external file, something like:
INSERT INTO images
(name, data)
VALUES
('foobar', {$blobfile='blob_r1_c2.data'});
The above statement can only be executed with SQL Workbench again. It is not compatible with any other SQL client.
Use utl_raw
Another alternative is to use a "blob literal", but due to Oracle's limit on 4000 bytes for a character literal, this only works for really small blob values:
INSERT INTO images
(name, data)
VALUES
('foobar', to_blob(utl_raw.cast_to_raw('......')));
where the character literal for the cast_to_raw call would contain the hex values of the BLOB. As this requires 2 characters per "blob byte", you can't handle BLOBs larger than 2000 bytes with that. But that syntax would work for nearly all Oracle SQL tools (if they can handle scripts with very long lines).
SQL*Loader input file
The third alternative is to export the data into a text file that can be imported using SQL*Loader:
The text file would contain something like this:
NAME DATA
foobar blob_r1_c2.data
Together with the following SQL*Loader control file:
OPTIONS (skip=1)
LOAD DATA CHARACTERSET 'WE8ISO8859P15'
INFILE 'images.txt'
APPEND
INTO TABLE IMAGES
FIELDS TERMINATED BY '\t' TRAILING NULLCOLS
(
NAME,
lob_file_data FILLER,
DATA LOBFILE(lob_file_data) TERMINATED BY EOF
)
This can be loaded using SQL*Loader and is thus doesn't need SQL Workbench to import the data.
More details are in the manual
Edit
As Alex has pointed out in his comment, you can also use a DataPump export - but that requires that you have access to the file system on the server. The above solutions all store the data on the client.
If you absolutely need to use a single .sql file to import the BLOB you can generate the script using PL/SQL:
set serveroutput on
declare
lob_in blob;
i integer := 0;
lob_size integer;
buffer_size integer := 1000;
buffer raw(32767);
begin
select
data, dbms_lob.getlength(data)
into lob_in, lob_size
from images
where name = 'example.png';
for i in 0 .. (lob_size / buffer_size) loop
buffer := dbms_lob.substr(lob_in, buffer_size, i * buffer_size + 1);
dbms_output.put('dbms_lob.append(lob_out, hextoraw(''');
dbms_output.put(rawtohex(buffer));
dbms_output.put_line('''));');
end loop;
end;
Its output will be the BLOB's content encoded like:
dbms_lob.append(lob_out, hextoraw('FFD8FFE0...0000'));
dbms_lob.append(lob_out, hextoraw('00000000...0000'));
...
dbms_lob.append(lob_out, hextoraw('007FFFD9'));
Which you can load into an already inserted row with PL/SQL:
declare
lob_out blob;
begin
select data into lob_out
from images
where name = 'example.png'
for update;
dbms_lob.append(lob_out, hextoraw('FFD8FFE0...0000'));
dbms_lob.append(lob_out, hextoraw('00000000...0000'));
...
dbms_lob.append(lob_out, hextoraw('007FFFD9'));
end;
Just remember the resulting .sql file will be huge.
Thx for your answer. I used the third alternative.
First I downloaded SQL Workbench/J. Then I used the following command to make an export:
WbExport -type=text -file='c:\temp\Images' delimiter='|' -decimal=',' -sourcetable=Images -formatfile=oracle;
This produced a Images.txt file and many Images_r*_c2.data files and a Images.ctl file.
I could then use the following command to import:
sqlldr myuser#myhost/mypassword control=Images.ctl
This is definitely possible in SQL developer.
First you need to export the table in the source location choosing
appropriate table(s).
Tools > Database Export
Select output format as loader rather than insert , excel which we
normally use.
Following these steps would create sqlldr control files and data files and also the create table ddl if you chose the option.You can use them to import(sqlldr) data in the destination.
This is a better solution and is portable in terms of extraction and distribution . It gives the flexibility of delivering components to be deployed through code repositories.
Here is a link that explains it step by step.
Exporting Multiple BLOBs with Oracle SQL Developer
SQL workbench uses a special file format for blob data, in addition to .sql. If you can accept such files, an even simpler solution is to use Oracle's Original import and export. (It is deprecated, but unlike Oracle's DataPump, it does not require access rights on the server.)
Here is a nice tutorial on the export part.
I'm running a pretty large query against Oracle database using SAS for windows 9.2. This query is pretty large where in I wrote a sub-query in WITH clause and used it 4 times. This runs fine on SQL PLUS and SQL Developer, but when i run it using SAS, the program hangs up after 20 mins and I can't even see the log window. I have never worked with SAS and not sure how to proceed but tried following option:
I created a SAS code file and ran it from windows batch file hoping to get log written to windows file system, but even this runs in-definitely and I don't see anything written to log file
Can some one direct me here. How can i use ALTLOG command to get log file written to windows file system so that i can understand the exact error message. By the way DBA's have mentioned that query runs fine and rows are returned from server side, but for some reason SAS program is not able to show this data. I get about 45,000 records from the query.
Thanks
I'll break it into two points:
1) running an existing Oracle SQL query in SAS without ever using SAS:
best way for you is to embed your Oracle SQL code in so called PROC SQL explicit pass-through:
proc sql;
connect to oracle as db1 (user=user1 pw=pasw1 path=DB1);
create table test_table as
select *
from connection to db1
( /* here we're in oracle */
select * from test.table1 where rownum <20
)
;
disconnect from db1;
quit;
(borrowed from my answer to another question Limiting results in PROC SQL)
The point is not to try to translate it to SAS SQL (don't know if you tried or not).
Also make sure you're creating a SAS table (as in the example) from query result, not writing it to SAS OUTPUT window.
2) Regarding getting the log: the log about an action is in general written once it's done, so if the query is really running for a long time, you won't see any intermediate logs.
Anyway, log buffering is the default setting for batch jobs, so log messages are written after the buffer is full.
To get log messages written immediately to the log file set LOGPARM option:
-LOGPARM= “WRITE=IMMEDIATE”
the opposite option is BUFFERED.
To find out the config file(s) used run following in your SAS session:
proc options option=config;run;
Then enter the option above on separate line in the config file.