Using Powershell to export big output from Oracle to CSV - oracle

Question

I need to export an quite a big CSV file from Oracle once a week.
I tried two approaches.
Adapter.fill(dataset)
Looping through columns and rows to save into a CSV file one line at a time.
The first one is running out of memory when running (the server machine has only 4 GB of RAM), the second one takes about an hour as there are over 4 milion rows to export.
Here's code #1:
#Your query. It cannot contain any double quotes otherwise it will break.
$query = "SELECT manycolumns FROM somequery"
#Oracle login credentials and other variables
$username = "username"
$password = "password"
$datasource = "database address"
$output = "\\NetworkLocation\Sales.csv"
#creates a blank CSV file and make sure it's in ASCI
Out-File $output -Force ascii
#This here will look for "Oracle.ManagedDataAccess.dll" file inside "C:\Oracle" folder. We usually have two versions of Oracle installed so the adaptor can be in different locations. Needs changing if the Oracle is installed elsewhere.
$location = Get-ChildItem -Path C:\Oracle -Filter Oracle.ManagedDataAccess.dll -Recurse -ErrorAction SilentlyContinue -Force
#Establishes connection to Oracle using the DLL file
Add-Type -Path $location.FullName
$connectionString = 'User Id=' + $username + ';Password=' + $password + ';Data Source=' + $datasource
$connection = New-Object Oracle.ManagedDataAccess.Client.OracleConnection($connectionString)
$connection.open()
$command=$connection.CreateCommand()
$command.CommandText=$query
#Creates a table in memory and fills it with results from the query. Then, export the virtual table into CSV.
$DataSet = New-Object System.Data.DataSet
$Adapter = New-Object Oracle.ManagedDataAccess.Client.OracleDataAdapter($command)
$Adapter.Fill($DataSet)
$DataSet.Tables[0] | Export-Csv $output -NoTypeInformation
$connection.Close()
And here's #2
#Your query. It cannot contain any double quotes otherwise it will break.
$query = "SELECT manycolumns FROM somequery"
#Oracle login credentials and other variables
$username = "username"
$password = "password"
$datasource = "database address"
$output = "\\NetworkLocation\Sales.csv"
$tempfile = $env:TEMP + "\Temp.csv"
#creates a blank CSV file and make sure it's in ASCI
Out-File $tempfile -Force ascii
#This here will look for "Oracle.ManagedDataAccess.dll" file inside "C:\Oracle" folder. Needs changing if the Oracle is installed elsewhere.
$location = Get-ChildItem -Path C:\Oracle -Filter Oracle.ManagedDataAccess.dll -Recurse -ErrorAction SilentlyContinue -Force
#Establishes connection to Oracle using the DLL file
Add-Type -Path $location.FullName
$connectionString = 'User Id=' + $username + ';Password=' + $password + ';Data Source=' + $datasource
$connection = New-Object Oracle.ManagedDataAccess.Client.OracleConnection($connectionString)
$connection.open()
$command=$connection.CreateCommand()
$command.CommandText=$query
#Reads results column by column. This way you don't have to specify how many columns it has.
$reader = $command.ExecuteReader()
while($reader.Read()) {
$props = #{}
for($i = 0; $i -lt $reader.FieldCount; $i+=1) {
$name = $reader.GetName($i)
$value = $reader.item($i)
$props.Add($name, $value)
}
#Exports each line to CSV file. Works best when the file is on local drive as it saves it after each line.
new-object PSObject -Property $props | Export-Csv $tempfile -NoTypeInformation -Append
}
Move-Item $tempfile $output -Force
$connection.Close()
Ideally, I would like to use the first code as it is way faster than the second one but to somehow avoid running out of memory.
Do you guys and gals know if there's some way to "fill" first 1 milion records, append them to CSV, clean the "DataSet" table, next 1 milion etc? After the code finishes running the CSV weights ~1.3 GB but when it runs, even 8 GB of Memory is not enough for it (my laptop has 8 but server only has 4 GB and it really hits it hard).
Any tips will be appreciated.

Answer 1

In the *nix community we love one-liners!
You can set markup to 'csv on' in sqlplus (>= 12)
Create the query file
cat > query.sql <<EOF
set head off
set feed off
set timing off
set trimspool on
set term off
spool output.csv
select
object_id,
owner,
object_name,
object_type,
status,
created,
last_ddl_time
from dba_objects;
spool off
exit;
EOF
Spool the output.csv like this:
sqlplus -s -m "CSV ON DELIM ',' QUOTE ON" user/password#\"localhost:1521/<my_service>\" #query.sql
Another option is SQLcl (the SQL Developer CLI tool. Binary name: 'sql' renamed by me to 'sqlcl')
Create the query file (Note! term on|off)
cat > query.sql <<EOF
set head off
set feed off
set timing off
set term off
set trimspool on
set sqlformat csv
spool output.csv
select
object_id,
owner,
object_name,
object_type,
status,
created,
last_ddl_time
from dba_objects
where rownum < 5;
spool off
exit;
EOF
Spool the output.csv like this:
sqlcl -s system/oracle#\"localhost:1521/XEPDB1\" #query.sql
Viola!
cat output.csv
9,"SYS","I_FILE#_BLOCK#","INDEX","VALID",18.10.2018 07:49:04,18.10.2018 07:49:04
38,"SYS","I_OBJ3","INDEX","VALID",18.10.2018 07:49:04,18.10.2018 07:49:04
45,"SYS","I_TS1","INDEX","VALID",18.10.2018 07:49:04,18.10.2018 07:49:04
51,"SYS","I_CON1","INDEX","VALID",18.10.2018 07:49:04,18.10.2018 07:49:04
And the winner is sqlplus for 77k rows! (removed filter rownum < 5)
time sqlcl -s system/oracle#\"localhost:1521/XEPDB1\" #query.sql
real 0m23.776s
user 0m39.542s
sys 0m1.293s
time sqlplus -s -m "CSV ON DELIM ',' QUOTE ON" system/oracle#localhost/XEPDB1 #query.sql
real 0m3.066s
user 0m0.700s
sys 0m0.265s
wc -l output.csv
77480 output.csv
You can experiment with formats in SQL Developer.
select /*CSV|HTML|JSON|TEXT|<TONSOFOTHERFORMATS>*/ from dba_objects;
If you are loading CSV into the database, this tool will do it!
https://github.com/csv2db/csv2db
Best of luck!

Answer 2

Thank you all for the responses, I learned about Oracle scripts and sql*plus that I never knew existed. I probably will use them in the future but I guess I will have to update my Oracle Developer package.
I have found a way to edit my code to work using the documentation here:
https://docs.oracle.com/database/121/ODPNT/OracleDataAdapterClass.htm#i1002865
It's not perfect as it pauses every 1 milion rows, saves the output and re-run the query which re-evaluates it (the one I'm running takes about 1-2 minutes to evaluate).
It's basically the same as running one code x times (where x is a ceiling of number of rows in milions) doing "fetch first 1'000'000 rows only" then "Offset 1'000'00 rows Fetch next 1'000'000 rows only" etc. and saving it into CSV appending at the bottom.
Here's the code:
#Your query. It cannot contain any double quotes otherwise it will break.
$query = "SELECT
A lot of columns
FROM
a lot of tables joined together
WHERE
a lot of conditions
"
#Oracle login credentials and other variables
$username = myusername
$password = mypassword
$datasource = TNSnameofmyDatasource
$output = "$env:USERPROFILE\desktop\Sales.csv"
#creates a blank CSV file and make sure it's in ASCII as that's what the output of my query is
Out-File $output -Force ascii
#This here will look for "Oracle.ManagedDataAccess.dll" file inside "C:\Oracle" folder. Needs changing if the Oracle is installed elsewhere.
$location = Get-ChildItem -Path C:\Oracle -Filter Oracle.ManagedDataAccess.dll -Recurse -ErrorAction SilentlyContinue -Force
#Establishes connection to Oracle using the DLL file
Add-Type -Path $location.FullName
$connectionString = 'User Id=' + $username + ';Password=' + $password + ';Data Source=' + $datasource
$connection = New-Object Oracle.ManagedDataAccess.Client.OracleConnection($connectionString)
$connection.open()
$command=$connection.CreateCommand()
$command.CommandText=$query
#Creates a table in memory to be filled up with results from the query using ODAC
$DataSet = New-Object System.Data.DataSet
$Adapter = New-Object Oracle.ManagedDataAccess.Client.OracleDataAdapter($command)
#Declaring variables for the loop
$fromrecord = 0
$numberofrecords = 1000000
$timesrun = 0
#Loop as long as the number of Rows in the virtual table are equal to specified $numberofrecords
while(($timesrun -eq 0) -or ($DataSet.Tables[0].Rows.Count -eq $numberofrecords))
{
$DataSet.Clear()
$Adapter.Fill($DataSet,$fromrecord,$numberofrecords,'*') | Out-Null #Suppresses writing to console the number of rows filled
Write-progress "Saved: $fromrecord Rows"
$DataSet.Tables[0] | Export-Csv $output -Append -NoTypeInformation
$fromrecord=$fromrecord+$numberofrecords
$timesrun++
}
$connection.Close()