How do i split each row using powershell - oracle

I'm trying to insert an table from 1 database to another (rows) using powershell. I already know how to get the data out but when i try to insert it into a new table I get an error that the string is too long (I think it means that i am trying to put all the data into an new row). I already searched the web for an answer. The only thing I could find is a command -split but then i need something to split with and the data is only numbers.
below you will find the script.
$query = “SELECT ZAMNIEUW.MACHINENAME FROM ZAMNIEUW WHERE MACHINENAME not in (SELECT PATCHGROEPEN.MACHINENAME FROM PATCHGROEPEN ) "
$command = $con.CreateCommand()
$command.CommandText = $query
$result = $command.ExecuteReader()
$table = new-object “System.Data.DataTable”
$table.Load($result)
$format = #{Expression={$_.WAARDE_PCNAME};Label=”ID”;width=20}
echo $table.MACHINENAME
$table| foreach {
$qu= " INSERT into TESTZEN (MACHINENAME) VALUES('$table')"
$cmd = $con.CreateCommand()
$cmd.CommandText = $qu
$cmd.ExecuteNonQuery() |out-null
}

apperantly oracle does use something to split each row (;)
the code below
$query = “SELECT ZAMNIEUW.MACHINENAME FROM ZAMNIEUW WHERE MACHINENAME not in (SELECT PATCHGROEPEN.MACHINENAME FROM PATCHGROEPEN ) "
$command = $con.CreateCommand()
$command.CommandText = $query
$result = $command.ExecuteReader()
$table = new-object “System.Data.DataTable”
$table.Load($result)
$format = #{Expression={$_.WAARDE_PCNAME};Label=”ID”;width=20}
$tables =echo $table.MACHINENAME
$tables| foreach {
$tables = $_ -split ';'
$qu= " INSERT into TESTZEN (MACHINENAME) VALUES('$tables')"
$cmd = $con.CreateCommand()
$cmd.CommandText = $qu
$cmd.ExecuteNonQuery() |out-null
}

Related

Powershell - How to export to CSV file a query with Oracle Data Provider for .NET (ODP.NET)

In Powershell, I want to export the results of a query ODP.NET to a CSV file
This is my code:
$connection = New-Object Oracle.ManagedDataAccess.Client.OracleConnection($connectionString)
$connection.open()
$cmd = New-Object Oracle.ManagedDataAccess.Client.OracleCommand -ArgumentList $query
$cmd.Connection = $connection
$reader = $cmd.ExecuteReader()
This code is working with the correct values of $connectionString and $query variables. If I add this code I can read the correct result of my query:
while ($reader.Read()) {
$col1 = $reader["Col1"]
$col2 = $reader["Col2"]
$col3 = $reader["Col3"]
#Write-Host $col1, $col2, $col3
}
Those 3 columns are an example, in my real case I have many more. Then I want to export directly to a CSV file, something like this:
XXXXXXX | export-csv -Delimiter ";" -Path "E:\export.csv"
How can I do this? The expected result is a CSV file similar than this:
"4581";"6";26867;"191057";"BH02 - 26867 - ";"30/05/2019";"";"2040";1991,04;"2040";2,4;"00";"";348;"";"1";"1";"";"";"BRL";2040;"";1
"4581";"4";28313;"747990";"BH02 - 28313 - ";"30/05/2019";"";"140";137,13;"140";2,05;"00";"";459;"";"1";"1";"";"";"BRL";140;"";1
"4581";"1";28316;"881411";"BH02 - 28316 - ";"30/05/2019";"";"140";137,13;"140";2,05;"00";"";460;"";"1";"1";"";"";"BRL";140;"";1
"4581";"1";;"878676";"BH02 - - 275650/PF19";"28/05/2019";"";"103";100,8885;"103";2,05;"00";"";305;"";"1";"1";"";"";"BRL";103;"";1
"4581";"6";28168;"006778";"BH02 - 28168 - 275714/PF19";"30/05/2019";"";"848";828,92;"848";2,25;"00";"";429;"";"1";"1";"";"";"BRL";848;"";1
"4581";"3";29080;"641559";"BH02 - 29080 - ";"30/05/2019";"";"3424,14";3338,5365;"3424,14";2,5;"00";"";488;"";"1";"1";"";"";"BRL";3424,14;"";1
"4581";"4";;"602483";"BH02 - - 23443";"28/05/2019";"";"157";153,7815;"157";2,05;"00";"";329;"";"1";"1";"";"";"BRL";157;"";1
What is the raw output of this Oracle command?
If it is a list, then use -Join and use the Csv cmdlets to export to convert to a csv file.
This also would indicate that you are new or never used PowerShell before since csv use case is a daily thing. That's OK, but jump on YouTube, MSDN Channel9 and go through the videos on PowerShell, beginning/intermediate/advanced, PowerShell and Databases, PowerShell and Csv files and get ramped up to limit/avoid, misconceptions, frustrations, bad habits, etc.
It should be straightforward as doing something like this ( of course don't put passwords in scripts)...
Add-Type -Path 'C:\oracle\instantclient_10_2\odp.net\managed\common\Oracle.ManagedDataAccess.dll'
$username = "USERID"
$password = "Password"
$datasource = "HOST:PORT/Instance"
$connectionString = "User Id = $username;Password = $password;Data Source = $datasource"
$query = "SELECT DATA1, DATA2, DATA3, DATA4, DATA5, DATA6, DATA7, DATA8
FROM TABLE WHERE NOT REGEXP_LIKE (EMAIL_ID, '#domain.com','i') order by DATA2"
$connection = New-Object Oracle.ManagedDataAccess.Client.OracleConnection("$connectionString")
$connection.open()
$command = New-Object Oracle.ManagedDataAccess.Client.OracleCommand
$command.Connection = $connection
$command.CommandText = $query
$ds = New-Object system.Data.DataSet
$da = New-Object Oracle.ManagedDataAccess.Client.OracleDataAdapter($command)
[void]$da.fill($ds)
return $ds.Tables[0] |
SELECT DATA1, DATA2, DATA3, DATA4, DATA5, DATA6, DATA7, DATA8 |
Export-CSV "C:\test.csv" -NoTypeInformation
$connection.Close()

Faster iteration

I have this code, which is part of a function that returns a list of SQL rows based on a time range.
The query itself (1st line of code) is quite fast. But the foreach loop that extract the relevant data takes a while to complete.
I have around 350.000 lines to iterate, and despite it's has to take a while, I was wondering if there is any change I could make in order to make it faster.
$SqlDocmasterTableResuls = $this.SqlConnection.GetSqlData("SELECT DOCNUM, DOCLOC FROM MHGROUP.DOCMASTER WHERE ENTRYWHEN between '" + $this.FromDate + "' and '" + $this.ToDate + "'")
[System.Collections.ArrayList]$ListOfDocuments = [System.Collections.ArrayList]::New()
if ($SqlDocmasterTableResuls.Rows.Count)
{
foreach ($Row in $SqlDocmasterTableResuls.Rows)
{
$DocProperties = #{
"DOCNUM" = $Row.DOCNUM
"SOURCE" = $Row.DOCLOC
"DESTINATION" = $Row.DOCLOC -replace ([regex]::Escape($this.iManSourceFileServerName + ":" + $this.iManSourceFileServerPath.ROOTPATH)),
([regex]::Escape($this.iManDestinationFileServerName + ":" + $this.iManDestinationFileServerPath.ROOTPATH))
}
$DocObj = New-Object -TypeName PSObject -Property $DocProperties
$ListOfDocuments.Add($DocObj)
}
return $ListOfDocuments
Avoid appending to an array in a loop. The best way to capture loop data in a variable is to simply collect the loop output in a variable:
$ListOfDocuments = foreach ($Row in $SqlDocmasterTableResuls.Rows) {
New-Object -Type PSObject -Property #{
"DOCNUM" = $Row.DOCNUM
"SOURCE" = $Row.DOCLOC
"DESTINATION" = $Row.DOCLOC -replace ...
}
}
You don't need the surrounding if conditional, because if the table doesn't have any rows the loop should skip right over it, leaving you with an empty result.
Since you want to return the list anyway, you don't even need to collect the loop output in a variable. Just leave the output as it is and it will get returned anyway.
Also avoid repeating operations in a loop when their result doesn't change. Calculate the escaped source and destination paths once before the loop:
$srcPath = [regex]::Escape($this.iManSourceFileServerName + ':' + $this.iManSourceFileServerPath.ROOTPATH)
$dstPath = [regex]::Escape($this.iManDestinationFileServerName + ':' + $this.iManDestinationFileServerPath.ROOTPATH)
and use the variables $srcPath and $dstPath inside the loop.
Something like this should do:
$SqlDocmasterTableResuls = $this.SqlConnection.GetSqlData("SELECT ...")
$srcPath = [regex]::Escape($this.iManSourceFileServerName + ':' + $this.iManSourceFileServerPath.ROOTPATH)
$dstPath = [regex]::Escape($this.iManDestinationFileServerName + ':' + $this.iManDestinationFileServerPath.ROOTPATH)
foreach ($Row in $SqlDocmasterTableResuls.Rows) {
New-Object -Type PSObject -Property #{
'DOCNUM' = $Row.DOCNUM
'SOURCE' = $Row.DOCLOC
'DESTINATION' = $Row.DOCLOC -replace $srcPath, $dstPath
}
}
return
[edit - per Ansgar Wiechers, the PSCO accelerator is only available with ps3+.]
one other thing that may help is to replace New-Object with [PSCustomObject]. that is usually somewhat faster to use. something like this ...
$DocObj = [PSCustomObject]$DocProperties
another way to use that type accelerator is to do what Ansgar Wiechers did in his code sample, but use the accelerator instead of the cmdlet. like this ...
[PSCustomObject]#{
'DOCNUM' = $Row.DOCNUM
'SOURCE' = $Row.DOCLOC
'DESTINATION' = $Row.DOCLOC -replace $srcPath, $dstPath
}
hope that helps,
lee

Can't export more than 2 lines to the csv

[void][System.Reflection.Assembly]::LoadWithPartialName("Oracle.DataAccess")
$username = "USER"
$password = "PW"
$datasource = "XXX.XXX.XXX.XXX:YYYY/NODE1"
function getConnectionString()
{
#Below is a one line code
$connectionstring = "Data Source=$datasource;User Id=$($username); Password=$($password)"
return $connectionstring
}
$query = get-content "C:\Test.sql";
$connectionstring = getConnectionString
$DataTable = New-Object System.Data.DataTable
$command = New-Object Oracle.DataAccess.Client.OracleCommand($query,$connectionstring)
$dataAdapter = New-Object Oracle.DataAccess.Client.OracleDataAdapter($command)
[void]$dataAdapter.Fill($DataTable)
#To display data on console
$DataTable | format-table -auto
#To export to csv
$DataTable | Export-Csv "C:\Test.csv" -notype
The "Test.sql contains:
SELECT ACCOUNTNUMBER, ACCOUNTID, USERID FROM account
WHERE ACCOUNTNUMBERLIKE '%0011111%'
That quesy brings back 3 rows.
Issue at hand: My powershell script only gives me the first 2 rows (1 being the headers, the other being the first data row)
Any advice?

Powershell API Post Variable to Ducksboard

Trying to use the following Powershell script which I've taken from Github to push data into a Ducksboard dashboard. The function works excellently however I need to feed in a variable as part of the data. Of the two calls to the function included below the function which pushes in the actual value 44 works fine, however if I substitute it for my variable $qtybord the function falls over. I've tried a number of options to overcome the var being within the single quotes but cannot get it to work - can anyone help me?
# Squirt data to Duscksboard
function Execute-DucksboardApi
{
param(
[string] $url = $null,
[string] $data = $null,
[string] $apikey = $null,
[string] $contentType = "application/json",
[string] $codePageName = "UTF-8",
[string] $userAgent = $null
);
if ($url -and $data -and $apikey)
{
[System.Net.WebRequest]$webRequest = [System.Net.WebRequest]::Create($url);
$webRequest.ServicePoint.Expect100Continue = $false;
[System.Net.NetworkCredential]$credentials = New-Object System.Net.NetworkCredential($apikey, 'ignored');
$webRequest.Credentials = $credentials.GetCredential($url, 'Basic');
$webRequest.PreAuthenticate = $true;
$webRequest.ContentType = $contentType;
$webRequest.Method = "POST";
if ( $userAgent )
{
$webRequest.UserAgent = $userAgent;
}
$enc = [System.Text.Encoding]::GetEncoding($codePageName);
[byte[]]$bytes = $enc.GetBytes($data);
$webRequest.ContentLength = $bytes.Length;
[System.IO.Stream]$reqStream = $webRequest.GetRequestStream();
$reqStream.Write($bytes, 0, $bytes.Length);
$reqStream.Flush();
$resp = $webRequest.GetResponse();
$rs = $resp.GetResponseStream();
[System.IO.StreamReader]$sr = New-Object System.IO.StreamReader -argumentList $rs;
$sr.ReadToEnd();
}
}
$qtybord = 44
Execute-DucksboardApi -url 'https://push.ducksboard.com/v/123752/' -data '{"value": $qtybord}' -apikey 'tu2j3d3epqytWZD1haHnjJSJ1NqBrmvPe5SONc0VYge4BbIPi0'
Execute-DucksboardApi -url 'https://push.ducksboard.com/v/123752/' -data '{"value": 44}' -apikey 'tu2j3d3epqytWZD1haHnjJSJ1NqBrmvPe5SONc0VYge4BbIPi0'
*
try this:
-data "{`"value`": $qtybord}"
or
-data "{""value"": $qtybord}"
in single quote variable aren't expanded, in double quote you need to escape the double quote inside the string.
So make the -data parameter take $data instead and make $data = '{"value": ' + $qtyboard + '}' or just wrap that whole thing in parens after -data.
Also, if you're on PowerShell v3 you can play with something like this:
$data = New-Object -Type PSObject -Property #{
value = $qtyboard
} | ConvertTo-JSON

formatting csv files and powershell

Ok so we have a manual process that runs through PL/SQL Developer to run a query and then export to csv.
I am trying to automate that process using powershell since we are working in a windows environment.
I have created two files that seems to be exact duplicates from the automated and manual process but they don't work the same so I assume I am missing some hidden characters but I can't find them or figure out how to remove them.
The most obvious example of them working differently is opening them in excel. The manual file opens in excel automatically putting each column in it's own seperate column. The automated file instead puts everything into one column.
Can anybody shed some light? I am hoping that by resolving this or at least getting some info will help with the bigger problem of it not processing correctly.
Thanks.
ex one column
"rownum","year","month","batch","facility","transfer_facility","trans_dt","meter","ticket","trans_product","trans","shipper","customer","supplier","broker","origin","destination","quantity"
ex seperate column
"","ROWNUM","RPT_YR","RPT_MO","BATCH_NBR","FACILITY_CD","TRANSFER_FACILITY_CD","TRANS_DT","METER_NBR","TKT_NBR","TRANS_PRODUCT_CD","TRANS_CD","SHIPPER_CD","CUSTOMER_NBR","SUPPLIER_NBR","BROKER_CD","ORIGIN_CD","DESTINATION_CD","NET_QTY"
$connectionstring = "Data Source=database;User Id=user;Password=password"
$connection = New-Object System.Data.OracleClient.OracleConnection($connectionstring)
$command = New-Object System.Data.OracleClient.OracleCommand($query, $connection)
$connection.Open()
Write-Host -ForegroundColor Black " Opening Oracle Connection"
Start-Sleep -Seconds 2
#Getting data from oracle
Write-Host
Write-Host -ForegroundColor Black "Getting data from Oracle"
$Oracle_data=$command.ExecuteReader()
Start-Sleep -Seconds 2
if ($Oracle_data.read()){
Write-Host -ForegroundColor Green "Connection Success"
while ($Oracle_data.read()) {
#Variables for recordset
$rownum = $Oracle_data.GetDecimal(0)
$rpt_yr = $Oracle_data.GetDecimal(1)
$rpt_mo = $Oracle_data.GetDecimal(2)
$batch_nbr = $Oracle_data.GetString(3)
$facility_cd = $Oracle_data.GetString(4)
$transfer_facility_cd = $Oracle_data.GetString(5)
$trans_dt = $Oracle_data.GetDateTime(6)
$meter_nbr = $Oracle_data.GetString(7)
$tkt_nbr = $Oracle_data.GetString(8)
$trans_product_cd = $Oracle_data.GetString(9)
$trans_cd = $Oracle_data.GetString(10)
$shipper_cd = $Oracle_data.GetString(11)
$customer_nbr = $Oracle_data.GetString(12)
$supplier_nbr = $Oracle_data.GetString(13)
$broker_cd = $Oracle_data.GetString(14)
$origin_cd = $Oracle_data.GetString(15)
$destination_cd = $Oracle_data.GetString(16)
$net_qty = $Oracle_data.GetDecimal(17)
#Define new file
$filename = "Pipeline" #Get-Date -UFormat "%b%Y"
$filename = $filename + ".csv"
$fileLocation = $newdir + "\" + $filename
$fileExists = Test-Path $fileLocation
#Create object to hold record
$obj = new-object psobject -prop #{
rownum = $rownum
year = $rpt_yr
month = $rpt_mo
batch = $batch_nbr
facility = $facility_cd
transfer_facility = $transfer_facility_cd
trans_dt = $trans_dt
meter = $meter_nbr
ticket = $tkt_nbr
trans_product = $trans_product_cd
trans = $trans_cd
shipper = $shipper_cd
customer = $customer_nbr
supplier = $supplier_nbr
broker = $broker_cd
origin = $origin_cd
destination = $destination_cd
quantity = $net_qty
}
$records += $obj
}
}else {
Write-Host -ForegroundColor Red " Connection Failed"
}
#Write records to file with headers
$records | Select-Object rownum,year,month,batch,facility,transfer_facility,trans_dt,meter,ticket,trans_product,trans,shipper,customer,supplier,broker,origin,destination,quantity |
ConvertTo-Csv |
Select -Skip 1|
Out-File $fileLocation
Why are you skipping the first row(usually the headers)? Also, try using Export-CSV instead:
#Write records to file with headers
$records | Select-Object rownum, year, month, batch, facility, transfer_facility, trans_dt, meter, ticket, trans_product, trans, shipper, customer, supplier, broker, origin, destination, quantity |
Export-Csv $fileLocation -NoTypeInformation

Resources