I have two sets of data that I am trying to manipulate for an expected output using PowerShell. Any ideas on how I can achieve this? Thanks for the help in advance
Data 1:
StartTime Url
2022-01-28T04:44:28.111542Z Url a
2022-02-02T13:35:15.1040997Z Url b
2022-02-02T15:33:57.961112Z Url c
Data2:
Endtime Url
2022-01-28T22:11:39.1086189Z Url m
2022-02-04T02:49:08.6644804Z Url d
Goal is table below
Startime Endtime Url of Starttime duration
2022-01-28T04:44:28.111542Z 2022-01-28T22:11:39.1086189Z Url a
2022-02-02T15:33:57.961112Z 2022-02-04T02:49:08.6644804Z Url c
If your input data sets are powershell object arrays
You can do:
$result = foreach ($item in $data1) {
$matchingItem = $data2 | Where-Object { $_.Url -eq $item.Url }
if ($matchingItem) {
# choose the resolution of the duration. Here I'm using TotalSeconds
# but you can also opt for TotalDays, TotalHours, TotalMinutes or TotalMilliseconds
$duration = ([datetime]$matchingItem.EndTime - [datetime]$item.StartTime).TotalSeconds
# return a new object with properties combined
[PsCustomObject]#{
StartTime = $item.StartTime
EndTime = $matchingItem.EndTime
Url = $item.Url
'Duration (Seconds)' = $duration
}
}
}
# show on screen
$result | Format-Table -AutoSize
# save as new CSV
$result | Export-Csv -Path 'X:\SomePath\durations.csv' -NoTypeInformation
Output on screen:
StartTime EndTime Url Duration (Seconds)
--------- ------- --- ------------------
2022-01-28T04:44:28.111542Z 2022-01-28T22:11:39.1086189Z Url a 62830.9970769
2022-02-02T15:33:57.961112Z 2022-02-04T02:49:08.6644804Z Url c 126910.7033684
Related
`$RDPAuths = Get-WinEvent -LogName 'Microsoft-Windows-TerminalServices-`RemoteConnectionManager/Operational' -FilterXPath '<QueryList><Query Id="0"><Select>*[System[EventID=1149]]</Select></Query></QueryList>'
[xml[]]$xml=$RDPAuths|Foreach{$_.ToXml()}
$EventData = Foreach ($event in $xml.Event)
{ New-Object PSObject -Property #{
TimeCreated = (Get-Date ($event.System.TimeCreated.SystemTime) -Format 'dd-MM-yyyy hh:mm:ss K')
User = $event.UserData.EventXML.Param1
Domain = $event.UserData.EventXML.Param2
Client = $event.UserData.EventXML.Param3
}
}
actually i want filter logs based on from date and To date in powershell >
> in Time created line i don't know how to write that >
I'm using FromFile to get the image out of files, and it has the following error for the png's on the FromFile line:
Exception calling "FromFile" with "1" argument(s): "The given path's
format is not supported."
So, I'm trying to convert the bmp's to jpg, (see convert line above FromFile below) but all the examples I see (that seem usable) are saving the file. I don't want to save the file in the dir. All I need is the image format, so FromFile can use it like this example. I saw ConvertTo-Jpeg, but I don't think this is a standard powershell module, or don't see how to install it.
I saw this link, but I don't think that would leave the image in the format needed by FromFile.
This is my code:
$imageFile2 = Get-ChildItem -Recurse -Path $ImageFullBasePath -Include #("*.bmp","*.jpg","*.png") | Where-Object {$_.Name -match "$($pictureName)"} #$imageFile | Select-String -Pattern '$($pictureName)' -AllMatches
Write-Host $imageFile2
if($imageFile2.Exists)
{
if($imageFile2 -Match "png")
{
$imageFile2 | .\ConvertTo-Jpeg #I don't think this will work with FromFile below
}
$image = [System.Drawing.Image]::FromFile($imageFile2) step
}
else {
Write-Host "$($imageFile2) does not exist"
}
And then I put it in excel:
$xlsx = $result | Export-Excel -Path $outFilePath -WorksheetName $errCode -Autosize -AutoFilter -FreezeTopRow -BoldTopRow -PassThru # -ClearSheet can't ClearSheet every time or it clears previous data ###left off
$ws = $xlsx.Workbook.Worksheets[$errCode]
$ws.Dimension.Columns #number of columns
$tempRowCount = $ws.Dimension.Rows #number of rows
#only change width of 3rd column
$ws.Column(3).Width
$ws.Column(3).Width = 100
#Change all row heights
for ($row = 2 ;( $row -le $tempRowCount ); $row++)
{
#Write-Host $($ws.Dimension.Rows)
#Write-Host $($row)
$ws.Row($row).Height
$ws.Row($row).Height = 150
#place the image in spreadsheet
#https://github.com/dfinke/ImportExcel/issues/1041 https://github.com/dfinke/ImportExcel/issues/993
$drawingName = "$($row.PictureID)_Col3_$($row)" #Name_ColumnIndex_RowIndex
Write-Host $image
$picture = $ws.Drawings.AddPicture("$drawingName",$image)
$picture.SetPosition($row - 1, 0, 3 - 1, 0)
if($ws.Row($row).Height -lt $image.Height * (375/500)) {
$ws.Row($row).Height = $image.Height * (375/500)
}
if($ws.Column(3).Width -lt $image.Width * (17/120)){
$ws.Column(3).Width = $image.Width * (17/120)
}
}
Update:
I just wanted to reiterate that FromFile can't be used for a png image. So where Hey Scripting Guy saves the image like this doesn't work:
$image = [drawing.image]::FromFile($imageFile2)
I figured out that the $imageFile2 path has 2 filenames in it. It must be that two met the Get-ChildItem/Where-Object/match criteria. The images look identical, but have similar names, so will be easy to process. After I split the names, it does FromFile ok.
I have a ruby script that basically checks for few keys in registry that are flags for our application. The script would check if the flag was 'on'(DWORD value 1) for longer than 14 days, it will turn it off(0) and send out an email.
To track this, what the script does is during the first run, it creates a local DB and store the flag name, date time and so on of every flag that is on in the registry. During next run, it checks this DB to find if the flag has been on for more than 14 days.
I wanted to know if there is a better way to handle this using powershell or should i create a local DB and follow the same flow?
Further info:
So i will have different accounts under '\HKEY_LOCAL_MACHINE\SOFTWARE\Company\Clients' and each client will have a debug folder which will have the flags as seen below
HKEY_LOCAL_MACHINE\SOFTWARE\Company\Clients
\Client1\Debug
IsEmail
IsShip
\Client2\Debug
IsEmail
IsPack
So i need to effectively iterate through each client, maybe create an XML/JSON or DB with the client name, flags defined and current date time. Then during a later run, i should be accessing this info, compare the registry entries and turn off flags that have been turned for longer than a days specified. I will use the timestamp saved early for this.
Update 1:
Few updates based on my updated understanding of how powershell scripts work. I will have a master array or sort with a list of flags i need. For examples $MasterFlagList=['IsEmail','IsShip','IsNew']. Now during my first run of the application, i would like to create a json string like #tukan mentioned in this answer or xml file that will be something like:
<xml>
<Client1>
<IsEmail>
<value>1</value>
<time>*current time*</time>
</IsEmail>
<IsShip>
<value>1</value>
<time>*current time*</time>
</IsShip>
<IsPack>
<value>0</value>
<time>*current time*</time>
</IsPack>
</Client1>
<Client2>
<IsEmail>
<value>0</value>
<time>*current time*</time>
</IsEmail>
<IsShip>
<value>0</value>
<time>*current time*</time>
</IsShip>
<IsPack>
<value>0</value>
<time>*current time*</time>
</IsPack>
</Client2>
</xml>
So basically what happens here is that the XML string will have all the flags from masterlist, but if there is an entry for those in registry, that value will be loaded(0 or 1) and for those not mentioned in registry, default value of 0 will be set.
Next time i run, i will load this XML from directory into an object and for each client, i need to check if the XML already has a record, if so check the date time and see if the flag has been on for an amount of time, if so turn the flag off.
A better solution to implement the idea is also welcome. I am new to powershell and that why i thought of this approach.
That is really a nasty hack. Why don't you store the date also in the registry with the client record?
I think Powershell does better job in querying registry than ruby.
To get values from a registry path:
Get-ItemProperty -Path Registry::HKEY_LOCAL_MACHINE\SOFTWARE\company\Clients\Client1\Debug
This yields result:
IsEmail : 0
IsShip : 0
PSPath : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SOFTWARE\company\Clients\Client1\Debug
PSParentPath : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SOFTWARE\company\Clients\Client1
PSChildName : Debug
PSProvider : Microsoft.PowerShell.Core\Registry
Alternative is to do it this way:
Get-Item -Path Registry::HKEY_LOCAL_MACHINE\SOFTWARE\company\Clients\Client1\Debug
Which gives:
Hive: HKEY_LOCAL_MACHINE\SOFTWARE\company\Clients\Client1
Name Property
---- --------
Debug IsEmail : 0
IsShip : 0
If you want get the value directly, you have to do it this way:
(Get-ItemProperty -Path Registry::HKEY_LOCAL_MACHINE\SOFTWARE\company\Clients\Client1\Debug -Name IsEmail).IsEmail
To set a value (don't forget that you probably need admin rights for that!):
Set-ItemProperty -Path Registry::HKEY_LOCAL_MACHINE\SOFTWARE\company\Clients\Client1\Debug -Name 'IsEmail' -Type Dword -Value '1'
Now to the core of your question (and probably where you have problems)
I had to test it as i had different solution on my mind so it took quite a while.
# %Y ... year
# %m ... month
# %d ... day
# %R ... 24 hour time and minutes
# %S ... seconds
# example output: 20180226_10:38:23
$time_stamp= Get-Date -UFormat "%Y%m%d_%R:%S"
Write-Output $time_stamp
Get-ChildItem 'HKLM:\SOFTWARE\company\Clients' -Recurse | ForEach-Object {
$regkey = (Get-ItemProperty $_.pspath)
$regkey.PSObject.Properties | ForEach-Object {
# you could filter it via -> If($_.Name -like 'Is*'){
# $regkey is a HashTable
# Printing some output so you can check
Write-Output $regkey.PSParentPath
Write-Output $regkey.PSPath
Write-Output $_.Name ' = ' $_.Value
Write-Output ''
# convert to JSON - will contain more information than you need -> format it as you wish
$convert_registry_to_json = $regkey | ConvertTo-Json
#}
}
}
# Printing JSON output
Write-Output $convert_registry_to_json
Any further filtering or formatting I leave to the discretion.
Edit - edited question
This made me remember why I hate XML :). I have create a simple solution to your formatting. This code just a proof of concept, I recommend using functions to shorten your code (also avoiding duplicity)
$time_stamp= Get-Date -UFormat "%Y%m%d_%R:%S"
$result_hash = #{}
$result_array= #()
$first_level_hash = #{}
$second_level_hash = #{}
Get-ChildItem 'HKLM:\SOFTWARE\company\Clients' -Recurse | ForEach-Object {
$regkey = (Get-ItemProperty $_.pspath)
$regkey.PSObject.Properties | ForEach-Object {
# a HashTable
#Write-host $regkey.PSParentPath
$parent_key = $regkey.PSParentPath -Match '\w+$'
#Write-host "Found Parent:" $matches[0]
$parent_value = $matches[0]
#Write-Host $regkey.PSPath
#Write-Host $_.Name ' = ' $_.Value
#Write-host ''
$hash_key = $_.Name
$hash_value = $_.Value
If ($hash_key -like 'IsEmail'){
$second_level_hash.Add('value',$hash_value)
$second_level_hash.Add('time',$time_stamp)
$first_level_hash.Add($hash_key,$second_level_hash)
$result_hash.Add($parent_value,$first_level_hash)
$result_array += ($result_hash)
$second_level_hash=#{}
$first_level_hash=#{}
$result_hash=#{}
} ElseIf ($hash_key -like 'IsShip'){
$second_level_hash.Add('value',$hash_value)
$second_level_hash.Add('time',$time_stamp)
$first_level_hash.Add($hash_key,$second_level_hash)
$result_hash.Add($parent_value,$first_level_hash)
$result_array += ($result_hash)
$second_level_hash=#{}
$first_level_hash=#{}
$result_hash=#{}
} ElseIf ($hash_key -like 'IsPack'){
$second_level_hash.Add('value',$hash_value)
$second_level_hash.Add('time',$time_stamp)
$first_level_hash.Add($hash_key,$second_level_hash)
$result_hash.Add($parent_value,$first_level_hash)
$result_array += ($result_hash)
$second_level_hash=#{}
$first_level_hash=#{}
$result_hash=#{}
}
#$convert_registry_to_json = $regkey | ConvertTo-Json
#}
}
}
#Write-Output $result_hash
#Write-Output $result_hash.Item('IsEmail').Keys
#Write-Output $result_hash.Item('Client1').item('IsEmail')
Write-Output $result_array
ForEach ($result in $result_array) {
ForEach ($entry in $result.GetEnumerator()) {
$first_level_name = $($entry.Value)
ForEach ($second_level in $first_level_name.GetEnumerator()) {
$second_level_name = $($second_level.Value)
ForEach ($third_level in $second_level_name.GetEnumerator()) {
Write-Host "$($entry.Name) -> $($second_level.Name) -> $($third_level.Name): $($third_level.Value)"
}
}
}
}
# You can convert it to JSON
# FSpecifies how many levels of contained objects are included in the JSON representation. The default value is 2.
$convert_result_to_json = $result_array | ConvertTo-Json -Depth 3
Write-Output $convert_result_to_json
The output now is, which can be easily converted to JSON or XML if you want :
Now it yields result:
Name Value
---- -----
Client1 {IsEmail}
Client1 {IsShip}
Client2 {IsEmail}
Client2 {IsPack}
Client1 -> IsEmail -> time: 20180226_13:31:58
Client1 -> IsEmail -> value: 1
Client1 -> IsShip -> time: 20180226_13:31:58
Client1 -> IsShip -> value: 0
Client2 -> IsEmail -> time: 20180226_13:31:58
Client2 -> IsEmail -> value: 1
Client2 -> IsPack -> time: 20180226_13:31:58
Client2 -> IsPack -> value: 0
with JSON format:
[
{
"Client1": {
"IsEmail": {
"time": "20180226_14:40:02",
"value": 1
}
}
},
{
"Client1": {
"IsShip": {
"time": "20180226_14:40:02",
"value": 0
}
}
},
{
"Client2": {
"IsEmail": {
"time": "20180226_14:40:02",
"value": 1
}
}
},
{
"Client2": {
"IsPack": {
"time": "20180226_14:40:02",
"value": 0
}
}
}
]
Second edit
As for the DB. Perhaps the best solution would be to use Sqlite, check the technet wiki about it PowerShell: Accessing SQLite databases
Below I have some code to get the values of instances of performance counters (which are instantiated once a page is visited) and send them to Graphite to display graphs in the following format:
[Path in Graphite (e.g., metric.pages.Counter1)] [value of counter] [epoch time]
To do this I made the following code where the writer is configured correctly to work:
# Get all paths to MultipleInstance counters and averages that start with "BLABLA" and
# put them into an array and get the epoch time
$pathsWithInstances = (get-counter -ListSet BLABLA*) | select -ExpandProperty PathsWithInstances
$epochtime = [int][double]::Parse((Get-Date -UFormat %s))
# This functions splits the path (e.g., \BLABLA Web(welcome)\Page Requests) into three
# parts: the part before the
# opening brace (the CounterCategory, e.g., "\BLABLA Web"), the part in between the braces
# (the page or
# service, e.g., "welcome"), and the part after the closing brace (the name of the test,
# e.g.,
# "\Page Requests"). We obtain the metric out of this information and send it to
# Graphite.
enter code here
foreach ($pathWithInstance in $pathsWithInstances)
{
$instanceProperties = $pathWithInstance.Split('()')
$counterCategory = $instanceProperties[0]
if ($counterCategory -eq ("\BLABLA Web") )
{
# Replace the * with nothing so that counters that are used to display the
# average (e.g., \BLABLAWeb(*)\Page Requests) are displayed on top in the
# Graphite directory.
$pagePath = $instanceProperties[1].Replace('*','')
$nameOfTheTest = $instanceProperties[2]
# Countername which is used in Graphite path gets whitespace and backslash
# removed in the name used for the path in Graphite (naming conventions)
$counterName = $nameOfTheTest.Replace(' ','').Replace('\','')
$pathToPerfCounter = $pathWithInstance
$pathInGraphite = "metrics.Pages." + $pagePath + $counterName
#Invoked like this since otherwise the get-counter [path] does not seem to work
$metricValue = [int] ((Get-Counter "$pathToPerfCounter").countersamples | select -
property cookedvalue).cookedvalue
$metric = ($pathInGraphite + " " + $metricValue + " " + $epochTime)
$writer.WriteLine($metric)
$writer.Flush()
}
}
Unfortunately this code is very slow. It takes about one second for every counter to send a value. Does someone see why it is so slow and how it can be improved?
You're getting one counter at a time, and it takes a second for Get-Counter to get and "Cook" the values. Get-Counter will accept an array of counters, and will sample, "cook" and return them all in that same second. You can speed it up by sampling them all at once, and then parsing the values from the array of results:
$CounterPaths = (
'\\Server1\Memory\Page Faults/sec',
'\\Server1\Memory\Available Bytes'
)
(Measure-Command {
foreach ($CounterPath in $CounterPaths)
{Get-Counter -counter $counterpath}
}).TotalMilliseconds
(Measure-Command {
Get-Counter $CounterPaths
}).TotalMilliseconds
2017.4693
1012.3012
Example:
foreach ($CounterSample in (Get-Counter $CounterPaths).Countersamples)
{
"Path = $($CounterSample.path)"
"Metric = $([int]$CounterSample.CookedValue)"
}
Path = \\Server1\memory\page faults/sec
Metric = 193
Path = \\Server1\memory\available bytes
Metric = 1603678208
Use the Start-Job cmdlet, to create separate threads for each counter.
Here is a simple example of how to take the Counter Paths and pass them into an asynchronous ScriptBlock:
$CounterPathList = (Get-Counter -ListSet Processor).PathsWithInstances.Where({ $PSItem -like '*% Processor Time' });
foreach ($CounterPath in $CounterPathList) {
Start-Job -ScriptBlock { (Get-Counter -Counter $args[0]).CounterSamples.CookedValue; } -ArgumentList $CounterPath;
}
# Call Receive-Job down here, once all jobs are finished
IMPORTANT: The above example uses PowerShell version 4.0's "method syntax" for filtering objects. Please make sure you're running PowerShell version 4.0, or change the Where method to use the traditional Where-Object instead.
Ok so we have a manual process that runs through PL/SQL Developer to run a query and then export to csv.
I am trying to automate that process using powershell since we are working in a windows environment.
I have created two files that seems to be exact duplicates from the automated and manual process but they don't work the same so I assume I am missing some hidden characters but I can't find them or figure out how to remove them.
The most obvious example of them working differently is opening them in excel. The manual file opens in excel automatically putting each column in it's own seperate column. The automated file instead puts everything into one column.
Can anybody shed some light? I am hoping that by resolving this or at least getting some info will help with the bigger problem of it not processing correctly.
Thanks.
ex one column
"rownum","year","month","batch","facility","transfer_facility","trans_dt","meter","ticket","trans_product","trans","shipper","customer","supplier","broker","origin","destination","quantity"
ex seperate column
"","ROWNUM","RPT_YR","RPT_MO","BATCH_NBR","FACILITY_CD","TRANSFER_FACILITY_CD","TRANS_DT","METER_NBR","TKT_NBR","TRANS_PRODUCT_CD","TRANS_CD","SHIPPER_CD","CUSTOMER_NBR","SUPPLIER_NBR","BROKER_CD","ORIGIN_CD","DESTINATION_CD","NET_QTY"
$connectionstring = "Data Source=database;User Id=user;Password=password"
$connection = New-Object System.Data.OracleClient.OracleConnection($connectionstring)
$command = New-Object System.Data.OracleClient.OracleCommand($query, $connection)
$connection.Open()
Write-Host -ForegroundColor Black " Opening Oracle Connection"
Start-Sleep -Seconds 2
#Getting data from oracle
Write-Host
Write-Host -ForegroundColor Black "Getting data from Oracle"
$Oracle_data=$command.ExecuteReader()
Start-Sleep -Seconds 2
if ($Oracle_data.read()){
Write-Host -ForegroundColor Green "Connection Success"
while ($Oracle_data.read()) {
#Variables for recordset
$rownum = $Oracle_data.GetDecimal(0)
$rpt_yr = $Oracle_data.GetDecimal(1)
$rpt_mo = $Oracle_data.GetDecimal(2)
$batch_nbr = $Oracle_data.GetString(3)
$facility_cd = $Oracle_data.GetString(4)
$transfer_facility_cd = $Oracle_data.GetString(5)
$trans_dt = $Oracle_data.GetDateTime(6)
$meter_nbr = $Oracle_data.GetString(7)
$tkt_nbr = $Oracle_data.GetString(8)
$trans_product_cd = $Oracle_data.GetString(9)
$trans_cd = $Oracle_data.GetString(10)
$shipper_cd = $Oracle_data.GetString(11)
$customer_nbr = $Oracle_data.GetString(12)
$supplier_nbr = $Oracle_data.GetString(13)
$broker_cd = $Oracle_data.GetString(14)
$origin_cd = $Oracle_data.GetString(15)
$destination_cd = $Oracle_data.GetString(16)
$net_qty = $Oracle_data.GetDecimal(17)
#Define new file
$filename = "Pipeline" #Get-Date -UFormat "%b%Y"
$filename = $filename + ".csv"
$fileLocation = $newdir + "\" + $filename
$fileExists = Test-Path $fileLocation
#Create object to hold record
$obj = new-object psobject -prop #{
rownum = $rownum
year = $rpt_yr
month = $rpt_mo
batch = $batch_nbr
facility = $facility_cd
transfer_facility = $transfer_facility_cd
trans_dt = $trans_dt
meter = $meter_nbr
ticket = $tkt_nbr
trans_product = $trans_product_cd
trans = $trans_cd
shipper = $shipper_cd
customer = $customer_nbr
supplier = $supplier_nbr
broker = $broker_cd
origin = $origin_cd
destination = $destination_cd
quantity = $net_qty
}
$records += $obj
}
}else {
Write-Host -ForegroundColor Red " Connection Failed"
}
#Write records to file with headers
$records | Select-Object rownum,year,month,batch,facility,transfer_facility,trans_dt,meter,ticket,trans_product,trans,shipper,customer,supplier,broker,origin,destination,quantity |
ConvertTo-Csv |
Select -Skip 1|
Out-File $fileLocation
Why are you skipping the first row(usually the headers)? Also, try using Export-CSV instead:
#Write records to file with headers
$records | Select-Object rownum, year, month, batch, facility, transfer_facility, trans_dt, meter, ticket, trans_product, trans, shipper, customer, supplier, broker, origin, destination, quantity |
Export-Csv $fileLocation -NoTypeInformation