Speed up CSV Powershell script - performance

I've got a Powershell script that functions but it takes ages to complete. I'm a newbie with Powershell and i can't find a solution to speed up the proces. Hopefully somebody can show me to the right direction.
Example. I've got 2 csv files.
CSV 1:
CI Name,Last Logon Account
Computer1, User1
Computer2, User2
Computer3, User3
CSV 2:
Device Display Label,Subscriber Employee Id
Computer1, User1
Computer2, User2
Computer3, User6
I want to have all the Ci names in the first column with the last logon account in the second column and match subscriber employee id with ci name from the first file.
Resulting in:
Ci name, Last logon Account, Subscriber Employee Id
Computer1,User1,User1
Computer2,User2,User2
Computer3,User3,User6
I have the following script in Powershell:
$Data = Import-csv 'C:\Temp\Excel\CSV\file1.csv'
$Data2 = Import-Csv 'C:\Temp\Excel\CSV\file2.csv'
$combine = #()
foreach ($first in $Data) {
foreach ($second in $Data2) {
if ($second.'Device Display Label' -eq $first.'CI Name') {
$match = New-Object PSObject
$match | Add-Member Noteproperty "Ci Name" $first.'CI Name'
$match | Add-Member Noteproperty "Last Logon Account" $first.'Last Logon Account'
$match | Add-Member Noteproperty "Subscriber Employee Id" $second.'Subscriber Employee Id'
$combine += $match
}
}
}
$Combine
It works and it gives the desired result.
The only problem is that both csv files have 15000 lines. So it takes ages to finish the script.
Is there a way to speed up the proces. I hope somebody can point me to the right direction.

Use a hashtable to build an index out of one of the CSV files - this way you don't need the nested loops and the runtime should drop significantly:
# Build index/reference table from first data set
$DataTable = #{}
Import-csv 'C:\Temp\Excel\CSV\file1.csv' |ForEach-Object {
$DataTable[$_.'CI Name'] = $_
}
# No need to store the second data set in an intermediate variable
$combine = Import-Csv 'C:\Temp\Excel\CSV\file2.csv' |ForEach-Object {
if($DataTable.ContainsKey($_.'Device Display Label')){
# Take the existing object from the first data set
# and add the subscriber from the second data set
$DataTable[$_.'Device Display Label'] |Add-Member NoteProperty "Subscriber Employee Id" $_.'Subscriber Employee Id' -PassThru
}
}

Related

Active Directory - list full subgroup dependency of a given group and omit subgroups without any user

I have to list full subgroup dependency of specific group - filter -> only subgroups which contain at least 1 user.
I have tried this approach:
dsquery group -samid <specific_group> | dsget group -members -expand | dsquery * -filter "(&(objectclass=group))"
This is an adapted version of my answer here to omit groups without any user object.
Unfortunately, using Get-ADGroupMember together with switch -Recursive will not return members that are groups.
As the docs state:
If the Recursive parameter is specified, the cmdlet gets all members
in the hierarchy of the group that do not contain child objects.
To get an array of nested group objects within a certain parent group, you will need a recursive function like below:
function Get-NestedADGroup {
Param (
[Parameter(Mandatory = $true, ValueFromPipeline = $true, Position = 0)]
[ValidateNotNullOrEmpty()]
[Alias ('Identity')]
[string]$Group,
# the other parameters are optional
[string]$Server = $null,
[string]$SearchBase = $null,
[ValidateSet('Base', 'OneLevel', 'Subtree')]
[string]$SearchScope = 'Subtree'
)
$params = #{
Identity = $Group
SearchScope = $SearchScope
Properties = 'Members'
ErrorAction = 'SilentlyContinue'
}
if (![string]::IsNullOrWhiteSpace($Server)) { $params['Server'] = $Server }
if (![string]::IsNullOrWhiteSpace($SearchBase)) { $params['SearchBase'] = $SearchBase }
$adGroup = Get-ADGroup #params
if ($adGroup) {
if (-not $script:groupsHash.ContainsKey($Group)) {
# output this group object only if it has at least one user object
if (#($adGroup.Members | Where-Object {$_.objectClass -eq 'user'}).Count -gt 0) {
$adGroup
}
# avoid circular group references
$script:groupsHash[$Group] = $true
# and recurse to get the nested groups
foreach ($group in ($adGroup.Members | Where-Object {$_.objectClass -eq 'group'})) {
Get-NestedADGroup -Group $group.DistinguishedName -Server $Server -SearchBase $SearchBase
}
}
}
else {
Write-Warning "Group '$($Group)' could not be found.."
}
}
# create a Hashtable to avoid circular nested groups
$groupsHash = #{}
# call the function
$result = Get-NestedADGroup -Group 'SpecificGroup'
# output just the names if you like
$result.Name
# save to CSV
$result | Export-Csv -Path 'X:\Somewhere\SubgroupsWithAtLeastOneUser.csv' -NoTypeInformation
Here is an easy alternative leveraging Active Directory filtering capabilities. See inline comments to understand the logic.
Do note, this answer requires the ActiveDirectory Module available.
# Get the DN of the parent group (this is the initial group)
$parent = (Get-ADGroup parentGroup).distinguishedName
$param = #{
LDAPFilter = "(memberof:1.2.840.113556.1.4.1941:=$parent)"
Properties = "member"
}
# Find, recursively, all child groups Members Of this parent group
# and filter
Get-ADGroup #param | Where-Object {
# For each member of this child group,
# check if this member is an object of the Class `user`,
# if it is, we already can break the pipeline and return
# this child group, since we already know it has
# at least 1 user member
$_.member | Get-ADObject | Where-Object ObjectClass -EQ user |
Select-Object -First 1
}

How to get the highest value from list of Device Names

Generating a list of windows workstation computer names by reading the active directory and I need to find the highest number so that I can then assign a new device with the next available number - I am not having any success in doing this - how to do it? And as you can see from the list of names, I also have missing numbers in the sequence that ideally, I would like to fill in with new devices also...
The code I am using to get the list from AD is below.
((Get-ADComputer -Filter {operatingsystem -notlike "*server*" -and Name -like $NamingConvention -and enabled -eq "true"} -Credential $credential -server $ADServerIP).Name)
List of device names
PC01
PC28
PC29
PC30
PC31
PC32
PC33
PC34
PC35
PC36
PC37
PC38
PC40
PC41
PC42
PC43
PC44
PC45
PC46
PC47
PC27
PC48
PC26
PC24
PC179
PC18
PC180
PC181
PC182
PC183
PC184
PC185
PC186
PC187
PC188
PC189
PC19
PC190
PC191
PC192
PC21
PC22
PC23
PC25
PC178
PC49
PC51
PC77
PC78
PC79
PC80
PC81
PC83
PC84
PC85
PC87
PC88
PC89
PC90
PC91
PC92
PC93
PC94
PC95
PC96
PC97
PC76
PC50
PC75
PC72
PC52
PC53
PC54
PC55
PC56
PC57
PC59
PC60
PC61
PC62
PC63
PC64
PC65
PC66
PC67
PC68
PC69
PC70
PC71
PC73
PC98
PC177
PC175
PC115
PC116
PC117
PC118
PC119
PC12
PC120
PC121
PC122
PC123
PC124
PC125
PC126
PC127
PC128
PC129
PC13
PC130
PC131
PC114
PC132
PC113
PC111
PC02
PC03
PC04
PC06
PC08
PC09
PC10
PC100
PC101
PC102
PC103
PC104
PC105
PC106
PC107
PC108
PC109
PC11
PC110
PC112
PC176
PC133
PC135
PC158
PC159
PC16
PC160
PC161
PC162
PC163
PC164
PC165
PC166
PC167
PC168
PC169
PC17
PC170
PC171
PC172
PC173
PC174
PC157
PC134
PC156
PC154
PC136
PC137
PC138
PC139
PC14
PC140
PC141
PC142
PC143
PC144
PC145
PC146
PC147
PC148
PC149
PC150
PC151
PC152
PC153
PC155
PC99
Sort the pc names on their numeric values and select the last one:
$lastPC = (Get-ADComputer -Filter {operatingsystem -notlike "*server*" -and Name -like $NamingConvention -and enabled -eq "true"} -Credential $credential -server $ADServerIP).Name |
Sort-Object { [int]($_ -replace '\D+')} | Select-Object -Last 1
Here's a solution that will give you the highest number ($dataMax), the missing numbers ($dataMissing), and the next number to use ($dataNext). The next number to use will be either the 1st missing number, or if there are no missing numbers then it will be the highest number + 1
# load the computers list
$data = ((Get-ADComputer -Filter {operatingsystem -notlike "*server*" -and Name -like $NamingConvention -and enabled -eq "true"} -Credential $credential -server $ADServerIP).Name)
# create an array by splitting the data text using the "space" character as a delimiter
$data = $data.Split(" ")
# remove all the alpha characters ("PC"), leaving only the number values so it can be sorted easier
$dataCleaned = $data -replace "[^0-9]" , '' | sort { [int]$_ }
# after sorting the data, [-1] represents the last element in the array which will be the highest number
[int]$dataMax = $dataCleaned[-1]
# create a number range that represents all the numbers from 1 to the highest number
$range = 1..$dataMax | foreach-object { '{0:d2}' -f $_ }
# compare the created range against the numbers actually in the computer array to find the missing numbers
$dataMissing = #(compare $range $dataCleaned -PassThru)
# if there's a missing value, [0] represents the first element in the array of missing numbers
if ($dataMissing)
{
$dataNext = $dataMissing[0]
}
# if there's no missing values, the next value is the max value + 1
else
{
$dataMissing = "none"
$dataNext = $dataMax + 1
}
Write-Host "The highest number is:"('{0:d2}' -f $dataMax)
Write-Host "The missing numbers are: $dataMissing"
Write-Host "The next number to use is:" ('{0:d2}' -f $dataNext)
Assuming your list is exactly as it appears to be, then this appears to be one way to do it:
$List = 'PC01 PC28 PC29 PC30 PC31 PC32 PC33 PC34 PC35 PC36 PC37 PC38 PC40 PC41 PC42 PC43 PC44 PC45 PC46 PC47 PC27 PC48 PC26 PC24 PC179 PC18 PC180 PC181 PC182 PC183 PC184 PC185 PC186 PC187 PC188 PC189 PC19 PC190 PC191 PC192 PC21 PC22 PC23 PC25 PC178 PC49 PC51 PC77 PC78 PC79 PC80 PC81 PC83 PC84 PC85 PC87 PC88 PC89 PC90 PC91 PC92 PC93 PC94 PC95 PC96 PC97 PC76 PC50 PC75 PC72 PC52 PC53 PC54 PC55 PC56 PC57 PC59 PC60 PC61 PC62 PC63 PC64 PC65 PC66 PC67 PC68 PC69 PC70 PC71 PC73 PC98 PC177 PC175 PC115 PC116 PC117 PC118 PC119 PC12 PC120 PC121 PC122 PC123 PC124 PC125 PC126 PC127 PC128 PC129 PC13 PC130 PC131 PC114 PC132 PC113 PC111 PC02 PC03 PC04 PC06 PC08 PC09 PC10 PC100 PC101 PC102 PC103 PC104 PC105 PC106 PC107 PC108 PC109 PC11 PC110 PC112 PC176 PC133 PC135 PC158 PC159 PC16 PC160 PC161 PC162 PC163 PC164 PC165 PC166 PC167 PC168 PC169 PC17 PC170 PC171 PC172 PC173 PC174 PC157 PC134 PC156 PC154 PC136 PC137 PC138 PC139 PC14 PC140 PC141 PC142 PC143 PC144 PC145 PC146 PC147 PC148 PC149 PC150 PC151 PC152 PC153 PC155 PC99'
$NextNumber = ($List -split "\s" | ForEach-Object { if ($_ -match 'PC(?<Number>\d+)') { $Matches.Number } } | Measure-Object -Maximum).Maximum + 1
$NextNumber
"PC$NextNumber"

How to change the loop Powershell content from the csv file automatically?

I want to change the loop PowerShell content from the CSV file automatically, is there someone know how to achieve the goal?
I want to run the loop PowerShell as below,I export the AD Group Members UPN with CSV file like One#contoso.com. I want to change the Userlist with the UPN automatically because there have more than 1000 members in the group and I have to change the upn from one#contoso.com to 'one#contoso.com',it’s easy to make a miss, is there has a smart way can achieve the goal,thanks.
Export-ADGroupmember UPN:
Get-ADGroupmember -identity adgroup | % { get-aduser $_.samaccountname | select userprincipalname } | export-csv upn.csv -notypeinformation
Loop PowerShell:
Loop PowerShell:
$UserList = #(
'One#contoso.com'
'Two#contoso.com'
'Three#contoso.com'
'Four#contoso.com'
'Five#contoso.com'
)
foreach ($UL_Item in $UserList)
{
$ARAGU_Params = #{
TenantName = "contoso"
HostPoolName = "contosoHostPool"
AppGroupName = "Desktop Application Group"
UserPrincipalName = $UL_Item
}
Add-RdsAppGroupUser #ARAGU_Params
}
You cant change content in your $UserList .More precisely, it is possible but affect the performance since You will need to read from the file and break it into segments. The best way is to download all users at once.You can do it like this:
$UserList=Get-ADGroupmember -identity adgroup | % { get-aduser $_.samaccountname | select userprincipalname }
foreach ($UL_Item in $UserList)
{
$ARAGU_Params = #{
TenantName = "contoso"
HostPoolName = "contosoHostPool"
AppGroupName = "Desktop Application Group"
UserPrincipalName = $UL_Item.userprincipalname
}
Add-RdsAppGroupUser #ARAGU_Params
}

Avoid duplicate Bulk-New-ADUser Creation via a csv file

I have some modifications for this script. UserID is normally a user’s first name followed by their last name.
What I need to do is to compare the proxy address & SAMAccountName attribute associated with each Username before it create.
So I mean lets say Jack Sparrow , if jsparrow already in use then script will try as jasparrow (first and second letter of firstname) and in use jasparrow as well , will be jacsparrow and so on. I want to avoid duplicate usernames.
2-I decided that it would be better to make 'fun' passwords that use the first two letters of the FirstName , day/month , first two letters of the Lastname. The end result is that users get a password like "Ja1009Sp".
Firstname,LastName,Department,Manager,MobilePhone
Jack,Sparrow,IT,jsmith,1 88 635 5254-0551
John Smith,Sparrow,Finance,jsmith,188 635 5254-0554
Script :
Import-Module ActiveDirectory
$UserList = Import-CSV -Path C:\Temp\CreateUsers.csv
$targetOU='OU=usersOU,DC=My,DC=Domain,DC=org'
$upnDomain='sec.local'
foreach($Person in $UserList){
$useritems=#{
GivenName=$Person.Firstname
Surname=$Person.LastName
Department=$Person.Department
AccountPassword=ConvertTo-SecureString -String $Person.Password -AsPlainText -force
ChangePasswordAtLogon=$false
Enabled=$true
DisplayName="$($Person.Firstname) $($Person.Lastname)"
Manager=$Person.Manager
MobilePhone=$Person.MobilePhone
Name="$($Person.Firstname) $($Person.Lastname)"
SamAccountName="$($Person.Firstname+$Person.LastName.Substring(0,1))"
UserPrincipalName="$($Person.FirstName+$Person.LastName.Substring(0,1))#$upnDomain"
Company="Contoso"
}
New-ADUser #useritems -Path $targetOU
}
Try something like this.. I don't have AD available atm. to test the Get-ADUser-query used to look for existing account so it might need some tuning.
foreach ($Person in $UserList) {
#Reset counters
$i = 1
$n = 1
do {
if($i -le $person.Firstname.Length) {
$user = "$($Person.Firstname.Substring(0,$i)+$Person.LastName)"
$i++
} else {
#All combinations in use, adding number
$user = "$($Person.Firstname.Substring(0,1)+$Person.LastName+$n)"
$n++
}
} while ((Get-ADUser -Filter "(samAccountName -eq '$user') -or (proxyaddresses -like '$user*')"))
#Result username
#$user
#$useritems = #{
#.....
#SamAccountName=$user
#UserPrincipalName="$user#$upnDomain"
#....
#}
}
If all combinations are in use including jacksparrow, it tries jsparrow1 ++ until it finds a free number.
The password can be generated using:
$Password = "{0}{1}{2}" -f $Person.Firstname.Substring(0,2), (Get-Date).ToString("ddMM"), $Person.Lastname.Substring(0,2)

formatting csv files and powershell

Ok so we have a manual process that runs through PL/SQL Developer to run a query and then export to csv.
I am trying to automate that process using powershell since we are working in a windows environment.
I have created two files that seems to be exact duplicates from the automated and manual process but they don't work the same so I assume I am missing some hidden characters but I can't find them or figure out how to remove them.
The most obvious example of them working differently is opening them in excel. The manual file opens in excel automatically putting each column in it's own seperate column. The automated file instead puts everything into one column.
Can anybody shed some light? I am hoping that by resolving this or at least getting some info will help with the bigger problem of it not processing correctly.
Thanks.
ex one column
"rownum","year","month","batch","facility","transfer_facility","trans_dt","meter","ticket","trans_product","trans","shipper","customer","supplier","broker","origin","destination","quantity"
ex seperate column
"","ROWNUM","RPT_YR","RPT_MO","BATCH_NBR","FACILITY_CD","TRANSFER_FACILITY_CD","TRANS_DT","METER_NBR","TKT_NBR","TRANS_PRODUCT_CD","TRANS_CD","SHIPPER_CD","CUSTOMER_NBR","SUPPLIER_NBR","BROKER_CD","ORIGIN_CD","DESTINATION_CD","NET_QTY"
$connectionstring = "Data Source=database;User Id=user;Password=password"
$connection = New-Object System.Data.OracleClient.OracleConnection($connectionstring)
$command = New-Object System.Data.OracleClient.OracleCommand($query, $connection)
$connection.Open()
Write-Host -ForegroundColor Black " Opening Oracle Connection"
Start-Sleep -Seconds 2
#Getting data from oracle
Write-Host
Write-Host -ForegroundColor Black "Getting data from Oracle"
$Oracle_data=$command.ExecuteReader()
Start-Sleep -Seconds 2
if ($Oracle_data.read()){
Write-Host -ForegroundColor Green "Connection Success"
while ($Oracle_data.read()) {
#Variables for recordset
$rownum = $Oracle_data.GetDecimal(0)
$rpt_yr = $Oracle_data.GetDecimal(1)
$rpt_mo = $Oracle_data.GetDecimal(2)
$batch_nbr = $Oracle_data.GetString(3)
$facility_cd = $Oracle_data.GetString(4)
$transfer_facility_cd = $Oracle_data.GetString(5)
$trans_dt = $Oracle_data.GetDateTime(6)
$meter_nbr = $Oracle_data.GetString(7)
$tkt_nbr = $Oracle_data.GetString(8)
$trans_product_cd = $Oracle_data.GetString(9)
$trans_cd = $Oracle_data.GetString(10)
$shipper_cd = $Oracle_data.GetString(11)
$customer_nbr = $Oracle_data.GetString(12)
$supplier_nbr = $Oracle_data.GetString(13)
$broker_cd = $Oracle_data.GetString(14)
$origin_cd = $Oracle_data.GetString(15)
$destination_cd = $Oracle_data.GetString(16)
$net_qty = $Oracle_data.GetDecimal(17)
#Define new file
$filename = "Pipeline" #Get-Date -UFormat "%b%Y"
$filename = $filename + ".csv"
$fileLocation = $newdir + "\" + $filename
$fileExists = Test-Path $fileLocation
#Create object to hold record
$obj = new-object psobject -prop #{
rownum = $rownum
year = $rpt_yr
month = $rpt_mo
batch = $batch_nbr
facility = $facility_cd
transfer_facility = $transfer_facility_cd
trans_dt = $trans_dt
meter = $meter_nbr
ticket = $tkt_nbr
trans_product = $trans_product_cd
trans = $trans_cd
shipper = $shipper_cd
customer = $customer_nbr
supplier = $supplier_nbr
broker = $broker_cd
origin = $origin_cd
destination = $destination_cd
quantity = $net_qty
}
$records += $obj
}
}else {
Write-Host -ForegroundColor Red " Connection Failed"
}
#Write records to file with headers
$records | Select-Object rownum,year,month,batch,facility,transfer_facility,trans_dt,meter,ticket,trans_product,trans,shipper,customer,supplier,broker,origin,destination,quantity |
ConvertTo-Csv |
Select -Skip 1|
Out-File $fileLocation
Why are you skipping the first row(usually the headers)? Also, try using Export-CSV instead:
#Write records to file with headers
$records | Select-Object rownum, year, month, batch, facility, transfer_facility, trans_dt, meter, ticket, trans_product, trans, shipper, customer, supplier, broker, origin, destination, quantity |
Export-Csv $fileLocation -NoTypeInformation

Resources