I have hundreds of file in working directory that needs to be processed. It looks similar to this
a.txt
b.txt
c.txt
d.txt
All of these file can be processed manually like this
$lines = Get-Content "a.txt"
foreach ($line in $lines){
Out-File -FilePath "a-done.txt" -Encoding UTF8 -Append -InputObject ($line.Split(","))[0]
}
How to automate this process using loop by passing all filename to variable above.
I have tried foreach loop but it's not working
$lines = Get-Content "path/*.txt"
foreach ($line in $lines){
Out-File -FilePath "$lines-processed.txt" -Encoding UTF8 -Append -InputObject ($line.Split(","))[0]
}
What do I miss here?
To discover the files themselves, you'll want to use Get-ChildItem instead of Get-Content! To reference the file name without the extension (ie. a from a.txt), reference the BaseName property:
foreach($file in Get-ChildItem .\path\ -Filter *.txt){
foreach ($line in $file |Get-Content){
Out-File -FilePath "$($file.BaseName)-done.txt" -Encoding UTF8 -Append -InputObject ($line.Split(","))[0]
}
}
Youre looping through the lines of get-content, not where the filenames are saved. You need probably an extra step e.g.
$items = Get-ChildItem 'C:\Users\Alex\Desktop\oop'
foreach ($item in $items) {
<#your processin with get-content here#>
echo $item.name
echo "$item-processed.txt"
}
I misunderstodd in the first time. I hope I am right now:
You want so save one done-File per one input file.
The Problem with your code is that you are collection all the content of all Files in your $lines-Variable. And there is no Information about the underlying File(-names) any more.
Instead you have to loop over all the files an handle them seperately.
The solution suggested:
$files = dir *.txt -Exclude *done.txt
foreach ($f in $files) {
Get-Content $f | % {$_.split(',')[0]} |
Out-File ($f.DirectoryName + '\' + $f.Basename + '-done.txt') -Encoding UTF8}
Regards Martin
Here's an example of how I combine csv files into one big CSV.
$txtFilter = "D:\Temp\*.csv"
$fileOutputSummary = "D:\Temp\Summary.csv"
$list = Get-ChildItem -Path $txtFilter | select FullName
$iItems = $list.Count
$i = 0
ForEach($file in $list){
$i++
Write-Host "Combining ($i of $iItems) `r"
Write-Progress -Activity "Combining CSV files" -PercentComplete ($i / $iItems*100)
Import-Csv -Path $file.FullName | Export-Csv -Path $fileOutputSummary -Append -NoTypeInformation
Sleep 1
} # end ForEach file
I hope my example helps.
Related
consider I have a below CSV file.
input:
ID;ITEM_ID;STATUS;
001;;RELEASED;
002;36530;RELEASED;
003;86246;RELEASED;
004;;RELEASED;
I want to remove the row that has ;; (ITEM_ID) missing and save it.I tried doing it on one sample file and it worked as expected.
Import-Csv -Path ".\TestFile.CSV" | where {$_.ITEM_ID -ne ""} | Export-Csv -Path ".\TestFile-temp.CSV" -NoTypeInformation
Remove-Item -Path '.\TestDir\TestFile.csv'
Rename-Item -Path '.\TestDir\TestFile-temp.csv' -NewName 'TestFile.csv'
output:
ID;ITEM_ID;STATUS;
002;36530;RELEASED;
003;86246;RELEASED;
The challenge is, i have multiple csv files and it doesn't has value in different columns, but in single column when i opened in excel file.
so it's not taking the condition < where {$_.ITEM_ID -ne ""} >.
Now i have to search/parse each row of each csv file, search special character (;;) in that row and delete the line and save the file.
i am good at shell scripting but, i am very new to powershell scripting. can anybody please help me to get the logic here or use other cmdlet that can do the job?
$fileDirectory = "C:\Users\Administrator\Documents\check";
foreach($file in Get-ChildItem $fileDirectory)
{
$csvFileToCheck = Import-Csv -Path $fileDirectory\$file
$noDoubleSemiComma = foreach($line in $csvFileToCheck)
{
if(Select-String << i want the logic here>>)
{
$line
}
}
$noDoubleSemiComma | Export-Csv -Path $fileDirectory\tmp.csv -NoTypeInformation
Remove-Item -Path $fileDirectory\$file
Rename-Item -Path $fileDirectory\tmp.csv -NewName $file
}
As commented, you need to add parameter -Delimiter ';' to the cmdlet otherwise a comma is used to parse the fields in the CSV.
As I understand, you also want to remove the quotes Export-Csv outputs around all fields and headers and for PowerShell version 7 you have the option to use parameter -UseQuotes AsNeeded.
As this is not available for version 5.1, I made a function ConvertTo-CsvNoQuotes some time ago to remove the quotes in a safe way. (simply replacing them all with an empty string is dangerous, because sometimes values do need quotes)
Copy that function into your script at the top, then below that, your code could be simplified like this:
$fileDirectory = "C:\Users\Administrator\Documents\check"
Get-ChildItem -Path $fileDirectory -Filter '*.csv' -File | ForEach-Object {
# for better readability store the full path of the file in a variable
$filePath = $_.FullName
(Import-Csv -Path $filePath -Delimiter ';') | ConvertTo-CsvNoQuotes -Delimiter ';' | Set-Content $filePath -Force
Write-Host "File '$filePath' modified"
}
After all helpful suggestion, i finally nailed it down. AS my power-shell version was 5.1 , i had to use logic for trimming double quotes after export-csv. Powershell version 7 and later has -UseQuotes that could have solve that too.
Hope this help others.
$fileDirectory = "C:\Users\Administrator\Documents\check";
foreach($file in Get-ChildItem $fileDirectory)
{
Import-Csv -Path $fileDirectory\$file -Delimiter ';' | where {$_..ITEM_ID -ne ""} | Export-Csv -Path $fileDirectory\temp.csv -Delimiter ';' -NoTypeInformation
$Test = Get-Content $fileDirectory\temp.csv
$Test.Replace('";"',";").TrimStart('"').TrimEnd('"') | Out-File $fileDirectory\temp.csv -Force -Confirm:$false
Remove-Item -Path $fileDirectory\$file
Rename-Item -Path $fileDirectory\temp.csv -NewName $file
Write-Output "$file file modified."
}
Any suggestion to trim down number of lines of code is welcomed.
The below code searches 400+ numbers from a list.txt file to see if it exists within any files within the folder path specified.
The script is very slow and has yet to complete as it did not complete after 25 minutes of running. The folder we are searching is 507 MB (532,369,408 bytes) and it contains 1,119 Files & 480 Folders. Any help to improve the speed of the search and the efficiency is greatly appreciated.
$searchWords = (gc 'C:\temp\list.txt') -split ','
$results = #()
Foreach ($sw in $searchWords)
{
$files = gci -path 'C:\Users\david.craven\Dropbox\Asset Tagging\_SJC Warehouse_\_Project Completed_\2018\A*' -filter "*$sw*" -recurse
foreach ($file in $files)
{
$object = New-Object System.Object
$object | Add-Member -Type NoteProperty –Name SearchWord –Value $sw
$object | Add-Member -Type NoteProperty –Name FoundFile –Value $file.FullName
$results += $object
}
}
$results | Export-Csv C:\temp\output.csv -NoTypeInformation
The following should speed up your task substantially:
If the intent is truly to look for the search words in the file names:
$searchWords = (Get-Content 'C:\temp\list.txt') -split ','
$path = 'C:\Users\david.craven\Dropbox\Facebook Asset Tagging\_SJC Warehouse_\_Project Completed_\2018\A*'
Get-ChildItem -File -Path $path -Recurse -PipelineVariable file |
Select-Object -ExpandProperty Name |
Select-String -SimpleMatch -Pattern $searchWords |
Select-Object #{n='SearchWord'; e='Pattern'},
#{n='FoundFile'; e={$file.FullName}} |
Export-Csv C:\temp\output.csv -NoTypeInformation
If the intent is to look for the search words in the files' contents:
$searchWords = (Get-Content 'C:\temp\list.txt') -split ','
$path = 'C:\Users\david.craven\Dropbox\Facebook Asset Tagging\_SJC Warehouse_\_Project Completed_\2018\A*'
Get-ChildItem -File -Path $path -Recurse |
Select-String -List -SimpleMatch -Pattern $searchWords |
Select-Object #{n='SearchWord'; e='Pattern'},
#{n='FoundFile'; e='Path'} |
Export-Csv C:\temp\output.csv -NoTypeInformation
The keys to performance improvement:
Perform the search with a single command, by passing all search words to Select-String. Note: -List limits matching to 1 match (by any of the given patterns).
Instead of constructing custom objects in a script block with New-Object and Add-Member, let Select-Object construct the objects for you directly in the pipeline, using calculated properties.
Instead of building an intermediate array iteratively with += - which behind the scenes recreates the array every time - use a single pipeline to pipe the result objects directly to Export-Csv.
So there are definitely some basic things in the PowerShell code you posted that can be improved, but it may still not be super fast. Based on the sample you gave us I'll assume you're looking to match the file names against a list of words. You're looping through the list of words (400 iterations) and in each loop you're looping through all 1,119 files. That's a total of 447,600 iterations!
Assuming you can't reduce the number of iterations in the loop, let's start by making each iteration faster. The Add-Member cmdlet is going to be really slow, so switch that approach up by casting a hashtable to the [PSCustomObject] type accelerator:
[PSCustomObject]#{
SearchWord = $Word
File = $File.FullName
}
Also, there is no reason to pre-create an array object and then add each file to it. You can simply capture the ouptut of the foreach loop in a variable:
$Results = Foreach ($Word in $Words)
{
...
So a faster loop might look like this:
$Words = Get-Content -Path $WordList
$Files = Get-ChildItem -Path $Path -Recurse -File
$Results = Foreach ($Word in $Words)
{
foreach ($File in $Files)
{
if ($File.BaseName -match $Word)
{
[PSCustomObject]#{
SearchWord = $Word
File = $File.FullName
}
}
}
}
A simpler approach might be to use Where-Object on the files array:
$Results = Foreach ($Word in $Words)
{
$Files | Where-Object BaseName -match $Word
}
Try both and test out the performance.
So if speeding up the loop doesn't meet your needs, try removing the loop entirely. You could use regex and join all the words together:
$Words = Get-Content -Path $WordList
$Files = Get-ChildItem -Path $Path -Recurse -File
$WordRegex = $Words -join '|'
$Files | Where basename -match $WordRegex
I have 50 sub-Folders inside a Single Parent Folder.
Inside each sub folders there are multiple .txt files. I want to merge all the text files in a single sub-folder into 1 .txt file.
But I want a command so that it can be done in one go for all the subfolder, like i don't want to write command for each sub-folder.
For example:-
ABCD (Parent Folder ):-
A
B ; Here A and B are sub-folder
A\0001.txt
A\0002.txt
I want to merge and make a single text file A\0001.txt.
B\0001.txt
B\0002.txt
I want to merge both the text files in B Folder.
Can it be done in one go ?
This is probably a lot easier using powershell.
Try the following and change the basedir to the parent folder of all your subdirectories.
$basedir = "C:\Basedir"
$folderlist = Get-childitem -Path $basedir
foreach ($folder in $folderlist)
{
$dir = $folder
$outFile = Join-Path $dir "merged.txt"
# Build the file list
$fileList = Get-ChildItem -Path $dir -Filter File*.txt -File
# Get the header info from the first file
Get-Content $fileList[0] | select -First 2 | Out-File -FilePath $outfile -Encoding ascii
# Cycle through and get the data (sans header) from all the files in the list
foreach ($file in $filelist)
{
Get-Content $file | select -Skip 2 | Out-File -FilePath $outfile -Encoding ascii -Append
}
}
Maybe old but useful: this version works with folders and subfolders recursively:
$basedir = "..."
$folderlist = Get-childitem -Path $basedir -Recurse -Directory | Select-Object FullName
foreach ($folder in $folderlist)
{
Write-Host $folder.FullName
$dir = $folder.FullName
$outFile = Join-Path $basedir "merged.txt"
# Build the file list
$fileList = Get-ChildItem -Path $dir -Filter *.log | Select-Object FullName
# Get the header info from the first file
#Get-Content $fileList[0] | select -First 2 | Out-File -FilePath $outfile -Encoding ascii
# Cycle through and get the data (sans header) from all the files in the list
foreach ($file in $filelist)
{
Write-Host $file.FullName
Get-Content $file.FullName | Out-File -FilePath $outfile -Encoding ascii -Append
}
}
I'm working on a script that checks folders in specific directory. For example, I run the script for first time, it generates me a txt file containing folders in the directory.
I need the script to add any new directories that are found to the previously created txt file when the script is run again.
Does anyone have any suggestions how to make that happen?
Here is my code so far:
$LogFolders = Get-ChildItem -Directory mydirectory ;
If (-Not (Test-Path -path "txtfilelocated"))
{
Add-Content txtfilelocated -Value $LogFolders
break;
}else{
$File = Get-Content "txtfilelocatedt"
$File | ForEach-Object {
$_ -match $LogFolders
}
}
$File
something like this?
You can specify what directory to check adding path to get-childitem cmdlet in first line
$a = get-childitem | ? { $_.psiscontainer } | select -expand fullname #for V2.0 and above
$a = get-childitem -Directory | select -expand fullname #for V3.0 and above
if ( test-path .\list.txt )
{
compare-object (gc list.txt) ($a) -PassThru | Add-Content .\list.txt
}
else
{
$a | set-content .\list.txt
}
I'm trying to copy one file to any subfolder in a directory that has a specific name. I am part way there, but just can't quite get it to work.
I am able to find all of the subfolders called "help" using:
Get-ChildItem -Path Y:\folder1\subfolder -Directory -Recurse | ? { ($_.PSIsContainer -eq $true) -and ($_.Name -like 'help')}
That will get any folder in Y:\folder1\subfolder named help. So have been trying:
$folder = Get-ChildItem -Path Y:Y:\folder1\subfolder -Directory -Recurse | ? { ($_.PSIsContainer -eq $true) -and ($_.Name -like 'help')}
foreach ($f in $folder){
Copy-Item Y:\Info.html -Destination $folder[$f]
}
and that does not work. Bonus points if you can also tell me how to have it write out to a csv file all of the directories it copies the file to.
Thanks
I wrote this with version 3, but I think it will work with 1 and 2 since I used Set-StrictMode -Version <number> to test them.
The CSV output will look something like this for every line: Y:\Info.html,Y:\folder1\subfolder\help
$logpath = 'C:\log.csv'
$logopts = #{filepath=$logpath; append=$true; encoding='ascii'}
$file = 'Y:\Info.html'
$path = 'Y:\folder1\subfolder'
$search = 'help'
gci $path -d -s `
| ?{ $_.psIsContainer -and $_.name -match $search } `
| %{
cp $file $_.fullName; # copy file
$line = $file, $_.fullName -join ','; # build output
$line | out-file #logopts; # write output
}
Version 1
$folders = #(
(gci Y:\folder1\subfolder -dir -r | ? {$_.Name -like 'help'}).fullname
)
ForEach ($f in $folders) {
Copy-Item Y:\Info.html $f
}
Version 2
(gci Y:\folder1\subfolder -dir -r | ? {$_.Name -like 'help'}).fullname | % {cp Y:\Info.html $_}