While trying to transfer file from Windows to Unix Azure environment, I am getting error dos2unix format error
dos2unix -o /xyz/home/ABC_efg.txt failed to execute dos2unix format change.
I tried to run a PS script to fix it but does seem to work .
Get-ChildItem -File -Recurse *.txt | % { $x = get-content -raw -path $_.fullname; $x -replace "`r`n","`n" | set-content -NoNewline -path $_.fullname }
Instead of using -replace, I would prefer to read the content(s) as string array and join these strings with "`n".
Something like this:
$files = Get-ChildItem -File -Recurse -Filter '*.txt' | Select-Object -ExpandProperty FullName
$files | ForEach-Object {
(Get-Content -Path $_) -join "`n" | Set-Content -Path $_ -NoNewline -WhatIf
}
Remove the -WhatIf switch if you are satisfied with the outout shown in the console.
Well, part of the issue is that you are piping a string to Set-Content and then trying to use that string to determine where to save the file. Try changing the last part from:
$x -replace "`r`n","`n" | set-content -NoNewline -path $_.fullname
to this:
set-content -NoNewline -path $_.fullname -value ($x -replace "`r`n","`n")
If that doesn't update the formatting like you expect it to you may need to use the -Encoding parameter for Set-Content. I'm not real familiar with encoding though, so I am not sure about that.
Related
I tried following Remove leading spaces in Windows file names but it's not working for my use case.
I have a lot of folders and filenames that either have a blank space at the front or at the end. How would I go about removing those spaces in bulk?
This was the command-line command I used after following the linked post:
for /R %A IN ("* ") do #for /F "tokens=*" %B IN ("%~nxA") do #ren "%A" "%B"
But it didn't work out.
Update: thank you to all who replied trying to help. I think there is just a Windows-level glitch in the file system. I ended up just having to manually create new folders without leading and trailing spaces and then dragging all the files over manually then renaming those to non-trailing and leading names as well.
It's unclear whether or not you want a PowerShell solution, but there's a reasonable assumption to be made you might.
If you wanted a PowerShell solution, you could try this:
function Test-LeadingTrailingWhitespace {
param(
[Parameter(Mandatory)]
[String]$String
)
$String[0] -eq ' ' -Or $String[-1] -eq ' '
}
Get-ChildItem -Path "<path_to_folder>" | ForEach-Object {
if ($_.PSIsContainer -And (Test-LeadingTrailingWhitespace -String $_.Name)) {
$Destination = Split-Path -Path $_.FullName -Parent
$NewName = $_.Name.Trim()
Move-Item -Path $_ -Destination (Join-Path -Path $Destination -ChildPath $NewName)
}
elseif (Test-LeadingTrailingWhitespace -String $_.BaseName) {
$Destination = Split-Path -Path $_.FullName -Parent
$NewName = $_.BaseName.Trim() + $_.Extension
Move-Item -Path $_ -Destination (Join-Path -Path $Destination -ChildPath $NewName)
}
}
To be on the safe side, you could add -WhatIf or -Confirm on the Move-Item cmdlet. The former will tell you what would have changed without that parameter without actually making any changes (like a 'dry run'). The latter will prompt you for confirmation before making each change, giving you a chance to validate incrementally and not make changes en masse from the moment you hit enter.
Trim() is a method available for all strings in PowerShell:
Returns a new string in which all leading and trailing occurrences of a set of specified characters from the current string are removed.
You can loop over files and folder and check if they actually have a leading or trailing whitespace before renaming, this would avoid errors like:
Rename-Item: Source and destination path must be different.
We can use the -match matching operator with a simple regex ^\s|\s$ (starts with whitespace or ends with whitespace - regex101 link for a simple example) to see if the file or folder should be renamed:
Get-ChildItem path\to\startingfolder -Recurse | ForEach-Object {
$newName = switch($_) {
# handle folders
{ $_.PSIsContainer -and $_.Name -match '^\s|\s$' } {
$_.Name.Trim()
break
}
# handle files
{ $_.BaseName -match '^\s|\s$' -or $_.Extension -match '^\s|\s$' } {
$_.BaseName.Trim() + $_.Extension.Trim()
break
}
# if none of the above conditions were true, continue with next item
Default {
return
}
}
Rename-Item -LiteralPath $_.FullName -NewName $newName
}
Personally, I'd do this in two steps to rename folders and files separately. This to overcome the problem that when a folder is renamed, the items inside that folder all have a new path.
Using switch -Force allows renaming items such as hidden or read-only files
Using -ErrorAction SilentlyContinue swallows the error when the new name is equal to the existing name
$rootPath = 'X:\thepath'
# first the folders and subfolders (deepest nesting first)
(Get-ChildItem -Path $rootPath -Directory -Recurse | Sort-Object FullName -Descending) |
Rename-Item -NewName {$_.Name.Trim()} -Force -ErrorAction SilentlyContinue
# next the files
(Get-ChildItem -Path $rootPath -File -Recurse) |
Rename-Item -NewName {'{0}{1}' -f $_.BaseName.Trim(), $_.Extension.Trim()} -Force -ErrorAction SilentlyContinue
consider I have a below CSV file.
input:
ID;ITEM_ID;STATUS;
001;;RELEASED;
002;36530;RELEASED;
003;86246;RELEASED;
004;;RELEASED;
I want to remove the row that has ;; (ITEM_ID) missing and save it.I tried doing it on one sample file and it worked as expected.
Import-Csv -Path ".\TestFile.CSV" | where {$_.ITEM_ID -ne ""} | Export-Csv -Path ".\TestFile-temp.CSV" -NoTypeInformation
Remove-Item -Path '.\TestDir\TestFile.csv'
Rename-Item -Path '.\TestDir\TestFile-temp.csv' -NewName 'TestFile.csv'
output:
ID;ITEM_ID;STATUS;
002;36530;RELEASED;
003;86246;RELEASED;
The challenge is, i have multiple csv files and it doesn't has value in different columns, but in single column when i opened in excel file.
so it's not taking the condition < where {$_.ITEM_ID -ne ""} >.
Now i have to search/parse each row of each csv file, search special character (;;) in that row and delete the line and save the file.
i am good at shell scripting but, i am very new to powershell scripting. can anybody please help me to get the logic here or use other cmdlet that can do the job?
$fileDirectory = "C:\Users\Administrator\Documents\check";
foreach($file in Get-ChildItem $fileDirectory)
{
$csvFileToCheck = Import-Csv -Path $fileDirectory\$file
$noDoubleSemiComma = foreach($line in $csvFileToCheck)
{
if(Select-String << i want the logic here>>)
{
$line
}
}
$noDoubleSemiComma | Export-Csv -Path $fileDirectory\tmp.csv -NoTypeInformation
Remove-Item -Path $fileDirectory\$file
Rename-Item -Path $fileDirectory\tmp.csv -NewName $file
}
As commented, you need to add parameter -Delimiter ';' to the cmdlet otherwise a comma is used to parse the fields in the CSV.
As I understand, you also want to remove the quotes Export-Csv outputs around all fields and headers and for PowerShell version 7 you have the option to use parameter -UseQuotes AsNeeded.
As this is not available for version 5.1, I made a function ConvertTo-CsvNoQuotes some time ago to remove the quotes in a safe way. (simply replacing them all with an empty string is dangerous, because sometimes values do need quotes)
Copy that function into your script at the top, then below that, your code could be simplified like this:
$fileDirectory = "C:\Users\Administrator\Documents\check"
Get-ChildItem -Path $fileDirectory -Filter '*.csv' -File | ForEach-Object {
# for better readability store the full path of the file in a variable
$filePath = $_.FullName
(Import-Csv -Path $filePath -Delimiter ';') | ConvertTo-CsvNoQuotes -Delimiter ';' | Set-Content $filePath -Force
Write-Host "File '$filePath' modified"
}
After all helpful suggestion, i finally nailed it down. AS my power-shell version was 5.1 , i had to use logic for trimming double quotes after export-csv. Powershell version 7 and later has -UseQuotes that could have solve that too.
Hope this help others.
$fileDirectory = "C:\Users\Administrator\Documents\check";
foreach($file in Get-ChildItem $fileDirectory)
{
Import-Csv -Path $fileDirectory\$file -Delimiter ';' | where {$_..ITEM_ID -ne ""} | Export-Csv -Path $fileDirectory\temp.csv -Delimiter ';' -NoTypeInformation
$Test = Get-Content $fileDirectory\temp.csv
$Test.Replace('";"',";").TrimStart('"').TrimEnd('"') | Out-File $fileDirectory\temp.csv -Force -Confirm:$false
Remove-Item -Path $fileDirectory\$file
Rename-Item -Path $fileDirectory\temp.csv -NewName $file
Write-Output "$file file modified."
}
Any suggestion to trim down number of lines of code is welcomed.
I want to search all drives using PowerShell on windows machine to get the list of all files along with their extensions -
Based on desired extension we pass in it like - *.mp3 or
Fetch all files with multiple extensions like - *.txt, *.mp3 etc.
I tried below script but its giving only information from where we are running it. But I want to scan whole machine.
Get-ChildItem -Path .\ -Filter ***.doc** -Recurse -File| Sort-Object Length -Descending | ForEach-Object { $_.BaseName }
Checkout the Get-PSDrive cmdlet. It returns a list of drives, and you can specify just disk drives with the -PSProvider FileSystem parameter:
foreach ( $drive in $(Get-PSDrive -PSProvider FileSystem) ) {
Get-ChildItem -Path $drive.Root -Filter ***.doc** -Recurse -File |
Sort-Object Length -Descending |
ForEach-Object { $_.BaseName }
}
Didn't test that but you get the idea.
Using -Include on Get-ChildItem will allow you to specify a list of extensions. The -ErrorAction will cause it to skip drives that are not available such as an unmounted CD drive.
Get-PSDrive -PSProvider FileSystem |
ForEach-Object {
Get-ChildItem -Path $_.Root -Recurse -Include '*.doc*', '*.txt' -ErrorAction SilentlyContinue |
ForEach-Object { $_.Name }
} |
ForEach-Object {[PSCustomObject]#{HashCode = $_.GetHashCode(); FullName = $_.FullName}}
} |
Export-Csv -Path $TempFile -NoTypeInformation -Encoding ASCII
Update:
Here is a better way. It will prevent unknown extensions from getting into the mix such as "Microsoft.NET.Sdk.Publish.Docker.targets."
$ExtensionList = #('.txt', '.doc', '.docx', '.mp3')
$TempFile = Join-Path -path $Env:TEMP -ChildPath "$($pid.ToString()).tmp"
Get-PSDrive -PSProvider FileSystem |
ForEach-Object {
Get-ChildItem -Path $_.Root -Recurse -ErrorAction SilentlyContinue |
Where-Object { $ExtensionList -contains $_.Extension } |
ForEach-Object {
[PSCustomObject]#{
HashCode = $_.GetHashCode();
DirectoryName = $_.DirectoryName
Name = $_.Name
}
}
} |
Export-Csv -Path $TempFile -Delimiter ';' -NoTypeInformation -Encoding ASCII
Write-Host "The temp file is $TempFile"
This is more than what the original question asked, but if you are going to go through the trouble of listing all your files, I suggest getting the filehash as well so you can determine if you have duplicates. A simple file name search will not detect if the same file has been saved with a different name. Adding to what #lit (https://stackoverflow.com/users/447901/lit) has posted:
$ExtensionList = #('.txt', '.doc', '.docx', '.mp3')
Get-PSDrive -PSProvider FileSystem |
ForEach-Object {
Get-ChildItem -Path $_.Root -Recurse -ErrorAction SilentlyContinue |
Where-Object { $ExtensionList -eq $_.Extension } |
## ForEach-Object { $_.Name, $_.FullName, $_.GetHashCode() }
Select-Object #{Name="Name";Expression={$_.Name}}, #{Name="Hash";Expression={$_.GetHashCode()}}, #{Name="FullName";Expression={$_.FullName}} |
Export-Csv -Path C:\Temp\testing.csv -NoTypeInformation -Append
}
The addition of the file hash will allow you to see if you have duplicates and the full name will allow you to see where they are located.
I have this script that stores filenames into a text file:
Get-ChildItem -Path "C:\Users\MyUser\Documents\Files" -Name |
Out-File "C:\Users\MyUser\Documents\Files\FileList.txt"
The result is:
file1.txt
file2.txt
file3.txt
file4.txt
I would like it to produce a list of PowerShell statements in a text file so that I could use them later like this:
Add-Content -Path "C:\Users\MyUser\Documents\Files\file1.txt" -Value "foo"
Add-Content -Path "C:\Users\MyUser\Documents\Files\file2.txt" -Value "foo"
Add-Content -Path "C:\Users\MyUser\Documents\Files\file3.txt" -Value "foo"
Add-Content -Path "C:\Users\MyUser\Documents\Files\file4.txt" -Value "foo"
Is it possible to concatenate a string to the file name the way I need?
I am also not sure whether I understand your question correctly but maybe this is what you are looking for:
Get-ChildItem -Path "C:\Users\MyUser\Documents\Files" |
% {
"Add-Content -Path `"{0}`" -Value `"foo`"" -f $_.FullName
} | Out-File "C:\Users\MyUser\Documents\Files\FileList.txt"
I'm a little unsure about what you're asking, but if I have it correct you want to get a list of files, add a line to the end of the file, and put the names of the files in text file.
Get-ChildItem -Path "C:\Users\MyUser\Documents\Files" |
ForEach-Object {
$_ | Add-Content -Value "foo"
$_.Name
} | Out-File "C:\Users\MyUser\Documents\Files\FileList.txt"
This is one possible way. It passes each item to ForEach-Object, where a value is added to the file, and the file name is sent to the pipeline. That gets piped into Out-File which writes the list.
There are many ways to achieve this; it's best to study and play with pipelining and then look at the various cmdlets and how they handle pipeline input.
I have following Power Shell script I need to modify.
$filepath = "F:\feeds\Amazon\"
$temppath = $filepath+"temp.csv"
$files = ls -Path $filepath *.csv
foreach ($file in $files){
gc $file.FullName |
% { if($_.indexOf("|~|") -eq -1) {$_ -replace "`"((?:`"`"|.)*?)`"(?!`")", "|~|`$1|~|" -replace "`"`"", "`""} else {$_ -replace " ", " "}} |
sc $temppath
ri $file.fullName
rni -Path $temppath -NewName $file.fullName
}
This script loops through all .csv files in a defined folder and change the text qualifier. Now I need to change this a bit and I am stucked.
Basically my CSV files are spitted into multiple folders like. Amazon1, Amazon2, Amazon3 .. so on. Is there anything wild card match sort of things I can do here so that it looks into all folders whose name starting with Amazon?
I Don't want to loop through folders.
... The * character? Try this:
$filepath = "F:\feeds\Amazon*"
$files = ls -Path $filepath *.csv -recurse
foreach ($file in $files){
$temppath = $file.directoryName+"\temp.csv"
gc $file.FullName |
% { if($_.indexOf("|~|") -eq -1) {$_ -replace "`"((?:`"`"|.)*?)`"(?!`")", "|~|`$1|~|" -replace "`"`"", "`""} else {$_ -replace " ", " "}} |
sc $temppath
ri $file.fullName
rni -Path $temppath -NewName $file.fullName
}
Yes withe Get-child-Item Cmdlet (dir) you can use wild card match on folders :
Get-ChildItem "C:\temp\Amazon?\" -include *.csv -Recurse