I have generated the list of MD5 checksum values from a directory within my project using Powershell's Get-FileHash function and then I exported the values to a .csv file.
$path = "C:\Users\Krishnaa\Documents\Visual Studio 2012\Projects\NamePrint\NamePrint\obj\Debug"
$hash = Get-FileHash -Path $path\* -Algorithm MD5
$export = $hash | Export-csv $path\hashfile.csv
This is how the output looks like if I call on $hash: http://i.stack.imgur.com/Owi0Q.png
Then I imported the .csv file back to the Powershell console.
$import = Import-csv $path\hashfile.csv | Format-Table
And when I call on $import, it outputs this : http://i.stack.imgur.com/cqvsO.png
When I created a simple function of my own to compare both the contents, I encounter problem whereby it says the the contents do not match. I do understand that each line in a .csv is treated as an object by Powershell. How to compare object-to-object in Powershell?
One problem with you above code is your use of Import-CSV. You aren't assigning the objects returned by Import-CSV to $import, you're assigning the array of formatting objects returned by Format-Table. If you drop the Format-Table, you should be able to compare $import.hash with $hash.hash (although you may need to loop through and compare row by row).
Related
I am having a hard time with powershell (because I am learning it in the run). I have huuuge amount of data and I am trying to find a unique identifier for every folder with data. I wrote a script which is just MD5-ing every folder recursively and saving the hash value for every folder. But as you might have already thought it is super slow. So I thought that I will hash only the metadata. But I have no idea how to do this in powershell. The ideas from the internet are not working and they return always the same hash value. Has anyone had similar problem? Is there a magic powershell trick to perform such task?
Sorry for lack of precision.
I have a big ~20000 list of folders. In every folder there are unique data, photos, files etc. I iterated through every folder and counted hash of every file (I actually made a crypto-stream here so I had a one hash for the data). This solution is taking ages.
The solution I wanted to adopt was using the metadata. Like those from this command:
Get-ChildItem -Path $Env:USERPROFILE\Desktop -Force | Select-Object -First 1 | Format-List *
But hashing this always gives me the same value even when something changed. I have to have a possibility to chceck if nothing has changed in those files.
First, create an MD5 class that does not create a new instance of System.Security.Cryptography.MD5 every time we create an MD5 from a string.
class MD5 {
static hidden [System.Security.Cryptography.MD5]$_md5 = [System.Security.Cryptography.MD5]::Create()
static [string]Create([string]$inputString) {
return [BitConverter]::ToString([MD5]::_md5.ComputeHash([Text.Encoding]::ASCII.GetBytes($inputString)))
}
}
Second, figure out a way to use each child items Name, Length, CreationTimeUtc, and LastWriteTimeUtc to create unique ID text per each child in the folder, merge into a single string and create an MD5 based on that resulting string.
Get the child objects of a folder.
Select only certain properties, returning the content as a string array.
Join the string array into a single string. No need for joining with newline.
Convert the string into an MD5.
Output the newly created MD5.
$ChildItems = Get-ChildItem -Path $Env:USERPROFILE\Desktop -Force
$SelectProperties = [string[]]($ChildItems | Select-Object -Property Name, Length, CreationTimeUtc, LastWriteTimeUtc)
$JoinedText = $SelectProperties -join ''
$MD5 = [MD5]::Create($JoinedText)
$MD5
Alternately, join the above lines into a very long command.
$AltMD5 = [MD5]::Create([string[]](Get-ChildItem -Path $Env:USERPROFILE\Desktop -Force | Select-Object -Property Name, Length, CreationTimeUtc, LastWriteTimeUtc) -join '')
$AltMD5
This resulting MD5 should be a unique signature of a folder's contents, not the folder itself, but only of the contents. So, you could in theory change the name of the folder itself and this MD5 would remain the same.
Not exactly sure how you aim to use this, but be aware that if any file, or sub-folder, in the folder changes, the MD5 for the folder will also change.
Continuing from my comment.
As per this resource
3rdP tool: http://www.idrix.fr/Root/Samples/DirHash.zip
function Get-FolderHash ($folder)
{
dir $folder -Recurse | ?{!$_.psiscontainer} |
%{[Byte[]]$contents += [System.IO.File]::ReadAllBytes($_.fullname)}
$hasher = [System.Security.Cryptography.SHA1]::Create()
[string]::Join("",$($hasher.ComputeHash($contents) |
%{"{0:x2}" -f $_}))
}
Note, that I've not tested/validated either of the above and will leave that to you.
Lastly, this is not the first time this kind of question has been asked via SO, using the default cmdlet and some .Net. So, this could be seen/markerd as a duplicate.
$HashString = (Get-ChildItem C:\Temp -Recurse |
Get-FileHash -Algorithm MD5).Hash |
Out-String
Get-FileHash -InputStream ([IO.MemoryStream]::new([char[]]$HashString))
Original, faster but less robust, method:
$HashString = Get-ChildItem C:\script\test\TestFolders -Recurse | Out-String
Get-FileHash -InputStream ([IO.MemoryStream]::new([char[]]$HashString))
could be condensed into one line if wanted, although it starts getting
harder to read:
Get-FileHash -InputStream ([IO.MemoryStream]::new([char[]]"$(Get-ChildItem C:\script\test\TestFolders -Recurse|Out-String)"))
Whether it's faster or fast enough for your use case is a different matter. Yet, it does address ensuring you get a different hash based on target folder changes.
What I'm trying to accomplish:
Create a PS script to run from a single Admin machine, but search against C$ on all Windows servers in AD.
Search for a specific list of paths\filenames that I provide.
If ANY of the specific list of paths\filenames are found on a server, THEN output the server name, and paths\filenames to a *.CSV file titled "Badfiles.csv" on the Admin machine.
Was trying to build from the following syntax but admittedly my old brain is not good at this stuff, and I've only specified a single file here - how do I refer to a list of multiple paths\files? Thank you for helping an old lady out. :)
$name= gc env:computername
$computers= get-content -path C:\Windows\Temp\v.bat
$csvfile = "c:\temp\$badfiles.csv"
foreach ($computer in $computers) {
"\$computer\C$\" | Get-ChildItem -recurse -filter "*.bat"
}
To refer to a list of items whether those are files or computer names you will need to use what is called an Array.
You can create an array in many ways, in your case it might best to create a list in a txt file and afterwards in Powershell you read the list contents using get-content, save the result in a variable and it will automatically be saved as an array!
Then iterate through each of them using what is called a foreach loop, that basically lets you take each of the items in the array and do something with it, then move to the next item and so on until every item has been dealt with.
Now the most important part of what you want to achieve is not clear. Let me explain.
To check if a file exists you can use test-path. That will return true or false and you can then act upon the result of that. You need to define an exact path and name of a file to check for this to work.
If you don't know the exact names and paths of files that need to be checked, you can use Get-ChildItem similarly as you have done in the code you provided. The caveat here is that you have to narrow down the scope of the file search as much as you can. In your example you search for the .bat file extension on the whole machine and that can result in some issues. A typical C drive will have hundreds of thousands if not millions of files and folders. Parsing all of them can take a long time.
So this is an important distinction to understand and what causes confusion for me is that you say in "2. Search for a specific list of paths\filenames that I provide..." yet in the code you use Get-ChildItem to get all files instead of providing a list of filenames.
Further I will assume you have a list of filenames with exact known paths.
Now in your given code I can see you have found some of the right commands but they need to be arranged differently to produce the results you need.
Please review this example code that might help you further:
Example ComputerList.txt file content(list of computer hostnames to check):
computer1
serverXYZ
laptop123
Example FileList.txt file content(List of files to check for in each of the above computers):
c:\temp\virus.bat
c:\games\game.exe
c:\Pictures\personal.jpg
Now the PowerShell code:
# Gets the list of items from TXT files and saves them as arrays in variables
$ComputerNames = Get-Content 'c:\temp\ComputerList.txt'
$FileList = Get-Content 'c:\temp\FileList.txt'
# Define the path and name of CSV report
$BadFiles = "c:\temp\badfiles.csv"
# Define the foreach loop that will iterate through each hostname in computer list
foreach($computer in $ComputerNames){
# Define foreach loop that will iterate through each file in the list and test their path in the current computer in the current iteration
foreach($file in $FileList){
# Test the path of the current file in the loop and append the CSV file if it was found
# Convert the file path to C$ share path
$file = $file -replace("c:","c$")
# Define path of file to test
$FileToTest = "\\$computer\$file"
if (test-path $FileToTest -ErrorAction SilentlyContinue){
# This block will run only when a bad file has been found
# This part can be tricky but it is used to make sure we properly format the current bad file entry when we append it to the resulting CSV file
$BadFile = "" | select Computer,File
# Save information about current computer
$BadFile.computer = $computer
# Save information about current file
$BadFile.file = $file
# Append the entry to an array of found bad files
$BadFileList += $badfile
}
}
}
# When done iterating through every computer and file, save the results in a CSV file
$BadFileList | ConvertTo-Csv -NoTypeInformation | Out-File $BadFiles
The above is a full code snippet you can test and run in your environment. First please create the two TXT files and make sure you run PowerShell with the appropriate permissions to access the C$ network shares of the servers.
The snippet should work but I have not tested it myself. Let me know if there are any errors.
Please test and feel free to ask if you have any follow up questions.
What I'm looking for might be a variation of this solution: Windows batch file to sort files into separate directories based on types specified in a csv
My Situation: a batch process in a server creates files that look like this: S0028513-010716-0932.txt. S stands for summary, the first five digits stand for a supplier, the last two before the hyphen stand for the Distribution Center. After the hyphen, there is the date and after the second hyphen the timestamp.
What I need to do is:
set a variable for the month/year (e.g. 0716) (this has been set with "set /P c:Please enter MMYY:"). This part is done.
create a folder with subfolders (e.g. 0716\PHARMA, 0716\MEDICAL, etc). I've done this part.
look up the supplier number in a CSV file (e.g. S00285 above) and
move the file to the corresponding folder based on MMYY\PHARMA, etc.
Points 3 and 4 are obvioulsy missing. A practical example: there are three folders where the files can be moved: PHARMA, MEDICAL and CONSUMER
The CSV file looks like this:
S00285 CONSUMER
S00286 PHARMA
S00287 MEDICAL
...
What I want the script to do is to look up the month/year combination in variable c and take all files that correspond to this month/year and move them to the three folders according to the assignment in the CSV file.
Can this be done with standard Windows scripting? Sorry guys, I'm a novice as you can tell. I have only some very basic knowledge of BASH scripting.
Thank you a lot for any advice.
BR
Marcio
This can fairly easily be accomplished with PowerShell
$FolderRoot = "E:\Target\Directory"
Set-Location $FolderRoot
# 1. Have user input month/year string
do{
$MMYY = $(Read-Host 'Please enter MMYY').Trim()
} until ($MMYY -match '\d{4}')
# 2. Create directory
mkdir $MMYY
# ?. Gather input files for that year
$Files = Get-ChildItem -Filter S*.txt |Where-Object {$_.BaseName -match "S\d{7}-\d{2}$MMYY-\d{4}"}
# ?. load CSV file into hash table to easily look up supplier numbers
$SupplierLookupTable = #{}
# Assuming the csv has headers: Supplier,Industry
Import-Csv -Path E:\path\to\suppliers.csv |ForEach-Object {
$SupplierLookupTable[$_.Supplier] = $_.Industry
}
foreach($File in $Files)
{
# Grab the S and first 5 digits from the file name
$Supplier = $File.BaseName.Substring(0,6)
# 3. Look up the industry
$Industry = $SupplierLookupTable[$Supplier]
$Destination = Join-Path $MMYY $Industry
# Create folder if it doesn't already exist
if(-not (Test-Path $Destination))
{
mkdir $Destination
}
# 4. Move the file
Move-Item $File.Fullname -Destination $Destination
}
Good day,
I am trying to rename/organize files based on the match/lookup found in the text file.
I have a couple of hundred Cyrillic(Russian) named media files in a folder like this:
файл 35.avi
файл34.avi
файл2 4.avi
файл14.avi
*note that some files have spaces
The text file, with the desired names, looks like this:
файл 35.avi| 4. файл 35.avi
файл34.avi| 3. файл34.avi
файл2 4.avi| 1. файл2 4.avi
файл14.avi| 2. файл14.avi
The reason it looks that way (with | as a separator) is because I tried using "Bulk Renaming Utility" which uses pipe | as a separator for "Rename Pairs" function. So essentially, the filename to the right of pipe | is the final product. Unfortunately, that function does not work with Cyrillic(Russian) or other non standard characters.
I found PowerShell script HERE which appears to be almost what I need except that it does not match file names before renaming.
Similarly, I found this Python script HERE which does what i need but it's for Ubuntu. Unfortunately, I am on a Windows7 and not sure it applies to me.
Any recommendations?
Thank you very much for your time!
You could read the text file into a hashtable, where the key is the old name (the value on the left hand side of the |), and the value is the new name:
$RenameTable = #{}
Get-Content textfile.txt |ForEach-Object {
$OldName,$NewName = $_.Split('|')
$RenameTable[$OldName] = $NewName
}
Then rename the files based on what is in the hashtable:
Get-ChildItem .\folder\with\avi\files |Rename-Item -NewName {
if($RenameTable.ContainsKey($_.Name)){
$RenameTable[$_.Name]
} else {
$_.Name
}
}
I already know batch renaming is possible via Powershell, but so far I didn't find an answer that solves my problem:
I have a staple of pdfs named with an ID:
Eg.
332906.pdf
331339.pdf
343807.pdf
...
I produced a simple CSV out of my spreadsheet which contains the id in the first column and the corresponding filename i want in the second one:
appid;Fullname with I.D.
332906;Mike Miller_332906;
331339;Tom Hanks_331339;
343807;Scarlett Jo_343807;
....
....
I have to do that on my office computer so I can't download additional programms or use programming lanugages, all I have is the cmd box and Powershell.
Can I do that?
Thank you for your help!
Edit: Spreadsheet changed to CSV; CSV Example added
You may have a problem with the column heading as-is, but if you change it to below:
appid,Fullname_ID
332906,Mike Miller_332906,
331339,Tom Hanks_331339,
343807,Scarlett Jo_343807,
You can run this simple script to do the file rename:
$c = import-csv .\path\to.csv
$c | % { mv "$($_.appid).pdf" "$($_.Fullname_ID).pdf" }
What this does is create a PowerShell object containing the data from the csv file.
$c | % { .. } means iterate over all the rows in the csv and run the code in the blocks {...}
$_ represents each row, and from that you can access the columns using .appid and .Fullname_ID. The move (mv) command is what you will use to do the rename.