custom protocol handler via powershell script - windows

QUESTION ANSWERED.. I have edited this question to the working solution.
Here's the scenerio.
Windows 10 workstation with Jitsi VOIP software installed.
I made a protocol handler for SIP: with this registry entry..
Windows Registry Editor Version 5.00
[HKEY_CLASSES_ROOT\sip]
#="URL: SIP Protocol handler"
"URL Protocol"=""
[HKEY_CLASSES_ROOT\sip\DefaultIcon]
#="C:\\Program Files (x86)\\Jitsi\\sc-logo.ico"
[HKEY_CLASSES_ROOT\sip\shell]
[HKEY_CLASSES_ROOT\sip\shell\open]
[HKEY_CLASSES_ROOT\sip\shell\open\command]
#="\"C:\\Program Files (x86)\\Jitsi\\Jitsi.exe\" %1"
This part works. Entering sip:1234567890 as a run command dials the number.
What I want to do is create a new protocol named CHK: that does a http request to a local webserver, and if the webserver
responds with 0, dial the number. If the response is 1, show a message "this number can't be dialed"
Here is the registry entry I made for this new chk protocol
Windows Registry Editor Version 5.00
[HKEY_CLASSES_ROOT\chk]
#="URL: CHK Protocol handler"
"URL Protocol"=""
[HKEY_CLASSES_ROOT\chk\DefaultIcon]
#="C:\\Program Files (x86)\\Jitsi\\sc-logo.ico"
[HKEY_CLASSES_ROOT\chk\shell]
[HKEY_CLASSES_ROOT\chk\shell\open]
[HKEY_CLASSES_ROOT\chk\shell\open\command]
#="\"C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe\"
-File C:\\DNC\\dnc.ps1 %1"
Even though this is nearly identical to SIP reg entry, when I try running chk:1234567890 I get an error "Apllication not found" , so
something is not right with the open comand....
edit: I was right, it was the open command.. I had the quotes in wrong place
and the content of the dnc.ps1 script...
$w=$args[0]
$chprot,$num = $w.split(':',2)
$url = "http://server/numchk.php?ph=$num"
$webclient = New-Object System.Net.WebClient
$webpage = $webclient.DownloadString($url)
if ($webpage -match "0"){
$launch = "C:\Program Files (x86)\Jitsi\Jitsi.exe"
$prot = 'sip:'
$arguments = $prot + $num
start-process $launch $arguments
} Else {
$wshell = New-Object -ComObject Wscript.Shell
$wshell.Popup("CANT DIAL $num ",0,"",0x0)
}
If I run the script via run command powershell -noexit -File c:\DNC\dnc.ps1 chk:1234567890
I can see the script is doing the right thing, dialing numbers if response is zero, show can't dial message if response is 1.
Again.. I think the problem is with the registry entry... specifilly the command/open part...
[HKEY_CLASSES_ROOT\chk\shell\open\command]
#="\"C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe -File C:\\DNC\\dnc.ps1\" %1"
Some trick to passing an argument to an argument I am missing :(

I think you have a quote in the wrong place, so it's not looking for "powershell.exe", it's looking for a file named "powershell.exe -File C:\DNC\dnc.ps1".
Does this work?
[HKEY_CLASSES_ROOT\chk\shell\open\command]
#="\"C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe\" -File C:\\DNC\\dnc.ps1 %1"

Related

How to query and add (if not listed) a location to Windows 10 Search Index using PowerShell

I have to re-mount removable drives (which require authentication) each time I boot the computer and Windows Indexing keeps removing the removable drives (perhaps because the removable drives are not available when the computer boots). In an ideal world Windows Indexing would keep these locations and just list them as 'Unavailable' (which it sometimes does). However because it doesn't I am interested in executing a script that queries the Windows Indexing locations and if it does not list the removable drives then add them. At the bottom of this thread I pasted the Batch script that I setup to run at boot (via Start Up folder) to search for a specific folder that is available thereafter mounting one of the removable drives.
I have found several examples of how to do this on Windows 7 (links pasted below) but I can't figure out how to do it in Windows 10. The links provided to the DLL (Microsoft.Search.Interop.dll) no longer resolve.
When searching for the latest Windows Search SDK for Windows 10 I was lead to the Windows SDK here:
https://learn.microsoft.com/en-us/windows/win32/search/-search-developers-guide-entry-page
I installed the C++ related portion of the Windows SDK then searched for Microsoft.Search.Interop.dll but I couldn't find it. Perhaps the DLL has changed?
From How to rebuild Windows Search Index by using PowerShell?
Load DLL containing classes & interfaces
Add-Type -path "C:\Temp\SearchIndexSdk\Microsoft.Search.Interop.dll"
#Provides methods for controlling the Search service. This
interface manages settings and objects that affect the search engine
across catalogs.
https://msdn.microsoft.com/en-us/library/bb231485(v=vs.85).aspx
$sm = New-Object Microsoft.Search.Interop.CSearchManagerClass
#Retrieves a catalog by name and creates a new ISearchCatalogManager
object for that catalog.
$catalog = $sm.GetCatalog("SystemIndex")
#Resets the underlying catalog by rebuilding the databases and performing a full indexing.
#https://msdn.microsoft.com/en-us/library/bb266414(v=vs.85).aspx
$catalog.Reset()
From How to add a location to windows 7/8 search index using batch or vbscript?
#Code copied from "Powershell Tackles Windows Desktop Search" http://powertoe.wordpress.com/2010/05/17/powershell-tackles-windows-desktop-search/
#Microsoft.Search.Interop.dll is needed, download from http://www.microsoft.com/en-us/download/details.aspx?id=7388
#Load the dll
Add-Type -path "D:\Unattend\UserFiles\Tools\Microsoft.Search.Interop.dll"
#Create an instance of CSearchManagerClass
$sm = New-Object Microsoft.Search.Interop.CSearchManagerClass
#Next we connect to the SystemIndex catalog
$catalog = $sm.GetCatalog("SystemIndex")
#Get the interface to the scope rule manager
$crawlman = $catalog.GetCrawlScopeManager()
#add scope
$crawlman.AddUserScopeRule("file:///D:*",$true,$false,$null)
$crawlman.SaveAll()
I would add a comment to the existing threads but I am not able to because I don't have reputation of 50 (dumb rule IMO).
Last... I found this site which lists the DLL along with some code but it hasn't been updated in a long time.
https://github.com/FileMeta/WindowsSearchSample
Thanks in advance!
Batch script that runs at boot:
#echo off
echo Windows Search is being restarted to recognize the Z drive
:while
if EXIST Z:\Watch (
I WANT TO CALL POWERSHELL SCRIPT TO ADD THE LOCATION TO THE INDEX IF NEEDED HERE
sc stop WMPNetworkSvc
ping 127.0.0.1 -n 5 > nul
sc stop WSearch
ping 127.0.0.1 -n 5 > nul
sc start WSearch
ping 127.0.0.1 -n 5 > nul
sc start WMPNetworkSvc
echo Exiting this script in 5 seconds
ping 127.0.0.1 -n 5 > nul
exit
) else (
echo Waiting 60 seconds to check if Z drive is available
ping 127.0.0.1 -n 60 > nul
goto :while
)
When I do a search for Searchdll in what I believe to be the folder where the Windows SDK installed to (C:\Program Files (x86)\Windows Kits\10) I find the following. If I had to guess which DLL is the Windows 10 equivalent of Windows 7's Microsoft.Search.Interop.dll I would guess that it's the 1st one i.e. interop.searchapi.dll.
Add-Type -Path "C:\Program Files (x86)\Windows Kits\10\bin\10.0.19041.0\x64\interop.searchapi.dll" does return without error... however $sm = New-Object Microsoft.Search.Interop.CSearchManagerClass returns with error that it cannot find the class in the assembly.
When I cd to "C:\Program Files (x86)\Windows Kits\10\bin\10.0.19041.0\x64" and enter ([appdomain]::currentdomain.GetAssemblies() | Where-Object Location -Match 'interop.searchapi').gettypes() I get the following
When I enter (([appdomain]::currentdomain.GetAssemblies() | Where-Object location -match 'interop.searchapi.dll').gettypes() | Where-Object name -eq 'CSearchManagerClass').getmembers() | Format-Table name, membertype I get
From the list of commands in the previous threads I do see GetCatalog and I presume that the members GetCrawlScopeManager, AddUserScopeRule, Reset, and SaveAll exist.
I don't know how to find the fully qualified class name or I'm doing something else wrong (unknowingly).
When I enter ([appdomain]::currentdomain.GetAssemblies() | Where-Object Location -Match 'interop.searchapi').fullname I get the following
Interop.SearchAPI, Version=10.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
But when I enter $sm = New-Object Interop.SearchAPI.CSearchManagerClass I get an error that it can't find the type Interop.SearchAPI.CSearchManagerClass.

New-WebBinding: Cannot retrieve the dynamic parameters for the cmdlet

We're using Windows 2012 Server R2.
We're trying to automate the creation of LetsEncrypt certificates. We're using LetsEncrypt-Win-Simple (https://github.com/Lone-Coder/letsencrypt-win-simple).
Once the cert is created (via LetsEncrypt.exe) we have a .bat script that gets called (using the --script and --scriptparameters flags). This runs powershell.exe and tries to create the necessary IIS binding. The line in the .bat file is:
powershell.exe -file c:\temp\SSLIISBinding.ps1 %1 %2 %3 %4
The %1-4 are args passed in by LetsEncrypt. In the powershell script, the command we're trying to run is:
$iis_host_name = $args[0]
$iis_site_name = $args[1]
$certificate_hash = $args[2]
$certificate_store = $args[3]
"IIS Host Name: " + $iis_host_name
"IIS Site Name: " + $iis_site_name
"Certificate Hash: " + $certificate_hash
"Certificate Store: " + $certificate_store
$guid = [guid]::NewGuid().ToString("B")
netsh http add sslcert hostnameport="${iis_host_name}:443" certhash=$certificate_hash certstorename=$certificate_store appid="$guid"
New-WebBinding -name $iis_site_name -Protocol https -HostHeader $iis_host_name -Port 443 -SslFlags 1
The args are passed into the .bat fine, as we output them and they are showing correctly.
If we run the .bat file on its own, it works perfectly. If it gets called by LetsEncrypt.exe it fails, reporting the following issue:
New-WebBinding : Cannot retrieve the dynamic parameters for the cmdlet.
Retrieving the COM class factory for component with CLSID
{688EEEE5-6A7E-422F-B2E1-6AF00DC944A6} failed due to the following error:
80040154 Class not registered (Exception from HRESULT: 0x80040154
(REGDB_E_CLASSNOTREG)).
At C:\temp\SSLIISBinding.ps1:13 char:1
+ New-WebBinding -name $iis_site_name -Protocol https -HostHeader
$iis_host_name ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~
+ CategoryInfo : InvalidArgument: (:) [New-WebBinding], Parameter
BindingException
+ FullyQualifiedErrorId : GetDynamicParametersException,Microsoft.IIs.Powe
rShell.Provider.NewWebBindingCommand
I've googled, some mentioning something about 32bit vs 64bit powershell, but I've tried using all the different powershell.exe available.
Anyone hit this issue, or know to resolve.
If we call .bat directly from command line it works fine, just as part of being called via LetsEncrypt.exe. A permission problem? Wrong powershell.exe?
That part of your question:
I've googled, some mentioning something about 32bit vs 64bit powershell
is already half of an answer. Some commands do not run properly if bitness of PowerShell process does not match bitness of operation system. So, you need to run powershell.exe, which located in this %windir%\System32\WindowsPowerShell\v1.0\ directory. But there is a little problem described in this documentation topic:
In most cases, whenever a 32-bit application attempts to access %windir%\System32, the access is redirected to %windir%\SysWOW64.
Thus, if 32-bit program on 64-bit OS invoke %windir%\System32\WindowsPowerShell\v1.0\powershell.exe, it will actually invoke 32-bit version of PowerShell from here %windir%\SysWOW64\WindowsPowerShell\v1.0\ instead of 64-bit one. To actually invoke 64-bit PowerShell from 32-bit application you need to use this trick:
32-bit applications can access the native system directory by substituting %windir%\Sysnative for %windir%\System32. WOW64 recognizes Sysnative as a special alias used to indicate that the file system should not redirect the access.
I've got the same error when running the following cmdlet:
PS> Remove-WebAppPool -Name 'Test'
Remove-WebAppPool : Cannot retrieve the dynamic parameters for the cmdlet. Retrieving the COM class factory for
component with CLSID {688EEEE5-6A7E-422F-B2E1-6AF00DC944A6} failed due to the following error: 80040154 Class not
registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).
At line:1 char:1
+ Remove-WebAppPool -Name 'Test'
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [Remove-WebAppPool], ParameterBindingException
+ FullyQualifiedErrorId : GetDynamicParametersException,Microsoft.IIs.PowerShell.Provider.RemoveAppPoolCommand
The reason was because I ran it using Windows PowerShell (x86) on my Windows 10 x64 machine.
When I tried the same but using Windows PowerShell, which is 64 bit version, it worked just fine.
I think your $guid is the issue. The GUID needs to be the GUID of the program to bind the cert to. For your example port 443 is only bound to a random GUID, and not your program's GUID. IIS and other apps have a static GUID that you will want to use. If the GUID for a powershell script then Get-host is the powershell host executing code so that's the GUID you need. It changes for every powershell session and the netsh binding needs to as well.
$appid = "appid={"+(get-host).InstanceId.guid+"}"
$certhash = ls Cert:\LocalMachine\my | where {$.EnhancedKeyUsageList -Match 'Server' -and $.subject -match (hostname)}|sort-object $_.NotAfter|select -expand Thumbprint -last 1
$cmdline='netsh http add sslcert ipport=0.0.0.0:443 certhash=' + $certhash + ' "' + $appid + '"'
netsh http delete sslcert ipport=0.0.0.0:443
Invoke-Expression $cmdline
A google search for "Cannot retrieve the dynamic parameters for the cmdlet" brought me here but my issue was using powershell from the command line, and the answer was to escape the double quotes on the command...
I've got a problem with the same error. This happens when i'm trying to Add-WebBinding to my IIS site remotely, using Invoke-Command from different agent machines at time.
It's worked for me, maybe it helps someone too:
$Mutex = New-Object -TypeName System.Threading.Mutex($false, "Global\Mutex")
if ($Mutex.WaitOne(300000)) {
#For example
#$Command = {
#New-WebBinding -name $iis_site_name -Protocol https -HostHeader
#$iis_host_name -Port 443 -SslFlags 1
#}
#Invoke-Command -Command $Command
} else {
Write-Warning "Timed out acquiring mutex!"
}
$Mutex.Dispose()

Download web content through CMD

I want some way of getting an online content on the command prompt window (Windows CMD).
Imagine some content online stored either on a hosting service or the MySql database provided to you by that hosting service. It can be any data stored in any form. I just want to remotely view it with the help of a CMD window anywhere in the world. What command should I be using?
To make it more clear, lets just say you have one day to prepare for your exam. Rather than preparing for it, you are making a plan to cheat.
Now your exam is going to be conducted on a computer that has been allotted to you and you are not allowed to use a browser or download any new application on the PC. But you can use Command Prompt, so your task being a cheat is to put the answers somewhere online. You cannot install anything new.
How will you go about if you are stuck in the above scene?
P.S This is for educational purpose only. I have no such such exams.
It depends entirely on how the content is stored and accessed. CMD knows how to get things from SMB network shares (\computername\folder), but you need to run a program to access most other stuff. (eg: sqlcmd for a database)
CMD does not support downloading web content. You will need to use another program to connect to the website and download the page. Obviously a browser would work. Another option is to download wget.exe from UnxUtils. Or use another scripting language like PowerShell or Wscript.
If you have access to launch PowerShell (pre-installed on Windows 7-10), you can use the .NET library to download web resources.
PS> Invoke-Webrequest 'www.mywebsite.com/myfile.txt' -OutFile '.\myfile.txt'
This will use .NET to connect to the page and download it into the local directory. To download to another directory or filename, change the -OutFile argument.
To launch this from CMD, go into a PowerShell prompt by simply typing powershell in CMD, and running the PS commands from there. Alternatively, you can run PS commands from CMD using the powershell -c command.
powershell.exe -c "invoke-webrequest 'www.mywebsite.com/myfile.txt' -outfile .\myfile.txt"
You can check this question.The easiest way is with bitsadmin command:
bitsadmin /transfer myDownloadJob /download /priority normal http://downloadsrv/10mb.zip c:\10mb.zip
You can try also winhttpjs.bat:
call winhhtpjs.bat https://example.com/files/some.zip -saveTo c:\somezip.zip
Windows 10 build 17063 and later ships curl.exe (ref: https://techcommunity.microsoft.com/t5/containers/tar-and-curl-come-to-windows/ba-p/382409). Assuming you don't need to support earlier Windows versions, including earlier Windows 10 builds, you can use it in your batch files like this
curl http://example.com/ --output content.txt
notepad content.txt
Set Arg = WScript.Arguments
set WshShell = createObject("Wscript.Shell")
Set Inp = WScript.Stdin
Set Outp = Wscript.Stdout
On Error Resume Next
Set File = WScript.CreateObject("Microsoft.XMLHTTP")
File.Open "GET", Arg(1), False
File.setRequestHeader "User-Agent", "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 1.1.4322; .NET CLR 3.5.30729; .NET CLR 3.0.30618; .NET4.0C; .NET4.0E; BCD2000; BCD2000)"
File.Send
txt=File.ResponseText
'Putting in line endings
Outp.write txt
If err.number <> 0 then
Outp.writeline ""
Outp.writeline "Error getting file"
Outp.writeline "=================="
Outp.writeline ""
Outp.writeline "Error " & err.number & "(0x" & hex(err.number) & ") " & err.description
Outp.writeline "Source " & err.source
Outp.writeline ""
Outp.writeline "HTTP Error " & File.Status & " " & File.StatusText
Outp.writeline File.getAllResponseHeaders
Outp.writeline LCase(Arg(1))
End If
General Use
Filter is for use in a command prompt. Filter.vbs must be run with cscript.exe. If you just type filter it will run a batch file that will do this automatically.
filter subcommand [parameters]
Filter reads and writes standard in and standard out only. These are only available in a command prompt.
filter <inputfile >outputfile
filter <inputfile | other_command
other_command | filter >outputfile
other_command | filter | other_command
Web
filter web webaddress
filter ip webaddress
Retrieves a file from the web and writes it to standard out.
webaddress - a web address fully specified including http://
Example
Gets Microsoft's home page
filter web http://www.microsoft.com
Filter with 19 example command prompt scripts avaialable at https://onedrive.live.com/redir?resid=E2F0CE17A268A4FA!121&authkey=!AAFg7j814-lJtmI&ithint=folder%2cres
Apart from VBScript & JScripts, there's a utility (resides with CMD) on Windows which can be run from CMD (if you have write access):
set url=https://www.nsa.org/content/hl-images/2017/02/09/NSA.jpg
set file=file.jpg
certutil -urlcache -split -f %url% %file%
Cmdlets in Powershell:
$url = "https://www.nsa.org/content/hl-images/2017/02/09/NSA.jpg"
$file = "file.jpg"
$ProgressPreference = "SilentlyContinue";
Invoke-WebRequest -Uri $url -outfile $file
.Net under PowerShell:
$url = "https://www.nsa.org/content/hl-images/2017/02/09/NSA.jpg"
$file = "file.jpg"
# Add the necessary .NET assembly
Add-Type -AssemblyName System.Net.Http
# Create the HttpClient object
$client = New-Object -TypeName System.Net.Http.Httpclient
$task = $client.GetAsync($url)
$task.wait();
[io.file]::WriteAllBytes($file, $task.Result.Content.ReadAsByteArrayAsync().Result)
C# application built on command-line with csc.exe:
https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/compiler-options/command-line-building-with-csc-exe
using System;
using System.IO;
using System.Net.Http;
using System.Threading.Tasks;
namespace DownloadImage
{
class Program
{
static async Task Main(string[] args)
{
using var httpClient = new HttpClient();
var url = "https://www.nsa.org/content/hl-images/2017/02/09/NSA.jpg";
byte[] imageBytes = await httpClient.GetByteArrayAsync(url);
using var fs = new FileStream("file.jpg", FileMode.Create);
fs.Write(imageBytes, 0, imageBytes.Length);
}
}
}
Built in Windows applications. No need for external downloads.
Tested on Win 10

Not able to spawn off email message from windows search in powershell

I have a powershell script that is searching the Windows Search index directly for emails and files. I have the following code:
$searchme="my thing to find"
$sql="SELECT System.FileName, System.ItemPathDisplay, System.DateCreated, System.DateModified, system.itemurl, system.itemtypetext FROM SYSTEMINDEX WHERE Contains(System.FileName, '"+$searchme+"') OR Contains('"+$searchme+"')"
$adapter = new-object system.data.oledb.oleDBDataadapter -argumentlist $sql, "Provider=Search.CollatorDSO;Extended Properties=’Application=Windows’;"
$ds = new-object system.data.dataset
$adapter.Fill($ds)
foreach ($record in $ds.Tables[0].Rows)
{
$exeparams = $record[4]
write-host $exeparams
write-host $exeparams.split(":")[0]
if ($exeparams.split(":")[0] -eq "mapi15")
{
$exeparams2="mapi://" + $exeparams.substring(8)
}
write-host $exeparams
write-host "start"
$exe="start"
$exe+" "+$exeparams | Out-File 'file.txt' -encoding Unicode
write-host "start-process"
Start-Process $exeparams
Start-Process $exeparams2
write-host "andpersand process"
&$exe $exeparams
&$exe $exeparams2
write-host "dotnet"
$proc = [Diagnostics.Process]::Start($exeparams)
$proc.WaitForExit()
$proc = [Diagnostics.Process]::Start($exeparams2)
$proc.WaitForExit()
}
There are several "shell" calls because I was trying to figure out if it was a process spawning issue. Files work with no issue. Emails however fail with either No such interface if i leave mapi15, or Unspecified error if i change mapi15 to mapi. I believe that Open mails in outlook from java using the protocol "mapi://" may be the solution, but if it is I am not sure how to apply this in powershell. Thank you for your help.
Ok, that took more work than I expected, and I blame Office 2013 for it. Here's the short answer:
$exeparams2 = $exeparams -replace "^mapi15", "mapi"
& start $exeparams2
That is the code that now opens an email for me. That code did not do that yesterday, all it did is tell me:
Either there is no default mail client or the current mail client cannot fulfill the messaging request. Please run Microsoft Outlook and set it as the default mail client.
Infuriating is what this was, because I did have Outlook, in fact it was running, and was the default mail application for everything email related. This annoyed me, and sent me on a quest to figure out WTF was wrong, and if I could fix it. The answers to that are "I'm not real sure WTF was wrong, except maybe a naming change on MS's part", and "yes, yes I can fix it".
I finally found the fix that worked for me (and I believe that this is probably Office 2013/Office 365 specific) was found at the bottom of this thread:
https://social.technet.microsoft.com/Forums/office/en-US/64c0bedf-2bcd-40aa-bb9c-ad5de20c84be/cannot-send-email-with-microsoft-outlook-2010-64-bit-from-adobe-outlook-not-recognised-as?forum=outlook
The process is simple. Change 2 Registry Entries, then re-set your default mail application. Registry entries:
HKLM:\Software\Clients\Mail(default)
HKLM:\Software\Clients\Mail\Microsoft Outlook(default)
Change the value of (Default) from "Microsoft Outlook" to simply "Outlook".
After that I set Outlook to be the default for everything it could be ( in Win8 that's Control Panel > All Control Panel Items > Default Programs > Set Default Programs then select Outlook, and choose the first option to make it default for all extensions that it is registered for). Once that was done I was able to run the modified code above to launch an email that I had search the index for.

FTP: copy, check integrity and delete

I am looking for a way to connect to a remote server with ftp or lftp and make sure the following steps:
Copy files from FTP server to my local machine.
Check if the downloaded files are fine (i.e. md5checksum).
If the download was fine then delete the downloaded files from the FTP server.
This routine will be executed each day from my local machine. What would be the best option to do this? Is there a tool that makes abstraction of all the 3 steps ?
I am running Linux on both client and server machines.
Update: Additionally, I have also a text file that contains the association between the files on the FTPserver and their MD5sum. They were computed at the FTP server side.
First, make sure your remote server supports the checksum calculation at all. Many do not. I believe there's even no standard FTP command to calculate a checksum of a remote file. There were many proposals and there are many proprietary solutions.
The latest proposal is:
https://datatracker.ietf.org/doc/html/draft-bryan-ftpext-hash-02
So even if your server supports checksum calculation, you have to find a client that supports the same command.
Some of the commands that can be used to calculate checksum are: XSHA1, XSHA256, XSHA512, XMD5, MD5, XCRC and HASH.
You can test that with WinSCP. The WinSCP supports all the previously mentioned commands. Test its checksum calculation function or the checksum scripting command. If they work, enable logging and check, what command and what syntax WinSCP uses against your server.
Neither the ftp (neither Windows nor *nix version) nor the lftp support checksum calculation, let alone automatic verification of downloaded file.
I'm not even aware of any other client that can automatically verify downloaded file.
You can definitely script it with a help of some feature-rich client.
I've wrote this answer before OP specified that he/she is on Linux. I'm keeping the Windows solution in case it helps someone else.
On Windows, you could script it with PowerShell using WinSCP .NET assembly.
param (
$sessionUrl = "ftp://username:password#example.com/",
[Parameter(Mandatory)]
$localPath,
[Parameter(Mandatory)]
$remotePath,
[Switch]
$pause = $False
)
try
{
# Load WinSCP .NET assembly
Add-Type -Path (Join-Path $PSScriptRoot "WinSCPnet.dll")
# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions
$sessionOptions.ParseUrl($sessionUrl);
$session = New-Object WinSCP.Session
try
{
# Connect
$session.Open($sessionOptions)
Write-Host "Downloading $remotePath to $localPath..."
$session.GetFiles($remotePath, $localPath).Check();
# Calculate remote file checksum
$buf = $session.CalculateFileChecksum("sha-1", $remotePath)
$remoteChecksum = [BitConverter]::ToString($buf)
Write-Host "Remote file checksum: $remoteChecksum"
# Calculate local file checksum
$sha1 = [System.Security.Cryptography.SHA1]::Create()
$localStream = [System.IO.File]::OpenRead($localPath)
$localChecksum = [BitConverter]::ToString($sha1.ComputeHash($localStream))
Write-Host "Downloaded file checksum: $localChecksum"
# Compare cheksums
if ($localChecksum -eq $remoteChecksum)
{
Write-Host "Match, deleting remote file"
$session.RemoveFiles($remotePath).Check();
$result = 0
}
else
{
Write-Host "Does NOT match"
$result = 1
}
}
finally
{
# Disconnect, clean up
$session.Dispose()
}
}
catch [Exception]
{
Write-Host "Error: $($_.Exception.Message)"
$result = 1
}
# Pause if -pause switch was used
if ($pause)
{
Write-Host "Press any key to exit..."
[System.Console]::ReadKey() | Out-Null
}
exit $result
You can run it like:
powershell -file checksum.ps1 -remotePath ./file.dat -localPath C:\path\file.dat
This is partially based on WinSCP example for Verifying checksum of a remote file against a local file over SFTP/FTP protocol.
(I'm the author on WinSCP)
The question was later edited to say that OP has a text file with a checksum. That makes it a completely different question. Just download the file, calculate local checksum and compare it to the checksum you have in the text file. If they match, delete the remote file.
That's a long shot, but if the server supports php, you can exploit that.
Save the following as a php file (say, check.php), in the same folder as your name_of_file.txt file:
<? php
echo md5_file('name_of_file.txt');
php>
Then, visit the page check.php, and you should get the md5 hash of your file.
Related questions:
Calculate file checksum in FTP server using Apache FtpClient
How to perform checksums during a SFTP file transfer for data integrity?
https://serverfault.com/q/98597/401691

Resources