Hoping someone can guide me / help me.
The issue, I have 2 servers one running a Ubuntu which has a website for clients to login and download / view reports. The other is a windows server 2012 R2 which creates / stores the reports. I need to move the files from the windows to the Ubuntu server so clients can view. The data is large currently 7gb and growing at 3 gb a year.
I need a batch file to connect using ftp and then copy the folder to a local folder. However it only needs to copy those files which have modified.
I have only ever written one batch file and I cant seem to find any ftp batch scripts which only copies modifed files.
Your my last resort as I cant seem to find a coder who knows batch script (its a dieing art). I have never used powershell so would not know where to start here.
Any help or advice please let me know.
Thanks
John
You can do it with PowerShell with winscp. Exemple :
try
{
# Load WinSCP .NET assembly
Add-Type -Path "WinSCPnet.dll"
# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property #{
Protocol = [WinSCP.Protocol]::Sftp
HostName = "example.com"
UserName = "user"
Password = "mypassword"
SshHostKeyFingerprint = "ssh-rsa 2048 xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx"
}
$session = New-Object WinSCP.Session
try
{
# Connect
$session.Open($sessionOptions)
# Upload files
$transferOptions = New-Object WinSCP.TransferOptions
$transferOptions.TransferMode = [WinSCP.TransferMode]::Binary
$transferResult = $session.PutFiles("d:\toupload\*", "/home/user/", $False, $transferOptions)
# Throw on any error
$transferResult.Check()
# Print results
foreach ($transfer in $transferResult.Transfers)
{
Write-Host ("Upload of {0} succeeded" -f $transfer.FileName)
}
}
finally
{
# Disconnect, clean up
$session.Dispose()
}
exit 0
}
catch [Exception]
{
Write-Host ("Error: {0}" -f $_.Exception.Message)
exit 1
}
This would be a way to do it in PowerShell. It would take files that are older then 31 days and upload them.
function FTP-Upload {
[CmdletBinding()]
param(
[Parameter(Mandatory=$true)]
[string]$Source_File,
[Parameter(Mandatory=$true)]
[string]$Target_File,
[Parameter(Mandatory=$true)]
[string]$Target_Server,
[Parameter(Mandatory=$true)]
[string]$Target_Username,
[Parameter(Mandatory=$true)]
[string]$Target_Password
)
$FTP = [System.Net.FTPWebRequest]::Create("ftp://$Target_Server/$Target_File")
$FTP = [System.Net.FTPWebRequest]$FTP
$FTP.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile
$FTP.Credentials = New-Object System.Net.NetworkCredential($Target_Username,$Target_Password)
$FTP.UseBinary = $true
$FTP.UsePassive = $true
# read in the file to upload as a byte array
$content = [System.IO.File]::ReadAllBytes($Source_File)
$FTP.ContentLength = $content.Length
# get the request stream, and write the bytes into it
$rs = $FTP.GetRequestStream()
$rs.Write($content, 0, $content.Length)
# be sure to clean up after ourselves
$rs.Close()
$rs.Dispose()
}
$Upload_Server = "server.network.tld"
$Upload_Location = "/data/"
$Upload_Username = "ftpuser"
$Upload_Password = "ftppassword"
$Files_To_Upload = Get-ChildItem E:\Path\To\Files -Recurse | Where-Object {($_.CreationTime -le (Get-Date).AddDays(-31)) -and (!$_.PSIsContainer)}
Foreach ($File in $Files_To_Upload) {
FTP-Upload -Source_File $File.FullName -Target_File ($Upload_Location + $File.Name) -Target_Server $Upload_Server -Target_Username $Upload_Username -Target_Password $Upload_Password
}
Related
I frequently have to copy a single file to multiple destinations, so i'm trying to write a script to make that go faster. it seems to work fine when i'm dealing with local files, but fails without any errors when running on a file that is on a mapped network drive.
at first I was using copy-item, and I couldn't make that work, so i used robocopy. that does the trick, but if the file already exists, i have an if statement using test-path which is supposed to skip to a user input that asks if you want to overwrite.. this is not working. i should say the one that checks the folder exists is working, but the one that checks for the file name always comes back true. for now, i have it just forcing an overwrite with robocopy because most of the time that's what i'll want to do anyway.
here's what i have right now.. "K:" is the mapped network drive i'm copying to, and i'm usually copying files from another mapped network drive "T:". I also should mention i have this set up to run from the context menu in windows (7) explorer, and it passes the file path to the script via %L and $args.
any advice is appreciated. (i apologize in advance, i know it's rather rough.. This is somewhat new to me.)
$Folders = #("K:\OKKHM 800" , "K:\OKKHM 1000" , "K:\OKKHM 1002" , "K:\OKKHM 1003" , "K:\OKKHM 1004", "K:\OKKHM 1250")
$source = $args[0]
$Filename = Split-Path -Path $source -Leaf
$sourcefolder= split-path -path $source -parent
$COUNTER = 0
$successful=0
$CONFIRMATION=0
foreach($Folder in $Folders){
$newpath = $folder + "\" + $filename
WRITE-HOST $NEWPATH
if(-not(test-path -path $newpath)) {
if((test-path -path $folder)) {
WRITE-HOST 'TEST 2'
robocopy $sourcefolder $folder $filename -is -it
$successful=1
}
else{
write-host 'folder does not exist'
}
}
else {
$title = 'Existing File Will Be Overwritten'
$question = 'Are you sure you want to proceed?'
$choices = New-Object Collections.ObjectModel.Collection[Management.Automation.Host.ChoiceDescription]
$choices.Add((New-Object Management.Automation.Host.ChoiceDescription -ArgumentList '&Yes'))
$choices.Add((New-Object Management.Automation.Host.ChoiceDescription -ArgumentList '&No'))
$decision = $Host.UI.PromptForChoice($title, $question, $choices, 1)
if ($decision -eq 0) {
Write-Host 'confirmed'
$CONFIRMATION=1
}
else {
Write-Host 'cancelled'
$CONFIRMATION=0
}
IF ($CONFIRMATION -EQ 1) {
try {
robocopy $sourcefolder $folder $filename
$successful=1
}
catch {
throw "NO GOOD"
}
}
}
$COUNTER++
}
if ($successful -eq 1) {
WRITE-HOST 'SUMMARY: ' $COUNTER ' FILES COPIED SUCCESSFULLY.'
}
Start-Sleep 5
Not able transfer multiple file though inputted correct path
######Transfer file
Write-Host ("Start script.")
try
{
$todaysDate = (Get-Date).ToString('yyyy-MM-dd')
Add-Type -Path "D:\WinSCP\WinSCPnet.dll"
# Setup session options
$session = New-Object WinSCP.Session
$sessionOptions = New-Object WinSCP.SessionOptions
$sessionOptions.Protocol = [WinSCP.Protocol]::Sftp
$SessionOptions.Timeout = New-TimeSpan -Seconds 90
$sessionOptions.HostName = "hostname"
$sessionOptions.UserName = "username"
$sessionOptions.PortNumber = "portnumber"
$sessionOptions.Password = ""
$sessionOptions.SshPrivateKeyPath = "D:\privatekey"
$sessionOptions.SshHostKeyFingerprint = "ssh-rsa 2048 xxxxxx"
#Write-Host ("Connecting.")
$session = New-Object WinSCP.Session
$session.SessionLogPath = "D:\WinSCPSessionLog_$todaysDate.log"
#Upload files
try
{
# Connect
$session.Open($sessionOptions)
#File list:
Write-Host ("File list: ")
#transferoptions
$transferOptions = New-Object WinSCP.TransferOptions
#$transferOptions.FileMask = "*.*"
$transferOptions.FilePermissions = $Null # This is default
$transferOptions.PreserveTimestamp = $False # if Timestamp on file is enable
$localPath = Get-ChildItem "D:\dq\*.csv" | Where-Object {($_.LastWriteTime -ge [datetime]::today)}
$remotePath = "/Outbox/"
# Upload files, collect results
$transferResult = $session.PutFiles($localPath, $remotePath, $False, $transferOptions)
# Upload files, collect results
#$transferResult = $session.PutFiles(($localPath + "*.*"), ($remotePath, + "*.*") $False, $transferOptions).Check()
# Iterate over every transfer
foreach ($transfer in $transferResult.Transfers)
{
# Success or error?
if ($transfer.Error -eq $Null)
{
#$transferResult = $session.PutFiles($localPath, $remotePath, $False, $transferOptions).Check()
Write-Host ("Upload of {0} succeeded, moving to save" -f $transfer.FileName)
}
else
{
Write-Host ("Upload of {0} failed: {1}" -f $transfer.FileName, $transfer.Error.Message)
}
}
#End of files:
Write-Host ("End of files. ")
}
finally
{
# Disconnect, clean up
$session.Dispose()
Write-Host ("Disconnected.")
}
#exit 0
}
catch [Exception]
{
$todaysDate = (Get-Date).ToString('yyyy-MM-dd')
Set-Content -Path "D:\WinSCPError_$todaysDate.log" $_.Exception.Message
#exit 1
}
Log File:
File list:
Upload of D:\dq\ADD.csv D:\dq\MINUS.csv D:\dq\DIVIDE.csv failed: File or folder 'D:\dq\ADD.csv D:\dq\MINUS.csv D:\dq\DIVIDE.csv' does not exist.
System Error. Code: 123.
The filename, directory name, or volume label syntax is incorrect
Error:
not able transfer file due to file not exists. I have inputted correct
Tools:
Using superb old version of .NET
Using superb old version of window server
Expected result:
able transfer multiple files in "D:\dq*.csv" and the file modified date must be today's date
I have inputted correct
Nope, you haven't. From the WinSCP documentation for the localPath argument:
Full path to local file or directory to upload. Filename in the path can be replaced with Windows wildcard1 to select multiple files. To upload all files in a directory, use mask *.
That is, it expects one path, not multiple.
Given your requirement for filtering on LastWriteTime, passing "D:\dq\*.csv" as the argument is not a viable option, so the solution is to upload the files one-by-one:
# ...
try{
# Connect
$session.Open($sessionOptions)
#File list:
Write-Host ("File list: ")
#transferoptions
$transferOptions = New-Object WinSCP.TransferOptions
#$transferOptions.FileMask = "*.*"
$transferOptions.FilePermissions = $Null # This is default
$transferOptions.PreserveTimestamp = $False # if Timestamp on file is enable
# Loop through each relevant file
foreach($localFile in Get-ChildItem "D:\dq\*.csv" | Where-Object {($_.LastWriteTime -ge [datetime]::today)}){
$remotePath = "/Outbox/"
# Upload files, collect results
$transferResult = $session.PutFileToDirectory($localFile.FullName, $remotePath, $false, $transferOptions)
# Success or error?
if ($transferResult.Error -eq $Null) {
#$transferResult = $session.PutFiles($localPath, $remotePath, $False, $transferOptions).Check()
Write-Host ("Upload of {0} succeeded, moving to save" -f $transferResult.FileName)
}
else {
Write-Host ("Upload of {0} failed: {1}" -f $transferResult.FileName, $transferResult.Error.Message)
}
}
}
finally{
# ...
}
The answer by #Martias is correct. You cannot pass multiple paths to Session.PutFiles.
But you can let WinSCP itself pick the today's *.csv files with just two lines of code, using todays constraint.
Using Session.PutFilesToDirectory would also make the code somewhat simpler.
$transferOptions = New-Object WinSCP.TransferOptions
# (your other transfer options)
$transferOptions.FileMask = ">=today"
$transferResult =
$session.PutFilesToDirectory(
$localPath, $remotePath, "*.csv", $False, $transferOptions).Check()
I am using below script to Verify checksum of a remote file against a local file. The server I installed on my machine is freeSSHd.
When I tried to execute the below script using PowerShell ISE I get an error message saying:
Your shell is probably incompatible with the application (BASH is recommended)
I've granted shell access in the FreeSSHd Server User properties:
Script:
param (
# Use Generate URL function to obtain a value for -sessionUrl parameter.
$sessionUrl = "sftp://user:mypassword;fingerprint=ssh-rsa-xx-xx-xx#example.com/",
[Parameter(Mandatory = $True)]
$localPath,
[Parameter(Mandatory = $True)]
$remotePath,
[Switch]
$pause = $False
)
try
{
Write-Host $localPath -foregroundcolor Gray
# Calculate local file checksum
$localChecksum = ((CertUtil -hashfile $localPath SHA1)[1] -replace " ","")
# Write-Host "Local Checksum:"
Write-Host $localChecksum
# Load WinSCP .NET assembly
#Add-Type -Path (Join-Path $PSScriptRoot "WinSCPnet.dll")
[Reflection.Assembly]::LoadFrom("\\c:\Program Files (x86)\WinSCP\WinSCPnet.dll") | Out-Null
# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions
$sessionOptions.ParseUrl($sessionUrl)
$session = New-Object WinSCP.Session
try
{
# Connect
$session.Open($sessionOptions)
Write-Host $remotePath -foregroundcolor Gray
# Calculate remote file checksum
$sha1Command = "bash sha1sum -b $remotePath | awk '{print `$1}'"
$result = $session.ExecuteCommand($sha1Command)
$result.Check()
$remoteChecksum = $result.Output;
#$remoteChecksum =
[System.BitConverter]::ToString($session.CalculateFileChecksum("sha-1", $remotePath))
# Write-Host "Remote Checksum:"
Write-Host $remoteChecksum
}
finally
{
# Disconnect, clean up
$session.Dispose()
}
# Compare cheksums
if ($localChecksum -eq $remoteChecksum)
{
Write-Host
Write-Host "Match" -foregroundcolor "green"
$result = 0
}
else
{
Write-Host
Write-Host "Does NOT match" -foregroundcolor "red"
$result = 1
}
}
catch [Exception]
{
Write-Host $_.Exception.Message
$result = 1
}
# Pause if -pause switch was used
if ($pause)
{
Write-Host "Press any key to exit..."
[System.Console]::ReadKey() | Out-Null
}
exit $result
FreeSSHd server does not support any "bash". Its "shell" is Windows cmd.exe.
Your code cannot work. Windows cmd.exe is not compatible with WinSCP.
Moreover FreeSSHd is pretty buggy, do not use it.
You should use another Windows SSH server.
You can use Windows build of OpenSSH. It would allow you to execute a PowerShell script on the server to calculate the checksum.
If you install Windows Subsystem for Linux, you may even get the sha1sum (but I'm not sure).
You can use Cygwin, if you need to simulate *nix environment on Windows.
You can use Bitvise SSH Server for personal use for free. Its SFTP server supports checksum calculation on its own, so you would be able to use WinSCP Session.CalculateFileChecksum method directly.
There are lot of other options.
I am working on a PowerShell script that is run from another application that is called after a success event and executes a series of commands:
First it zips a designated file (ExampleFile)
Then deleted the original
Uploads the zipped file to a server through FTP
My problem starts during the FTP process: as it exists below - the ZIP file is created correctly in the local system, and a ZIP file is created on the FTP server but it's stuck with a filesize of 0 bytes.
From the server, it looks like the upload hangs?
So: Lines 1-3 all work fine and the local zipped file has a non-zero size.
Add-Type -A 'System.IO.Compression.FileSystem';
[IO.Compression.ZipFile]::CreateFromDirectory("ExampleFolder\ExampleFile", "ExampleFolder\ExampleFile_Datestamp.zip");
Remove-Item -Recurse -Force "ExampleFolder\ExampleFile";
$Username = "exampleusername";
$Password = "examplepassword";
$LocalFile = "C:\Users\example_path\ExampleFolder\ExampleFile.zip";
$RemoteFile = "ftp://exampleserver/examplepath2/ExampleFile_Datestamp.zip";
$FTPRequest = [System.Net.FtpWebRequest]::Create("$RemoteFile");
$FTPRequest = [System.Net.FtpWebRequest]$FTPRequest;
$FTPRequest.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile;
$FTPRequest.Credentials = new-object System.Net.NetworkCredential($Username, $Password);
$FTPRequest.UseBinary = $true;
$FTPRequest.UsePassive = $true;
$FileContent = gc -en byte $LocalFile;
$FTPRequest.ContentLength = $FileContent.Length;
$Run = $FTPRequest.GetRequestStream();
$Run.Write($FileContent, 0, $FileContent.Length);
$Run.Close();
$Run.Dispose();
This has me pretty well stumped, so any ideas or thoughts appreciated.
powershell.exe -nologo -noprofile -command "Add-Type -A 'System.IO.Compression.FileSystem'; [IO.Compression.ZipFile]::CreateFromDirectory(\"ExampleFolder\ExampleFile\", \"ExampleFolder\ExampleFile_Datestamp.zip\"); Remove-Item -Recurse -Force \"ExampleFolder\ExampleFile\"; $Username = \"exampleusername\"; $Password = \"examplepassword\"; $LocalFile = \"C:\Users\example_path\ExampleFolder\ExampleFile.zip\"; $RemoteFile = \"ftp://exampleserver/examplepath2/ExampleFile_Datestamp.zip\"; $FTPRequest = [System.Net.FtpWebRequest]::Create(\"$RemoteFile\"); $FTPRequest = [System.Net.FtpWebRequest]$FTPRequest; $FTPRequest.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile; $FTPRequest.Credentials = new-object System.Net.NetworkCredential($Username, $Password); $FTPRequest.UseBinary = $true; $FTPRequest.UsePassive = $true; $FileContent = gc -en byte $LocalFile; $FTPRequest.ContentLength = $FileContent.Length; $Run = $FTPRequest.GetRequestStream(); $Run.Write($FileContent, 0, $FileContent.Length); $Run.Close(); $Run.Dispose();
You are missing a call to FtpWebRequest.GetResponse:
...
$Run = $FTPRequest.GetRequestStream();
$Run.Write($FileContent, 0, $FileContent.Length);
$Run.Close();
$FTPResponse = $FTPRequest.GetResponse()
Write-Out $FTPResponse.StatusCode
Write-Out $FTPResponse.StatusDescription
See How to: Upload Files with FTP.
I have created several SharePoint hosted apps. Is it possible to create bulk app catalogs because the app is created for a little 100 clients. I don't find a PowerShell command that can do it for Sharepoint office 365 site catalogs.
Is it also possible to upload bulk apps with PowerShell. Don't find much info on the web.
Maybe anyone here has some advice ?
Cheers,
Kris
it is possible to bulk upload apps to an Office 365 environment as it is for SharePoint on-premise.
The tricky part here is that you have to upload the apps to the app catalog, which is in fact a document library.
Recently I created a function using scripts from the internet. Posted the function below:
function UploadAppToCatalog
{
[CmdletBinding()]
Param(
[Parameter(Mandatory=$true,ValueFromPipeline=$true)]
[string]$webUrl,
[Parameter(Mandatory=$true)]
[string]$DocLibName,
[Parameter(Mandatory=$true)]
[string]$FilePath
)
Start-SPAssignment -Global
$spWeb = Get-SPWeb -Identity $webUrl
$spWeb.AllowUnsafeUpdates = $true;
$List = $spWeb.Lists[$DocLibName]
$folder = $List.RootFolder
$FileName = $FilePath.Substring($FilePath.LastIndexOf("\")+1)
$File= Get-ChildItem $FilePath
[Microsoft.SharePoint.SPFile]$spFile = $spWeb.GetFile("/" + $folder.Url + "/" + File.Name)
$flagConfirm = 'y'
if($spFile.Exists -eq $true)
{
$flagConfirm = Read-Host "File $FileName already exists in library $DocLibName, do you want to upload a new version(y/n)?"
}
if ($flagConfirm -eq 'y' -or $flagConfirm -eq 'Y')
{
$fileStream = ([System.IO.FileInfo] (Get-Item $File.FullName)).OpenRead()
#Add file
write-host -NoNewLine -f yellow "Copying file " $File.Name " to " $folder.ServerRelativeUrl "..."
[Microsoft.SharePoint.SPFile]$spFile = $folder.Files.Add($folder.Url + "/" + $File.Name, [System.IO.Stream]$fileStream, $true)
write-host -f Green "...Success!"
#Close file stream
$fileStream.Close()
write-host -NoNewLine -f yellow "Update file properties " $spFile.Name "..."
$spFile.Item["Title"] = "Document Metrics Report"
$spFile.Item.Update()
write-host -f Green "...Success!"
}
$spWeb.AllowUnsafeUpdates = $false;
Stop-SPAssignment -Global
}