I'm building a (for now pretty minimal) network sync system for some of our users, involving a samba server on one end and an rsync cron job which is "installed" for OSX or Linux clients by running a simple bash script linked from our intranet.
I need to do the same thing for Windows clients. I know there are several rsync implementations on Windows (I used cwRsync ages ago), but are there any (off the top of your head) that I can silently pass a config to during install? As it is, I guess I'm going to have to write a crappy old batchfile to interface with Windows Task Scheduler, but I'd at least like for clients installing this to not have to input any more than their username and password.
Thanks!
I've had success with
RichCopy
RoboCopy
Cygwin rsync.exe
All using scheduled tasks.
RichCopy (and maybe robocopy) have options to save config files from the GUI. All worked well for me from a batch file.
All three have restartable/incremental modes. Most are highly aware of specific features think
NTFS encryption
NTFS compression
permissions (ACLs)
alternate NTFS streams
junctions/reparse points
hardlinks/symlinks
etc.
Related
Our team has ~80 Windows development machines, and activities of each developer are logged as text files on the local storage of those machines.
To analyze the logged activities, I want to gather all log files from those machines. Additionally, the log files are updated constantly, so It is desirable to gather files with the command-line from my machine.
I’ve searched and found some solutions, but all of those are not suitable for our situation:
We cannot use PsExec, because tcp/135 and tcp/445 are both closed (countermeasure for WannaCry).
Administrative share is disabled.
telnet service is not up and is banned by security reasons.
WinRM is disabled on those machines by default.
It is difficult to install new software like OpenSSH on those machines (because of the rule of this project)
RDP is the only way to connect those machines. (I have an account on all machines)
How can I copy files from remote Windows machines with command-line through RDP?
Or, at least, is there any way to execute a command on remote Windows machines with command-line through RDP?
I think you can do this, though it is very hacky :)
For a basic setup, which just copies files once, what you would need to do is
Run a script in the remote session when it logs in. I can think of three ways to do this:
Use the "Alternate Shell" RDP file property. This runs a specified program in place of explorer.exe on login; you can use it to run "cmd.exe /c [your script]" for instance.
If that doesn't work (e.g. the remote machine doesn't respect it), you might be able to use a scheduled task that runs the script on login, but perhaps only for a specified user, or maybe the script could check the WinStation type to make sure this is actually an RDP connection before doing anything.
It's also possible to do this by connecting in RemoteApp mode and using the script as your "application", but that only works for Server and Enterprise editions of Windows.
Enable either drive redirection or clipboard redirection on the RDP connection, to give you a way to get data out.
Drive redirection is much simpler to script; you just have the remote script copy files to e.g. "\\tsclient\C\logs".
Clipboard redirection is theoretically possible - you have the remote script copy, then a local script paste - but would probably be a pain to get working in practice. I'm only mentioning it in case drive redirection isn't available for some reason.
You would probably want to script to then log the session off afterward.
You could then launch that from command-line by running "mstsc.exe [your RDP file]". The RDP files could be programmatically generated if needed (given you're working with 80 machines).
If you want a persistent connection you can execute commands over, that's more complicated, but still technically possible. Two ways I can think of:
Use the previous method to run a program on logon, but this time create a custom application that receives commands using a transport that isn't blocked and executes them in the session. I've done this with WCF over HTTP, for instance; it's not secure, of course.
Develop and install a service on the remote machine that opens an RDP virtual channel, and a corresponding RDP client plugin that communicates with it. You can then do whatever you want across the connection. While this solution would be the most likely to work, it's also the most heavyweight and time-consuming to implement so it's probably a last resort.
I need to pull log files from a Windows server. The log files appear every minute and I am trying to ftp to the server pull the file back to the linux box and process it. I have been looking around and found about inotify, I am not sure how to use it in a bash script after ftp to the Windows server. I am open to other implementation it does not have to be ftp/inotify but I am not sure how this could be done.
any ideas??
Two products come to mind depending on how you plan to approach the solution.
I personally use Splunk on a variety of platforms (Windows & Linux servers/local Linux & OSX dev environments). It is a real-time log aggregator that features an API and the ability to query. Even if this doesn't solve your problem, the free version has some very robust features that you should consider: http://www.splunk.com
The second approach would be synchronization of your web directories using something like RSync. I've used RSync on Linux boxes and always appreciated what it can do. I even see it now has a Windows port: https://www.itefix.no/i2/cwrsync
I am needed to move entire directories from one computer in the network to the other (In a platform independent way). Basically I am working on some automation tool to help the developers do Build Verification Tests, for this; I am directed to automate the installation and un-installation of the product on multiple platforms. So, I will need to first copy the files!
And this is where I needed some help in both conceptual and practical knowledge.
Firstly, let me mention that using something like FileZilla or WinSCP is out of the question since I need things to happen automatically and not through button clicks. But please let me know if these tools have any command line utilities!
I tried Perl's NET::FTP, and while it looked promising, I was wondering whether it was the best way to go. Also, I want to know what are the pre-requisites before I can run FTP, I mean would I need perl installed on the other end as well ? I constantly read that the commands from perl's FTP actually try to connect to a FTP host, does this mean its not going to work if I haven't configured the remote host in some way? And if I am right, then what is this extra piece of configuration to be done?
Apart from this, is there any other way I could solve my problem ? I mean I am looking for API's here that would help me do platform independent file transfers. But once again, I cannot use tools that would need button clicks and stuff, because I am doing automation and everything needs be dome programmatic-ally and automatically.
Also, I think this is a very generic problem-statement: "Moving files across a computers connected by LAN"; So, it would be wonderful if we can have a list of (possibly) many options (ways to solve the problem) in the form of answers to this post.
Thanks in advance for any help that you wish to provide.
If nearly all of the files in your directory have changed, creating an archive, sending it over the network, and unarchiving makes sense. Actually, if your LAN is fast enough, though, it may be faster not to compress the archive--just use tar.
If only some of the files have changed, rsync, a command line tool, will only download the changes. It can be used with ssh like this:
rsync -ae ssh username#hostname:/path/to/files /store/here/locally
http://www.thegeekstuff.com/2010/09/rsync-command-examples/
On Linux and OS X, cron and crontab allow you to schedule scripts to run periodically. Windows provides the Windows Task Scheduler.
FTP is fine if you don't care about encryption over your LAN. Otherwise, SSH would be preferable.
rsync is available on OS X and Linux, but I think you can use it on Windows through Cygwin.
I suggest making an archive (e.g. a .tar.gz file) on the source host, transferring it with scp, and unarchive it on the target host.
You could also use unison or rsync
I would suggest you to develop your own FTP client in .NET. This way you will have complete control over the application, and instead of button-clicks you can schedule it using windows-scheduler. Here is an article about how to create your own FTP client in VB.NET:
http://dot-net-talk.blogspot.com/2008/12/how-to-create-ftp-client-in-vbnet.html
I need to provide an update to application data as a download from a website. The update would actually just be the replacing of some data files with some updated ones.
The update, which I assume would be some sort of setup package type program, would need to be able to do the following:
access the file system and registry
to determine where files should be
copied to
supply the files to be copied
provide strong security so the data files cannot be downloaded or used by the wrong people
What would be best way to achieve all of the above?
I would suggest to use either
Java Web Start, for Java
ClickOnce Deployment, for .NET
These technologies are meant to distribute software over networks.
(I think that developing something similar yourself can turn to be something rather complicated.)
There are a number of comprehensive update manager and delivery services available from commercial software vendors if you don't want to write something from scratch yourself.
TrueUpdate
InstallShield
ClickOnce
... plus more, I'll leave you to research yourself :)
Your choice will depend on your chosen language, existing setup methods, targeted operating systems and budget.
We just use standard MSI installers/upgrades and wrote our own update manager/update delivery service. You'll need to weigh up own resources and development expense to decide if it's worth purchasing a commercial product, or developing something in-house.
I've used AdvancedInstaller - a very capable free version is available. It is a fully-fledged MSI installer, so it can read/write to the registry and install files to "Program Files" that normally requires admin privileges. It also supports application updates out of the box. You can publish updates and have them automatically downloaded and installed.
As to security, this is not really the realm of an Installer but part of the webserver where you host the files. You could use a password protected folder, or for stronger prevention, client SSL certificates. The latter would work well with automatic updates.
An alternative is to string together a solution from a number of readily available components:
use a batch file to perform the download that is executed regularly. You can schedule batch files using the Windows Task Scheduler.
The batch file will use the Windows REG command to read values from the registry, and use those values to invoke rsync. rsync is a file synchronization command that can be run through ssh. If you configure ssh with public key authentication, you restrict access to specific clients of your choosing, while allowing automated access (no password required.) The key with rsync is that if the files are up to date, no download is performed.
This article explains how to sync files from a server to a notebook using rsync and ssh, and explains how to set up rsync and ssh using Cygwin.
See also
Windows REG command
ssh (Wikipedia) and openssh manpages.
rsync manpage
My interactive 32-bit Windows app (now moving from Delphi [Ent] 2007 to 2009) uses command-line interactions to spawn child processes that do computationally-intensive tasks, which in turn write text files that the GUI parent app parses and analyzes - resulting in an interactive graphical display of the results.
I have access to a multiprocessor (multi-user) Linux cluster (via ssh), and would like to off-load the heavy lifting to that cluster. My question is how to spawn the processes in Linux from my Windows app. I can envision using secure FTP to put and get files, but not sure how to spawn the child processes in Linux.
Some leads for further reading would be fine - but code/pseudocode would be ideal. I can imagine that this may be more about Windows-Linux interaction than Delphi.
if you have access to ssh, one option is to issue commands through that.
For example:
ssh user#host ls -l ~
will in the ssh terminal show the files in the user's home directory. I'm not sure if this is what you really want. But it would likely work.
If you do this, you almost certainly want to setup SSH password less logins
However, A more ideal solution would likely be to setup a daemon on the linux boxes whose sole job is to run specific long running tasks in the background and let you fetch the results later.
You're going to have to install something on the Linux machine to run the process. You might find some kind of clustering or batch job submission API you can install and access from Windows. You might have to code a custom server. You might be able to run everything over ssh if you can drive an ssh process from Windows and if you have sshd installed on the Linux side. But my preference would be to write a webservice or simple CGI script on the Linux side designed to take your arguments and data and return the result over plain old http (or https as the case might be).
One way or another, this is going to encompass more than just coding on the Windows side.
I would download the full "putty" package.
As well as the excellent secure shell terminal, it includes PSCP to transfer files securely and PLINK to remote execute commands over SSH.
Hint: you will need to set up the full public/private key configuration for PLINK to work without an annoying password prompt. There is a useful guide http://unixwiz.net/techtips/putty-openssh.html>here.