I want to know how to house keep(delete) the files which there in remote server by connecting using SFTP based retention period for n customers and each customer will have different directory.
Note: We don't have privileges to write script and run in remote server. We need to write script in our server to connect remote by using SFTP protocol.
Thanks,
Dasaradha
If you are on Linux sftp can run in batch mode and delete files on remote site: http://linux.die.net/man/1/sftp.
Related
What is the best method for reading files from a Windows shared folder using a DAG on a remote Apache Airflow server?
My current process needs to be executed a remote Airflow server, but it needs to read and write files (which the client and my process update) that are stored on a shared net Windows folder.
Airflow is installed on remote Linux server (same network)
Windows folders are just standard UNC paths where people have access based on their NT ID. These users are saving files which I need to retrieve.
The files formats are .csv, .xls and .xlsx, which I plan to convert as Dataframes in my Airflow process
Best if I can save my user and password within an Airflow connection to access it with a conn_id
I am currently using windows telnet to submit files to the OpenVMS queue via a series of sendkeys/application waits through VBA. It works, up until the end-user shifts focus away from the telnet window. I would prefer to issue the SUBMITs using an FTP session, where I can script the commands into a batch file and shoot it across FTP. I was able to do something similar with IBM mainframes - through the quote site FTP command - setting the filetype=jes, followed by a JCL file that would be dropped into the work queue for immediate execution. I can't seem to find anything on the internet related to FTP, openVMS, and submit. I have tried using Quote submit/que=... but it does not recognize the command. (Submit works fine under telnet).
Maybe you can use Remote Shell Protocol (RSH) to execute a command in a remote node
You would need a rsh client on windows:
http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/rsh.mspx?mfr=true
And also enable RSH service on VMS via TCPIP$CONFIG
(See OpenVMS documentation http://h71000.www7.hp.com/doc/index.html)
This works best with a VMS username dedicated to processing inbound FTP files. If you put in the LOGIN.COM for that username to detect it's a network connection and submit a batch job to look for the expected file, get exclusive access to it with retries (the FTP is done), and then process the file - That has worked for me.
The other option is to put a security ACL on the directory and make an audit listener - it will get file creates via a mailbox message. Then it can do similar: get exclusive access to the file being created and then process it.
I'm currently in a situation where I receive flat files via FTP from my clients. A couple of clients have insisted on the need to use SSH Private Key SFTP rather than regular FTP.
What I want to do is setup a web server (preferably in linux/unix but I guess I can do it on a windows server and purchase SFTP server software) that will do the following:
Allow me to setup an SFTP directory for each client with unique user/pass. Each directory also has to have the public/private key SSH "stuff" I'm a little new to this but I've googled it.
Once the file is completely uploaded by the client, I want to kick off an event that ftp's that file via regular FTP to my Windows cloud.
These files can be up to 10mb so the even that ftp's to the other server can't fire until the file is completely uploaded.
Has anyone set something like this up? Any guidance would be appreciated.
Thanks!
In Linux, you can use incron to monitor the directory the files will be SFTP'd to and have it trigger your ftp job. It's kind of like cron except that instead of triggering jobs based on time, it does so based on filesystem modifications. In order to only trigger once the entire file has been written, I think you can use IN_CLOSE_WRITE in the inotify mask. Failing this, I suggest configuring events for each of the events individually to echo a message to a log file and see if you can identify one which reliably happens only at the end of the SFTP transfer.
If you're using RedHat, it's not in the standard distribution, but it is in EPEL.
On Windows you could use Titan FTP Server Enterprise Edition, which supports SFTP as well as allows you to define various types of events. When the event is triggered, you could kick off anything you need on a per folder/per account basis.
PS. AFAIK, when it comes to SFTP it is either password authentication or public key authentication (SSH key), but not both.
In your UNIX server, you can configure SSH to use a custom sftp server that instead of handling SFTP protocol itself, opens a new SSH connection to to the Windows SFTP server using password authentication and forwards the SFTP traffic there.
Writting the proxy is easy with the right tools, for instance, in Perl using the Net::OpenSSH module:
#!/usr/bin/perl
# this is the sftp-proxy-server
use Net::OpenSSH;
my $ssh = Net::OpenSSH->new($windows_server, $user, $passwd);
$ssh->system({ssh_opts => '-s'}, 'sftp');
$ssh->error and die $ssh->error;
You can instruct the SSH server to use that alternative SFTP server changing the configuration in /etc/ssh/sshd_config. For instance:
Subsystem sftp /usr/local/bin/sftp-proxy-server
Did you try apache FTP Serveur ?
I think you can do what you need with the ftplet API.
see :
http://mina.apache.org/ftpserver-project/index.html
is there a way of how to connect to mysql dbf on a remote server and run sql queries using windows command line?
Yes, you can connect to a different host by running mysql -h 123.45.67.89.
Please note that there are a few security implications:
You will have to grant yourself access. You will need to run something like GRANT ALL on db_name.table TO user#your_ip IDENTIFIED BY 'password'. db_name, table and your_ip can be * but beware of opening your server to hackers.
You will have to open your server's firewall if you are not on the same LAN. Again, ymmv and you should be aware not to open the door to exploits.
You may want to use SSL and use secure-auth in order to protect your traffic and credentials.
Hope that helps.
MySQL has a command-line client, where you can run queries. If you don't want to allow remote connections to the database on the server, you can still script things into a batch. There are command-line telnet/ssh clients, that either accept external file as a list of commands to run remotely, or you can pass it with the input stream redirection (less then symbol) to them.
When opening a connection to server - most clients are programmed so that the only way to specify the login password is by typing it in from keyboard (yeah, they don't use default input stream). Things like that make it hard to script it. However, it may be possible to set up a certificate based login on SSH - you'd actually have to research that.
If the server that's hosting the MySQL database is also a web server - you could also think about putting some script (PHP, Perl, Python, Ruby - whatever you like) on the password protected area, that would allow you to execute queries by simply making a HTTP(S) queries on that script. Although, Windows doesn't have a command-line HTTP(S) client, you can always get something like wget.exe and perform queries with it. Note, that if you choose this approach - I strongly advice to put that script under HTTPS - if discovered by malicious user, it could be lethal to your data.
You could use telnet, or SSH if you want to be more secure.
If the MySQL is running on Linux or BSD, you need a Telnet or SSH connection through something like putty
This will open a command line on the remote server. The command is mysql. There will be issues around authentication of remote users (as you would expect).
If the remote server is running Windows, you have a whole different set of issues.
I'm not sure you can connect to a remote Windows server and control it this way.
I should say I'm not sure HOW you could connect to a remote Windows server and use it this way. But no doubt it's possible.
I use Filezilla to do my FTP duties and I'm unable to copy files without making a copy to the my local computer.
I doubt Filezilla is bad software. Is this just an inherent quality of FTP?
The FTP Protocol does not define a "copy" command.
If your FTP client has a copy command, then it has to implement this itself using what's available through FTP. This works out to something like: GET/RETR , CWD , PUT/STOR .
Some FTP servers will implement proprietary extensions and offer a command like COPY. A client may or may not choose to implement these, in which case it doesn't involve a local file copy.
According to the error messages I get within WinSCP when I try and duplicate or copy.. apparently it is.
I'm guessing there's some FTP theory on it somewhere. Even to duplicate WinSCP wanted to temp dump stuff to my local machine.
There aren't any copy commands listed at the list of FTP commands, so it's not in the FTP definition. It's just something FTP clients add themselves to be more user-friendly.