I have a database transfer script, which uses bzip2 to minimise locking of large databases on a server.
First line is
ssh root#server "mysqldump db | bzip2 >/root/db.sql.bz2"
This works on many servers, but on a new Ubuntu 14.04 server the file created on the server has a question mark appended:
ls -la gt*
-rw-r--r-- 1 root root 2364190 Nov 21 00:25 db.sql.bz2?
Any idea why this may be happening?
Does your script have CR+LF line-endings? Make sure to use Unix (LF) line endings.
Related
I'd like to create a file, for example lets call it /tmp/not_running_pods, that when read cat /tmp/not_running_pods runs kubectl get pods -o wide | grep -v Running and gives the output to the reading process.
This use case is simplified for example's sake. Not really looking for alternatives, unless it fits this exact case: file that outputs without having a 'service' always running listening for readers
Having a hard time finding anything specific for this on searching. My local env is macos, but hoping for something generalizable to linux/bash/zsh
edit: finally found what was on the tip of my brain, something like inetd / super-server - still looking to see if this would work for this case
when read cat /tmp/not_running_pods runs
A file is static. It exists.
HTTP web servers runs a php script (and much more stuff) to generate the web page for you to view. An SSHD server runs a shell for you to connect with. MYSQL server serves a specific protocol that allows to execute queries. To "do something" when a connection is made typically sockets - network, tcp, but also file sockets - are used, that allow with accept() detect incoming connections and run actually an action on such event.
# in one terminal
$ f() { echo new >&2; echo Hello world; LC_ALL=C date; }; export -f f; socat UNIX-LISTEN:/tmp/file,fork SYSTEM:'bash -c f'
new
new
# in the second terminal
$ socat UNIX-CONNECT:/tmp/file -
Hello world
Tue Jul 19 21:29:03 CEST 2022
$ socat UNIX-CONNECT:/tmp/file -
Hello world
Tue Jul 19 21:29:19 CEST 2022
If you really want to "execute an action when reading from a file", then you have to create your own file system that does that. Primary examples are files in /proc /sys. For user space file systems, write a program using FUSE.
Instead of
$ cat /tmp/not_running_pods
just make ~/bin/not_running_pods:
#! /bin/bash
kubectl get pods -o wide | grep -v Running
with chmod 755 and do
$ not_running_pods
Easy, well-understood, well-supported.
I'm attempting to move multiple files on an FTP server to a different directory on the same server. So far, I've written a bash script that will login and retrieve any new files in the remote directory but the ftp command doesn't support a 'mv' command. Essentially the script would download the new file(s) and then once downloaded move the file(s) to a different directory on the same server. Please Note that the filenames will be different every time so the use of wildcards is important here.
Before you answer please note that this needs to be automated so using a GUI like Filezilla wouldn't help me as I would have to login to various ftp sites and move the files manually, also, keep in mind that I'm unable to ssh into any of the servers as they are managed by other company's and ftp access is all I'm able to get. Last thing, I won't know what the file names are so using a wildcard would be helpful.
Any help or guidance is truly appreciated.
Thank you!
Perhaps the rename command in ftp could work for you?
rename [from [to]]
Rename the file from on the remote machine, to the file to.
I gave it a bash with an old file I had sitting on a server and it seemed to do what you want:
ftp> ls tmp/test*
229 Entering Extended Passive Mode (|||55572|)
150 Accepted data connection
-rw-r--r-- 1 sinasohn sinasohn 21 Mar 31 16:37 tmp/testfile01
226-Options: -a -l
226 1 matches total
ftp> ls tmp2/test*
229 Entering Extended Passive Mode (|||64715|)
150 Accepted data connection
226-Options: -a -l
226 0 matches total
ftp> rename tmp/testfile01 tmp2/testfile01
350 RNFR accepted - file exists, ready for destination
250 File successfully renamed or moved
ftp> ls tmp/test*
229 Entering Extended Passive Mode (|||56698|)
150 Accepted data connection
226-Options: -a -l
226 0 matches total
ftp> ls tmp2/test*
229 Entering Extended Passive Mode (|||50239|)
150 Accepted data connection
-rw-r--r-- 1 sinasohn sinasohn 21 Mar 31 16:37 tmp2/testfile01
226-Options: -a -l
226 1 matches total
ftp>
I put blank lines in between commands here for clarity.
Hope this helps!
full script to achieve move more than one file
1. get file list from ftp server with mls command
2. generate to do list file
2.1 get file
2.2 rename (move file)
3. execute ftp command with to do list file
#!/bin/sh
clear
# change local directory
cd [local-directory]
#collect file names
ftp -ni ftp.abccompany.com <<EOF
user [user] [password]
cd /OUT
mls abc*.* list.txt
quit
EOF
# create ftp action list
echo >>todo.lst user [user] [password]
while read N
do
echo >>todo.lst cd /OUT
echo >>todo.lst get $N
echo >>todo.lst rename $N ARCHIVE/$N
done <list.txt
echo >>todo.lst quit
# ftp transfer process
ftp -nv ftp.abccompany.com <todo.lst
# cleanup
rm todo.lst
Looking for a script what will show all logged users sorted by FreeBSD jails where they're logged in. So, need run the who command in all currently running FreeBSD jails and in the main host too.
I make this:
who #main host
jls | grep -v JID | while read jid ip host path
do
echo $jid $host
jexec $jid who
done
but the jexec need root execution and i'm logging in usually as non-root and make su everytime is painfull...
Is here any other simple way?
The who command in FreeBSD knows a file argument from where read informations about the logged-in users, the default is /var/run/utx.active - and the file is usually world-readable...
Probably will be enough the next script:
#!/usr/local/bin/bash
while read jpath
do
echo JWHO: ${jpath:-$(hostname)}
who "${jpath}/var/run/utx.active"
done < <( jls -h path | sed '1s:.*::' )
example output:
JWHO: marvin.example.com
smith pts/0 7 nov 20:55 (adsl2343-some-another.example.com)
JWHO: /jails/jail1
JWHO: /jails/testjail
root pts/2 7 nov 20:55 (someother.example.com)
JWHO: /jails/dbjail
steps:
show the path to "root filesystem" for all running jails
run the who for the /var/run/utx.active for the given jail
skip the header line from the jls, - so the 1st output will be the host.
Maybe someone know much simpler solution, e.g. by sorting the ps output or something like...
Comments: you usually don't want to use constructions like command | while read - the pipe forks new shell and you losing values of the variables set inside of the loop, the done < <( commands ) is usually better...
You can enable sudo in your system change your script just a little to:
sudo jexec $jid who
Then your srcipt can run as normal user.
How could I use grep and ls in FTP client...
I mean if I want to find some specific file I could use:
ls -l | grep pattern
With the usual Unix commandline interactive ftp, one approach is:
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> dir . foobar
output to local-file: foobar [anpqy?]? y
500 Unknown command
227 Entering Passive Mode (62,149,140,15,195,159)
150 Accepted data connection
11966 5.26 KB/s
226-Options: -a -l
226 156 matches total
ftp> !grep con foobar
-rwxr-xr-x 1 11050207 users 911007 Sep 13 2007 accu_pyconc.pdf
-rwxr-xr-x 1 11050207 users 9805405 Mar 25 2009 pycon_abst.pdf
i.e., get the dir results into a local file first, then run grep locally. Incidentally, this lets you run multiple greps after paying for just one dir data transfer;-).
lftp can, exactly the way you typed.
On Windows, you can do this with WinSCP scripting for any protocol, including the FTP:
winscp.com /command ^
"open ftp://username:password#example.com" ^
"ls /path/*.txt" ^
"exit"
References:
https://winscp.net/eng/docs/scripting
https://winscp.net/eng/docs/scriptcommand_ls
You can mount an sftp folder to a host where you have permissions to use grep or any tool.
After downloading files from a remote UNIX FTP server, you want to verify that you have downloaded all the files correctly. Minimal you will get information similar to "dir /s" command in Windows command prompt. The FTP client runs on Windows.
Sadly this was written for Unix/Linux users :/
Personally, I would install CYGWIN just to get Linux binaries of LFTP/RSYNC to work on windows, as there appears not to be anything that competes with it.
As #zadok.myopenid.com
mentioned rsync, this appears to be a windows build for it using CYGWIN ( if you manage to be able to get ssh access to the box eventually )
http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp
Rsync is handy in that it will compare everything with check sums, and optimally transfer partial change blocks.
If you get CYGWIN/Linux:
http://lftp.yar.ru/ is my favorite exploration tool for this.
It can do almost everything bash can do, albeit remotely.
Example:
$ lftp mirror.3fl.net.au
lftp mirror.3fl.net.au:~> ls
drwxr-xr-x 14 root root 4096 Nov 27 2007 games
drwx------ 2 root root 16384 Apr 13 2006 lost+found
drwxr-xr-x 15 mirror mirror 4096 Jul 15 05:20 pub
lftp mirror.3fl.net.au:/> cd games/misc
lftp mirror.3fl.net.au:/games/misc>find
./
./dreamchess/
./dreamchess/full_game/
./dreamchess/full_game/dreamchess-0.2.0-win32.exe
./frets_on_fire/
./frets_on_fire/full_game/
./frets_on_fire/full_game/FretsOnFire-1.2.451-macosx.zip
./frets_on_fire/full_game/FretsOnFire-1.2.512-win32.zip
./frets_on_fire/full_game/FretsOnFire_ghc_mod.zip
./gametap_setup.exe
......
lftp mirror.3fl.net.au:/games/misc> du gametap_setup.exe
32442 gametap_setup.exe
lftp mirror.3fl.net.au:/games/misc> du -sh gametap_setup.exe
32M gametap_setup.exe
lftp mirror.3fl.net.au:/games/misc>
Do this :
ls -lR
..................
If you have ssh access, use rsync instead. It is a far better data transfer app.
Grab fuse for your OS and load ftpfs. This will let you mount the remote ftp directory locally and you can use dir /s or any other application you want on it.
Assuming you are using simple ftp via command line,
Use dir command with -Rl option to search recursively and copy it to a file and then search the file using grep, find or whatever way is supported on your OS.
ftp> dir -Rl education.txt
output to local-file: education.txt? y
227 Entering Passive Mode (9,62,119,15,138,239)
150 Opening ASCII mode data connection for file list
226 Transfer complete
You can use ftp.listFiles("directory") from apache-commons-net and can write your own BFS or DFS to fetch all the files recursively.