I have the following:
mysqldump -u xxxx
-h localhost
--password=xxxxx databasename |
ssh username#00.000.00.202 "dd of=httpdocs/backup`date +'%Y-%m-%d-%H-%M-%S'`.sql"
...which SSH's a mysqldump to a remote machine.
I need to compress the mysqldump before it is SSH'd as the dump is 500mb and its eating up my bandwidth allowance.
mysqldump ... | gzip -9 | ssh ...
or
mysqldump ... | bzip2 -9 | ssh ...
or, if you want it uncompressed on the other end
mysqldump ... | bzip2 -9 | ssh machine "bzip2 -d >..."
mysqldump ... | gzip -9 | ssh machine "gzip -d >..."
You can add the -C flag to the ssh call to automatically compress the transmitted data.
You need to call gzip between mysqldump and ssh, like:
mysqldump [mysql options] | gzip | ssh [ssh options]
I would recommend changing the saved file extension to ".sql.gz" as well.
This has already been answered and accepted, but I thought you might find this an interesting alternative.
Percona's OpenSource xtrabackup application will perform compressed (TAR) backups on the fly - along with lots of other interesting things.
I couldn't find an anchor on the page, but scroll down to "Compressed Backups".
Related
This is a 2 part question.
Ive made a bash script that logs into a remote server makes a list.txt and saves that locally.
#!/bin/bash
sshpass -p "xxxx" ssh user#pass ls /path/to/files/ | grep "^.*iso" > list.txt
It then starts a for loop using the list.txt
for f in $(cat list.txt); do
The next command splits the target file and saves it locally
sshpass -p "xxxx" ssh user#pass tar --no-same-owner -czf - /path/to/files/$f | split -b 10M - "$f.tar.bz2.part"
Question 1
I need help understanding the above command, why is it saving the *part files locally? Even though that is what I intend to do I would like to understand it better, How would I do this the other way round, tar and split files saving output to remote directory (flip around what is happening in the above command using the same tools sshpass is a requirement)
Question 2
When running the above command even though I have made it not verbose it still prints this message
tar: Removing leading `/' from member names
How do I get rid of it as I have my own echo output as part of the script I have tried the following after searching online but I think me piping a few commands together confuses tar and breaks the operation.
I have tried these with no luck
sshpass -p "xxxx" ssh user#pass tar --no-same-owner -czfP - /path/to/files/$f | split -b 10M - "$f.tar.bz2.part
sshpass -p "xxxx" ssh user#pass tar --no-same-owner -czf -C /path/to/files/$f | split -b 10M - "$f.tar.bz2.part
sshpass -p "xxxx" ssh user#pass tar --no-same-owner -czf - /path/to/files/$f | split -b 10M - "$f.tar.bz2.part > /dev/null 2>&1
sshpass -p "xxxx" ssh user#pass tar --no-same-owner -czf - /path/to/files/$f > /dev/null 2>&1 | split -b 10M - "$f.tar.bz2.part
All of the above break the operation and I would like it to not display any messages at all. I suspect it has something to do with regex and how the pipe passes through arguments. Any input is appreciated.
Anyways this is just part of the script the other part uploads the processed file after tar and splitting it but Ive had to break it up into a few commands a 'tar | split' locally, then uploading via rclone. It would be way more efficient if I could pipe the output of split and save it remotely via ssh.
First and foremost, you must consider the security vulnerabilities when using sshpass.
About question 1:
Using tar with -f - option will create the tar on the fly and will send to stdout.
The | separates the commands.
sshpass -p "xxxx" ssh user#pass tar --no-same-owner -czf - /path/to/files/$f - Runs remotely
split -b 10M - "$f.tar.bz2.part" - Runs in local shell
The second command reads the stdin from the first command (the tar output) and it creates the file locally.
If you want to perform all the operations in the remote machine, you could enclose the rest of the commands in quotes like this (read other sources about qouting).
sshpass -p "xxxx" ssh user#pass 'tar --no-same-owner -czf - /path/to/files/$f | split -b 10M - "$f.tar.bz2.part"'
About question 2.
tar: Removing leading '/' from member names is generated by tar command which sends errors/warnings to STDERR which in the terminal, STDERR defaults to the user's screen.
So you can suppress tar errors by adding 2>/dev/null:
sshpass -p "xxxx" ssh user#pass tar --no-same-owner -czf - /path/to/files/$f 2 > /dev/null | split -b 10M - "$f.tar.bz2.part
Hi I want to automate my workflow of creating Postgres dump and downloading it. I want to do it from my local machine, for now I figured it out so far in two seperate commands:
sshpass -p "FuckinHardPass" ssh andi#1.2.3.4 "pg_dump -U andi andi_some_db -f /home/andi/PSQL_DUMPS/andi_some_db.sql"
sshpass -p "FuckinHardPass" scp -r andi#1.2.3.4:/home/andi/PSQL_DUMPS/andi_some_db.sql .
how can I join it in one command using pipes etc?
Thanks James Hightower for hint, using your answer I complete it in one command:
sshpass -p "FuckinHardPass" ssh andi#1.2.3.4 "pg_dump -U andi andi_some_db" > andi_some_db.sql
As pg_dump defaults to stdout as it's output file, and ssh displays the command's stdout on it's own stdout, you could do something like:
ssh andi#1.2.3.4 'pg_dump -U andi andi_some_db' > andi_some_db.sql
which would save the output from the command on your local disk as andi_some_db.sql
Depending on the size of your dump and the speed of your connection, you could perhaps benefit from pre-compressing your output:
ssh andi#1.2.3.4 'pg_dump -U andi andi_some_db | gzip' > andi_some_db.sql.gz
And so on.
I am using ssh from a program which sends commands to ssh and parses answers. However, each time I log in, I get the welcome banner like:
Linux mymachine 3.2.0-4-686-pae #1 SMP Debian 3.2.54-2 i686
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
...
I do not want this banner, because my parser would need to deal with it. Is it possible to login with ssh and not to get this banner at the beginning?
You should be able to silence this banner, and other diagnostic messages, by passing -q to SSH:
ssh -q user#remote_host
If you want to make -q permanent for all your SSH sessions, do:
echo "LogLevel QUIET" >> ~/.ssh/config
What works here seems to depend on the operating system, SSH version, and the server-side configuration of sshd.
For connecting to a stock Ubuntu 18 server ssh -q didn't work for me, and neither did ssh -o LogLevel=error that is suggested elsewhere.
What did work is the comment posted under the question about creating a .hushlogin file in the remote user's home directory:
$ ssh myuser#myhost
Welcome to Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-55-generic x86_64)
<snip>
Last login: Thu Aug 1 14:04:26 2019 from 1.2.3.4
myuser#myhost$ touch .hushlogin
myuser#myhost$ exit
Then:
$ ssh myuser#myhost echo 'Test'
Test
This will run command1 command2 and command3 on the remote_host.
ssh user#remote_host 'command1; command2; command3'
No banners are displayed.
Try ssh -q to supress the banner message
If you expect more than 1000 lines in the server answer then replace 1000 with a corresponding number or the server answer will be truncated.
# Demo script file creation \
DIVIDER="___"; echo "echo $DIVIDER; echo 100; echo 200; echo 300;" > "./test.sh"; \
# \
# Getting the answer without the banner \
ssh -q login#server.name < "./test.sh" | grep -A1000 -e "^$DIVIDER" | tail -n +2
Success
100
200
300
The same command without
| grep -A1000 -e "^$DIVIDER" | tail -n +2
gives
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux...
[...]
Run 'do-release-upgrade' to upgrade to it.
___
100
200
300
You can replace "___" (three underscores) with any exotic sign(s) or even password (which can't be found in the beginnings of lines of the banner).
To avoid the replacing 1000 with a corresponding number (and possible truncation of big server answers) search something about "how to grep all lines after match" and modify my code.
For running commands remotely:
#!/bin/bash
SCRIPT='
#Your commands
'
sshpass -p<pass> ssh -o 'StrictHostKeyChecking no' -p <port> user#host "$SCRIPT"
I answer my own question with the solution based on Keith Reynolds answer. I am using:
ssh my_host bash
allowing bash interaction without banner and without prompt.
How to write a echo server bash script using tools like nc, echo, xargs, etc capable of simultaneously processing requests from multiple clients each with durable connection?
The best that I've came up so far is
nc -l -p 2000 -c 'xargs -n1 echo'
but it only allows a single connection.
If you use ncat instead of nc your command line works fine with multiple connections but (as you pointed out) without -p.
ncat -l 2000 -k -c 'xargs -n1 echo'
ncat is available at http://nmap.org/ncat/.
P.S. with the original the Hobbit's netcat (nc) the -c flag is not supported.
Update: -k (--keep-open) is now required to handle multiple connections.
Here are some examples. ncat simple services
TCP echo server
ncat -l 2000 --keep-open --exec "/bin/cat"
UDP echo server
ncat -l 2000 --keep-open --udp --exec "/bin/cat"
In case ncat is not an option, socat will also work:
socat TCP4-LISTEN:2000,fork EXEC:cat
The fork is necessary so multiple connections can be accepted. Adding reuseaddr to TCP4-LISTEN may be convenient.
netcat solution pre-installed in Ubunutu
The netcat pre-installed in Ubuntu 16.04 comes from netcat-openbsd, and has no -c option, but the manual gives a solution:
sudo mknod -m 777 fifo p
cat fifo | netcat -l -k localhost 8000 > fifo
Then client example:
echo abc | netcat localhost 8000
TODO: how to modify the input string value? The following does not return any reply:
cat fifo | tr 'a' 'b' | netcat -l -k localhost 8000 > fifo
The remote shell example however works:
cat fifo | /bin/sh -i 2>&1 | netcat -l -k localhost 8000 > fifo
I don't know how to deal with concurrent requests simply however.
what about...
#! /bin/sh
while :; do
/bin/nc.traditional -k -l -p 3342 -c 'xargs -n1 echo'
done
I'm currently using mySQLdump to backup my dev machine and servers.
There is one project I just started, however, that has a HUUUUUGE database that I don't really need backed up, and i'll be a big problem to add it to the rest of the backup cycle.
I'm currently doing this:
"c:\Program Files\mysql\MySQL Server 5.1\bin\mysqldump" -u root -pxxxxxx --all-databases > g:\backups\MySQL\mysqlbackup.sql
Is it possible to somehow specify "except this database(s)"?
I wouldn't like to have to specify the list of DBs manually, since that would mean that I'd have to remember updating my backup batch file every time I create a new DB, and I know that's not gonna happen.
EDIT: As you probably guessed from my command line above, i'm doing this on Windows, so I can't do any kind of fancy bash stuff, only wimpy .bat things.
Alternatively, if you have other ideas to solve this same issue, they are more than welcome, of course!
mysql ... -N -e "show databases like '%';" |grep-v -F databaseidontwant |xargsmysqldump ... --databases > out.sql
echo 'show databases;' | mysql -uroot -proot | grep -v ^Database$ | grep -v ^information_schema$ | grep -v ^mysql$ | grep -v -F db1 | xargs mysqldump -uroot -proot --databases > all.sql
dumps all databases except: mysql, information_schema, mysql and db1.
Or if you'd like to review the list before dumping:
echo 'show databases;' | mysql -uroot -proot > databases.txt
edit databases.txt and remove any you don't want to dump
cat databases.txt | xargs mysqldump -uroot -proot --databases > all.sql
What about
--ignore-table=db_name.tbl_name
Do not dump the given table, which must be specified using both the database and table names. To ignore multiple tables, use this option multiple times.
Maybe you'll need to specify a few to completely ignore the big database.
I created the following one line solution avoiding multiple grep commands.
mysql -e "show databases;" | grep -Ev "Database|DatabaseToExclude1|DatabaseToExclude2" | xargs mysqldump --databases >mysql_dump_filename.sql
The -E in grep enables extended regex support which allowed to provide different matches separated by the pipe symbol "|". More options can be added to the mysqldump command. But only before the "--databases" parameter.
Little side note, i like to define the filename for the dump like this ...
... >mysql_dump_$(hostname)_$(date +%Y-%m-%d_%H-%M).sql
This will automatically ad the host name, date and time to the filename. :)
Seeing as your using Windows you should have PowerShell available to use.
Here is a short PowerShell script to get a list of all Databases, remove unwanted ones from the list & then use mysqldump to backup the others.
$MySQLPath = "."
$Hostname = "localhost"
$Username = "root"
$Password = ""
# Get list of Databases
$Databases = [System.Collections.Generic.List[String]] (
& $MySQLPath\mysql.exe -h"$Hostname" -u"$Username" -p"$Password" -B -N -e"show databases;"
)
# Remove databases from list we don't want
[void]$Databases.Remove("information_schema")
[void]$Databases.Remove("mysql")
# Dump database to .SQL file
& $MySQLPath\mysqldump.exe -h"$HostName" -u"$Username" -p"$Password" -B $($Databases) | Out-File "DBBackup.sql"
Create a backup user and only grant that user access to the databases that you want to backup.
You still need to remember to explicitly grant the privileges but that can be done in the database and doesn't require a file to be edited.
It took me a lot of finagling to come up with this but I've used it for a few years now and it works well...
mysql -hServerName -uUserName -pPassword -e "SELECT CONCAT('\nmysqldump -hServerName -uUserName -pPassword --set-gtid-purged=OFF --max_allowed_packet=2048M --single-transaction --add-drop-database --opt --routines --databases ',DBList,' | mysql -hServerName2 -uUserName2 -pPAssword2 ' ) AS Cmd FROM (SELECT GROUP_CONCAT(schema_name SEPARATOR ' ') AS DBList FROM information_schema.SCHEMATA WHERE LEFT(schema_name, 8) <> 'cclegacy' AND schema_name NOT IN ('mysql','information_schema','performance_schema','test','external','othertoskip')) a \G" | cmd
Instead of the pipe over to mysql where I'm moving from serverName to Servername2 you could redirect to a file but this allows me to tailor what I move. Sometimes i even OR the list so I can say LIKE 'Prefix%' etc.
You can use this one for production
It excludes 'performance_schema\|information_schema\|mysql\|sys'...modify for your needs
MYSQL_USER=
MYSQL_PASS=
MYSQL_HOST=
MYSQL_CONN="-u${MYSQL_USER} -p${MYSQL_PASS} -h${MYSQL_HOST}"
MYSQLDUMP_OPTIONS="--routines --triggers --single-transaction"
DBLIST=`mysql -s --host=$MYSQL_HOST --user=$MYSQL_USER --password=$MYSQL_PASS \
--execute="SHOW DATABASES;" | grep -v \
'performance_schema\|information_schema\|mysql\|sys' | awk '{printf("\"%s\" ",$0)}'`
mysqldump ${MYSQL_CONN} ${MYSQLDUMP_OPTIONS} --databases ${DBLIST} | gzip >all-dbs.sql.gz