I want to change the name of a file if it is already present on a remote server via SSH.
I tried this from here (SuperUser)
bash
ssh user#localhost -p 2222 'test -f /absolute/path/to/file' && echo 'YES' || echo 'NO'
This works well with a prompt, echoes YES when the file exists and NO when it doesn't. But I want this to be launched from a crontab, then it must be in a script.
Let's assume the file is called data.csv, a condition is set in a loop such as if there already is a data.csv file on the server, the file will be renamed data_1.csv and then data_2.csv, ... until the name is unique.
The renaming part works, but the detection part doesn't :
while [[ $fileIsPresent!='false' ]]
do
((appended+=1))
newFileName=${fileName}_${appended}.csv
remoteFilePathname=${remoteFolder}${newFileName}
ssh pi#localhost -p 2222 'test -f $remoteFilePathname' && fileIsPresent='true' || fileIsPresent='false'
done
always returns fileIsPresent='true' for any data_X.csv. All the paths are absolute.
Do you have any idea to help me?
This works:
$ cat replace.sh
#!/usr/bin/env bash
if [[ "$1" == "" ]]
then
echo "No filename passed."
exit
fi
if [[ ! -e "$1" ]]
then
echo "no such file"
exit
fi
base=${1%%.*} # get basename
ext=${1#*.} # get extension
for i in $(seq 1 100)
do
new="${base}_${i}.${ext}"
if [[ -e "$new" ]]
then
continue
fi
mv $1 $new
exit
done
$ ./replace.sh sample.csv
no such file
$ touch sample.csv
$ ./replace.sh sample.csv
$ ls
replace.sh
sample_1.csv
$ touch sample.csv
$ ./replace.sh sample.csv
$ ls
replace.sh
sample_1.csv
sample_2.csv
However, personally I'd prefer to use a timestamp instead of a number. Note that this sample will run out of names after 100. Timestamps won't. Something like $(date +%Y%m%d_%H%M%S).
As you asked for ideas to help you, I thought it worth mentioning that you probably don't want to start up to 100 ssh processes each one logging into the remote machine, so you might do better with a construct like this that only establishes a single ssh session that runs till complete:
ssh USER#REMOTE <<'EOF'
for ((i=0;i<10;i++)) ; do
echo $i
done
EOF
Alternatively, you can create and test a bash script locally and then run it remotely like this:
ssh USER#REMOTE 'bash -s' < LocallyTestedScript.bash
Im trying to check connections for a list of servers. I want to loop through the list, check if a connection works and if yes, do some stuff, if not, echo out a problem message.
My problem is:
the script stops at the first node without echoing the $?.
So, whats wrong with my for-loop?
These vars are included from a config file:
$nodes is a list of server IPs like 1.1.1.1,2.2.2.2,10.10.10.10
$user is one string
for node in $(echo $nodes | sed "s/,/ /g")
do
echo "Checking Node: $node"
ssh -q -o ConnectTimeout=3 $user#$node echo ok
echo $?
if [[ $? != 0 ]]
then
echo "Problem in logging into $node"
else
# do some stuff here
fi
done
EDIT #1:
for node in $(echo $nodes | sed "s/,/ /g")
do
echo "Checking Node: $node"
ssh -q -t -o ConnectTimeout=3 $user#$node "echo ok"
retcode=$?
echo $retcode
if [[ "$retcode" -ne 0 ]]
then
echo "Problem in logging into $node"
else
echo "OK"
fi
done
It is because ssh first asks you to validate The authority of the host and If you accept the authority it will ask for password. That is why your command does not return to shell and waits for input.
If your intention is just validating ssh connection, then you may consider to use
telnet <your_host> <port> < /dev/null
But if your intend is to run some commands you need a trust relationship between hosts. In that case you can use:
Execute this commands:
ssh-keygen -t rsa
then
ssh-copy-id -i root#ip_address
Now you can connect with
ssh <user>#<host>
Furher information
You can add -tto make virtual terminal and add quotes on command:
ssh -q -t -o ConnectTimeout=3 ${user}#${node} "echo ok"
Also use -ne instead of != which is for compare strings
if [[ "$?" -ne 0 ]]
Also echo $? mess the return code. You should use something like:
ssh -q -t -o ConnectTimeout=3 ${user}#${node} "echo ok"
retcode=$?
echo $retcode
if [[ "$retcode" -ne 0 ]]
You can rewrite ssh command like this to avoid problems with ssh host keys
ssh -q -t -o StrictHostKeyChecking=no -o ConnectTimeout=3 ${user}#${node} "echo ok"
I am creating a bash script to automate some commands and I am having some trouble writing my error checking to the same file.
#!/bin/bash
touch ErrorLog.txt
bro-cut service < conn.log | sort | uniq -c > ProtocolHierarchy.txt
if [ $? -eq 0 ]; then
echo -e "OK Protocol Hierarchy Created\n" > ErrorLog.txt
else
echo -e "FAILED Creating Protocol Hierarchy\n" > ErrorLog.txt
fi
bro-cut id.orig_h < dns.log | sort | uniq -c > AllIPAddresses.txt
if [ $? -eq 0 ]; then
echo -e "OK Created all IP Addresses\n" > ErrorLog.txt
else
echo -e "FAILED Creating all IP Addresses\n" > ErrorLog.txt
fi
The goal being to have a file I can open and see that all the commands worked or failed, currently the file looks like this
-e OK Created all IP Addresses
When I would like it to look like this
OK Protocol Hierarchy Created
OK Created all IP Addresses
I am really new to bash scripting so any tips would be greatly appreciated!
Open it once, and write to that file descriptor multiple times.
# Open (creating or truncating) the output file (only once!)
exec 3>ErrorLog.txt
# Write a line to that already-open file
echo "something" >&3
# Write a second line to that already-open file
echo "something else" >&3
# Optional: close the output file (can also be implicit when the script exits)
exec 3>&-
The other common idiom is to open in append mode using >>, but doing that once per line is considerably less efficient.
# Open ErrorLog.txt, truncating if it exist, write one line, and close it
echo "something" >ErrorLog.txt
# Reopen ErrorLog.txt, write an additional line to the end, and close it again
echo "something else" >>ErrorLog.txt
Putting this practice to work in your script (and making some other best-practice improvements) looks like the following:
#!/bin/bash
# not related to file output, but to making sure we detect errors
# only works correctly if run with bash, not sh!
set -o pipefail ## set exit status based on whole pipeline, not just last command
# picking 3, since FD numbers 0-2 are reserved for stdin/stdout/stderr
exec 3>ErrorLog.txt
if bro-cut service <conn.log | sort | uniq -c >ProtocolHierarchy.txt; then
echo "OK Protocol Hierarchy Created" >&3
else
echo "FAILED Creating Protocol Hierarchy" >&3
fi
if bro-cut id.orig_h <dns.log | sort | uniq -c >AllIPAddresses.txt; then
echo "OK Created all IP Addresses" >&3
else
echo "FAILED Creating all IP Addresses" >&3
fi
I have a file forwarding system where a bunch of files are downloaded to a directory, de-multiplexed and copied to individual machines.
The files are forwarded when they are received by the master server. And files normally arrive in bursts. (Auth by ssh keys)
This script creates the sftp session, and uses a pipe to watch the head of a fifo pipe.
HOST=$1
pipe=/tmp/pipes/${HOST%%.*}
ps aux | grep -v grep | grep sftp | grep "user#$HOST" > /dev/null
if [[ $? == 0 ]]; then
echo "FTP is Running on this Server"
exit
else
pid=`ps aux | grep -v grep | grep tail | tr -s ' ' | grep $pipe`
[[ $? == 0 ]] && kill -KILL `echo $pid | cut -f2 -d' '`
fi
if [[ ! -p $pipe ]]; then
mkfifo $pipe
fi
tail -n +1 -f $pipe | sftp -o 'ServerAliveInterval 60' user#$HOST > /dev/null &
echo cd /tmp/data >>$pipe #Sends Command to Host
echo "Started FTP to $HOST"
Update: I ended up changing the cleanup code to use "ps aux" to see if an ftp session is running, and subsequently if the tail -f is still running. Grep by user#host and the name of the pipe respectively. This is done when the script is called, and the script is called whenever I try to upload a file.
IE:
FILENAME=`basename $1`
function transfer {
echo cd /apps/data >> $2 # For Safety
echo put $1 .$FILENAME >> $2
echo rename .$FILENAME $FILENAME >> $2
echo chmod 0666 $FILENAME >> $2
}
./ftp.sh host
[ -p $pipedir/host ] && transfer $1 $pipedir/host
Files received on the master server are caught by Incron which writes a put command and the available file's location to the fifo pipe, to be sent by sftp (rename is also preformed).
My question is, is this safe? Could this crash on ftp errors/events. Not really worried about login errors.
The goal is to reduce the number of ftp logins. Single Session/Minute(or more) intervals.
And allow files to be forwarded as they're received. Dynamic Commands.
I'd prefer to use standard ubuntu libraries, if possible.
EDIT: After testing and working through some issues the server simply runs with
[[ -p $pipe ]] && echo FTP is Running on this Server
ln -s $pipe $lock &> /dev/null || (echo FTP is Running on this Server && exit)
[[ ! -p $pipe ]] && mkfifo $pipe
( tail -n +1 -F $pipe & echo $! > $pipe.pid ) | tee >
( sed "/tail:/ q" >/dev/null && kill $(cat $pipe.pid) |& rm -f $pipe >/dev/null; )
| sftp -i ~/.ssh/$HOST.rsa -oServerAliveInterval=60 user#$HOST &
rm -f $lock
Its rather simple but works nicely.
you might be intrested in setting up a more simpler(and robust) syncronization infrastructure:
if a given host is not connected when a file arrives...it never recieves it (if i understand correctly your code)
i would do something like
rsync -a -e ssh user#host:/apps/data pathToLocalDataStore
on the client machines either periodically or by event...rsync is intelligently syncronizes the files by their timestamp and size (-a contains -t)
the event would be some process termination like:
client does(configure private key usage in ~/.ssh/config for host):
#!/bin/bash
while :;do
ssh user#host /srv/bin/sleepListener 600
rsync -a -e ssh user#host:/apps/data pathToLocalDataStore
done
on the server
/srv/bin/sleepListener is a symbolic link to /bin/sleep
server after recieving new file:
killall sleepListener
note: every 10 minutes a full check is performed...if nodes go offline/online it doesn't matter...
How can an internet connection be tested without pinging some website?
I mean, what if there is a connection but the site is down? Is there a check for a connection with the world?
Without ping
#!/bin/bash
wget -q --spider http://google.com
if [ $? -eq 0 ]; then
echo "Online"
else
echo "Offline"
fi
-q : Silence mode
--spider : don't get, just check page availability
$? : shell return code
0 : shell "All OK" code
Without wget
#!/bin/bash
echo -e "GET http://google.com HTTP/1.0\n\n" | nc google.com 80 > /dev/null 2>&1
if [ $? -eq 0 ]; then
echo "Online"
else
echo "Offline"
fi
Ping your default gateway:
#!/bin/bash
ping -q -w 1 -c 1 `ip r | grep default | cut -d ' ' -f 3` > /dev/null && echo ok || echo error
Super Thanks to user somedrew for their post here: https://bbs.archlinux.org/viewtopic.php?id=55485 on 2008-09-20 02:09:48
Looking in /sys/class/net should be one way
Here's my script to test for a network connection other than the loop back.
I use the below in another script that I have for periodically testing if my website is accessible. If it's NOT accessible a popup window alerts me to a problem.
The script below prevents me from receiving popup messages every five minutes whenever my laptop is not connected to the network.
#!/usr/bin/bash
# Test for network conection
for interface in $(ls /sys/class/net/ | grep -v lo);
do
if [[ $(cat /sys/class/net/$interface/carrier) = 1 ]]; then OnLine=1; fi
done
if ! [ $OnLine ]; then echo "Not Online" > /dev/stderr; exit; fi
Note for those new to bash: The final 'if' statement tests if NOT [!] online and exits if this is the case. See man bash and search for "Expressions may be combined" for more details.
P.S. I feel ping is not the best thing to use here because it aims to test a connection to a particular host NOT test if there is a connection to a network of any sort.
P.P.S. The Above works on Ubuntu 12.04 The /sys may not exist on some other distros. See below:
Modern Linux distributions include a /sys directory as a virtual filesystem (sysfs, comparable to /proc, which is a procfs), which stores and allows modification of the devices connected to the system, whereas many traditional UNIX and Unix-like operating systems use /sys as a symbolic link to the kernel source tree.[citation needed]
From Wikipedia https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard
This works on both MacOSX and Linux:
#!/bin/bash
ping -q -c1 google.com &>/dev/null && echo online || echo offline
In Bash, using it's network wrapper through /dev/{udp,tcp}/host/port:
if : >/dev/tcp/8.8.8.8/53; then
echo 'Internet available.'
else
echo 'Offline.'
fi
(: is the Bash no-op, because you just want to test the connection, but not processing.)
The top answer misses the fact that you can have a perfectly stable connection to your default gateway but that does not automatically mean you can actually reach something on the internet. The OP asks how he/she can test a connection with the world. So I suggest to alter the top answer by changing the gateway IP to a known IP (x.y.z.w) that is outside your LAN.
So the answer would become:
ping -q -w 1 -c 1 x.y.z.w > /dev/null && echo ok || echo error
Also removing the unfavored backticks for command substitution[1].
If you just want to make sure you are connected to the world before executing some code you can also use:
if ping -q -w 1 -c 1 x.y.z.w > /dev/null; then
# more code
fi
I've written scripts before that simply use telnet to connect to port 80, then transmit the text:
HTTP/1.0 GET /index.html
followed by two CR/LF sequences.
Provided you get back some form of HTTP response, you can generally assume the site is functioning.
make sure your network allow TCP traffic in and out, then you could get back your public facing IP with the following command
curl ifconfig.co
Execute the following command to check whether a web site is up, and what status message the web server is showing:
curl -Is http://www.google.com | head -1 HTTP/1.1 200 OK
Status code ‘200 OK’ means that the request has succeeded and a website is reachable.
The top voted answer does not work for MacOS so for those on a mac, I've successfully tested this:
GATEWAY=`route -n get default | grep gateway`
if [ -z "$GATEWAY" ]
then
echo error
else
ping -q -t 1 -c 1 `echo $GATEWAY | cut -d ':' -f 2` > /dev/null && echo ok || echo error
fi
tested on MacOS High Sierra 10.12.6
If your local nameserver is down,
ping 4.2.2.1
is an easy-to-remember always-up IP (it's actually a nameserver, even).
This bash script continuously check for Internet and make a beep sound when the Internet is available.
#!/bin/bash
play -n synth 0.3 sine 800 vol 0.75
while :
do
pingtime=$(ping -w 1 8.8.8.8 | grep ttl)
if [ "$pingtime" = "" ]
then
pingtimetwo=$(ping -w 1 www.google.com | grep ttl)
if [ "$pingtimetwo" = "" ]
then
clear ; echo 'Offline'
else
clear ; echo 'Online' ; play -n synth 0.3 sine 800 vol 0.75
fi
else
clear ; echo 'Online' ; play -n synth 0.3 sine 800 vol 0.75
fi
sleep 1
done
Similarly to #Jesse's answer, this option might be much faster than any solution using ping and perhaps slightly more efficient than #Jesse's answer.
find /sys/class/net/ -maxdepth 1 -mindepth 1 ! -name "*lo*" -exec sh -c 'cat "$0"/carrier 2>&1' {} \; | grep -q '1'
Explenation:
This command uses find with -exec to run command on all files not named *lo* in /sys/class/net/. These should be links to directories containing information about the available network interfaces on your machine.
The command being ran is an sh command that checks the contents of the file carrier in those directories. The value of $interface/carrier has 3 meanings - Quoting:
It seems there are three states:
./carrier not readable (for instance when the interface is disabled in Network Manager).
./carrier contain "1" (when the interface is activated and it is connected to a WiFi network)
./carrier contain "0" (when the interface is activated and it is not connected to a WiFi network)
The first option is not taken care of in #Jesse's answer. The sh command striped out is:
# Note: $0 == $interface
cat "$0"/carrier 2>&1
cat is being used to check the contents of carrier and redirect all output to standard output even when it fails because the file is not readable.
If grep -q finds "1" among those files it means there is at least 1 interface connected. The exit code of grep -q will be the final exit code.
Usage
For example, using this command's exit status, you can use it start a gnubiff in your ~/.xprofile only if you have an internet connection.
online() {
find /sys/class/net/ -maxdepth 1 -mindepth 1 ! -name "*lo*" -exec sh -c 'cat "$0"/carrier 2>&1 > /dev/null | grep -q "1" && exit 0' {} \;
}
online && gnubiff --systemtray --noconfigure &
Reference
Help testing special file in /sys/class/net/
find -exec a shell function?
shortest way: fping 4.2.2.1 => "4.2.2.1 is alive"
i prefer this as it's faster and less verbose output than ping, downside is you will have to install it.
you can use any public dns rather than a specific website.
fping -q google.com && echo "do something because you're connected!"
-q returns an exit code, so i'm just showing an example of running something you're online.
to install on mac: brew install fping; on ubuntu: sudo apt-get install fping
Ping was designed to do exactly what you're looking to do. However, if the site blocks ICMP echo, then you can always do the telnet to port 80 of some site, wget, or curl.
Checking Google's index page is another way to do it:
#!/bin/bash
WGET="/usr/bin/wget"
$WGET -q --tries=20 --timeout=10 http://www.google.com -O /tmp/google.idx &> /dev/null
if [ ! -s /tmp/google.idx ]
then
echo "Not Connected..!"
else
echo "Connected..!"
fi
For the fastest result, ping a DNS server:
ping -c1 "8.8.8.8" &>"/dev/null"
if [[ "${?}" -ne 0 ]]; then
echo "offline"
elif [[ "${#args[#]}" -eq 0 ]]; then
echo "online"
fi
Available as a standalone command: linkStatus
Pong doesn't mean web service on the server is running; it merely means that server is replying to ICMP echo.
I would recommend using curl and check its return value.
If your goal is to actually check for Internet access, many of the existing answers to this question are flawed. A few things you should be aware of:
It's possible for your computer to be connected to a network without that network having internet access
It's possible for a server to be down without the entire internet being inaccessible
It's possible for a captive portal to return an HTTP response for an arbitrary URL even if you don't have internet access
With that in mind, I believe the best strategy is to contact several sites over an HTTPS connection and return true if any of those sites responds.
For example:
connected_to_internet() {
test_urls="\
https://www.google.com/ \
https://www.microsoft.com/ \
https://www.cloudflare.com/ \
"
processes="0"
pids=""
for test_url in $test_urls; do
curl --silent --head "$test_url" > /dev/null &
pids="$pids $!"
processes=$(($processes + 1))
done
while [ $processes -gt 0 ]; do
for pid in $pids; do
if ! ps | grep "^[[:blank:]]*$pid[[:blank:]]" > /dev/null; then
# Process no longer running
processes=$(($processes - 1))
pids=$(echo "$pids" | sed --regexp-extended "s/(^| )$pid($| )/ /g")
if wait $pid; then
# Success! We have a connection to at least one public site, so the
# internet is up. Ignore other exit statuses.
kill -TERM $pids > /dev/null 2>&1 || true
wait $pids
return 0
fi
fi
done
# wait -n $pids # Better than sleep, but not supported on all systems
sleep 0.1
done
return 1
}
Usage:
if connected_to_internet; then
echo "Connected to internet"
else
echo "No internet connection"
fi
Some notes about this approach:
It is robust against all the false positives and negatives I outlined above
The requests all happen in parallel to maximize speed
It will return false if you technically have internet access but DNS is non-functional or your network settings are otherwise messed up, which I think is a reasonable thing to do in most cases
If you want to handle captive portals, you can do this oneliner:
if [[ $(curl -s -D - http://www.gstatic.com/generate_204 2>/dev/null | head -1 | cut -d' ' -f 2) == "204" ]]; then
echo 'online'
else
echo 'offline'
fi
Or if you want something more readable that can differentiate captive portals from lack of signal:
function is_online() {
# Test signal
local response
response=$(curl --silent --dump-header - http://www.gstatic.com/generate_204 2> /dev/null)
if (($? != 0)); then return 2; fi
# Test captive portal
local status=$(echo $response | head -1 | cut -d' ' -f 2)
((status == "204"))
}
is_online && echo online || echo offline