I'm trying to monitor a website using curl but the output doesn't seem to work, please see commands below:
#!/bin/bash
varDate=$(date '+%Y-%m-%d %H:%M:%S')
varCurlError=$(curl -sSf https://website.com > /dev/null)
varHttpCode=$(curl -Is https://website.com | head -n 1)
varResponseTime=$(curl -s -w '%{time_total}' -o /dev/null website.com)
varOutput="$varDate | $varCurlError | $varHttpCode | $varResponseTime"
echo $varOutput
The output looks like this :
| 0.07323 18:51:40 | | HTTP/1.1 200 OK
What can I change or add to fix the output.
Much appreciated.
#!/bin/bash
varDate=$(date '+%Y-%m-%d %H:%M:%S')
varCurlError=$(curl -sSf https://website.com 2>&1 >/dev/null)
varHttpCode=$(curl -Is https://website.com | head -n 1)
varResponseTime=$(curl -s -w '%{time_total}' -o /dev/null website.com | tr -d \\r )
varOutput="$varDate | $varCurlError | $varHttpCode | $varResponseTime"
echo $varOutput
There are two corrections:
tr -d \r was added as per glenn jackman. The CR is causing your varResponseTime to be printed at the beginning of the line. The tr command deletes the CR.
You need to first redirect stderr to stdout before you close file descriptor 1 in your varCurlError statement. Now, errors reported by curl to stderr will be sent to stdout (and captured by your $() enclosure). The output curl sends to stdout will go to the bitbucket. Order is important. >/dev/null 2>&1 doesn't work - it sends stdout and stderr to /dev/null.
#glenn jackman is correct about the need to pipe the curl output to | tr -d '\r'
That is, change your code to
#!/bin/bash
varDate=$(date '+%Y-%m-%d %H:%M:%S' | tr -d '\r')
varCurlError=$(curl -sSf https://website.com | tr -d '\r' > /dev/null)
varHttpCode=$(curl -Is https://website.com | tr -d '\r' | head -n 1)
varResponseTime=$(curl -s -w '%{time_total}' -o /dev/null website.com | tr -d '\r')
varOutput="$varDate | $varCurlError | $varHttpCode | $varResponseTime"
echo "$varOutput"
It can be done with wget so you see if you can get any data and it can be simple like this:
#!/bin/bash
dt=$(date '+%d/%m/%Y %H:%M:%S');
wget domain/yourindex
if [ -f /home/$USER/yourindex ] ; then
#echo $dt GOOD >> /var/log/fix.log
echo GOOD >/dev/null 2>&1
else
#counter measures like sudo systemctl restart php7.2-fpm.service && sudo systemctl restart nginx
echo $dt BROKEN >> /var/log/fix.log
fi
rm login*
exit
Consider the following example:
{ echo 1 | tee /dev/stderr 2> >(sed -e 's|1|err|' >&2) 1> >(sed -e 's|1|out|') ; }
which prints
out
err
Note that out out is printed on stdout and err on stderr.
Question: How to do this remotely via ssh?
More precisely, how to run
ssh host 'echo 1 | tee /dev/stderr SOME_MAGIC_HERE'
st. again out/err pops up on stdout/stderr (for an appropriate bash magic SOME_MAGIC_HERE).
Clearly, the following works:
ssh host 'echo 1 | tee /dev/stderr' 2> >(sed -e 's|1|err|' >&2) 1> >(sed -e 's|1|out|')
But that executes sed locally, and I'd rather want to do that remotely on host.
after the update:
ssh host 'echo 1 | tee >(cat - | sed -e "s|1|err|" >&2) | sed -e "s|1|out|"'
out
err
The idea is to use <pipes> | for processing /dev/stdout and use process substitution in combination with tee to create the /dev/stderr part.
Now it works as expected:
$ ssh host 'echo 1 | tee >(cat - | sed -e "s|1|err|" >&2) | sed -e "s|1|out|"' > /dev/null
err
$ ssh host 'echo 1 | tee >(cat - | sed -e "s|1|err|" >&2) | sed -e "s|1|out|"' 2> /dev/null
out
original answer:
The following command executes by changing your <single quotes> into <double quotes> :
ssh host 'echo 1 | tee /dev/stderr 2> >(sed -e "s|1|err|") 1> >(sed -e "s|1|out|")'
but this has everything in /dev/stdout. Example:
$ ssh host 'echo 1 | tee /dev/stderr 2> >(sed -e "s|1|err|") 1> >(sed -e "s|1|out|")' > /dev/null
$ ssh host 'echo 1 | tee /dev/stderr 2> >(sed -e "s|1|err|") 1> >(sed -e "s|1|out|")' 2> /dev/null
out
err
and this is exactly what your original command does on the host system:
{ echo 1 | tee /dev/stderr 2> >(sed -e "s|1|err|") 1> >(sed -e "s|1|out|") ; } >/dev/null
{ echo 1 | tee /dev/stderr 2> >(sed -e "s|1|err|") 1> >(sed -e "s|1|out|") ; } 2>/dev/null
out
err
The ssh program normally handles the passing of /dev/stdout,/dev/stderr and /dev/stdin correctly:
$ ssh host "echo 1; echo 2 > /dev/stderr" > /dev/null
2
$ ssh host "echo 1; echo 2 > /dev/stderr" 2> /dev/null
1
I have a txt file (which often gets updated) called hitlist.txt containing a list of words/strings I want to grep a directory against ... like:
# This is just a comment and will not be part of the search
* Blah - this is a category
foo
bar
sibilance
# A new category
* Meh - another category
snakefish
sex panther
My list is typically > 100 strings, and each is on its own line. Today, because of a deadline, I simply went through the list and executed the following command for each word:
find -iname "*" -type f -print0 | xargs -0 -HniI "foo" >> results.txt
As indicated in the command above, I am interested in the file path and name, as well as the line the contains the matched text. There are multiple categories list in the file (denoted by *) and I would like to be able to run my script against one, more, or all categories.
I would also like to be able to turn off the -i flag (case sensitivity) as an option. I have a script that recursively finds/lists all files in a directory, and the command I have been using above. Lastly the hitlist format can be changed completely if necessary.
Setup a ghl() (grep hitlist) shell function to do the work, (depends on GNU grep's -o switch, plus a little sed loop), the output is a list of words from hitlist.txt (or <filename>):
# usage ghl <glob> <filename>
ghl() { grep -o '\* '"$1"' -' "$2" | grep -o '[[:alpha:]]*' | \
while read x ; do \
sed -n '/\* '"$1"'/{:show ;n;/^[^ ]/{p;b show;}}' "$2" ; \
done ; }
Pipe the word list output of ghl with an ".*ah" wildcard, (which matches the Blah category), into grep -f -, plus some ad hoc bash process substitution to generate input text:
ghl '.*ah' hitlist.txt | grep -i -f - <(echo bar) <(echo foo) <(echo Foo)
Output:
/dev/fd/63:bar
/dev/fd/62:foo
/dev/fd/61:Foo
The 2nd grep above can be passed switches as desired, (see man grep). Example, the same thing, but case sensitive, (i.e. remove the -i switch):
ghl '.*ah' hitlist.txt | grep -f - <(echo bar) <(echo foo) <(echo Foo)
Output, (note missing uppercase item):
/dev/fd/63:bar
/dev/fd/62:foo
Since grep already has options to handle recursive searches, the rest is only a matter of adding switches as required.
Your question is extremely vague, but I'm imagining this is more or less what you are looking for.
awk -v cat='Blah|Meh' 'NR==FNR && /^#/ { next } # Skip comments
NR==FNR && /^\*/ { if ($0~cat) c=1; else c=0; next }
NR==FNR { if(c) a[$0]=1; next }
lower($0) in a { print FILENAME ":" FNR ":" $0 }' Hits.txt files to search
Figuring out how to selectively disable lower() and rigging it to read a list of file names other than Hits.txt from find should be fairly obvious.
This is what I ended up with:
hitlist format:
# MEH
never,going,to give,you up
# blah
word to,your,mother
Script:
# Set defaults
OUTPUT_FILE="hits.txt"
HITLIST_FILE="hitlist.txt"
# Hold on to the args
ARGLIST=($*)
# Declare any functiions
help ()
{
echo "--------------------------------- Luffa --------------------------------"
echo "Usage: luffa.sh [DIRTOSCRUB]"
echo ""
echo "Searches DIRTOSCRUB for category specific words in $HITLIST_FILE."
echo ""
echo "EXAMPLE: luffa.sh dirtoscrub"
echo ""
echo "--help display this help and exit"
echo "--version display version information and exit"
}
version ()
{
echo "luffa.sh v1.0"
}
process ()
{
if [ ${#FILEARG} -lt 1 ] # check for proper number of args
then
echo "ERROR: Specify directory to be searched."
help
exit 1
else
SEARCH_DIR=${ARGLIST[0]}
fi
echo ""
echo "--------------------------------------------------------- Luffa ---------------------------------------------------" | tee -a "$OUTPUT_FILE"
echo "search command: find [DIRTOSCRUB] -type f -print0 | xargs -0 grep -HniI --color=always $word | tee -a ../hits.txt | more" | tee -a "$OUTPUT_FILE"
echo
echo " .,,:::::." | tee -a "$OUTPUT_FILE"
echo " .,,::::~:::::.." | tee -a "$OUTPUT_FILE"
echo " ,,::::~~~~~~::~~:::." | tee -a "$OUTPUT_FILE"
echo " ,:,:~:~~~~~~~~~~~~~~::." | tee -a "$OUTPUT_FILE"
echo " ,,:::~:~~~~~~~~~~~~~~~~~~," | tee -a "$OUTPUT_FILE"
echo " .,,::::~~~~~~~~~~~~~~~~~~~~~~::" | tee -a "$OUTPUT_FILE"
echo " .,::~:~~~~~=~~~~=~~~~~~~~~~~=~~~~." | tee -a "$OUTPUT_FILE"
echo " ,::::~~:~~~=~~~~~~~~=~~=~~~===~~~~~~." | tee -a "$OUTPUT_FILE"
echo " ..:::~~~~=~~=~~~~~~=~~~~=~~===~~==~~~~~~," | tee -a "$OUTPUT_FILE"
echo " .,:::~~~~~~~~~~~~~~~~=~=~~~=~====~===~~~~~~~." | tee -a "$OUTPUT_FILE"
echo " .,::~~~~~~~~~~~~~~=~=~~~~~=~======~=~~~~=~=~~~:" | tee -a "$OUTPUT_FILE"
echo " ..,::~:~~~~~~=~~~=~~~~~~~~=~====+======~===~~~~~~~." | tee -a "$OUTPUT_FILE"
echo " ..,:,:~~~~~~=~::~~=~=~~~=~~=~=~=~======~~~==~~~~~~::." | tee -a "$OUTPUT_FILE"
echo " ,,.::~:=~~~~~~~~~~~~=~=~===~~~====+==~=====~~~~~::,." | tee -a "$OUTPUT_FILE"
echo " ,,,,:I++=:~==~=~~~~~~=~:==~=~+~====~=~===~~~~:~::,:" | tee -a "$OUTPUT_FILE"
echo " .,:+++?77+?=~~~~=~~=~=~~=~~+=~+~~+====~=~~~:::::,::," | tee -a "$OUTPUT_FILE"
echo " ..++++?++?II?=~~=~~~=~~~====~===~=====~~~:~::::::::,." | tee -a "$OUTPUT_FILE"
echo " ..=++?++++++???7+~~~~~~~~+~=~=====~~~~~~~~~::::~:::,,.." | tee -a "$OUTPUT_FILE"
echo " .=+++++++++++++++===:~~=~==+~~=~=~~:~~=~:~:::~::::,,.." | tee -a "$OUTPUT_FILE"
echo " .++++++?++++++?++=?~:~~~~===~==~==~~~~~:::::::::,,,..." | tee -a "$OUTPUT_FILE"
echo " ..=?+++++??+++++++===~::~~~~~~=~~~~~~:~~:::::,:,,,,,." | tee -a "$OUTPUT_FILE"
echo " ...=+?+++++++++=====~:,::,~:::~~~~~:~~~~::::~::::,,,,.." | tee -a "$OUTPUT_FILE"
echo " .=+++++++++++===~==::::,::~~,,,::~~~~~~::::::~:,:,,.." | tee -a "$OUTPUT_FILE"
echo " ..++++++++++=+===~,.,,:::,:~~~~~,.,:~:~::::::,::,:,.." | tee -a "$OUTPUT_FILE"
echo " ...++?++++++++=+=~~. ..,,,,,:,~,::~,:::,:,:,~::::,,.." | tee -a "$OUTPUT_FILE"
echo " .++++++++?++====~. ...,,:,~::~=::,::,:,:::,,,,.." | tee -a "$OUTPUT_FILE"
echo ".++?+++++?++++==~.. .,.:,,:::~,:,,,:::::,,,." | tee -a "$OUTPUT_FILE"
echo "++++++?+???==~=. ...,::~~~:,,:,:::,,." | tee -a "$OUTPUT_FILE"
echo "?+++?????+==~. ..,,,,::,:,,,,,." | tee -a "$OUTPUT_FILE"
echo "+?+++??+==~. ..,,,,,,,,." | tee -a "$OUTPUT_FILE"
echo "+I???+==~. ..,,.." | tee -a "$OUTPUT_FILE"
echo "??++==~." | tee -a "$OUTPUT_FILE"
echo "+===~." | tee -a "$OUTPUT_FILE"
echo "+=~." | tee -a "$OUTPUT_FILE"
echo "~" | tee -a "$OUTPUT_FILE"
echo "--------------------------------------------------------------------------------------------------------------------------" | tee -a "$OUTPUT_FILE"
echo "" | tee -a "$OUTPUT_FILE"
# Loop through hitlist
while read -re hitList || [[ -n "$hitList" ]]
do
# If first character is "#" it's a comment, or line is blank, skip
if [ "$(echo $hitListWords | head -c 1)" != "#" ]; then
if [ ! -z "$hitListWords" -a "$hitListWords" != "" ]; then
# Parse comma delimited category specific hitlist
IFS=',' read -ra categoryWords <<< "$hitListWords"
# Search for occurences/hits for the hitList word
for categoryWord in "${categoryWords[#]}"; do
echo "---------------------------------------------------" | tee -a "$OUTPUT_FILE"
echo "$category - \"$categoryWord"\" | tee -a "$OUTPUT_FILE"
echo "---------------------------------------------------" | tee -a "$OUTPUT_FILE"
eval 'find "$SEARCH_DIR" -type f -print0 | xargs -0 grep -HniI "$categoryWord" >> "$OUTPUT_FILE"'
eval 'find "$SEARCH_DIR" -type f -print0 | xargs -0 grep -HniI --color=always "$categoryWord" | more'
echo "" | tee -a "$OUTPUT_FILE"
done
fi
else
category="$(echo "$hitListWords" | cut -d "#" -f 2)"
fi
done < "$HITLIST_FILE"
exit $?
}
# Process the options
while [[ "${ARGLIST[0]}" == -* ]]; do
OPTION="${ARGLIST[0]}"
NUM_OPTS=1;
case $OPTION in
--version)
version
exit 0
;;
--help)
help
exit 0
;;
*)
help
exit 1
;;
esac
ARGLIST=(${ARGLIST[#]:$NUM_OPTS})
done
FILEARG=${ARGLIST[#]}
process
I am learning to customize wget in a bash function and having trouble. I would like to display Downloading (file):% instead of the messy output of wget. The function below seems close I am having trouble calling it for my specific needs.
For example, my standard wget is:
cd 'C:\Users\cmccabe\Desktop\wget'
wget -O getCSV.txt http://xxx.xx.xxx.xxx/data/getCSV.csv
and that downloads the .csv as a .txt in the directory specified with all the messy wget output.
This function seems like it will do more-or-less what I need, but I can not seem to get it to function correctly using my data. Below is what I have tried. Thank you :).
#!/bin/bash
download() {
local url=$1 wget -O getCSV.txt http://xxx.xx.xxx.xxx/data/getCSV.csv
local destin=$2 'C:\Users\cmccabe\Desktop\wget'
echo -n " "
if [ "$destin" ]; then
wget --progress=dot "$url" -O "$destin" 2>&1 | grep --line-buffered "%" | \
sed -u -e "s,\.,,g" | awk '{printf("\b\b\b\b%4s", $2)}'
else
wget --progress=dot "$url" 2>&1 | grep --line-buffered "%" | \
sed -u -e "s,\.,,g" | awk '{printf("\b\b\b\b%4s", $2)}'
fi
echo -ne "\b\b\b\b"
echo " DONE"
}
EDITED CODE
#!/bin/bash
download () {
url=http://xxx.xx.xxx.xxx/data/getCSV.csv
destin='C:\Users\cmccabe\Desktop\wget'
echo -n " "
if [ "$destin" ]; then
wget -O getCSV.txt --progress=dot "$url" -O "$destin" 2>&1 | grep --line-buffered "%" | \
sed -u -e "s,\.,,g" | awk '{printf("\b\b\b\b%4s", $2)}'
else
wget -O getCSV.txt --progress=dot $url 2>&1 | grep --line-buffered "%" | \
sed -u -e "s,\.,,g" | awk '{printf("\b\b\b\b%4s", $2)}'
fi
echo -ne "\b\b\b\b"
echo " DONE"
menu
}
menu() {
while true
do
printf "\n Welcome to NGS menu (v1), please make a selection from the MENU \n
==================================\n\n
\t 1 Patient QC\n
==================================\n\n"
printf "\t Your choice: "; read menu_choice
case "$menu_choice" in
1) patient ;;
*) printf "\n Invalid choice."; sleep 2 ;;
esac
done
}