I got a bash script wich reads urls from diffrent .txt files (ven Array defines the txt files) and I curl them and if the Response Code is not 200 it writes an error in error.txt
Sadly it always does it at the end, even if there is no error in any of the links, any idea why?
for i in "${ven[#]}"; do
while IFS='' read -r line || [[ -n "$line" ]]; do
IP=$(curl --write-out '%{http_code}' --silent --output /dev/null $line?upstream=$1)
if [ $IP != 200 ]
then
counter=$((counter + 1))
echo $(date +"%d.%m.%y %T") : $line >> error.txt
fi
done < $i
done
Your script has some quoting issues, although the main issue is the inconsistent use of test expression brackets (double vs. single) and how you are checking for the number.
if [[ "$IP" -ne 200 ]]
-ne means "not equal", and since you've already used double brackets stay consistent.
The other stuff is more preventative in the way you quote the variables:
for i in "${ven[#]}"; do
while IFS='' read -r line || [[ -n "$line" ]]; do
IP=$(curl --write-out '%{http_code}' --silent --output /dev/null "$line?upstream=$1")
if [[ "$IP" -ne 200 ]]; then
counter=$((counter + 1))
echo "$(date +'%d.%m.%y %T')" : "$line" >> error.txt
fi
done < "$i"
done
NOTE: If a site is redirecting ( 301 ) then it will show an error — something to maybe consider.
Related
I'm trying to automate a build process by replacing .js chunks for particular lines in my main.config.php file. When I run the following code:
declare -a js_strings=("footer." "footerJQuery." "headerCSS." "headerJQuery.")
build_path="./build/build"
config_path="./system/Config/main.config.php"
while read -r line;
do
for js_string in ${js_strings[#]}
do
if [[ $line == *$js_string* ]]
then
for js_file in "$build_path"/*
do
result="${js_file//[^.]}"
if [[ $js_file == *$js_string* ]] && [[ ${#result} -eq 3 ]]
then
sed -i "s/$line/$line$(basename $js_file)\";/g" $config_path
fi
done
fi
done
done < "$config_path"
I get this message back, and file has not been updated/edited:
sed: 1: "./system/Config/main.co ...": invalid command code .
I haven't been able to find anything in my searches that pertain to this specific message. Does anyone know what I need to change/try to get the specific lines replaced in my .php file?
Updated script with same message:
declare -a js_strings=("footer." "footerJQuery." "headerCSS." "headerJQuery.")
build_path="./build/build"
config_path="./system/Config/main.config.php"
while read -r line;
do
for js_string in ${js_strings[#]}
do
if [[ $line == *$js_string* ]]
then
for js_file in "$build_path"/*
do
result="${js_file//[^.]}"
if [[ $js_file == *$js_string* ]] && [[ ${#result} -eq 3 ]]
then
filename=$(basename $js_file)
newline="${line//$js_string*/$filename\";}"
echo $line
echo $newline
sed -i "s\\$line\\$newline\\g" $config_path
echo ""
fi
done
fi
done
done < "$config_path"
Example $line:
$config['public_build_header_css_url'] = "http://localhost:8080/build/headerCSS.js";
Example $newline:
$config['public_build_header_css_url'] = "http://localhost:8080/build/headerCSS.7529a73071877d127676.js";
Updated script with changes suggested by #Vercingatorix:
declare -a js_strings=("footer." "footerJQuery." "headerCSS." "headerJQuery.")
build_path="./build/build"
config_path="./system/Config/main.config.php"
while read -r line;
do
for js_string in ${js_strings[#]}
do
if [[ $line == *$js_string* ]]
then
for js_file in "$build_path"/*
do
result="${js_file//[^.]}"
if [[ $js_file == *$js_string* ]] && [[ ${#result} -eq 3 ]]
then
filename=$(basename $js_file)
newline="${line//$js_string*/$filename\";}"
echo $line
echo $newline
linenum=$(grep -n "^${line}\$" ${config_path} | cut -d':' -f 1 )
echo $linenum
[[ -n "${linenum}" ]] && sed -i "${linenum}a\\
${newline}
;${linenum}d" ${config_path}
echo ""
fi
done
fi
done
done < "$config_path"
Using sed's s command to replace a line of that complexity is a losing proposition, because whatever delimiter you choose may appear in the line and mess things up. If these are in fact entire lines, it is better to delete them and insert a new one:
linenum=$(fgrep -nx -f "${line}" "${config_path}" | awk -F : "{print \$1}" )
[[ -n "${linenum}" ]] && sed -i "" "${linenum}a\\
${newline}
;${linenum}d" "${config_path}"
What this does is search for the line number of the line that matches $line in its entirety, then extracts the line number portion. fgrep is necessary otherwise the symbols in your file are interpreted as regular expressions. If there was a match, then it runs sed, appending the new line (a) and deleting the old one (d).
I have a shell script that download urls one by one and check for updates in static sites.Here is the code:
#!/bin/bash
input="file.in"
while IFS= read -r line
do
# check if input line is a comment
case "$line" in \#*) continue ;; esac
line2="${line//\//\\}"
if wget -q "$line" -O index.html > /dev/null; then
if [ -f "$line2.html" ]; then
cmp --silent "$line2".html index.html || echo "$line"
else
echo "$line INIT"
fi
else
echo "$line FAIL"
fi
mv index.html "$line2".html
done <"$input"
file.in is the list with urls. Example:
#List of addresses
http://www.google.com
http://www.spotify.com
http://www.flickr.com
http://www.soundcloud.com
https://www.facebook.com
https://en.wikipedia.org/wiki/Linux
And i want to change the script to download all urls at once and save them with the same way with wget or curl. thanks!
A naive approach:
while IFS= read -r line
do
[[ "$line" == \#* ]] && continue
(
line2="${line//\//\\}"
if wget -q "$line" -O index.html > /dev/null; then
if [ -f "$line2.html" ]; then
cmp --silent "$line2".html index.html || echo "$line"
else
echo "$line INIT"
fi
else
echo "$line FAIL"
fi
mv index.html "$line2".html
) &
done <"$input"
wait
echo "all done"
When I try to match a string and do some conditions; it always fails to do so.
date=`date +%Y%m%d`
kol="/home/user/test_$date"
regex='Terminating the script'
if [ -f $kol ]; then
sudo tail -f $kol | while read line; do
if [[ $line = *"Terminating the"* ]]
then
echo "failed"
else
echo $line >> /home/user/test123_$date
fi
else
echo "File is not yet present"
exit 0
fi
I have also tried with regex and that to failed. So when ever I input the matching string into the file ($path) it wont output "failed"; Is there anything wrong in the code. Help is much appreciated.
You can try this. At least it works for me:
sudo tail -f $1 | while read line; do
if [[ $line = '' ]]
then
break
fi
if [[ $line = *"Terminating the"* ]]
then
echo "failed"
else
echo $line >> /home/nurzhan/test123_$date
fi
done < "$1"
$path is parameter to the running script. Also note that by default the command tail returns only 10 last lines.
i need check status of some link listen in a file called paths.txt, i have write a simple for loop but not work
for LINK in $(cat paths.txt)
do
CURL="$(curl -I $LINK 2>/dev/null | head -n 1 | cut -d$' ' -f2)"
if [ "${CURL}" = "200" ]
then
echo ${LINK} >> true.txt
else
echo ${LINK} >> false.txt
fi
done
If i launch the curl command from terminal i read the right status, instead if i make an echo of CURL variable i not have any output
Run the script like this: bash -x myscript and watch the executed commands.
don't use cat in bash scripts, because is bad practice and can be problems with big files.
SCRIPTDIR=`dirname -- "$0"`
for siteUrl in $( < "$SCRIPTDIR/paths.txt")
do
if [[ -z "$siteUrl" ]]; then break; fi
if [[ "$(curl --write-out "%{http_code}\n" --silent --output /dev/null ${siteUrl} )" = "200" ]]; then
echo ${siteUrl} >> true.txt
else
echo ${siteUrl} >> false.txt
fi
done
The following bash code, which reads from a single input file line-by-line and writes to a large number (~100) of output files, exhibits unreasonable performance -- on the scale of 30 seconds for 10,000 lines, when I want it to be usable on a scale of millions or billions of lines of input.
In the following code, batches is an already-defined associative array (in other languages, a map).
How can this be improved?
while IFS='' read -r line
do
x=`echo "$line" | cut -d" " -f1`;
y=`echo "$line" | cut -d" " -f2`;
# echo "find match between $x and $y";
a="${batches["$x"]}";
b="${batches["$y"]}";
if [ -z $a ] && [ -n $b ]
then echo "$line" >> Output/batch_$b.txt;
elif [ -n $a ] && [ -z $b ]
then echo "$line" >> Output/batch_$a.txt;
elif [ -z $a ] && [ -z $b ]
then echo "$line" >> Output/batch_0.txt;
elif [ $a -gt $b ]
then echo "$line" >> Output/batch_$a.txt;
elif [ $a -le $b ]
then echo "$line" >> Output/batch_$b.txt;
fi
done < input.txt
while IFS= read -r line; do
x=${line%%$'\t'*}; rest=${line#*$'\t'}
y=${rest%%$'\t'*}; rest=${rest#*$'\t'}
...
done <input.txt
That way you aren't starting two external programs every single time you want to split line into x and y.
Under normal circumstances you could use read to do string-splitting implicitly by reading the columns into different fields, but as read trims leading whitespace, that doesn't work correctly if (as here) your columns are whitespace-separated and the first can be empty; consequently, it's necessary to use parameter expanison. See BashFAQ #73 for details on how parameter expansion works, or BashFAQ #100 for a general introduction to string manipulation with bash's native facilities.
Also, re-opening your output files every time you want to write a single line to them is silly at this kind of volume. Either use awk, which will handle this for you automatically, or write a helper (note that the following requires a fairly new release of bash -- probably 4.2):
write_to_file() {
local filename content new_out_fd
filename=$1; shift
printf -v content '%s\t' "$#"
content=${content%$'\t'}
declare -g -A output_fds
if ! [[ ${output_fds[$filename]} ]]; then
exec {new_out_fd}>"$filename"
output_fds[$filename]=$new_out_fd
fi
printf '%s\n' "$content" >&"${output_fds[$filename]}"
}
...and then:
if [[ $a && ! $b ]]; then
write_to_file "Output/batch_$a.txt" "$line"
elif [[ ! $a ]] && [[ $b ]]; then
write_to_file "Output/batch_$b.txt" "$line"
elif [[ ! $a ]] && [[ ! $b ]]; then
write_to_file "Output/batch_0.txt" "$line"
elif (( a > b )); then
write_to_file "Output/batch_$a.txt" "$line"
else
write_to_file "Output/batch_$b.txt" "$line"
fi
Note that caching FDs only makes sense if you have few enough output files that you can maintain open file descriptors for each of them (and such that re-opening files receiving more than one write is a net benefit). Feel free to leave this out and only do faster string-splitting if it doesn't make sense for you.
Just for completion, here's another approach (also written using automatic fd management, thus requiring bash 4.2) -- running two cut invocations and letting them both run through the entirety of the input file.
exec {x_columns_fd}< <(cut -d" " -f1 <input.txt)
exec {y_columns_fd}< <(cut -d" " -f2 <input.txt)
while IFS='' read -r line && \
IFS='' read -r -u "$x_columns_fd" x && \
IFS='' read -r -u "$y_columns_fd" y; do
...
done <input.txt
This works because it's not cut itself that's inefficient -- it's starting it up, running it, reading its output and shutting it down all the time that costs. If you just run two copies of cut, and let each of them process the whole file, performance is fine.