So I'm writing a script that will take the output of a grep as an array and then iterate a filter over it to output to a file. I'm testing it on my own site and the wget works as expected and generates a list of URLS in the spider.queue. The grep command will also work in terms of filtering by a keyword, but when I add it into a while loop and use an if statement to check if it already exists I'll get the error;
./spider.sh: 19: ./spider.sh: Syntax error: "(" unexpected (expecting "done")
Which would lead me to believe it's a syntax problem with one of the loops.
#!/bin/sh
# Usage - ./spider.sh searchterm www.website.com
## Parameters
search=$1
URL=$2
## Spider WGET
wget -r -e robots=off --header="Accept: text/html" --user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:21.0) Gecko/20100101 Firefox/21.0" http://$URL 2>&1 | grep '^--' 2>&1 | awk '{ print $3 }' | grep -v '\.\(css\|js\|png\|gif\|jpg\|JPG\)$' >> spider.queue
## Keyword filter with grep
while true
do
PROFILES=($(grep -l -r "$search" $URL))
for x in ${PROFILES[*]}
do
if grep -q $x crawler.queue; then
echo "Already Exists"
else
$x >> crawler.queue
fi
done
done
/bin/sh does not support arrays, so the syntax error is caused by PROFILES=(....
Switch to #!/bin/bash.
Related
I am trying to create a bash script that will write to text files the output of some pacman queries mainly what packages are installed locally, packages that are installed as dependencies, packages that are orphans, and what packages require what dependencies. Currently, I am in the middle of solving an issue that is preventing my from writing what packages require what dependencies. I am using the following bash code:
#!/bin/bash
set -e -u
#Switch to PWD
cd /home/$USER/System/scripts/pacman-queries-output/
#Get the current date
DATE=`date +%m%d%Y`
#Pacman Queries
pacman --query -e >pacman_installed$DATE.txt
pacman --query -d >pacman_dependencies$DATE.txt
pacman -Qdt >pacman_orphans$DATE.txt
while read package_desc
do
package_name=$(echo $package_desc| cut -d' ' -f 1)
check_if_none=$(pacman -Qi $package_name | grep "Req" | sed -e 's/Required By : //g')
if $check_if_none != "Required By : None"
then
echo $package_name >>pacman_required_by$DATE.txt
pacman -Qi $package_name | grep "Req" | sed -e 's/Required By : //g' >>pacman_required_by$DATE.txt
fi
done < $PWD/pacman_installed$DATE.txt
echo 'Completed 'basename
However, the while loop doesn't seem to create and/or write to the text file I specified instead it echoes this multiple times in the terminal:
./pacman-queries.sh: line 20: Required: command not found
The following is one of the iterations of the while loop that is displayed when running bash -x pacman-queries.sh:
+ read package_desc
++ echo aesfix 1.0.1-4
++ cut '-d ' -f 1
+ package_name=aesfix
++ pacman -Qi aesfix
++ grep Req
++ sed -e 's/Required By : //g'
+ check_if_none='Required By : None'
+ Required By : None '!=' 'Required By : None'
pacman-queries.sh: line 20: Required: command not found
Could anyone suggest any solution that they might have to solve this issue? Thank you in advance.
if $check_if_none != "Required By : None"
If the check_if_none has the string:
check_if_none="Required By : None"
Then it gets expanded to:
if Required By : None != "Required By : None"
if tries to execute the command passed to it. So it will try to execute the command Required. And it does not find such command.
How to fix it:
use test [ .. ] or [[ ... ]] to do comparison in bash
always quote your variables "$varvar"
if [ "$check_if_none" != "Required By : None" ]
Also:
don't use backticks. They are less readable, can't be nested, and are deprecated.
You script after some fixing may look like this:
#!/bin/bash
set -e -u
#Switch to PWD
cd "/home/$USER/System/scripts/pacman-queries-output/"
#Get the current date
DATE=$(date +%m%d%Y)
#Pacman Queries
pacman --query -e >"pacman_installed$DATE.txt"
pacman --query -d >"pacman_dependencies$DATE.txt"
pacman -Qdt >"pacman_orphans$DATE.txt"
while IFS= read -r package_desc; do
package_name=$(echo "$package_desc" | cut -d' ' -f 1)
# renamed from check_if_none
# some newlines for readability
Required=$(
pacman -Qi "$package_name" |
grep "Req" |
sed -e 's/Required By : //g'
)
if [ "$Required" != "Required By : None" ]; then
echo "$package_name"
# runnung `pacman -Q` twice is just throwing cpu cycles...
echo "$Required"
fi
# All output goes into required_by - moved it here
# also changed `>>` into `>`
done < "$PWD/pacman_installed$DATE.txt" > "pacman_required_by$DATE.txt"
echo 'Completed 'basename
I'm having problems with this last part of my bash script. It receives input from 500 web addresses and is supposed to fetch the server information from each. It works for a bit but then just stops at like the 45 element. Any thoughts with my loop at the end?
#initializing variables
timeout=5
headerFile="lab06.output"
dataFile="fortune500.tsv"
dataURL="http://www.tech.mtu.edu/~toarney/sat3310/lab09/"
dataPath="/home/pjvaglic/Documents/labs/lab06/data/"
curlOptions="--fail --connect-timeout $timeout"
#creating the array
declare -a myWebsitearray
#obtaining the data file
wget $dataURL$dataFile -O $dataPath$dataFile
#getting rid of the crap from dos
sed -n "s/^m//" $dataPath$dataFile
readarray -t myWebsitesarray < <(cut -f3 -d$'\t' $dataPath$dataFile)
myWebsitesarray=("${myWebsitesarray[#]:1}")
websitesCount=${#myWebsitesarray[*]}
echo "There are $websitesCount websites in $dataPath$dataFile"
#echo -e ${myWebsitesarray[200]}
#printing each line in the array
for line in ${myWebsitesarray[*]}
do
echo "$line"
done
#run each website URL and gather header information
for line in "${myWebsitearray[#]}"
do
((count++))
echo -e "\\rPlease wait... $count of $websitesCount"
curl --head "$curlOptions" "$line" | awk '/Server: / {print $2 }' >> $dataPath$headerFile
done
#display results
echo "Results: "
sort $dataPath$headerFile | uniq -c | sort -n
It would certainly help if you actually passed the --connect-timeout option to curl. As written, you are currently passing the single argument --fail --connect-timeout $timeout rather than 3 distinct arguments --fail, --connect-timeout, and $timeout. This is one instance where you should not quote the variable. IOW, use:
curl --head $curlOptions "$line"
I am trying to write a script where I enumerate users by checking the HTTP reponse length. I want to get output "good" when response is not equal 23, however I get these errors now:
for ((i=113;i<=115;i++)); do
if [[curl -i -s -k -X 'GET' "http://myurl/some.asp?q=$i" |
grep Content-Length | cut -d' ' -f2 != 23]]
then
echo "good"
fi
done
Output:
bash: [[curl: command not found
cut: !=: No such file or directory
cut: 23]]: No such file or directory
cut: !=: No such file or directory
cut: 23]]: No such file or directory
bash: [[curl: command not found
cut: !=: No such file or directory
cut: 23]]: No such file or directory
bash: [[curl: command not found
If I simply make a script without if condition, then it works well:
for ((i=113;i<=115;i++)); do
curl -i -s -k -X 'GET' "http://myurl/some.asp?q=$i" |
grep Content-Length
done
I checked many examples but can't seem to figure out where I am doing wrong.
After updating your initial error, you may have a syntax like (suggestion: put some effort on format, so that it is more clear what you have and what may be wrong):
for ((i=113;i<=115;i++))
do
if [[ curl -i -s -k -X 'GET' "http://myurl/some.asp?q=$i" | grep Content-Length | cut -d' ' -f2 != 23 ]]
then
echo "good"
fi
done
This is returning you the error:
bash: conditional binary operator expected bash: syntax error near
-i'`
Which is normal, because you are basically saying:
if [[ command ]]; then ...
Where command is a set of multiple piped commands. However, in [[ you just can add expressions on the form "$var" -eq 23 or "$(command)" -ne 23.
So use $( ) to execute the command: if [[ "$(command)" -ne 23 ]]:
if [[ "$(curl -i -s -k -X 'GET' "http://myurl/some.asp?q=$i" | grep Content-Length | cut -d' ' -f2)" -ne 23 ]]
Note I am using -ne to perform an integer comparison that means "not equal to".
Finally, notice that awk alone can do what grep and cut do in two steps:
... | grep "Content-Length" | cut -d' ' -f2
This means: check the line containing "Content-Length" and print its second field. awk makes it simple by saying:
... | awk '/Content-Length/ {print $2}'
And last, but not least, your expression for ((i=113;i<=115;i++)) can be also written as for i in {113..115} using brace expansion.
If you want to test a result of the command execution you should put it into $() . So the resulting script should look as follows:
for i in {113..115}; do if [[ $(curl -i -s -k -X 'GET' "http://myurl/some.asp?q=$i" | grep Content-Length | cut -d' ' -f2) != 23 ]]; then echo "good" ; fi; done
Also I've changed a way you iterate the values. {a..b} in bash provides a sequence from 'a' to 'b'.
This code ouputs a http status of 000 - which seems to indicate something didn't connect properly but when I do this curl outside of the bash script it works fine and produces a 200 so something with this code is off... any guidance?
#!/bin/bash
URLs=$(< test.txt | grep Url | awk -F\ ' { print $2 } ')
# printf "Preparing to check $URLs \n"
for line in $URLs
do curl -L -s -w "%{http_code} %{url_effective}\\n" $line
done
http://beerpla.net/2010/06/10/how-to-display-just-the-http-response-code-in-cli-curl/
your script works on my vt.
I added in a couple of debugging lines, this may help you to see where any metacharacters are getting in, as I would have to agree with the posted coments.
I've output lines in the for to a file which is then printed out with od.
I have amended the curl line to grab the last line, just to get the response code.
#!/bin/bash
echo -n > $HOME/Desktop/urltstfile # truncate urltstfile
URLs=$(cat testurl.txt | grep Url | awk -F\ ' { print $2 } ')
# printf "Preparing to check $URLs \n"
for line in $URLs
do echo $line >> $HOME/Desktop/urltstfile;
echo line:$line:
curl -IL -s -w "%{http_code}\n" $line | tail -1
done
od -c $HOME/Desktop/urltstfile
#do curl -L -s -w "%{http_code} %{url_effective}\\n" "$line\n"
I am trying to write a script to test if my firewall is blocking multiple pages. When I try to read one or multiple lines from command line I get a -ne: unitary operator expected error.
This is my attempt:
sh test www.3232.com.pe
www.3232.com.pe
test: line 7: [: -ne: unary operator expected
$ cat test
#!/bin/sh
for var in "$#"
do
echo $var
res=`curl -s -I $var | grep HTTP/1.1 | awk {'print $2'}`
if [ $res -ne 200 ]
then
echo "Error on $var"
fi
done
Consider what happens if curl does not successfully connect:
res=`curl -s -I $var | grep HTTP/1.1 | awk {'print $2'}`
if [ $res -ne 200 ]
If curl does not get any headers, or does not get a HTTP/1.1 header, then the pipline in the first command outputs nothing and res is assigned to nothing. In that case, the test in the second line will fail with an unexpected operator error.
You need to first test that res is nonempty. And, if it is empty, take an appropriate action.