This question already has answers here:
Bash script processing limited number of commands in parallel
(4 answers)
Closed 6 years ago.
I wrote bash script to calculate the hashes of a files, when it's file path given and queries one among the file hash to get the result of that hash from a API service ( Parsing using jq ) .
#Scanner.sh
#!/bin/bash
for i in $(find $1 -type f);do
md5_checksum=$(md5sum $i|cut -d' ' -f 1)
sha1_checksum=$(sha1sum $i|cut -d' ' -f 1)
sha256_checksum=$(sha256sum $i|cut -d' ' -f 1)
json_result="$(curl --silent -H 'Authorization:Basic XXXX' 'https://XXXXX.com/api/databrowser/malware_presence/query/sha1/'$sha1_checksum'?format=json&extended=true'| jq -r '.rl.malware_presence.name,.rl.malware_presence.level' | awk -vORS=, '{print $1}' |sed 's/,$/\n/')"
echo "$md5_checksum,$sha1_checksum,$sha256_checksum,$json_result"
done
#Result :
c63a1576e4b416e6944a1c39dbdfc3b4,fd55dfdd9a432e34ce6c593cd09a48f457f7aab6,e2d1c1975c2406f60f1c0fe5255560c2cd06434e669933536a86296a2eb70020,Malware,5
Now, it's taking too much time to process one and get results for one file hash ( 10 Sec ). How can i send 5 request per second and get the results faster ?
Any suggestions please ?
You could put your code in a function and run it in the background, with something like this:
runCurl() {
md5_checksum=$(md5sum $1|cut -d' ' -f 1)
sha1_checksum=$(sha1sum $1|cut -d' ' -f 1)
sha256_checksum=$(sha256sum $1|cut -d' ' -f 1)
json_result="$(curl --silent -H 'Authorization:Basic XXXX' 'https://XXXXX.com/api/databrowser/malware_presence/query/sha1/'$sha1_checksum'?format=json&extended=true'| jq -r '.rl.malware_presence.name,.rl.malware_presence.level' | awk -vORS=, '{print $1}' |sed 's/,$/\n/')"
echo "$md5_checksum,$sha1_checksum,$sha256_checksum,$json_result"
}
for i in $(find $1 -type f);do
runCurl $i &
done
Related
I'm extracting two md5sums by using this code:
md5sum test{1,2} | cut -d' ' -f1-2
I'm receiving two md5sums as in example below:
02eace9cb4b99519b49d50b3e44ecebc
d8e8fca2dc0f896fd7cb4cb0031ba249
Afterwards I'm not sure how to compare them. I have tried using the xargs:
md5sum test{1,2} | cut -d' ' -f1-2 | xargs bash -c '$0 == $1'
However, it tries to execute md5sum as a command
Any advice?
Try using a command subsitution instead
#!/bin/bash
echo 1 > file_a
echo 2 > file_b
echo 1 > file_c
file1=file_a
# try doing "file2=file_b" as well
file2=file_c
if [[ $(sha1sum $file1 | cut -d ' ' -f1-2) = $(sha1sum $file2 | cut -d ' ' -f1-2) ]]; then
echo same
else
echo different
fi
I have a script such as :
cat list_id.txt | while read line; do for ACC in $line;
do
echo -n "$ACC\t"
curl -s "link=fasta&retmode=xml" |\
grep TSeq_taxid |\
cut -d '>' -f 2 |\
cut -d '<' -f 1 |\
tr -d "\n"
echo
sleep 0.25
done
done
This script allows me from a list of ID in list_id.txt to get the corresponding names in a database in https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=nuccore&id=${ACC}&rettype=fasta&retmode=xml
So from this script I get something like
CAA42669\t9913
V00181\t7154
AH002406\t538120
And what I would like is directly to print or echo this result in fiel call new_ids.txt, I tried echo >> new_ids.txt but the file is empty.
Thanks for your help.
A minimal refactoring of your script might look like
# Avoid useless use of cat
# Use read -r
# Don't use upper case for private variables
while read -r line; do
for acc in $line; do
echo -n "$acc\t"
# No backslash necessary after | character
curl -s "link=fasta&retmode=xml" |
# Probably use a proper XML parser for this
grep TSeq_taxid |
cut -d '>' -f 2 |
cut -d '<' -f 1 |
tr -d "\n"
echo
sleep 0.25
done
done <list_id.txt >new_ids.txt
This could probably still be simplified significantly, but without knowledge of what your input file looks like exactly, or what curl returns, this is somewhat speculative.
tr -s ' \t\n' '\n' <list_id.txt |
while read -r acc; do
curl -s "link=fasta&retmode=xml" |
awk -v acc="$acc" '/TSeq_taxid/ {
split($0, a, /[<>]/); print acc "\t" a[3] }'
sleep 0.25
done <list_id.txt >new_ids.txt
I have a simple bash script that uses an api to add itself into a database. But the script keeps adding ' ' to my variables and its breaking curl.
hostname=`hostname`
ip_address=`ip add show eth0 | grep 'inet ' | awk '{ print $2 }' | cut -d '/' -f1`
env=`hostname | cut -d '-' -f1`
os=`cat /etc/issue.net | head -1`
date=`date`
curl -H 'Content-Type: application/json' -PUT "https://10.10.10.10/database" -k -d '{"Environment":"'$env'","Hostname":"'$hostname'","IP_Address":"'$ip_address'","OS":"'$os'","State":"OK","Updated_Time":"'$date'"}'
exit $?
The output is this:
curl -H 'Content-Type: application/json' -PUT https://10.10.10.10/database -k -d '{"Environment":"ops","Hostname":"ex-example-host","IP_Address":"10.10.10.10","OS":"Ubuntu' 14.04 'LTS","State":"OK","Updated_Time":"Thu' Aug 14 15:27:55 PDT '2014"}'
Both the $date and $hostname put ' ' on the inside the format breaking the curl. Is there a way to fix this?
Thanks,
The problem is that you are leaving the parameter expansions unquoted in bash, so spaces in those values break up the word being passed to curl. It would be simpler to swap your use of double and single quotes, if JSON allowed you to use single quotes. That not being the case, I'd store the JSON in a variable using read and a here document first to simplify the quoting.
read -r data <<EOF
{"Environment":"$env","Hostname":"$hostname","IP_Address":"$ip_address","OS":"$os","State":"OK","Updated_Time":"$date"}
EOF
curl ... -d "$data"
This question already has answers here:
How do I set a variable to the output of a command in Bash?
(15 answers)
Closed 1 year ago.
So, lets say I have the following command:
curl -I http://google.com | head -n 1| cut -d $' ' -f2
This will capture the http status code??
Now I want to assign this to variable.. in bash script
like output = "curl -I http://localhost:8088/tracks?key=9 | head -n 1| cut -d $' ' -f2"
or something like that..
How do I assign the response of above command to a variable called output in bash?
Thanks
You have two options (see this StackOverflow answer here):
Preferred: Surround the invocation in $()
Surround the invocation in back ticks
NOTE: back ticks are legacy, the former method is preferred.
output=$(curl -I http://google.com | head -n 1| cut -d $' ' -f2)
echo "$output";
output=`curl -I http://google.com | head -n 1| cut -d $' ' -f2`
echo "$output";
My bash script contains the following line:
echo $(echo "$STRING_VAR" | cut -d' ' -f 2) >> $FILE
Here we have two echo calls, but are they really necessary ?
I wrote them, because otherwise the bash would think the string in first place is a command.
Simply echo "$STRING_VAR" | cut -d' ' -f 2 >> $FILE does the same thing.
echo "$STRING_VAR" | cut -d' ' -f 2 >> $FILE
should be all you need
Also, bash has the handy "here-string" redirection mode: you don't need echo at all:
cut -d' ' -f2 <<< "$STRING_VAR" >> "$FILE"