I have the following while loop in a bash script however I would like to run these in parallel (an failing) can anyone point me in to the right direction please?
Thanks!
while read LINE; do
RAYID=$(echo "$LINE" | jq -r .rayId)
LINE="$(echo $LINE | sed 's/\([[:digit:]]\{13\}\)[[:digit:]]\{6\}/\1/g')"
args=( -XPUT "localhost:9200/els/logs/$RAYID?pipeline=geoip-els" -H "Content-Type: application/json" -d "$LINE" )
curl "${args[#]}" > /dev/null 2>&1
done <<< "$ELS_LOGS"
** EDITED
Additionally to what #TomFenech stated which is correct, I want to add that it would be also nice if you add wait after done, so the script won't finish its execution, until all tasks are completed.
function doSomething(){
RAYID=$(echo "$1" | jq -r .rayId )
LINE="$(echo $1 | sed 's/\([[:digit:]]\{13\}\)[[:digit:]]\{6\}/\1/g' )"
args=( -XPUT "localhost:9200/els/logs/$RAYID?pipeline=geoip-els" -H "Content-Type: application/json" -d "$1" )
curl "${args[#]}" > /dev/null 2>&1
}
while read LINE; do
doSomething $LINE &
done <<< "$ELS_LOGS"
wait
Regards!
Related
i have this files
1.json2 - 2.json2 - 3.json2
1.json2.ml - 2.json2.ml -
Example ml file
1.json2.ml
{"message":"Validation error","error":"validation_error",...
2.json2.ml
{"Ok":"OK":"OK"...}
I want to search if *.json2.ml isnt execute a post and save.
If the file exist look if error is there and execute Post.
Here is the code i use for this
find . -type f -name '*.json2' | xargs bash -c 'for fname
do if [ ! -e ${fname}.ml ]
then curl -X POST -H "Content-Type: application/json" -d #${fname} https://web/api/post > ${fname}.ml
else
sed '1d' ${fname}.ml | while read line
do
FS=',' read pid pname
if [ "$var" -e ""error":"validation_error"" ]
then
curl -X POST -H "Content-Type: application/json" -d #${fname} https://web/api/post > ${fname}.ml
echo que ha y $pname
fi
done
' bash
I have this result
syntax error near unexpected token `fi'
What is the expected result
1 - Post 3.json2 ( file .ml no exist )
2- Post 1.json2 ( File .ml exist and have error in )
3- 2.json2 and 3.json2 ( do nothing because json2.ml is OK)
I found the solution whit this code
else
VAR1=$(head -n 1 ${fname}.ml)
IFS="," read -ra images <<< "$VAR1"
echo que ha y $images
Thx alot to Barmar and William
I have a script that tails a log file, and then uploads the line. I would like to have it exit as soon as the first line is read:
#!/bin/bash
tail -n0 -F "$1" | while read LINE; do
(echo "$LINE" | grep -e "$3") && curl -X POST --silent --data-urlencode \
"payload={\"text\": \"$(echo $LINE | sed "s/\"/'/g")\"}" "$2";
done
If you want to exit as soon as the first line is uploaded you can just add a break:
#!/bin/bash
tail -n0 -F "$1" | while read LINE; do
(echo "$LINE" | grep -e "$3") && curl -X POST --silent --data-urlencode \
"payload={\"text\": \"$(echo $LINE | sed "s/\"/'/g")\"}" "$2" && break;
done
The issue was the tail command wasn't getting killed. A slightly modified version of my script (I didn't end up needing the echo to stdout)
#!/bin/bash
tail -n0 -F "$1" | while read LINE; do
curl -X POST --data-urlencode "payload={\"text\": \"$(echo $LINE | sed "s/\"/'/g")\"}" "$2" && pkill -P $$ tail
done
This answer helped as well: https://superuser.com/questions/270529/monitoring-a-file-until-a-string-is-found
I'm using curl to retrieve the http_code size_header redirect_url and Website Title with:
#!/bin/bash
FILE="$1"
while read LINE; do
curl -H 'Cache-Control: no-cache' -i -s -k -o >(perl -l -0777 -ne 'print $1 if /<title.*?>\s*(.*?)\s*<\/title/si') --silent --max-time 2 --write-out '%{http_code} %{size_header} %{redirect_url} ' "$LINE"
echo " $LINE"
done < ${FILE}
but I like to also retrieve the first 20 characters from the response to have more information.
The idea is to get this output
%{http_code} %{size_header} %{redirect_url} $website_title $website_first_20_bytes
I only need to add the $website_first_20_bytes to the output. How can I achieve this?
PS: No the first 20 characters from header response. Only the source.
So you probably mean something like this then (I added a bunch of newlines etc. to the output which you can trim as you please):
#!/bin/bash
FILE="$1"
while read -r LINE; do
# read the response to a variable
response=$(curl -H 'Cache-Control: no-cache' -s -k --max-time 2 --write-out '%{http_code} %{size_header} %{redirect_url} ' "$LINE")
# get the title
title=$(sed -n 's/.*<title>\(.*\)<\/title>.*/\1/ip;T;q'<<<"$response")
# read the write-out from the last line
read -r http_code size_header redirect_url < <(tail -n 1 <<<"$response")
printf "Status: %s\n" "$http_code"
printf "Size: %s\n" "$size_header"
printf "Redirect-url: %s\n" "$redirect_url"
printf "Url: %s\n" "$LINE"
printf "Title: %s" "$title"
# -c 20 only shows the 20 first chars from response
printf "Body: %s" "$(head -c 20 <<<"$response")"
done < ${FILE}
I'm using the at command to schedule a job in the future.
DoCurlAt () {
if [ -n "${AuthToken:-}" ] ; then
$4 << 'EOF'
curl -s -H "${AuthHeader:-}" -H "$1" --data-urlencode "$2" "$3"
EOF
Exitcode=$?
fi
WriteLog Output Info "AT Output: $AtOutput Exitcode: $Exitcode"
}
How can I capture the result of the at in a variable called $AtOutput?
I tried with
AtOutput=$(bash $4 << EOF
curl -s -H "${AuthHeader:-}" -H "$1" --data-urlencode "$2" "$3"
EOF
)
But that does't really give any result.
Also tried with:
AtOutput=$(curl -s -H "${AuthHeader:-}" -H "$1" --data-urlencode "$2" "$3" | at "$4")
But I would prefer to use the HERE-doc.
The function is called with
DoCurlAt "$AcceptJson" "argString=$ArgString" "$ApiUrl/$ApiVersion/job/$JobUid/run" "$OneTime"
$OneTime ($4) could be for example "at 15:19 today" The output is mostly something like this:
job 7 at 2016-08-16 15:30
at writes to standard error, not standard output. Use the 2>&1 redirection to copy standard error to standard output first.
$ at_output=$( echo "cmd" | at "$when" 2>&1 )
This is what I have done so far;
#!/bin/bash
url="asdf.com/check "
for i in $(cat query.txt); do
content=$"curl --data "email=$i" "$url")"
echo "$content" >> output.txt
done
You can use your loop like this:
#!/bin/bash
url="asdf.com/check"
while read -r line; do
curl --data "email=$line" "$url"
done < query.txt > output.txt
You can use output redirection only once just after the done as shown.