Is it possible to put value from linux pipe into curl querystring param? - bash

I have a numeric output from command, let's say:
sh -c 'exit 1' ; echo $?
or
bc <<< "1 + 1"
And I need to send it in GET request via curl, like http://example.com/?value=1
I've tried this:
sh -c 'exit 1' ; echo $? | curl -G -d #- http://example.com/
but it just got param with a name 1 and empty value.
I know I can do something like:
result=`sh -c 'exit 129' ; echo $?` | curl -G -d value=${result} http://example.com
but I'd like to keep the first part of command unchanged and modify only part after pipe. Is it possible?


One possible solution I found:
sh -c 'exit 129'; echo $? | xargs -I '{}' curl -G "http://example.com?value={}";

Related

Quick search to find active urls

I'm trying to use cURL to find active rediractions and save results to file. I know the url is active, when it redirects at least once to specific website. So I came up with:
if (( $( curl -I -L https://mywebpage.com/id=00001&somehashnumber&si=0 | grep -c "/something/" ) > 1 )) ; then echo https://mywebpage.com/id=00001&somehashnumber&si=0 | grep -o -P 'id=.{0,5}' >> id.txt; else echo 404; fi
And it works, but how to modify it to check id range from 00001 to 99999?

you'll want to wrap the whole operation in a for loop and use a formatted sequence to print the ids you'd like to test. without know too much about the task at hand i would write something like this to test the ids
$ for i in $(seq -f "%06g" 1 100000); do curl --silent "example.com/id=$i" --write-out "$i %{response_code}\n" --output /dev/null; done

grep -c kills script when no match using set -e

Basic example:
#!/bin/bash
set -e
set -x
NUM_LINES=$(printf "Hello\nHi" | grep -c "How$")
echo "Number of lines: ${NUM_LINES}" # never prints 0
Output:
++ grep -c 'How$'
++ printf 'Hello\nHi'
+ NUM_LINES=0
If there are matches, it prints the correct number of lines. Also grep "How$" | wc -l works instead of using grep -c "How$".

You can suppress grep's exit code by running : when it "fails". : always succeeds.
NUM_LINES=$(printf "Hello\nHi" | grep -c "How$" || :)

xargs wget extract filename from URL with Parameter

I want to do parallel downloads but the problem wget output not correct filename.
url.txt
http://example.com/file1.zip?arg=tereef&arg2=okook
http://example.com/file2.zip?arg=tereef&arg2=okook
command
xargs -P 4 -n 1 wget <url.txt
output filename
file1.zip?arg=tereef&arg2=okook
file2.zip?arg=tereef&arg2=okook
expected output
file1.zip
file2.zip
I'm new with bash, please suggest me how to output correct filename, and please don't suggest for loop or & because it blocking.
Thank you

You can use a bash function that you have to export to be seen outside the current shell
function mywget()
{
wget -O ${1%%\?*} "'$1'"
}
export -f mywget
xargs -P 4 -n 1 -I {} bash -c "mywget '{}'" < url.txt

Process your input to produce the desired command, then run it through xargs.
perl -ne - iterate over the lines of the input file and execute the inline program
-e : Execute perl one-liner
-n : Loop over all input lines, assigning each to $_ in turn.
xargs -P 4 -n 1 -i -t wget "{}"
-P 4 : Max of 4 Processes at a time
-n 1 : Consume one input line at a time
-i : Use the replace string "{}"
-t : Print the command before executing it
perl -ne '
chomp(my ($url) = $_); # Remove trailing newline
my ($name) = $url =~ m|example.com/(.+)\?|; # Grab the filename
print "$url -O $name\n"; # Print all of the wget params
' url.txt | xargs -P 4 -n 1 -i -t wget "{}"
Output
wget http://example.com/file1.zip?arg=tereef&arg2=okook -O file1.zip
wget http://example.com/file2.zip?arg=tereef&arg2=okook -O file2.zip
--2016-07-21 22:24:44-- http://example.com/file2.zip?arg=tereef&arg2=okook%20-O%20file2.zip
--2016-07-21 22:24:44-- http://example.com/file1.zip?arg=tereef&arg2=okook%20-O%20file1.zip
Resolving example.com (example.com)... Resolving example.com (example.com)... 93.184.216.34, 2606:2800:220:1:248:1893:25c8:1946
93.184.216.34, Connecting to example.com (example.com)|93.184.216.34|:80... 2606:2800:220:1:248:1893:25c8:1946
Connecting to example.com (example.com)|93.184.216.34|:80... connected.
connected.
HTTP request sent, awaiting response... HTTP request sent, awaiting response... 404 Not Found
2016-07-21 22:24:44 ERROR 404: Not Found.
404 Not Found
2016-07-21 22:24:44 ERROR 404: Not Found.

With GNU Parallel it looks like this:
parallel -P 4 wget -O '{= s/\?.*//;s:.*/:: =}' {} <url.txt

Add floating numbers in bash

I'm trying something very easy but all code that I'm trying doesnt work.
I need add two float numbers in bash. I'm doing this:
result1=`$CURL -o /dev/null -s -w %{time_total} $url1`
result2=`$CURL -o /dev/null -s -w %{time_total} $url2`
result3=`$CURL -o /dev/null -s -w %{time_total} $url3`
total= `expr $result2 + $result3`
echo $total | $GAWK -F: '{ print "connection_1.value " $1 }'
but in the prompt I'm getting this output:
./http_response_2: line 12: 0,018+0,255: command not found
connection_1.value
I'm trying too do this:
result1=`$CURL -o /dev/null -s -w %{time_total} $url1`
result2=`$CURL -o /dev/null -s -w %{time_total} $url2`
result3=`$CURL -o /dev/null -s -w %{time_total} $url3`
total= `$result2 + $result3 | bc`
getting the same result.
Thanks in advance!

There are 3 issues:
There should be no space between total= & `
echo missing before $result2 + $result3
There is comma in your input, instead of decimal point.
Fixing all these issues:
total=$(tr ',' '.' <<< "$result2 + $result3" | bc -l)
If you are concerned about the leading 0 before decimal point, try:
total=$(tr ',' '.' <<< "$result2 + $result3" | bc -l | xargs printf "%g")

Instead of replacing commas with dots, don't produce commas in the first place.
They emerge from localization, so use LC_ALL=C as prefix, like:
LC_ALL=C curl -o /dev/null -s -w %{time_total} www.google.com
and abandon the outdated backticks, use $(...) instead:
result1=$(LC_ALL=C $CURL -o /dev/null -s -w %{time_total} $url1)

Bash error with subtraction

I've got some problem with substraction and I don't know why :(
it's my code:
#!/bin/bash
w3m http://www.weather.com/weather/hourbyhour/graph/PLXX0027 > plik.txt
image= grep -o 'http.*' plik.txt
t= cat plik.txt |sed '105q;d' | grep -o '[0-9][0-9]'
a=32
temp=$((t-a))
echo $temp
I've received sth like:
name#name ~/Desktop $ sh p.sh
http://s.imwx.com/v.20120328.084252//img/wxicon/70/14.png
25
-32
but i wan to receive substraction of 25-32... (of course 25 depends of value in webpage) but why it don't want substract it?

Try defining properly all variables, with $() surrounding them.
#!/bin/bash
w3m http://www.weather.com/weather/hourbyhour/graph/PLXX0027 > plik.txt
image=$(grep -o 'http.*' plik.txt)
t=$(cat plik.txt |sed '105q;d' | grep -o '[0-9][0-9]')
a=32
temp=$((t-a))
echo $temp

Resources