I want to do parallel downloads but the problem wget output not correct filename.
url.txt
http://example.com/file1.zip?arg=tereef&arg2=okook
http://example.com/file2.zip?arg=tereef&arg2=okook
command
xargs -P 4 -n 1 wget <url.txt
output filename
file1.zip?arg=tereef&arg2=okook
file2.zip?arg=tereef&arg2=okook
expected output
file1.zip
file2.zip
I'm new with bash, please suggest me how to output correct filename, and please don't suggest for loop or & because it blocking.
Thank you
You can use a bash function that you have to export to be seen outside the current shell
function mywget()
{
wget -O ${1%%\?*} "'$1'"
}
export -f mywget
xargs -P 4 -n 1 -I {} bash -c "mywget '{}'" < url.txt
Process your input to produce the desired command, then run it through xargs.
perl -ne - iterate over the lines of the input file and execute the inline program
-e : Execute perl one-liner
-n : Loop over all input lines, assigning each to $_ in turn.
xargs -P 4 -n 1 -i -t wget "{}"
-P 4 : Max of 4 Processes at a time
-n 1 : Consume one input line at a time
-i : Use the replace string "{}"
-t : Print the command before executing it
perl -ne '
chomp(my ($url) = $_); # Remove trailing newline
my ($name) = $url =~ m|example.com/(.+)\?|; # Grab the filename
print "$url -O $name\n"; # Print all of the wget params
' url.txt | xargs -P 4 -n 1 -i -t wget "{}"
Output
wget http://example.com/file1.zip?arg=tereef&arg2=okook -O file1.zip
wget http://example.com/file2.zip?arg=tereef&arg2=okook -O file2.zip
--2016-07-21 22:24:44-- http://example.com/file2.zip?arg=tereef&arg2=okook%20-O%20file2.zip
--2016-07-21 22:24:44-- http://example.com/file1.zip?arg=tereef&arg2=okook%20-O%20file1.zip
Resolving example.com (example.com)... Resolving example.com (example.com)... 93.184.216.34, 2606:2800:220:1:248:1893:25c8:1946
93.184.216.34, Connecting to example.com (example.com)|93.184.216.34|:80... 2606:2800:220:1:248:1893:25c8:1946
Connecting to example.com (example.com)|93.184.216.34|:80... connected.
connected.
HTTP request sent, awaiting response... HTTP request sent, awaiting response... 404 Not Found
2016-07-21 22:24:44 ERROR 404: Not Found.
404 Not Found
2016-07-21 22:24:44 ERROR 404: Not Found.
With GNU Parallel it looks like this:
parallel -P 4 wget -O '{= s/\?.*//;s:.*/:: =}' {} <url.txt
Related
I have a numeric output from command, let's say:
sh -c 'exit 1' ; echo $?
or
bc <<< "1 + 1"
And I need to send it in GET request via curl, like http://example.com/?value=1
I've tried this:
sh -c 'exit 1' ; echo $? | curl -G -d #- http://example.com/
but it just got param with a name 1 and empty value.
I know I can do something like:
result=`sh -c 'exit 129' ; echo $?` | curl -G -d value=${result} http://example.com
but I'd like to keep the first part of command unchanged and modify only part after pipe. Is it possible?
One possible solution I found:
sh -c 'exit 129'; echo $? | xargs -I '{}' curl -G "http://example.com?value={}";
I have a file containing values:
file.txt
value1
value2
value3
And I have this script:
#!/bin/bash
input="test.txt"
while IFS= read -r line ; do
curl -o /dev/null -s -w "%{http_code}\n" -u username:token "https://website.com/api/user"$line
done < "$input" > result.txt
At the moment this is my output:
result.txt
400
200
200
What I really want to achieve is to get this kind of output:
output.txt
value1,400
value2,200
value3,200
What my code is missing to achieve this?
Run and capture the output of the curl command using the command substitution syntax $(...).
Then, use echo or printf to show the captured output, along with the original input, in the form you want.
input="test.txt"
while IFS= read -r line ; do
res=$(curl -o /dev/null -s -w "%{http_code}\n" -u username:token "https://website.com/api/user$line")
printf "%s\n" "$line,$res"
done < "$input" > result.txt
Trying to run a command as a variable but I am getting strange results
Expected result "1" :
grep -i nosuid /etc/fstab | grep -iq nfs
echo $?
1
Unexpected result as a variable command:
cmd="grep -i nosuid /etc/fstab | grep -iq nfs"
$cmd
echo $?
0
It seems it returns 0 as the command was correct not actual outcome. How to do this better ?
You can only execute exactly one command stored in a variable. The pipe is passed as an argument to the first grep.
Example
$ printArgs() { printf %s\\n "$#"; }
# Two commands. The 1st command has parameters "a" and "b".
# The 2nd command prints stdin from the first command.
$ printArgs a b | cat
a
b
$ cmd='printArgs a b | cat'
# Only one command with parameters "a", "b", "|", and "cat".
$ $cmd
a
b
|
cat
How to do this better?
Don't execute the command using variables.
Use a function.
$ cmd() { grep -i nosuid /etc/fstab | grep -iq nfs; }
$ cmd
$ echo $?
1
Solution to the actual problem
I see three options to your actual problem:
Use a DEBUG trap and the BASH_COMMAND variable inside the trap.
Enable bash's history feature for your script and use the hist command.
Use a function which takes a command string and executes it using eval.
Regarding your comment on the last approach: You only need one function. Something like
execAndLog() {
description="$1"
shift
if eval "$*"; then
info="PASSED: $description: $*"
passed+=("${FUNCNAME[1]}")
else
info="FAILED: $description: $*"
failed+=("${FUNCNAME[1]}")
done
}
You can use this function as follows
execAndLog 'Scanned system' 'grep -i nfs /etc/fstab | grep -iq noexec'
The first argument is the description for the log, the remaining arguments are the command to be executed.
using bash -x or set -x will allow you to see what bash executes:
> cmd="grep -i nosuid /etc/fstab | grep -iq nfs"
> set -x
> $cmd
+ grep -i nosuid /etc/fstab '|' grep -iq nfs
as you can see your pipe | is passed as an argument to the first grep command.
I am writing a shell script that will ssh to multiple hosts, and perform some operations on them.
my script, test.sh, looks as follows:
cat < $1 | xargs -e -d ' ' -I % ssh % grep "example" /opt/example/test.log
I run it via the following command
./test.sh prod_hosts.txt
and the contents of prod_hosts.txt:
hostname1 hostname2 hostname3 hostname4
I have verified there is no carraige return at the end of this file, yet I am getting the following error:
[ryan#hostname1 ~]$ ./test.sh prod_hosts.txt
ssh: hostname4
: Name or service not known
xargs: ssh: exited with status 255; aborting
It looks like it successfully ssh's into the 4 hosts but then has a blank entry that it is attempting to ssh with, hence the error.
Any idea what I'm missing here? Seems like I'm missing something obvious!
echo '1 2' | xargs -d ' ' -I % echo % produces:
1
2
<blank line>
whereas echo -n '1 2' | xargs -d ' ' -I % echo % returns:
1
2
i.e. xargs decides to generate one more output entry if the input string is ended by newline.
Workarounds:
Use newline delimited entries in hosts.txt and: <hosts.txt xargs -I % <ssh-command>
If hosts.txt cannot be changed: < <(tr ' ' '\n' < hosts.txt) xargs -I % <ssh-command>
I've got some problem with substraction and I don't know why :(
it's my code:
#!/bin/bash
w3m http://www.weather.com/weather/hourbyhour/graph/PLXX0027 > plik.txt
image= grep -o 'http.*' plik.txt
t= cat plik.txt |sed '105q;d' | grep -o '[0-9][0-9]'
a=32
temp=$((t-a))
echo $temp
I've received sth like:
name#name ~/Desktop $ sh p.sh
http://s.imwx.com/v.20120328.084252//img/wxicon/70/14.png
25
-32
but i wan to receive substraction of 25-32... (of course 25 depends of value in webpage) but why it don't want substract it?
Try defining properly all variables, with $() surrounding them.
#!/bin/bash
w3m http://www.weather.com/weather/hourbyhour/graph/PLXX0027 > plik.txt
image=$(grep -o 'http.*' plik.txt)
t=$(cat plik.txt |sed '105q;d' | grep -o '[0-9][0-9]')
a=32
temp=$((t-a))
echo $temp