I want to use several instances of a spider object of scrapy with several instances of polipo proxies.
For this, it is quite simple:
for i in `seq 1 100`
do
numeroPortProxy=$(($i+30000))
command="scrapy crawl courses_PT ... -s HTTP_PROXY=http://127.0.0.1:${numeroPortProxy}"
eval $command&
done
My issue is I need to answer automatically "o" to the command.
I tried this eval $command&<<<"o", it does not work. The logfile indicates: EOFError: EOF when reading a line
I tried this too eval $command<<<"o" &, it does not work too. It gives this error in the terminal:
Usage
=====
scrapy crawl [options] <spider>
crawl: error: Invalid -s value, use -s NAME=VALUE
Well it does not appreciate.
So, how can I do to launch several instances of command at one time like & operator allows and to answer the prompt of each several instances like <<< allows.
Related
I have following scripts
#!/bin/bash
set -o xtrace
gluster_volume="a.example.com:/data/brick1/gv0 b.example.com:/data/brick1/gv0"
gluster volume create gv0 replica 2 ${gluster_volume} force
While executing second line this script throwing error related to gluster_volume that its unable read complete line and just considering a.example.com:/data/brick1/gv0 b.example.com(so look like gluster command not able parse it).
Then I run this script with trace and found that command is like
gluster volume create gv0 replica 2 'a.example.com:/data/brick1/gv0 b.example.com:/data/brick1/gv0' force
so script is adding single quotes which are creating problem. I was under the impression that it's due to set -o xtrace the single quotes are not part of the value, just part of the displayed command line. But its not true. if i run above command on command prompt it gives same error and on removing quotes it works.
How i can change my script so it don't add quotes? (I tried to remove quote with sed but not working, something happening at run time).
I need gluster_volume with dynamic values.
Update1:
workers=`echo "${WORKER_HOST_IP}"|sed "s/,$//"`
IFS=','
for worker in ${workers}; do
IFS='-' read -r -a array <<< "$worker"
gluster_volume+=${array[0]}':/data/brick1/gv0 '
done
You can try eval(Be careful if the list of bricks or the string comes from an external untrusted source)
#!/bin/bash
set -o xtrace
gluster_volume="a.example.com:/data/brick1/gv0 b.example.com:/data/brick1/gv0"
eval "gluster volume create gv0 replica 2 ${gluster_volume} force"
I'm working on a mac running El Capitan.
For a project I've been working on I'm trying to write a simple script to log ping times. I've come to the conclusion it isn't as simple as I'd thought. My first problem was "Ambiguous redirect" when using variables. I've corrected that, using quotes around variables, with help from $INPUT Ambiguous redirect
But now I get a different error when running the following script:
#!/bin/sh
set PINGDELAY=1.5
set PINGIP=google.nl
set PINGLOG=~/Library/Logs/doctorping.log
sudo ping -i "$PINGDELAY" "$PINGIP" | perl -nle 'use Time::Piece; BEGIN {$|++} print localtime->datetime, " ", $_' >> "$PINGLOG"
The error is
ping: invalid timing interval: `'
It's probably something I've overlooked but I'm a real noob when it comes to scripting and programming.
My goal isn't to process text or extract bits of it, it's just to monitor connections and have the results written to the log. It'd probably be wise to limit the number of log lines, but I'll get to that later. (of course would be appreciated if someone could point me in the right direction, but first things first)
Thanks!
The set command is not to set shell variables; it's used to set shell execution options and/or replace the script's argument list. Just leave it off. Also, it's best to use lowercase (or mixed-case) variable names to avoid accidentally using one of the variables that means something special to the shell. Here's what I get:
#!/bin/sh
pingdelay=1.5
pingIP=google.nl
pinglog=~/Library/Logs/doctorping.log
sudo ping -i "$pingdelay" "$pingIP" | perl -nle 'use Time::Piece; BEGIN {$|++} print localtime->datetime, " ", $_' >> "$pinglog"
It works fine if you use bash like this rather than sh:
#!/bin/bash -xv
PINGDELAY=1.5
PINGIP=google.nl
PINGLOG=~/Library/Logs/doctorping.log
sudo ping -i "$PINGDELAY" "$PINGIP"
The -xv is just for debugging - you can remove it safely.
I am running a bash script that takes hours. I was wondering if there is way to monitor what is it doing? like what part of the script is currently running, how long did it take to run the whole script, if it crashes at what line of the script stopped working, etc. I just want to receive feedback from the script. Thanks!!!
from man page for bash,
set -x
After expanding each simple command, for command, case command, select command, or arithmetic for command, display the expanded value of PS4, followed by the command and its expanded arguments or associated word list.
add these to the start of your script,
export PS4='+{${BASH_SOURCE}:$LINENO} '
set -x
Example,
#!/bin/bash
export PS4='+{${BASH_SOURCE}:$LINENO} '
set -x
echo Hello World
Result,
+{helloworld.sh:6} echo Hello World
Hello World
Make a status or log file. For example add this inside your script:
echo $(date) - Ok >> script.log
Or for a real monitoring you can use strace on linux for see system call, example:
$ while true ; do sleep 5 ; done &
[1] 27190
$ strace -p 27190
I would like to pass parameters to a perl script using positional parameters inside a bash script "tablecheck.sh". I am using an alias "tablecheck" to call "tablecheck.sh".
#!/bin/bash
/scripts/tables.pl /var/lib/mysql/$1/ /var/mysql/$1/mysql.sock > /tmp/chktables_$1 2>&1 &
Perl script by itself works fine. But when I do "tablecheck MySQLinstance", $1 stays $1. It won't get replaced by the instance. So I get the output as follows:
Exit /scripts/tables.pl /var/lib/mysql/$1/ /var/mysql/$1/mysql.sock > /tmp/chktables_$1 2>&1 &
The job exits.
FYI: alias tablecheck='. pathtobashscript/tablecheck.sh'
I have a bunch of aliases in another bash script. Hence . command.
Could anyone help me... I have gone till the 3rd page of Google to find an answer. Tried so many things with no luck.
I am a noob. But may be it has something to do with it being a background job or $1 in a path... I don't understand why the $1 won't get replaced...
If I copy your exact set up (which I agree with other commenters, is some what unusual) then I believe I am getting the same error message
$ tablecheck foo
[1]+ Exit 127 /scripts/tables.pl /var/lib/mysql/$1/ /var/mysql/$1/mysql.sock > /tmp/chktables_$1 2>&1
In the /tmp/chktables_foo file that it makes there is an additional error message, in my case "bash: /scripts/tables.pl: No such file or directory"
I suspect permissions are wrong in your case
I am writing a simple bash script (checkServs.sh) that will ssh into a list of servers and perform a health check on them.
I keep getting errors on the following line:
SERVERS=(blah1.example.com blah2.example.com blah3.example.com blah4.example.com)
Error is:
checkServs.sh: 3: checkServs.sh: Syntax error: "(" unexpected
I've checked online examples and this seems correct, isn't it? Thanks in advance!
I don't know about the syntax error, but this should work:
SERVERS="blah1.example.com blah2.example.com blah3.example.com blah4.example.com"
for server in $SERVERS
do
echo $server
done
EDIT: As noted by Jonathan Leffler in a comment, maybe you are not running the script with bash. Other shells, such as dash, may not recognize the array syntax. If that's the case, you can do:
SERVERS=(blah1.example.com blah2.example.com blah3.example.com blah4.example.com)
for i in $(seq 0 3)
do
echo ${SERVERS[$i]}
done
But if you just want to loop through the names and run an SSH command (ie if having an array won't provide useful functionality), the first method is more straightforward.
Your remote server probably calls a different shell when executing commands. Try to add bash -c to your arguments:
ssh user#server bash -c "<your commands>"
Or:
ssh user#server bash < yourscript.sh ## None in yourscript.sh must read input though.
An opening parenthesis starts a subshell, which is not a correct thing to have on the right side of an equals sign. It expects a string expression, not a command.
Quotation marks are used to keep a string expression together.