I'm trying to call a number of spiders from a shell script like so:
#!/bin/bash
source /home/pi/.bashrc
source /usr/local/bin/virtualenvwrapper.sh
workon email-scraper
cdvirtualenv
#Change to dir containing scrapy.cfg
cd ./sitecode/jobscraper
pwd
scrapy crawl site1
scrapy crawl site2
scrapy crawl site3
# Email me the jobs.csv file
python '/home/pi/Documents/jobscraper/scripts/jobs-mail.py'
# Delete the file so a new one is created on next scrape
# sudo rm /home/pi/Documents/jobscraper/csv/jobs.csv
The only part it executes properly is the python script at the end, which emails me an empty csv file as none of the scrapers have run. I get the following output when running the above script from BASH
(email-scraper)pi#raspberrypi ~/.virtualenvs/email-scraper/sitecode $ sudo sh runspiders.sh
ERROR: Environment 'email-scraper' does not exist. Create it with 'mkvirtualenv email-scraper'.
ERROR: no virtualenv active, or active virtualenv is missing
runspiders.sh: line 9: cd: ./sitecode/jobscraper: No such file or directory
/home/pi/.virtualenvs/email-scraper/sitecode
runspiders.sh: line 13: scrapy: command not found
runspiders.sh: line 14: scrapy: command not found
runspiders.sh: line 15: scrapy: command not found
runspiders.sh: line 16: scrapy: command not found
runspiders.sh: line 17: scrapy: command not found
runspiders.sh: line 18: scrapy: command not found
runspiders.sh: line 19: scrapy: command not found
runspiders.sh: line 20: scrapy: command not found
runspiders.sh: line 21: scrapy: command not found
runspiders.sh: line 22: scrapy: command not found
I'm very new to shell scripting. Can anyone illustrate how to make sure I activate the virtualenv and change into the correct dir before calling the first spider?
Edit In reply to #konsolebox
This is how I would run the scraper manually from the home dir:
First, source .bashrc as the raspberry pi doesn't do this automatically for some reason.
source .bashrc
This allows me access to virtualenvwrapper. I can then
pi#raspberrypi ~ $ workon email-scraper
(email-scraper)pi#raspberrypi ~ $ cdvirtualenv
which puts me in the virtualenv project directory /home/pi/.virtualenvs/email-scraper
I then do
cd sitecode/jobscraper
and ls -al puts me in the dir with access to scrapy.cfg, which I need in order to run the scraper.
drwxr-xr-x 3 pi pi 4096 Sep 9 19:40 .
drwxr-xr-x 5 pi pi 4096 Sep 8 19:41 ..
drwxr-xr-x 3 pi pi 4096 Sep 8 14:59 jobscraper
-rwxr-xr-x 1 pi pi 632 Sep 8 22:18 runspiders.sh
-rw-r--r-- 1 root root 12288 Sep 9 19:40 .runspiders.sh.swp
-rw-r--r-- 1 pi pi 381 Sep 7 23:34 scrapy.cfg
I can then do scrapy crawl site1 to run the scraper.
Perhaps you actually need to run the script in ~/.virtualenvs/email-scraper/. Do cd ~/.virtualenvs/email-scraper/ before running it.
When you're there run
sh sitecode/runspiders.sh
Or
sudo sh sitecode/runspiders.sh
#!/bin/bash
cd /home/pi
source ./bashrc
## source /usr/local/bin/virtualenvwrapper.sh ## You didn't run this.
workon email-scraper
cdvirtualenv
cd ./sitecode/jobscraper
scrapy crawl site1
Run as bash script.sh, not sudo sh script.sh.
Related
I have the following file xx.sh:
#!/bin/bash
echo "hi"
The permissions are defined as follows:
-rwxr-xr-x 1 root root 22 Nov 22 10:55 xx.sh*
but when I run the file ./xx.sh I get:
**-bash: ./xx.sh: Permission denied**
When running with override: bash xx.sh it runs correctly.
What is the permission issue here?
Your system is probably mounted with noexec. Verify by:
findmnt <folder>
If you have noexec in the OPTIONS, then you are not allowed to execute executable binaries in the mounted file system. You can solve using the mount command.
I'm trying to run a small bash script inside my CodeBuild process as per the AWS documentation and this.
deploy.sh can be found in the src/ directory:
#!/bin/bash
pwd=$PWD
for dir in */ ; do
target="$pwd/${dir%/}"
cd "$target"
npm install
sls deploy && sls s3deploy
done
buildspec file:
version: 0.2
phases:
install:
runtime-versions:
nodejs: 10
commands:
- cd backend/functions/src
- npm install serverless
build:
commands:
- ls
- ./deploy.sh
after the ls command above in the logs, I can see that deploy.sh is there in the current directory.
Error:
/codebuild/output/tmp/script.sh: 4: /codebuild/output/tmp/script.sh: ./deploy.sh: not found
I've also tried using the command /bin/bash deploy.sh but this results in another error:
deploy.sh: line 2: $'\r': command not found
deploy.sh: line 4: $'\r': command not found
deploy.sh: line 7: $'\r': command not found
deploy.sh: line 8: syntax error near unexpected token `$'do\r''
deploy.sh: line 8: `for dir in */ ; do
(First written as a comment, it solved the problem)
The files were made in Windows.
Change CRLF to LF and the \r is gone.
The first answer by Walter A worked for me, this is what i did .
Opened the file in VS Code
At bottom left corner for Select End Of Line Sequence it showed CRLF
clicked on CRLF and selected LF
Uploaded the file to github and it started working.
Using Windows, I installed Rakudo Star and Git and ensured that C:\rakudo\bin and C:\rakudo\share\perl6\site\bin are in my Path environment variable.
Now, typing perl6 inside Git Bash afterwards gives the command not found error, while the command does work inside powershell and cmd. Typing echo $PATH inside Git Bash confirms again that the folders above are in my path variable here as well.
How can I get the perl6 command working inside Git Bash?
Note: Using moar (moar.exe) which resides in the same folder as perl6 works as well in Git Bash. Also hitting Tab show the autocomplete suggestion for moar, it does not do that for perl6.
Bash doesn't run Windows batch files, so you'll have to work around that.
An easy solution might be to add something like this you your .bashrc:
alias perl6='cmd /c perl6.bat'
Alternatively, you can convert perl6.bat to a shell script and put it somewhere in your $PATH.
I use the following:
#!/bin/sh
PATH=/cygdrive/c/rakudo/bin:/cygdrive/c/rakudo/share/perl6/site/bin:$PATH
unset HOME
moar --execname="$0" \
--libpath='C:\rakudo\share\nqp\lib' \
--libpath='C:\rakudo\share\perl6\lib' \
--libpath='C:\rakudo\share\perl6\runtime' \
'C:\rakudo\share\perl6\runtime\perl6.moarvm' \
"$#"
This is using Cygwin; you may need to adapt it a bit for Git bash (I don't know, no experience with it).
Alternatively, if you're using Windows 10, I can recommend installing WSL, and using perl6 in a WSL bash prompt instead. This runs much smoother for me than the Windows version under Cygwin.
I tried to install perl6 from the link you provided and I can confirm the same behavior on Cygwin on Windows 10.
If I type in the Cygwin terminal window:
$ perl6
-bash: perl6: command not found
$ echo $PATH
/usr/local/bin:/usr/bin:/cygdrive/c/WINDOWS/system32:/cygdrive/c/WINDOWS:/cygdrive/c/WINDOWS/System32/Wbem:/cygdrive/c/WINDOWS/System32/WindowsPowerShell/v1.0:/cygdrive/c/WINDOWS/System32/OpenSSH:/cygdrive/c/Users/Bruker/AppData/Local/Microsoft/WindowsApps:/cygdrive/c/rakudo/bin:/cygdrive/c/rakudo/share/perl6/site/bin
$ cd /cygdrive/c/rakudo/bin
$ ls -l
-rwxrwx---+ 1 SYSTEM SYSTEM 930663 May 11 2017 libgcc_s_seh-1.dll
-rwxrwx---+ 1 SYSTEM SYSTEM 136146 Mar 30 20:55 libmoar.dll.a
-rwxrwx---+ 1 SYSTEM SYSTEM 56978 May 11 2017 libwinpthread-1.dll
-rwxrwx---+ 1 SYSTEM SYSTEM 7021172 Mar 30 20:55 moar.dll
-rwxrwx---+ 1 SYSTEM SYSTEM 64066 Mar 30 20:55 moar.exe
-rwxrwx---+ 1 SYSTEM SYSTEM 126 Mar 30 20:56 nqp.bat
-rwxrwx---+ 1 SYSTEM SYSTEM 126 Mar 30 20:56 nqp-m.bat
-rwxrwx---+ 1 SYSTEM SYSTEM 242 Mar 30 20:56 perl6.bat
-rwxrwx---+ 1 SYSTEM SYSTEM 248 Mar 30 20:56 perl6-debug-m.bat
-rwxrwx---+ 1 SYSTEM SYSTEM 242 Mar 30 20:56 perl6-m.bat
$ cat perl6.bat
# "C:\rakudo\bin\moar" --execname="%~dpf0" --libpath="C:\rakudo\share\nqp\lib" --libpath="C:\rakudo\share\nqp\lib" --libpath="C:\rakudo\share/perl6/lib" --libpath="C:\rakudo\share/perl6/runtime" C:\rakudo\share\perl6\runtime\perl6.moarvm %*
Notice that the paths in the bat file are not cygwin paths. So that might explain why it does not work..
For example:
$ "C:\rakudo\bin\moar"
-bash: C:\rakudo\bin\moar: command not found
$ /cygdrive/c/rakudo/bin/moar
ERROR: Missing input file.
USAGE: moar [--crash] [--libpath=...] input.moarvm [program args]
moar --dump input.moarvm
moar --help
[...]
Update:
I also tried install Git Bash, and then from the MINGW64 terminal window:
$ echo $PATH
/c/Users/Bruker/bin:/mingw64/bin:/usr/local/bin:/usr/bin:/bin:/mingw64/bin:/usr/bin:/c/Users/Bruker/bin:/c/WINDOWS/system32:/c/WINDOWS:/c/WINDOWS/System32/Wbem:/c/WINDOWS/System32/WindowsPowerShell/v1.0:/c/WINDOWS/System32/OpenSSH:/c/Users/Bruker/AppData/Local/Microsoft/WindowsApps:/usr/bin/vendor_perl:/usr/bin/core_perl
$ PATH=/c/rakudo/bin:$PATH
$ perl6
bash: perl6: command not found
$ moar
ERROR: Missing input file.
USAGE: moar [--crash] [--libpath=...] input.moarvm [program args]
moar --dump input.moarvm
moar --help
[...]
Note that moar is an .exe file while perl6 is a .bat file.
Also it seems perl6 is not "offical" for Cygwin yet according to this issue.
I'm getting the following error when systemctl attempts to run my startup script
/usr/bin/startup/status_start.sh: /usr/local/bin/procodile: /usr/bin/ruby: bad interpreter: No such file or directory
Here's my start script
#!/usr/bin/bash
cd /opt/staytus/staytus/
procodile start
Now I assume this has something to do with the bash shell and PATH's etc but I don't really understand that world :)
I have tried my start script with both #!/usr/bin/bash and #!/bin/bash
Any other suggestions?
UPDATE 1:
added a line to my script to print out 'which ruby' which resulted in the following error
Oct 06 17:31:02 status.sh[11014]: which: no ruby in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin)
which lead me to add more
echo "current path:" $PATH
PATH=/opt/staytus/.rbenv/shims/ruby:$PATH
echo "new path:" $PATH
BUT despite my attempts I still get errors when systemctl runs start...
Oct 06 17:45:37 start.sh[2878]: current path: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Oct 06 17:45:37 start.sh[2878]: new path: /opt/staytus/.rbenv/shims/ruby:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Oct 06 17:45:37 start.sh[2878]: /usr/bin/startup/start.sh: /usr/local/bin/procodile: /usr/bin/ruby: bad interpreter: No such file or directory
Update 2
here's the systemctl script
[Unit]
Description=Starts up procodile which runs staytus
[Service]
User=staytus
Type=simple
WorkingDirectory=/opt/staytus/staytus/
ExecStart=/usr/bin/startup/start.sh
ExecStop=/usr/bin/startup/stop.sh
# Restart=on-abort
[Install]
WantedBy=multi-user.target
When running version as root, staytus
bash: /usr/bin/ruby: No such file or directory
Ruby appears to be installed in
which ruby
~/.rbenv/shims/ruby
UPDATE 3:
OK so I decided to add what appeared to be missing path flows into my script by another script that prints the command to set the environment via the eval command
path.sh
#!/bin/sh
echo export PATH=/opt/staytus/bin/procodile:/opt/staytus/.rbenv/shims/ruby:/usr/lib64/ruby:/usr/share/ruby:/usr/bin/bash:$PATH
start.sh
#!/usr/bin/env bash
set -xv
whoami
echo "current path:" $PATH
cd /opt/staytus/staytus/
pwd
eval `/usr/bin/startup/path.sh`
echo "New path:" $PATH
procodile start
One of your scripts -- probably /usr/local/bin/procodile, but potentially something it invokes -- starts with #!/usr/bin/ruby.
However, your Ruby interpreter isn't installed as /usr/bin/ruby, so that breaks.
Find the responsible script, and change its first line to #!/usr/bin/env ruby, which will honor the PATH for finding the interpreter to run with.
I've recompiled vim with the ruby flag for command-T awesomeness, and I wrote a script that will automatically do this for me on any machine I ssh into (and run the script), but an issue that I'm running into is that when I alias the newly compiled vim, bash throws a no such file or directory at me. Here's what I have:
This works fine:
$ alias v="~/Code/vim"
But when I put this same line in my .bashrc, I get:
-bash: =~/Code/vim: No such file or directory
Even funnier, I put this in my .bashrc:
[ -e "~/Code/vim" ] && echo "found it!"
And that prints "found it!"
I'm not sure if it's a permission issue, but I'm just going to put them here anyway:
-rwxr-xr-x 1 buf staff 2374400 Sep 17 16:27 /Users/buf/Code/vim
What am I doing wrong? I'm running mac os x 10.7
-bash: =~/Code/vim: No such file or directory
It looks like you forgot remove the = sign