I'm having trouble running a VM's startup scripts for instance groups. Commands like touch, cat, chmod are not found.
ubuntu#instance-template-****-1:~$ sudo google_metadata_script_runner -d --script-type startup
startup-script: INFO Starting startup scripts.
startup-script: INFO Found startup-script-url in metadata.
startup-script: INFO Downloading url from https://storage.googleapis.com/bucket-*****/startup-image to /home/ubuntu/startup-scripts/script using authentication token.
startup-script: INFO startup-script-url: /home/ubuntu
startup-script: INFO startup-script-url: /home/ubuntu
startup-script: INFO startup-script-url: /home/ubuntu/startup-scripts/script: line 11: touch: command not found
startup-script: INFO startup-script-url: /home/ubuntu/startup-scripts/script: line 12: touch: command not found
startup-script: INFO startup-script-url: /home/ubuntu/startup-scripts/script: line 13: touch: command not found
startup-script: INFO startup-script-url: /home/ubuntu/startup-scripts/script: line 15: chmod: command not found
startup-script: INFO startup-script-url: /home/ubuntu/startup-scripts/script: line 16: chmod: command not found
startup-script: INFO startup-script-url: /home/ubuntu/startup-scripts/script: line 71: ssh-keyscan: command not found
startup-script: INFO startup-script-url: /home/ubuntu/startup-scripts/script: line 72: ssh-keygen: command not found
startup-script: INFO startup-script-url: /home/ubuntu/startup-scripts/script: line 74: cat: command not found
startup-script: INFO startup-script-url: /home/ubuntu/startup-scripts/script: line 76: git: command not found
startup-script: INFO startup-script-url: /home/ubuntu/startup-scripts/script: line 78: sh: command not found
startup-script: INFO startup-script-url: Return code 127.
startup-script: INFO Finished running startup scripts.
Script:
#!/bin/bash
PATH=/home/ubuntu
cd $PATH
touch $PATH/.ssh/id_rsa
touch $PATH/.ssh/id_rsa.pub
touch $PATH/.ssh/known_hosts
chmod 644 $PATH/.ssh/id_rsa.pub
chmod 600 $PATH/.ssh/id_rsa
echo $IDRSA > $PATH/.ssh/id_rsa
echo $IDRSA_PUB > $PATH/.ssh/id_rsa.pub
ssh-keyscan bitbucket.org >> bitbucketKey
ssh-keygen -lf bitbucketKey
cat bitbucketKey >> $PATH/.ssh/known_hosts
git clone git#bitbucket.org:repo/repo.git
sh ./repo/prepare
PROJECT_ID=$(curl "http://metadata.google.internal/computeMetadata/v1/project/numeric-project-id" -H "Metadata-Flavor: Google")
./repo/deploy/run --dev --project-id=$PROJECT_ID
When you set PATH=/home/ubuntu, it overwrites the value of PATH environment variable that you're supposed to get in the shell and it forces the subsequent commands to end up in "command not found" error, because the appropriate executable files for running those commands are not available on new path /home/ubuntu that you just set. You can workaround this by renaming that variable to any name other than PATH and all of its occurrences in your script. You can use the same variable name in lower case too. (like path=home/ubuntu).
The PATH environment variable is a colon-delimited list of directories that your shell searches through when you enter a command. Program files (executables) are kept in many different places on the Unix system. Your path tells the Unix shell where to look on the system when you request a particular program.
You modified the PATH environment in your script:
PATH=/home/ubuntu
However, PATH is a Shell Variables of bash:
PATH The search path for commands. It is a colon-separated list of directories in which the shell looks for commands (see
COMMAND EXECUTION below). A zero-length (null) directory name in the value of PATH indicates the current directory.
A null directory name may appear as two adjacent colons, or as an initial or trailing colon. The default path is
system-dependent, and is set by the administrator who installs bash. A common value is ``/usr/local/bin:
/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin''.
Therefore almost all commands cannot be found. (Unless they are under /home/ubuntu)
If you just want to set a variable as a user folder to operate with the script, please change the variable name to like MY_HOME.
#!/bin/bash
MY_HOME=/home/ubuntu
cd $MY_HOME
touch $MY_HOME/.ssh/id_rsa
touch $MY_HOME/.ssh/id_rsa.pub
touch $MY_HOME/.ssh/known_hosts
...
Related
I'm trying to run a small bash script inside my CodeBuild process as per the AWS documentation and this.
deploy.sh can be found in the src/ directory:
#!/bin/bash
pwd=$PWD
for dir in */ ; do
target="$pwd/${dir%/}"
cd "$target"
npm install
sls deploy && sls s3deploy
done
buildspec file:
version: 0.2
phases:
install:
runtime-versions:
nodejs: 10
commands:
- cd backend/functions/src
- npm install serverless
build:
commands:
- ls
- ./deploy.sh
after the ls command above in the logs, I can see that deploy.sh is there in the current directory.
Error:
/codebuild/output/tmp/script.sh: 4: /codebuild/output/tmp/script.sh: ./deploy.sh: not found
I've also tried using the command /bin/bash deploy.sh but this results in another error:
deploy.sh: line 2: $'\r': command not found
deploy.sh: line 4: $'\r': command not found
deploy.sh: line 7: $'\r': command not found
deploy.sh: line 8: syntax error near unexpected token `$'do\r''
deploy.sh: line 8: `for dir in */ ; do
(First written as a comment, it solved the problem)
The files were made in Windows.
Change CRLF to LF and the \r is gone.
The first answer by Walter A worked for me, this is what i did .
Opened the file in VS Code
At bottom left corner for Select End Of Line Sequence it showed CRLF
clicked on CRLF and selected LF
Uploaded the file to github and it started working.
I've configured Hadoop 3.1.1 on my MacPro running OSX 10.14.2, and I'm getting the following error when I run start-all.sh
$ sudo /usr/local/Cellar/hadoop/3.1.1/sbin/start-all.sh
Starting namenodes on [localhost]
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-functions.sh: line 398: syntax error near unexpected token `<'
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-functions.sh: line 398: ` done < <(for text in "${input[#]}"; do'
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 70: hadoop_deprecate_envvar: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 87: hadoop_bootstrap: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 104: hadoop_parse_args: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 105: shift: : numeric argument required
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 244: hadoop_need_reexec: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 252: hadoop_verify_user_perm: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/hdfs: line 213: hadoop_validate_classname: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/hdfs: line 214: hadoop_exit_with_usage: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 263: hadoop_add_client_opts: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 270: hadoop_subcommand_opts: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 273: hadoop_generic_java_subcmd_handler: command not found
Same issues starting the datanodes, secondary namenodes, resourcemanager, and nodemanagers.
I have found a similar bug reference online: https://issues.apache.org/jira/browse/HDFS-12571.
Update
After some debugging, the root cause is bash "< <(command)" syntax not being accepted for some reason. The bash versions on the system (/bin/bash and /usr/local/bin/bash from Homebrew) both work properly.
Maybe you should modify HDFS_NAMENODE_USER、HDFS_DATANODE_USER and so on in hadoop-env.sh to current user instead of root! Then before run the sudo ./start-all.sh command, you may need to recreate hdfs namenode with hdfs namenode -format.
I have been following http://saphanatutorial.com/hadoop-installation-on-windows-7-using-cygwin/ tutorial for hadoop installation in windows7
Everything went fine, however I am facing issues while formatting the name node.
I am failing in executing format command bin/hadoop namenode -format
As soon as I hit this command i get following error:
$ bin/hadoop namenode -format
bin/hadoop: line 2: $'\r': command not found
bin/hadoop: line 17: $'\r': command not found
bin/hadoop: line 18: $'\r': command not found
bin/hadoop: line 56: $'\r': command not found
bin/hadoop: line 58: cd: $'bin\r': No such file or directory
bin/hadoop: line 59: $'\r': command not found
bin/hadoop: line 67: syntax error near unexpected token `$'in\r''
'in/hadoop: line 67: `case "`uname`" in
I followed solution mentioned in export JAVA_HOME with spaces in Cygwin and made required changes in hadoop-env.sh file and systems variable path in computer properties.
Still the issue is not resolved. Could you please let me know where am i going wrong.
PFB the hadoop-env.sh file configuration:
# Set Hadoop-specific environment variables here.
# The only required environment variable is JAVA_HOME. All others are
# optional. When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.
# The java implementation to use. Required.
export JAVA_HOME="/cygdrive/c/PROGRA~1/Java/jdk1.7.0_80"
# Extra Java CLASSPATH elements. Optional.
# export HADOOP_CLASSPATH=
# The maximum amount of heap to use, in MB. Default is 1000.
# export HADOOP_HEAPSIZE=2000
# Extra Java runtime options. Empty by default.
# export HADOOP_OPTS=-server
# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS"
# export HADOOP_TASKTRACKER_OPTS=
# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
# export HADOOP_CLIENT_OPTS
# Extra ssh options. Empty by default.
# export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HADOOP_CONF_DIR"
# Where log files are stored. $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
# File naming remote slave hosts. $HADOOP_HOME/conf/slaves by default.
# export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
# host:path where hadoop code should be rsync'd from. Unset by default.
# export HADOOP_MASTER=master:/home/$USER/src/hadoop
# Seconds to sleep between slave commands. Unset by default. This
# can be useful in large clusters, where, e.g., slave rsyncs can
# otherwise arrive faster than the master can service them.
# export HADOOP_SLAVE_SLEEP=0.1
# The directory where pid files are stored. /tmp by default.
# NOTE: this should be set to a directory that can only be written to by
# the users that are going to run the hadoop daemons. Otherwise there is
# the potential for a symlink attack.
# export HADOOP_PID_DIR=/var/hadoop/pids
# A string representing this instance of hadoop. $USER by default.
# export HADOOP_IDENT_STRING=$USER
# The scheduling priority for daemon processes. See 'man nice'.
# export HADOOP_NICENESS=10
I have the following script:
https://gist.github.com/pc-magas/37ec5e668c12017eb32aa308a0cb8f35
And on the execution I have these error messages:
./scripts/install2.sh: line 84: /home/pcmagas/Kwdikas/Docker/ellakcy/scripts/../start.sh: Permission denied
./scripts/install2.sh: line 87: /home/pcmagas/Kwdikas/Docker/ellakcy/scripts/../stop.sh: Permission denied
./scripts/install2.sh: line 89: ${STARTUP_SCRIPT_PATH}: ambiguous redirect
chmod: missing operand after 'u+x'
Try 'chmod --help' for more information.
Startup script generated
./scripts/install2.sh: line 94: ${STOP_SCRIPT_PATH}: ambiguous redirect
chmod: missing operand after 'u+x'
Try 'chmod --help' for more information.
The source of the errors are these lines of script:
touch "${SCRIPT_PATH}/start.sh"
STARTUP_SCRIPT_PATH= "${SCRIPT_PATH}/start.sh"
touch "${SCRIPT_PATH}/stop.sh"
STOP_SCRIPT_PATH= "${SCRIPT_PATH}/stop.sh"
What I want/try to do is my script by given some parameters to auto-generate scripts in order to start and stop some docker container services.
But somehow the plan backfires to me right now.
May I have some help please.
You need sudo to get permission to write the script files, and there were spaces between the variable name and value; there must not be any spaces around the = signs:
sudo touch "${SCRIPT_PATH}/start.sh"
STARTUP_SCRIPT_PATH="${SCRIPT_PATH}/start.sh" # Note no spaces ..PATH="${..
sudo touch "${SCRIPT_PATH}/stop.sh"
STOP_SCRIPT_PATH="${SCRIPT_PATH}/stop.sh" # Again here.
I'm trying to call a number of spiders from a shell script like so:
#!/bin/bash
source /home/pi/.bashrc
source /usr/local/bin/virtualenvwrapper.sh
workon email-scraper
cdvirtualenv
#Change to dir containing scrapy.cfg
cd ./sitecode/jobscraper
pwd
scrapy crawl site1
scrapy crawl site2
scrapy crawl site3
# Email me the jobs.csv file
python '/home/pi/Documents/jobscraper/scripts/jobs-mail.py'
# Delete the file so a new one is created on next scrape
# sudo rm /home/pi/Documents/jobscraper/csv/jobs.csv
The only part it executes properly is the python script at the end, which emails me an empty csv file as none of the scrapers have run. I get the following output when running the above script from BASH
(email-scraper)pi#raspberrypi ~/.virtualenvs/email-scraper/sitecode $ sudo sh runspiders.sh
ERROR: Environment 'email-scraper' does not exist. Create it with 'mkvirtualenv email-scraper'.
ERROR: no virtualenv active, or active virtualenv is missing
runspiders.sh: line 9: cd: ./sitecode/jobscraper: No such file or directory
/home/pi/.virtualenvs/email-scraper/sitecode
runspiders.sh: line 13: scrapy: command not found
runspiders.sh: line 14: scrapy: command not found
runspiders.sh: line 15: scrapy: command not found
runspiders.sh: line 16: scrapy: command not found
runspiders.sh: line 17: scrapy: command not found
runspiders.sh: line 18: scrapy: command not found
runspiders.sh: line 19: scrapy: command not found
runspiders.sh: line 20: scrapy: command not found
runspiders.sh: line 21: scrapy: command not found
runspiders.sh: line 22: scrapy: command not found
I'm very new to shell scripting. Can anyone illustrate how to make sure I activate the virtualenv and change into the correct dir before calling the first spider?
Edit In reply to #konsolebox
This is how I would run the scraper manually from the home dir:
First, source .bashrc as the raspberry pi doesn't do this automatically for some reason.
source .bashrc
This allows me access to virtualenvwrapper. I can then
pi#raspberrypi ~ $ workon email-scraper
(email-scraper)pi#raspberrypi ~ $ cdvirtualenv
which puts me in the virtualenv project directory /home/pi/.virtualenvs/email-scraper
I then do
cd sitecode/jobscraper
and ls -al puts me in the dir with access to scrapy.cfg, which I need in order to run the scraper.
drwxr-xr-x 3 pi pi 4096 Sep 9 19:40 .
drwxr-xr-x 5 pi pi 4096 Sep 8 19:41 ..
drwxr-xr-x 3 pi pi 4096 Sep 8 14:59 jobscraper
-rwxr-xr-x 1 pi pi 632 Sep 8 22:18 runspiders.sh
-rw-r--r-- 1 root root 12288 Sep 9 19:40 .runspiders.sh.swp
-rw-r--r-- 1 pi pi 381 Sep 7 23:34 scrapy.cfg
I can then do scrapy crawl site1 to run the scraper.
Perhaps you actually need to run the script in ~/.virtualenvs/email-scraper/. Do cd ~/.virtualenvs/email-scraper/ before running it.
When you're there run
sh sitecode/runspiders.sh
Or
sudo sh sitecode/runspiders.sh
#!/bin/bash
cd /home/pi
source ./bashrc
## source /usr/local/bin/virtualenvwrapper.sh ## You didn't run this.
workon email-scraper
cdvirtualenv
cd ./sitecode/jobscraper
scrapy crawl site1
Run as bash script.sh, not sudo sh script.sh.