Recently installed the VM at the below link and was attempting to run the command sudo hdfs dfs -mkdir /user/vagrant as the HDFS user. However, Hadoop is throwing the below error message at me.
https://atlas.hashicorp.com/puppetlabs/boxes/centos-6.6-64-nocm
-bash-4.1$ hdfs dfs -mkdir /user/vagrant
: command not founddoop-env.sh: line 15:
: command not founddoop-env.sh: line 16:
: command not founddoop-env.sh: line 21:
: command not founddoop-env.sh: line 26:
: command not founddoop-env.sh: line 29:
: command not founddoop-env.sh: line 32:
: command not founddoop-env.sh: line 35:
: command not founddoop-env.sh: line 38:
: command not founddoop-env.sh: line 46:
: command not founddoop-env.sh: line 49:
: command not founddoop-env.sh: line 52:
: command not founddoop-env.sh: line 55:
: command not founddoop-env.sh: line 58:
: command not founddoop-env.sh: line 61:
: command not founddoop-env.sh: line 66:
: command not founddoop-env.sh: line 69:
: command not founddoop-env.sh: line 72:
: command not founddoop-env.sh: line 75:
: command not founddoop-env.sh: line 77:
: command not founddoop-env.sh: line 78:
mkdir: Call From bigtop1.vagrant/127.0.0.1 to bigtop1.vagrant:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
I believe this means that none of the Hadoop services are running (even running hadoop -fs returns the command not found errors), but am unsure of how to resolve this issue. As well, I am not even seeing a /hadoop folder under /usr or /usr/local.
I was hoping to resolve this issue by using vagrant destroy -> vagrant up, but to no avail.
Noticed line endings were windows format. Destroyed VM, correctly configured git for unix line endings git config --global core.autocrlf false, and remade. Works perfectly.
Related
I'm having trouble running a VM's startup scripts for instance groups. Commands like touch, cat, chmod are not found.
ubuntu#instance-template-****-1:~$ sudo google_metadata_script_runner -d --script-type startup
startup-script: INFO Starting startup scripts.
startup-script: INFO Found startup-script-url in metadata.
startup-script: INFO Downloading url from https://storage.googleapis.com/bucket-*****/startup-image to /home/ubuntu/startup-scripts/script using authentication token.
startup-script: INFO startup-script-url: /home/ubuntu
startup-script: INFO startup-script-url: /home/ubuntu
startup-script: INFO startup-script-url: /home/ubuntu/startup-scripts/script: line 11: touch: command not found
startup-script: INFO startup-script-url: /home/ubuntu/startup-scripts/script: line 12: touch: command not found
startup-script: INFO startup-script-url: /home/ubuntu/startup-scripts/script: line 13: touch: command not found
startup-script: INFO startup-script-url: /home/ubuntu/startup-scripts/script: line 15: chmod: command not found
startup-script: INFO startup-script-url: /home/ubuntu/startup-scripts/script: line 16: chmod: command not found
startup-script: INFO startup-script-url: /home/ubuntu/startup-scripts/script: line 71: ssh-keyscan: command not found
startup-script: INFO startup-script-url: /home/ubuntu/startup-scripts/script: line 72: ssh-keygen: command not found
startup-script: INFO startup-script-url: /home/ubuntu/startup-scripts/script: line 74: cat: command not found
startup-script: INFO startup-script-url: /home/ubuntu/startup-scripts/script: line 76: git: command not found
startup-script: INFO startup-script-url: /home/ubuntu/startup-scripts/script: line 78: sh: command not found
startup-script: INFO startup-script-url: Return code 127.
startup-script: INFO Finished running startup scripts.
Script:
#!/bin/bash
PATH=/home/ubuntu
cd $PATH
touch $PATH/.ssh/id_rsa
touch $PATH/.ssh/id_rsa.pub
touch $PATH/.ssh/known_hosts
chmod 644 $PATH/.ssh/id_rsa.pub
chmod 600 $PATH/.ssh/id_rsa
echo $IDRSA > $PATH/.ssh/id_rsa
echo $IDRSA_PUB > $PATH/.ssh/id_rsa.pub
ssh-keyscan bitbucket.org >> bitbucketKey
ssh-keygen -lf bitbucketKey
cat bitbucketKey >> $PATH/.ssh/known_hosts
git clone git#bitbucket.org:repo/repo.git
sh ./repo/prepare
PROJECT_ID=$(curl "http://metadata.google.internal/computeMetadata/v1/project/numeric-project-id" -H "Metadata-Flavor: Google")
./repo/deploy/run --dev --project-id=$PROJECT_ID
When you set PATH=/home/ubuntu, it overwrites the value of PATH environment variable that you're supposed to get in the shell and it forces the subsequent commands to end up in "command not found" error, because the appropriate executable files for running those commands are not available on new path /home/ubuntu that you just set. You can workaround this by renaming that variable to any name other than PATH and all of its occurrences in your script. You can use the same variable name in lower case too. (like path=home/ubuntu).
The PATH environment variable is a colon-delimited list of directories that your shell searches through when you enter a command. Program files (executables) are kept in many different places on the Unix system. Your path tells the Unix shell where to look on the system when you request a particular program.
You modified the PATH environment in your script:
PATH=/home/ubuntu
However, PATH is a Shell Variables of bash:
PATH The search path for commands. It is a colon-separated list of directories in which the shell looks for commands (see
COMMAND EXECUTION below). A zero-length (null) directory name in the value of PATH indicates the current directory.
A null directory name may appear as two adjacent colons, or as an initial or trailing colon. The default path is
system-dependent, and is set by the administrator who installs bash. A common value is ``/usr/local/bin:
/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin''.
Therefore almost all commands cannot be found. (Unless they are under /home/ubuntu)
If you just want to set a variable as a user folder to operate with the script, please change the variable name to like MY_HOME.
#!/bin/bash
MY_HOME=/home/ubuntu
cd $MY_HOME
touch $MY_HOME/.ssh/id_rsa
touch $MY_HOME/.ssh/id_rsa.pub
touch $MY_HOME/.ssh/known_hosts
...
I've configured Hadoop 3.1.1 on my MacPro running OSX 10.14.2, and I'm getting the following error when I run start-all.sh
$ sudo /usr/local/Cellar/hadoop/3.1.1/sbin/start-all.sh
Starting namenodes on [localhost]
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-functions.sh: line 398: syntax error near unexpected token `<'
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-functions.sh: line 398: ` done < <(for text in "${input[#]}"; do'
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 70: hadoop_deprecate_envvar: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 87: hadoop_bootstrap: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 104: hadoop_parse_args: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 105: shift: : numeric argument required
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 244: hadoop_need_reexec: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 252: hadoop_verify_user_perm: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/hdfs: line 213: hadoop_validate_classname: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/hdfs: line 214: hadoop_exit_with_usage: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 263: hadoop_add_client_opts: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 270: hadoop_subcommand_opts: command not found
/usr/local/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 273: hadoop_generic_java_subcmd_handler: command not found
Same issues starting the datanodes, secondary namenodes, resourcemanager, and nodemanagers.
I have found a similar bug reference online: https://issues.apache.org/jira/browse/HDFS-12571.
Update
After some debugging, the root cause is bash "< <(command)" syntax not being accepted for some reason. The bash versions on the system (/bin/bash and /usr/local/bin/bash from Homebrew) both work properly.
Maybe you should modify HDFS_NAMENODE_USER、HDFS_DATANODE_USER and so on in hadoop-env.sh to current user instead of root! Then before run the sudo ./start-all.sh command, you may need to recreate hdfs namenode with hdfs namenode -format.
I am trying to start hadoop by running ./start-dfs.sh, but i am getting some syntax errors. Could anybody please help?
Gurupads-MacBook-Air:sbin guru$ sudo ./start-dfs.sh
Starting namenodes on [localhost]
/Users/guru/homebrew/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-functions.sh: line 398: syntax error near unexpected token `<'
/Users/guru/homebrew/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-functions.sh: line 398: ` done < <(for text in "${input[#]}"; do'
/Users/guru/homebrew/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 70: hadoop_deprecate_envvar: command not found
/Users/guru/homebrew/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 87: hadoop_bootstrap: command not found
/Users/guru/homebrew/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 104: hadoop_parse_args: command not found
/Users/guru/homebrew/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 105: shift: : numeric argument required
/Users/guru/homebrew/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 244: hadoop_need_reexec: command not found
/Users/guru/homebrew/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 252: hadoop_verify_user_perm: command not found
/Users/guru/homebrew/Cellar/hadoop/3.1.1/libexec/bin/hdfs: line 213: hadoop_validate_classname: command not found
/Users/guru/homebrew/Cellar/hadoop/3.1.1/libexec/bin/hdfs: line 214: hadoop_exit_with_usage: command not found
/Users/guru/homebrew/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 263: hadoop_add_client_opts: command not found
/Users/guru/homebrew/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 270: hadoop_subcommand_opts: command not found
/Users/guru/homebrew/Cellar/hadoop/3.1.1/libexec/bin/../libexec/hadoop-config.sh: line 273: hadoop_generic_java_subcmd_handler: command not found
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [Gurupads-MacBook-Air.local]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
2018-09-18 21:51:24,380 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
The error states that, you are running hdfs datanode as root user and there is no HDFS_DATANODE_USER is defined.
Solution:
Edit /Users/guru/homebrew/Cellar/hadoop/3.1.1/etc/hadoop/hadoop-env.sh file by adding below lines at the end of file:
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
If you are using Yarn then you can add below lines in same file:
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
Restart hadoop using start-dfs.sh script.
In my Mac, I have installed expect:
$ brew install expect
Updating Homebrew...
Warning: expect 5.45.3 is already installed
I write a bash script on my Mac:
#!/usr/bin/expect -f (there I tried #!/bin/bash too)
set timeout 60
set host 103.212.12.76
set name root
set password my_password
spawn ssh $host -l $name
expect {
"(yes/no)?" {
send "yes\n"
expect "password:"
send "$password\n"
}
"password:" {
send "$password\n"
}
}
expect "#"
# test whether login to host
send "uname\n"
expect "Linux"
send_user "Now you can do some operation on this terminal\n"
interact
But when I execute my script on my Mac, I get error:
$ ./test01.sh
./test01.sh: line 11: spawn: command not found
couldn't read file "{": no such file or directory
./test01.sh: line 14: (yes/no)?: No such file or directory
./test01.sh: line 15: send: command not found
couldn't read file "password:": no such file or directory
./test01.sh: line 17: send: command not found
./test01.sh: line 18: syntax error near unexpected token `}'
./test01.sh: line 18: ` }'
It report spawn: command not found, but I installed the expect in my Mac.
I also tried change the test01.sh name to test01.exp, not work too.
./test01.exp
./test01.exp: line 11: spawn: command not found
couldn't read file "{": no such file or directory
./test01.exp: line 14: (yes/no)?: No such file or directory
./test01.exp: line 15: send: command not found
couldn't read file "password:": no such file or directory
./test01.exp: line 17: send: command not found
./test01.exp: line 18: syntax error near unexpected token `}'
./test01.exp: line 18: ` }'
Expect is not Bash. You need #!/usr/bin/expect or expect /your/script.exp.
While trying to install canopy I got following error
ubuntu#hari-desktop:~/Downloads$ sh canopy-2.1.3.rh6-x86_64-cp27.sh
canopy-2.1.3.rh6-x86_64-cp27.sh: 32: canopy-2.1.3.rh6-x86_64-cp27.sh: 0: not found
canopy-2.1.3.rh6-x86_64-cp27.sh: 46: canopy-2.1.3.rh6-x86_64-cp27.sh: 0: not found
canopy-2.1.3.rh6-x86_64-cp27.sh: 154: canopy-2.1.3.rh6-x86_64-cp27.sh: Syntax error: word unexpected (expecting ")")
Solution: run using bash: bash canopy-2.1.3.rh6-x86_64-cp27.sh