I am trying to setup hadoop from the blog : http://www.bogotobogo.com/Hadoop/BigData_hadoop_Install_on_ubuntu_single_node_cluster.php
I can do :
hduser#laptop:~$ wget http://mirrors.sonic.net/apache/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz
hduser#laptop:~$ tar xvzf hadoop-2.6.0.tar.gz
after that when I try to :
hduser#laptop:~/hadoop-2.6.0$ sudo mv * /usr/local/hadoop
and it gives an error like No Such file or directory.
What is the main problem here?
Thank you
Once the hadoop is untarred , check the actual path where it has done this operation and verify the path you are on is correct .
hduser#laptop:~$ tar xvzf hadoop-2.6.0.tar.gz .
hduser#laptop:~$ ls -lrt
And see if it is here
You should probably do
sudo mv ~/hadoop-2.6.0 /usr/local/hadoop
Give a try with,
hduser#laptop:~$ tar xvzf hadoop-2.6.0.tar.gz .
hduser#laptop:~$ sudo mkdir -p /usr/local/hadoop
hduser#laptop:~$ mv hadoop-2.6.0 /usr/local/hadoop
Here, I found a very simple tutorial to install apache hadoop single node cluster setup
Related
I automated my apache http logs to be stored in the tmp directory, which is running the script correctly until moving the file to the tmp directory. Now I have added few more lines of script to move those .tar files to the S3 bucket. When I perform this command manually they are getting moved to the s3 bucket but I do not want to do it everyday since its a daily job and I would want to automate them.
The .tar file is present in the /tmp/ directory when I manually go there and look but the awscl fails to locate to it.
The error I am getting is : the user provided path does not exist when I run the script.
the lines of code I added were these,
sudo apt update -y
sudo apt install apache2
sudo ufw allow 'Apache'
sudo systemctl start apache2
myname="abcd"
sudo tar -cvf $myname-httpd-logs-date +'%d%m%Y-%H%M%S'.tar /var/log/apache2/*.log
sudo mv *.tar /tmp/
sudo apt install awscli -y
s3_bucket="s3_test"
aws s3 \
cp /tmp/$myname-httpd-logs-$(date +'%d%m%Y-%H%M%S').tar \
s3://$s3_bucket/$myname-httpd-logs-date +'%d%m%Y-%H%M%S'.tar
Can anyone help me out in figuring out why this error has occurred and how to fix it.
Error
You appear to be calling datetime multiple times to construct different filenames. Rather than calling it multiple times, if you call it once, you will ensure that each time you refer to the tar filename, it uses the same name:
myname="abcd"
target_name=$myname-httpd-logs-`date +'%d%m%Y-%H%M%S'.tar`
sudo tar -cvf $target_name /var/log/apache2/*.log
sudo mv $target_name /tmp/
s3_bucket="s3_test"
aws s3 \
cp /tmp/$target_name \
s3://$s3_bucket/$target_name
There is not needed ' in your command in $(date +'%d%m%Y-%H%M%S). It should be:
aws s3 \
cp /tmp/$myname-httpd-logs-$(date +%d%m%Y-%H%M%S).tar \
s3://$s3_bucket/$myname-httpd-logs-${timestamp}.tar
And if you want the unneeded quote
aws s3 \
cp /tmp/$myname-httpd-logs-$(date +'%d%m%Y-%H%M%S').tar \
s3://$s3_bucket/$myname-httpd-logs-${timestamp}.tar
be sure to close it (the syntax color gives you a clue).
I'm trying to use a program that uses Linux versions of other programs.
I ran this in bash:
c3dpath=$( command -v c3d )
if [[ -z "${c3dpath}" ]]; then
echo "Command c3d was not found. Downloading and installing software to ${SCRIPTPATH}/depends/c3d. Path will be added to PATH environment variable."
mkdir -p "${SCRIPTPATH}/depends/c3d/"
wget https://downloads.sourceforge.net/project/c3d/c3d/Nightly/c3d-nightly-Linux-x86_64.tar.gz && \
tar -xzvf c3d-nightly-Linux-x86_64.tar.gz && mv c3d-1.1.0-Linux-x86_64/* "${SCRIPTPATH}/depends/c3d/" && \
rm c3d-nightly-Linux-x86_64.tar.gz
export PATH="${SCRIPTPATH}/depends/c3d/c3d-1.1.0-Linux-x86_64/bin/:$PATH"
fi
But it downloads the linux version of the program files into my directory -- yielding the error:
{..omitted}/HippMapp3r/depends/c3d/bin/c3d: cannot execute binary file
Return Code: 126
I understand in order for this to work on my computer, I need to use the version for Mac OS, however, I stuck on how one might go about "unarchiving" the files contained within a .dmg file so I can access c3d in bin.
I edited it to this, but I understand I cannot use the tar -xzvf command in this context -- is there an equivalent to tar -xzvf to "unpack" dmg files?
c3dpath=$( command -v c3d )
if [[ -z "${c3dpath}" ]]; then
echo "Command c3d was not found. Downloading and installing software to ${SCRIPTPATH}/depends/c3d. Path will be added to PATH environment variable."
mkdir -p "${SCRIPTPATH}/depends/c3d/"
wget https://downloads.sourceforge.net/project/c3d/c3d/Nightly/c3d-nightly-MacOS-x86_64.dmg && \
tar -xzvf c3d-nightly-MacOS-x86_64.dmg * "${SCRIPTPATH}/depends/c3d/" && \
rm c3d-nightly-MacOS-x86_64.dmg
export PATH="${SCRIPTPATH}/depends/c3d/c3d-nightly-MacOS-x86_64/bin/:$PATH"
fi
Try using 7-zip. You can install it on a Mac with homebrew using:
brew install p7zip
Then run it with:
7z
Please see comment by #StefanSchmidt about a possible alternative package.
First you have to mount the DMG file.
command: hdiutil mount test.dmg
I need a specific version of protocol buffer which is 3.14.0 on apt its not available like this
sudo apt install -y protobuf-compiler = 3.14.0
and If I follow these steps by binary :
$ PB_REL="https://github.com/protocolbuffers/protobuf/releases"
$ curl -LO $PB_REL/download/v3.14.0/protoc-3.14.0-linux-x86_64.zip
then
unzip protoc-3.14.0-linux-x86_64.zip -d $HOME/.local
then
$ export PATH="$PATH:$HOME/.local/bin"
from this source
and it's not getting install when I do protoc --version I am getting this error
bash: /usr/bin/protoc: No such file or directory
I think Because /usr/bin/protoc doesn't exist. When you unzipped you got folder named protoc-3.14.0.
which is /usr/bin/protoc-3.14.0
try doing this
PROTOC_ZIP=protoc-3.14.0-linux-x86_64.zip
curl -OL https://github.com/protocolbuffers/protobuf/releases/download/v3.14.0/$PROTOC_ZIP
sudo unzip -o $PROTOC_ZIP -d /usr/local bin/protoc
ref: http://google.github.io/proto-lens/installing-protoc.html
Please check out the following steps to Install protoc on Ubundu, Run the commands 1 to 4 on terminal. Then 5 and 6 to finalize the settings.
Get the latest version tag of protoc release and assign it to variable:
1.
PROTOC_VERSION=$(curl -s "https://api.github.com/repos/protocolbuffers/protobuf/releases/latest" | grep -Po '"tag_name": "v\K[0-9.]+')
curl -Lo protoc.zip "https://github.com/protocolbuffers/protobuf/releases/latest/download/protoc-${PROTOC_VERSION}-linux-x86_64.zip"
Run the following command to extract executable file from a ZIP archive:
sudo unzip -q protoc.zip bin/protoc -d /usr/local
Set execute permission:
sudo chmod a+x /usr/local/bin/protoc
Now protoc command is available for all users as a system-wide command.
We can now check protoc version:
protoc --version
Remove unnecessary ZIP archive:
rm -rf protoc.zip
I am using chef to write a recipe that installs mysql connector, extracts it and moves the .jar to the /lib folder.
bash "install_mysql-connector" do
user "root"
cwd "/opt/tomcat/lib/"
code <<-EOH
wget http://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.29.tar.gz
tar -zxvf mysql-connector-java-5.1.29.tar.gz
EOH
end
bash "setting_mysql-connector" do
user "root"
cwd "/opt/tomcat/lib/mysql-connector-java-5.1.29"
code <<-EOH
mv mysql-connector-java-5.1.29-bin.jar /opt/tomcat/lib/
rm -rf mysql-connector-java-5.1.29.tar.gz
EOH
end
Is there a way of combining these two so they run under the same bash?
I also tried extracting JUST the mysql-connector-java-5.1.29-bin.jar from the tar.gz file using
tar -zxvf mysql-connector-java-5.1.29.tar.gz -C /opt/tomcat/lib mysql-connector-java-5.1.29-bin.jar
Is something missing from this?
How about this? Extract the jar file out of .tar.gz and save under /opt/tomcat/lib directly?
tar -zxvf mysql-connector-java-5.1.29.tar.gz mysql-connector-java-5.1.29-bin.jar -O > /opt/tomcat/lib/mysql-connector-java-5.1.29-bin.jar
Anyone know why this line in these updating instructions don't work.
https://github.com/discourse/discourse/blob/master/docs/INSTALL-ubuntu.md#updating-discourse
DATESTAMP=$(TZ=UTC date +%F-%T)
pg_dump --no-owner --clean discourse_prod | gzip -c > ~/discourse-db-$DATESTAMP.sql.gz
tar cfz ~/discourse-dir-$DATESTAMP.tar.gz -C /var/www discourse
The first 2 lines work and I can see the .gz file being created in my home directory.
But when I run the third line tar cfz ~/discourse-dir-$DATESTAMP.tar.gz -C /mydiscourse directory it fails and give me an error:
tar: no files or directories specified
I even changed it to
tar cfz ~/discourse-db-$DATESTAMP.tar.gz -C /mydiscourse directory
because db is the name of the file. not dir but this still is giving me an error. Does anyone know what this could be?