i have a path on HDFS as
old_file_path="/user/projects/dir1/dir2/dir3/$variable1/$variable2"
where variable1 and variable2 are name of directories stored inside variable.
I want to create a file inside variable2 directory having same name as value stored in variable newMemCount
command that I am using is
hadoop fs -touch $old_file_path/$newMemCount
error that I am getting is
-touch: Unknown command
The hadoop fs command does not support any touch FS shell command, hence your error. It does however support a touchz command that behaves the same way *nix touch would if no file was present (it does however raise an error if a non-empty file exists, while *nix touch would just update its modification time).
Therefore, I think you're looking for the following :
hadoop fs -touchz $old_file_path/$newMemCount
Looking at the documentation seems it should be touchz
https://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/FileSystemShell.html#touchz
Related
This is a very basic question concerning the output files generated from running the Grep utility inside a HDFS directory. Essentially, I've included the grep command inside a simple shell script, which is supposed to search this directory for a given string - which is a parameter to the script. The contents of the script are as follows:
#!/bin/bash
set - e
cd $HADOOP_HOME
bin/hadoop org.apache.hadoop.examples.Grep
"hdfs://localhost:9000/user/hduser" "hdfs://localhost:9000/user/hduser/out" $1
bin/hadoop fs -get "hdfs://localhost:9000/user/hduser/out/*" "/opt/data/out/"
bin/hadoop fs -rm -r "hdfs://localhost:9000/user/hduser/out"
The results sent to the hdfs out directory are copied across to a local directory in the second last line. I've deliberately placed two files in this hdfs directory, only one of which contains multiple instances of the string I'm searching for. What ends up in my /opt/data/out directory are the following 2 files.
_SUCCESS
part-r-00000
The jobs look like they ran successfully, however the only content i'm seeing between both files, is in the "part-r-0000" file, and it's literally the following.
29472 e
I suppose I was naively hoping to see the filename where the string was located, and perhaps a count of the number of times it occurred.
My question is, how and where are these values typically returned from the hadoop grep command? I've looked through the console out while the map reduce jobs where running, and there's no reference to the file name where the search string is stored. Any pointers as to how I can access this information would be appreciated, as I'm unsure how to interpret "29472 e".
I understand like...
You have some jobs' output in HDFS, which you copy to your local.
You are then trying to get the count of a string in the files.
In that case, add the code after the below line
bin/hadoop fs -get "hdfs://localhost:9000/user/hduser/out/*" "/opt/data/out/"
grep -c $1 /opt/data/out/*
This command will do what is expected.
It will give the file name and also the count of strings found in the file.
Does hadoop filesystem shell moving of empty directory?
Assume that I have a below directory which is empty.
hadoop fs -mv /user/abc/* /user/xyz/*
When I am executing the above command , it is giving me the error
'/user/abc/*' does not exists.
However, If I put some data inside /user/abc/* , it is getting executed successfully.
Does anyone know how to handle for empty directory?
Is there any alternative to execute above command without giving error?
hadoop fs -mv /user/abc/* /user/xyz
The destination file doesn't need to add /*
I thinks you want to rename the file.
you also can use this ->
hadoop fs -mv /user/abc /user/xyz
Because you xyz file is empty，so you don't got error.
but if you xyz file has many file,you will get error as well.
This answer should be correct I believe.
hadoop fs -mv /user/abc /user/xyz
'*' is a wild card. So it's looking for any file inside the folder. When nothing found, it returns the error.
As per the command,
When you move a file, all links to otherfiles remain intact, except when youmove it to a different file system.
Created a folder [LOAN_DATA] with below command
hadoop fs -mkdir hdfs://masterNode:8020/tmp/hadoop-hadoop/dfs/LOAN_DATA
Now using the web UI when I list the contents of directory /tmp/hadoop-hadoop/dfs, it shows LOAN_DATA.
But when I want to store some Data from a TXT file to the LOAN_DATA directory using put or copyFromLocal I get
put: Unknown command
Command used:
hadoop fs –put '/home/hadoop/my_work/Acquisition_2012Q1.txt' hdfs://masterNode:8020/tmp/hadoop-hadoop/dfs/LOAN_DATA
How to resolve this issue?
This issue may occur when you copy-paste a command and use it. It is because of the change in font (or character set) used in the document from where it was copied.
For example:
If you copy/paste and execute the command -
hdfs dfs -put workflow.xml /testfile/workflow.xml
You may get-
–put: Unknown command
OR
–p-t: Unknown command
This happens because the copy is done from a UTF-8 file and the - or u (or any of the characters) copied may be of different character set.
So just type the command on the terminal (don't copy/paste) and you should be fine.
Alternatively, if you are running a shell script which was copied from
some other editor then run a dos2unix on the script before running it
on the Linux terminal.
Eg: dos2unix <shell_script.sh>
Tried your command and "it appears", there is a typo error in the above command 'hadoop fs –put ....'.
Instead of '–put', use '-put' or '-copyFromLocal'. Problem is with '–' but the correct character should be '-'. As such, the error is obvious :-)
Here is my example (using a get command instead of put):
$ hadoop fs –get /tmp/hadoop-data/output/* /tmp/hadoop-data/output/
–get: Unknown command
$ hadoop fs -get /tmp/hadoop-data/output/* /tmp/hadoop-data/output/
get: `/tmp/hadoop-data/output/part-r-00000': File exists
Anand's answer is, of course, correct. But it might not have been a typo but rather a subtle trap. Often when people are learning new technology, they copy and paste commands from websites and blogs. Often, what was originally entered as a dash will be copied as a hyphen. Hyphens differ from dashes only in that they are a tad longer, so the mistake is hard to spot, but since they are a completely different character the command is wrong, that is, "not found".
I am invoking a bash shell script using oozie editor in Hue.
I used the shell action in the workflow and tried below different options in shell command:
Uploaded the shell script using 'choose a file'
Gave local directory path where shell script is present
Gave HDFS path where shell script is present
But all these options gave following error:
Cannot run program "sec_test_oozie.sh" (in directory "/data/hadoop/yarn/local/usercache/user/appcache/application_1399542362142_0086/container_1399542362142_0086_01_000002"): java.io.IOException: error=2, No such file or directory
How should I give the shell script execution command?
Where the shell script file should be residing?
You need add file "sec_test_oozie.sh" in oozie shell step. In add files
I think you are creating the file from windows machine which is adding extra line break characters.You need to convert the shell script file to Unix format.I also faced the same issue.Then I created the file from a Linux system and it started working.The error is misguiding.
I want to extend the #SergioRG answer. Oozie, at least with Cloudera's Hue interface is very counterintuitive.
To run a script file, three conditions should be met:
the file is on the HDFS file system, in a folder accessible by Oozie
the file should be indicated in the shell command field
the file should be added with any other dependent file in the "Files+" part of the task card.
I wonder why they didn't add by default the script file you are calling.
Edit: please also check in advanced options (the gear in the left upper corner) if you need to set the path variable (eg. PATH=/usr/local/bin:/usr/bin).
Did you edit sec_test_oozie.sh with the Hue File Browser? Depending on your Hue version it might have corrupted it: hue-list
I encountered the same problem, and the problem was that the script echoed some irrelevant line while the workflow tried to parse it as a property line. Oozie gave a very irrelevant error message of java.io.IOException: error=2, No such file or directory which only added confusion.
You will need to use <file> to add your script.
If you used <capture-output/> then you must make sure that your script prints only "key=value" lines, like java properties, otherwise you will get the error you see java.io.IOException: error=2, No such file or directory with some path pointing to .../yarn/local/usercache/...
We had this issue on a test script, basically if you use an editor that adds wierd characters or line ending to the file, it'll throw this error because the script cannot be used in the container.
Try using nano file.sh to see if any strange characters appear. Then push it back to hdfs with hdfs dfs -put file.sh /path/you/need
Removing the #!/bin/bash from my shell script helped me
"No such a file or directory" oozie cannot locate the file. Please check the AddPath setting in the command.
In the edit node seciton, get the oozie application hdfs path.
Upload the shell script in hdfs oozie application path.
In the oozie edit node step, Shell command - specify the shell script name which is uploaded.
Below that there would be option to AddPath, then add files, add the shell script which was uploaded in the hdfs path.
I have created a folder to drop the result file from a Pig process using the Store command. It works the first time, but the second time it compains that the folder already exists. What is the best practice for this situiation? Documentation is sparse on this topic.
My next step will be to rename the folder to the original file name, to reduce the impact of this. Any thoughts?
You can execute fs commands from within Pig, and should be able to delete the directory by issuing a fs -rmr command before running the STORE command:
fs -rmr dir
STORE A into 'dir' using PigStorage();
The only subtly is the fs command doesn't expect quotes around the directory name, whereas the store command does expect quotes around the directory name.