A fast method for inspecting files on HDFS is to use tail:
~$ hadoop fs -tail /path/to/file
This displays the last kilobyte of data in the file, which is extremely helpful. However, the opposite command head does not appear to be part of the shell command collections. I find this very surprising.
My hypothesis is that since HDFS is built for very fast streaming reads on very large files, there is some access-oriented issue that affects head. This makes me hesitant to do things to access the head. Does anyone have an answer?
I would say it's more to do with efficiency - a head can easily be replicated by piping the output of a hadoop fs -cat through the linux head command.
hadoop fs -cat /path/to/file | head
This is efficient as head will close out the underlying stream after the desired number of lines have been output
Using tail in this manner would be considerably less efficient - as you'd have to stream over the entire file (all HDFS blocks) to find the final x number of lines.
hadoop fs -cat /path/to/file | tail
The hadoop fs -tail command as you note works on the last kilobyte - hadoop can efficiently find the last block and skip to the position of the final kilobyte, then stream the output. Piping via tail can't easily do this.
Starting with version 3.1.0 we now have it:
Usage: hadoop fs -head URI
Displays first kilobyte of the file to stdout.
See here.
hdfs -dfs /path | head
is a good way to solve the problem.
you can try the folowing command
hadoop fs -cat /path | head -n
where -n can be replace with number of records to view
In Hadoop v2:
hdfs dfs -cat /file/path|head
In Hadoop v1 and v3:
hadoop fs -cat /file/path|head
Related
I have a file in HDFS - /user//SimpleDir/SimpleFile.txt and I'm trying to use Grep command and search for "MapReduce" in that file and print the results to a different file (simpleoutput.txt) in the same directory.
Any help is much appreciated!
You can try the following command
hadoop fs -cat /user/SimpleDir/SimpleFile.txt | grep -i Mapreduce > /user/SimpleDir/simpleoutput.txt
How can I decompress and view few lines of a compressed file in hdfs.
The below command displays the last few lines of the compressed data
hadoop fs -tail /myfolder/part-r-00024.gz
Is there a way I can use the -text command and pipe the output to tail command? I tried this but this doesn't work.
hadoop fs -text /myfolder/part-r-00024.gz > hadoop fs -tail /myfolder/
The following will show you the specified number of lines without decompressing the whole file:
hadoop fs -cat /hdfs_location/part-00000.gz | zcat | head -n 20
The following will page the file, also without first decompressing the whole of it:
hadoop fs -cat /hdfs_location/part-00000.gz | zmore
Try the following, should work as long as your file isn't too big (since the whole thing will be decompressed):
hadoop fs -text /myfolder/part-r-00024.gz | tail
I ended up writing a pig script.
A = LOAD '/myfolder/part-r-00024.gz' USING PigStorage('\t');
B = LIMIT A 10;
DUMP B;
Use gunzip to view the compressed file contents:
hdfs dfs -cat /path/filename.gz | gunzip
I have one big file in hdfs bigfile.txt. I want to copy the first 100 lines of it into a new file on hdfs. I tried the following command:
hadoop fs -cat /user/billk/bigfile.txt |head -100 /home/billk/sample.txt
It gave me a "cat: unable to write output stream" error. I am on hadoop 1.
Are there other ways to do this? (note: copying 1st 100 line to local or another file on hdfs is OK)
Like this -
hadoop fs -cat /user/billk/bigfile.txt | head -100 | hadoop -put - /home/billk/sample.txt
I believe the "cat: unable to write output stream" is just because head closed the stream after it read its limit. see this answer about head for hdfs - https://stackoverflow.com/a/19779388/3438870
I am having a 2 GB data in my HDFS.
Is it possible to get that data randomly.
Like we do in the Unix command line
cat iris2.csv |head -n 50
Native head
hadoop fs -cat /your/file | head
is efficient here, as cat will close the stream as soon as head will finish reading all the lines.
To get the tail there is a special effective command in hadoop:
hadoop fs -tail /your/file
Unfortunately it returns last kilobyte of the data, not a given number of lines.
You can use head command in Hadoop too! Syntax would be
hdfs dfs -cat <hdfs_filename> | head -n 3
This will print only three lines from the file.
The head and tail commands on Linux display the first 10 and last 10 lines respectively. But, the output of these two commands is not randomly sampled, they are in the same order as in the file itself.
The Linux shuffle - shuf command helps us generate random permutations of input lines & using this in conjunction with the Hadoop commands would be helpful, like so:
$ hadoop fs -cat <file_path_on_hdfs> | shuf -n <N>
Therefore, in this case if iris2.csv is a file on HDFS and you wanted 50 lines randomly sampled from the dataset:
$ hadoop fs -cat /file_path_on_hdfs/iris2.csv | shuf -n 50
Note: The Linux sort command could also be used, but the shuf command is faster and randomly samples data better.
hdfs dfs -cat yourFile | shuf -n <number_of_line>
Will do the trick for you.Though its not available on mac os. You can get installed GNU coreutils.
My suggestion would be to load that data into Hive table, then you can do something like this:
SELECT column1, column2 FROM (
SELECT iris2.column1, iris2.column2, rand() AS r
FROM iris2
ORDER BY r
) t
LIMIT 50;
EDIT:
This is simpler version of that query:
SELECT iris2.column1, iris2.column2
FROM iris2
ORDER BY rand()
LIMIT 50;
Write this command
sudo -u hdfs hdfs dfs -cat "path of csv file" |head -n 50
50 is number of lines(this can be customize by the user based on the requirements)
Working code:
hadoop fs -cat /tmp/a/b/20200630.xls | head -n 10
hadoop fs -cat /tmp/a/b/20200630.xls | tail -3
I was using tail and cat for an avro file on HDFS cluster, but the result was not getting printed in correct encoding. I tried this and worked well for me.
hdfs dfs -text hdfs://<path_of_directory>/part-m-00000.avro | head -n 1
Change 1 to higher integer to print more samples from avro file.
hadoop fs -cat /user/hive/warehouse/vamshi_customers/* |tail
I think the head part is working as per the answer posted by #Viacheslav Rodionov works fine but for the tail part the one that I posted is working good.
How can I rename all files in a hdfs directory to have a .lzo extension? .lzo.index files should not be renamed.
For example, this directory listing:
file0.lzo file0.lzo.index file0.lzo_copy_1
could be renamed to:
file0.lzo file0.lzo.index file0.lzo_copy_1.lzo
These files are lzo compressed, and I need them to have the .lzo extension to be recognized by hadoop.
If you don't want to write Java Code for this - I think using the command line HDFS API is your best bet:
mv in Hadoop
hadoop fs -mv URI [URI …] <dest>
You can get the paths using a small one liner:
% hadoop fs -ls /user/foo/bar | awk '!/^d/ {print $8}'
/user/foo/bar/blacklist
/user/foo/bar/books-eng
...
the awk will remove directories from output..now you can put these files into a variable:
% files=$(hadoop fs -ls /user/foo/bar | awk '!/^d/ {print $8}')
and rename each file..
% for f in $files; do hadoop fs -mv $f $f.lzo; done
you can also use awk to filter the files for other criteria. This should remove files that match the regex nolzo. However it's untested. But this way you can write flexible filters.
% files=$(hadoop fs -ls /user/foo/bar | awk '!/^d|nolzo/ {print $8}' )
test if it works with replacing the hadoop command with echo:
$ for f in $files; do echo $f $f.lzo; done
Edit: Updated examples to use awk instead of sed for more reliable output.
The "right" way to do it is probably using the HDFS Java API .. However using the shell is probably faster and more flexible for most jobs.
When I had to rename many files I was searching for an efficient solution and stumbled over this question and thi-duong-nguyen's remark that renaming many files is very slow. I implemented a Java solution for batch rename operations which I can highly recommend, since it is orders of magnitude faster. The basic idea is to use org.apache.hadoop.fs.FileSystem's rename() method:
Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://master:8020");
FileSystem dfs = FileSystem.get(conf);
dfs.rename(from, to);
where from and to are org.apache.hadoop.fs.Path objects. The easiest way is to create a list of files to be renamed (including their new name) and feed this list to the Java program.
I have published the complete implementation which reads such a mapping from STDIN. It renamed 100 files in less than four seconds (the same time was required to rename 7000 files!) while the hdfs dfs -mv based approach described before requires 4 minutes to rename 100 files.
We created an utility to do bulk renaming of files in HDFS: https://github.com/tenaris/hdfs-rename. The tool is limited, but if you want you can contribute to improve it with recursive, awk regex syntax and so on.