I am having a 2 GB data in my HDFS.
Is it possible to get that data randomly.
Like we do in the Unix command line
cat iris2.csv |head -n 50
Native head
hadoop fs -cat /your/file | head
is efficient here, as cat will close the stream as soon as head will finish reading all the lines.
To get the tail there is a special effective command in hadoop:
hadoop fs -tail /your/file
Unfortunately it returns last kilobyte of the data, not a given number of lines.
You can use head command in Hadoop too! Syntax would be
hdfs dfs -cat <hdfs_filename> | head -n 3
This will print only three lines from the file.
The head and tail commands on Linux display the first 10 and last 10 lines respectively. But, the output of these two commands is not randomly sampled, they are in the same order as in the file itself.
The Linux shuffle - shuf command helps us generate random permutations of input lines & using this in conjunction with the Hadoop commands would be helpful, like so:
$ hadoop fs -cat <file_path_on_hdfs> | shuf -n <N>
Therefore, in this case if iris2.csv is a file on HDFS and you wanted 50 lines randomly sampled from the dataset:
$ hadoop fs -cat /file_path_on_hdfs/iris2.csv | shuf -n 50
Note: The Linux sort command could also be used, but the shuf command is faster and randomly samples data better.
hdfs dfs -cat yourFile | shuf -n <number_of_line>
Will do the trick for you.Though its not available on mac os. You can get installed GNU coreutils.
My suggestion would be to load that data into Hive table, then you can do something like this:
SELECT column1, column2 FROM (
SELECT iris2.column1, iris2.column2, rand() AS r
FROM iris2
ORDER BY r
) t
LIMIT 50;
EDIT:
This is simpler version of that query:
SELECT iris2.column1, iris2.column2
FROM iris2
ORDER BY rand()
LIMIT 50;
Write this command
sudo -u hdfs hdfs dfs -cat "path of csv file" |head -n 50
50 is number of lines(this can be customize by the user based on the requirements)
Working code:
hadoop fs -cat /tmp/a/b/20200630.xls | head -n 10
hadoop fs -cat /tmp/a/b/20200630.xls | tail -3
I was using tail and cat for an avro file on HDFS cluster, but the result was not getting printed in correct encoding. I tried this and worked well for me.
hdfs dfs -text hdfs://<path_of_directory>/part-m-00000.avro | head -n 1
Change 1 to higher integer to print more samples from avro file.
hadoop fs -cat /user/hive/warehouse/vamshi_customers/* |tail
I think the head part is working as per the answer posted by #Viacheslav Rodionov works fine but for the tail part the one that I posted is working good.
Related
So basically I used mapreduce for wordcount for a text file I've saved in hadoop, now I want to the view output.
Currently this is the only command I've seen online:
bin/hadoop fs -cat output/part-r-00000 | sort -k 2 -n -r | less
So far I'm just confused by this command, is it just sort the output? can I view the output without sorting it?
Is this command sorting the wordcount display everything in alphabetical order otherwise? Is there any other way you would recommend to sort the saved the text fie, a novel?
Also can I just view the outputfile of wordcount without sorting it?
Can I view the output without sorting it?
Just -cat it
bin/hadoop fs -cat output/part-r-00000 | less
Or copy the output file to the Local FS from HDFS and use it
bin/hadoop fs -get output/part-r-00000 /tmp/output
Is this command sorting the wordcount display everything in
alphabetical order otherwise?
sort -k 2 -n -r: Sort the 2nd column (-k 2) numerically (-n) in reverse (-r) order.
Assuming the second column contains the count, this would sort the words from most number of occurrences to the least. As for the different way of sorting, I feel this is the better one. If you want to sort the content alphabetically, just use sort. Refer sort manual.
I have a file on HDFS that I want to know how many lines are. (testfile)
In linux, I can do:
wc -l <filename>
Can I do something similar with "hadoop fs" command? I can print file contents with:
hadoop fs -text /user/mklein/testfile
How do I know how many lines do I have? I want to avoid copying the file to local filesystem then running the wc command.
Note: My file is compressed using snappy compression, which is why I have to use -text instead of -cat
Total number of files:
hadoop fs -ls /path/to/hdfs/* | wc -l
Total number of lines:
hadoop fs -cat /path/to/hdfs/* | wc -l
Total number of lines for a given file:
hadoop fs -cat /path/to/hdfs/filename | wc -l
1. Number of lines of a mapper output file:
`~]$ hadoop fs -cat /user/cloudera/output/part-m-00000 | wc -l`
2. Number of lines of a text or any other file on hdfs:
`~]$ hadoop fs -cat /user/cloudera/output/abc.txt | wc -l`
3. Top (Header) 5 lines of a text or any other file on hdfs:
`~]$ hadoop fs -cat /user/cloudera/output/abc.txt | head -5`
4. Bottom 10 lines of a text or any other file on hdfs:
`~]$ hadoop fs -cat /user/cloudera/output/abc.txt | tail -10`
You cannot do it with a hadoop fs command. Either you have to write a mapreduce code with the logic explained in this post or this pig script would help.
A = LOAD 'file' using PigStorage() as(...);
B = group A all;
cnt = foreach B generate COUNT(A);
Makesure you have the correct extension for your snappy file so that pig could detect and read it.
How can I decompress and view few lines of a compressed file in hdfs.
The below command displays the last few lines of the compressed data
hadoop fs -tail /myfolder/part-r-00024.gz
Is there a way I can use the -text command and pipe the output to tail command? I tried this but this doesn't work.
hadoop fs -text /myfolder/part-r-00024.gz > hadoop fs -tail /myfolder/
The following will show you the specified number of lines without decompressing the whole file:
hadoop fs -cat /hdfs_location/part-00000.gz | zcat | head -n 20
The following will page the file, also without first decompressing the whole of it:
hadoop fs -cat /hdfs_location/part-00000.gz | zmore
Try the following, should work as long as your file isn't too big (since the whole thing will be decompressed):
hadoop fs -text /myfolder/part-r-00024.gz | tail
I ended up writing a pig script.
A = LOAD '/myfolder/part-r-00024.gz' USING PigStorage('\t');
B = LIMIT A 10;
DUMP B;
Use gunzip to view the compressed file contents:
hdfs dfs -cat /path/filename.gz | gunzip
I have one big file in hdfs bigfile.txt. I want to copy the first 100 lines of it into a new file on hdfs. I tried the following command:
hadoop fs -cat /user/billk/bigfile.txt |head -100 /home/billk/sample.txt
It gave me a "cat: unable to write output stream" error. I am on hadoop 1.
Are there other ways to do this? (note: copying 1st 100 line to local or another file on hdfs is OK)
Like this -
hadoop fs -cat /user/billk/bigfile.txt | head -100 | hadoop -put - /home/billk/sample.txt
I believe the "cat: unable to write output stream" is just because head closed the stream after it read its limit. see this answer about head for hdfs - https://stackoverflow.com/a/19779388/3438870
A fast method for inspecting files on HDFS is to use tail:
~$ hadoop fs -tail /path/to/file
This displays the last kilobyte of data in the file, which is extremely helpful. However, the opposite command head does not appear to be part of the shell command collections. I find this very surprising.
My hypothesis is that since HDFS is built for very fast streaming reads on very large files, there is some access-oriented issue that affects head. This makes me hesitant to do things to access the head. Does anyone have an answer?
I would say it's more to do with efficiency - a head can easily be replicated by piping the output of a hadoop fs -cat through the linux head command.
hadoop fs -cat /path/to/file | head
This is efficient as head will close out the underlying stream after the desired number of lines have been output
Using tail in this manner would be considerably less efficient - as you'd have to stream over the entire file (all HDFS blocks) to find the final x number of lines.
hadoop fs -cat /path/to/file | tail
The hadoop fs -tail command as you note works on the last kilobyte - hadoop can efficiently find the last block and skip to the position of the final kilobyte, then stream the output. Piping via tail can't easily do this.
Starting with version 3.1.0 we now have it:
Usage: hadoop fs -head URI
Displays first kilobyte of the file to stdout.
See here.
hdfs -dfs /path | head
is a good way to solve the problem.
you can try the folowing command
hadoop fs -cat /path | head -n
where -n can be replace with number of records to view
In Hadoop v2:
hdfs dfs -cat /file/path|head
In Hadoop v1 and v3:
hadoop fs -cat /file/path|head