checking that the rows in a file have the same number of columns - bash

Question

I have a number of tsv files, and I want to check that each file is correctly formatted. primarily, I want to check that each row has the right number of columns. is there a way to do this? I'd love a command line solution if there is one.

Answer 1

Adding this here because these answers were all close but didn't quite work for me, in my case I needed to specify the field separator for awk.
The following should return with a single line containing the number of columns (if every row has the same number of columns).
$ awk -F'\t' '{print NF}' test.tsv | sort -nu
8
-F is used to specify the field separator for awk
NF is the number of fields
-nu orders the field count for each row numerically and returns only the unique ones
If you get more than one row returned, then there are some rows of your .tsv with more columns than others.
To check that the .tsv is correctly formatted with each row having the same number of fields, the following should return 1 (as commented by kmace on the accepted answer) however I needed to add the -F'\t'
$ awk -F'\t' '{print NF}' test.tsv | sort -nu | wc -l

Answer 2

awk '{print NF}' test | sort -nu | head -n 1
This gives you the lowest number of columns in the file on any given row.
awk '{print NF}' test | sort -nu | tail -n 1
This gives you the highest number of columns in the file on any given row.
The result should be the same, if all the columns are present.
Note: this gives me an error on OS X, but not on Debian... maybe use gawk.

Answer 3

(I'm assuming that by "tsv", you mean a file whose columns are separated with tab characters.)
You can do this simply with awk, as long as the file doesn't have quoted fields containing tab characters.
If you know how many columns you expect, the following will work:
awk -F '\t' -v NCOLS=42 'NF!=NCOLS{printf "Wrong number of columns at line %d\n", NR}'
(Of course, you need to change the 42 to the correct value.)
You could also automatically pick up the number of columns from the first line:
awk -F '\t' 'NR==1{NCOLS=NF};NF!=NCOLS{printf "Wrong number of columns at line %d\n", NR}'
That will work (with a lot of noise) if the first line has the wrong number of columns, but it would fail to detect a file where all the lines have the same wrong number of columns. So you're probably better off with the first version, which forces you to specify the column count.

Answer 4

Just cleaning up #snd answer above:
number_uniq_row_lengths=`awk '{print NF}' $pclFile | sort -nu | wc -l`
if [ $number_uniq_row_lengths -eq 1 ] 2>/dev/null; then
echo "$pclFile is clean"
fi

Answer 5

awk is a good candidate for this. If your columns are separated by tabs (I guess it is what tsv means) and if you know how many of them you should have, say 17, you can try:
awk -F'\t' 'NF != 17 {print}' file.tsv
This will print all lines in file.tsv that has not exactly tab-separated 17 columns. If my guess is incorrect, please edit your question and add the missing information (column separators, number of columns...) Note that the tsv (and csv) format is trickier than it seems. The fields can contain the field separator, records can span on several lines... If it is your case, do not try to reinvent the wheel and use an existing tsv parser.