detecting "duplicate" entries in a tab separated file using bash & commands - bash

Question

I have a tab-separated text file I need to check for duplicates. The layout looks roughly like so. (The first entries in the file are the column names.)
Sample input file:
+--------+-----------+--------+------------+-------------+----------+
| First | Last | BookID | Title | PublisherID | AuthorID |
+--------+-----------+--------+------------+-------------+----------+
| James | Joyce | 37 | Ulysses | 344 | 1022 |
| Ernest | Hemingway | 733 | Old Man... | 887 | 387 |
| James | Joyce | 872 | Dubliners | 405 | 1022 |
| Name1 | Surname1 | 1 | Title1 | 1 | 1 |
| James | Joyce | 37 | Ulysses | 345 | 1022 |
| Name1 | Surname1 | 1 | Title1 | 2 | 1 |
+--------+-----------+--------+------------+-------------+----------+
The file can hold up to 500k rows. What we're after is checking that there are no duplicates of the BookID and AuthorID values. So for instance, in the table above there can be no two rows with a BookID of 37 and AuthorID 1022.
It's likely, but not guaranteed, that the author will be grouped on consecutive lines. If it isn't, and it's too tricky to check, I can live with that. But otherwise, if the author is the same, we need to know if a duplicate BookID is there.
One complication-- we can have duplicate BookIDs in the file, but it's the combo of AuthorID + BookID that is not allowed.
Is there a good way of checking this in a bash script, perhaps some combo of sed and awk or another means of accomplishing this?
Raw tab-separated file contents for scripting:
First Last BookID Title PublisherID AuthorID
James Joyce 37 Ulysses 344 1022
Ernest Hemingway 733 Old Man... 887 387
James Joyce 872 Dubliners 405 1022
Name1 Surname1 1 Title1 1 1
James Joyce 37 Ulysses 345 1022
Name1 Surname1 1 Title1 2 1

Answer 1

If you want to find and count the duplicates you can use
awk '{c[$3 " " $6]+=1} END { for (k in c) if (c[k] > 1) print k "->" c[k]}'
which saves the combinations count in an associative array and then prints the counts if greater than 1

Answer 2

tab-separated text file
is checking that there are no duplicates of the BookID and AuthorID values
And from #piotr.wittchen answer the columns look like this:
First Last BookID Title PublisherID AuthorID
That's simple:
extract BookID AuthorID columns
sort
check for duplicates
cut -f3,6 input_file.txt | sort | uniq -d
If you gotta have the whole lines, we have to reorder the fields a bit for uniq to eat them:
awk '{print $1,$2,$4,$5,$3,$6}' input_file.txt | sort -k5 -k6 | uniq -d -f4
If you gotta have them in the initial order, you can number the lines, get the duplicates and re-sort them with the line numbers and then remove the line numbers, like so:
nl -w1 input_file.txt |
awk '{print $1,$2,$3,$5,$6,$4,$7}' input_file.txt | sort -k6 -k7 | uniq -d -f5 |
sort -k1 | cut -f2-

Answer 3

This is pretty easy with awk:
$ awk 'BEGIN { FS = "\t" }
($3,$6) in seen { printf("Line %d is a duplicate of line %d\n", NR, seen[$3,$6]); next }
{ seen[$3,$6] = NR }' input.tsv
It saves each bookid, authorid pair in a hash table and warns if that pair already exists.

Answer 4

As #Cyrus already said in the comment, your questions is not really clear, but looks interesting and I attempted to understand it and provide solution giving a few assumptions.
Assuming we have the following records.txt file:
First Last BookID Title PublisherID AuthorID
James Joyce 37 Ulysses 344 1022
Ernest Hemingway 733 Old Man... 887 387
James Joyce 872 Dubliners 405 1022
Name1 Surname1 1 Title1 1 1
James Joyce 37 Ulysses 345 1022
Name1 Surname1 1 Title1 2 1
we are going to remove lines, which has duplicated BookID (column 3) and AuthorID (Column 6) values at the same time. We assume that First, Last name and Title are also the same and we don't have to take it into consideration and PublisherID may be different or the same (it doesn't matter). Location of the records in the file doesn't matter (duplicated lines don't have to be grouped together).
Having these assumptions in mind, expected output for the input provided above will be as follows:
Ernest Hemingway 733 Old Man... 887 387
James Joyce 872 Dubliners 405 1022
James Joyce 37 Ulysses 344 1022
Name1 Surname1 1 Title1 1 1
Duplicated records of the same books of the same author for one publisher were removed.
Here's my solution for this problem in Bash
#!/usr/bin/env bash
file_name="records.txt"
repeated_books_and_authors_ids=($(cat $file_name | awk '{print $3$6}' | sort | uniq -d))
for i in "${repeated_books_and_authors_ids[#]}"
do
awk_statment_exclude="$awk_statment_exclude\$3\$6 != $i && "
awk_statment_include="$awk_statment_include\$3\$6 ~ $i || "
done
awk_statment_exclude=${awk_statment_exclude::-3}
awk_statment_exclude="awk '$awk_statment_exclude {print \$0}'"
not_repeated_records="cat $file_name | $awk_statment_exclude | sed '1d'"
eval $not_repeated_records
awk_statment_include=${awk_statment_include::-3}
awk_statment_include="awk '$awk_statment_include {print \$0}'"
repeated_records_without_duplicates="cat $file_name | $awk_statment_include | sort | awk 'NR % 2 != 0'"
eval $repeated_records_without_duplicates
It's probably not the best possible solution, but it works.
Regards,
Piotr