Why do these two GREP commands give different results? - sorting

Question

I have a large log file containing lines for a particular task as follows:
[info] My task : 123
[info] Other task : 111
[info] My task : 456
[info] My task : 456
[info] My task : 789
I want to count the number of unique "My task"s logged. Which in this case should be 3.
I have used these two commands which, in my opinion, should give the same and correct results:
grep 'My Task :' | uniq | wc -l
grep -E 'My Task :' | sort --unique | grep -cE 'My Task :'
The two commands give the same results on the small test files I create but different results on the large log file on the server. I cannot understand why. To be exact, the first command gives a count of ~33k while the second one gives ~15k. Which command of the two, if any is correct? And what should I ideally be doing?

Answer 1

It's possible it happens because uniq can only find consecutive
identical lines. Say, if your file looks like this:
[info] My task : 123
[info] Other task : 111
[info] My task : 456
[info] My task : 456
[info] My task : 789
[info] My task : 123
[info] Other task : 111
[info] My task : 456
[info] My task : 456
[info] My task : 789
[info] My task : 123
[info] Other task : 111
[info] My task : 456
[info] My task : 456
[info] My task : 789
[info] My task : 123
[info] Other task : 111
[info] My task : 456
[info] My task : 456
[info] My task : 789
[info] My task : 123
[info] Other task : 111
[info] My task : 456
[info] My task : 456
[info] My task : 789
results will be different:
$ grep 'My task :' FILE | uniq | wc -l
15
$ grep -E 'My task :' FILE | sort --unique | wc -l
3

Answer 2

Problem: uniq and sort -u
They should be equivalent in the simple case, but will behave
differently if you're using the -k option to define only certain
fields of the input line to use as sort keys. In that case, sort -u
will suppress lines which have the same key even if other parts of the
line differ, whereas uniq will only suppress lines that are exactly
identical.
**Difference Between the Two Commands **
1st Command grep 'My Task :' | uniq | wc -l: It prints the count
of lines which are unique for 'My Task'
2nd Command grep -E 'My Task :' | sort --unique | grep -cE 'My Task' : It prints the count of matching pattern 'My Task'
Discrepancy b/w the two depends on the contents of your log file.
To answer your question which one to use:
When using grep with -E like you did try to make an apt pattern and then count the lines.

Answer 3

No need multiple commands to count unique number.
awk '/My task/ {a[$NF]++;c+=a[$NF]==1?1:0} END {print c}' file
3
/My task/ do line contain My task, if yes:
a[$NF]++ create an array with number as a key. First time a value is found it will be 1, second time 2, third time same value found it will be 3 etc
c+=a[$NF]==1?1:0 if array a[number] is 1 (first time found), increment c with 1, else add 0
{print c} print number of unique number from variable c