Show duplicates in internal table - algorithm

Question

Each an every item should have an uniquie SecondNo + Drawing combination. Due to misentries, some combinations are there two times.
I need to create a report with ABAP which identifies those combinations and does not reflect the others.
Item: SecNo: Drawing:
121 904 5000 double
122 904 5000 double
123 816 5100
124 813 5200
125 812 4900 double
126 812 4900 double
127 814 5300
How can I solve this? I tried 2 approaches and failed:
Sorting the data and tried to print out each one when the value of the upper row is equal to the next value
counting the duplicates and showing all of them which are more then one.
Where do I put in the condition? in the loop area?
I tried this:
REPORT duplicates.
DATA: BEGIN OF lt_duplicates OCCURS 0,
f2(10),
f3(10),
END OF lt_duplicates,
it_test TYPE TABLE OF ztest WITH HEADER LINE,
i TYPE i.
SELECT DISTINCT f2 f3 FROM ztest INTO TABLE lt_duplicates.
LOOP AT lt_duplicates.
IF f2 = lt_duplicates-f2 AND f3 = lt_duplicates-f3.
ENDIF.
i = LINES( it_test ).
IF i > 1.
LOOP AT it_test.
WRITE :/ it_test-f1,it_test-f2,it_test-f3.
ENDLOOP.
ENDIF.
ENDLOOP.

Answer 1

From ABAP 7.40, you may use the GROUP BY constructs with the GROUP SIZE words so that to take into account only the groups with at least 2 elements.
ABAP statement LOOP AT ... GROUP BY ( <columns...> gs = GROUP SIZE ) ...
Loop at grouped lines:
Either LOOP AT GROUP ...
Or ... FOR ... IN GROUP ...
ABAP expression ... VALUE|REDUCE|NEW type|#( FOR GROUPS ... GROUP BY ( <columns...> gs = GROUP SIZE ) ...
Loop at grouped lines: ... FOR ... IN GROUP ...
For both constructs, it's possible to loop at the grouped lines in two ways:
* LOOP AT GROUP ...
* ... FOR ... IN GROUP ...
Line# Item SecNo Drawing
1 121 904 5000 double
2 122 904 5000 double
3 123 816 5100
4 124 813 5200
5 125 812 4900 double
6 126 812 4900 double
7 127 814 5300
You might want to produce the following table containing the duplicates:
SecNo Drawing Lines
904 5000 [1,2]
812 4900 [5,6]
Solution with LOOP AT ... GROUP BY ...:
TYPES: BEGIN OF t_line,
item TYPE i,
secno TYPE i,
drawing TYPE i,
END OF t_line,
BEGIN OF t_duplicate,
secno TYPE i,
drawing TYPE i,
num_dup TYPE i, " number of duplicates
lines TYPE STANDARD TABLE OF REF TO t_line WITH EMPTY KEY,
END OF t_duplicate,
t_lines TYPE STANDARD TABLE OF t_line WITH EMPTY KEY,
t_duplicates TYPE STANDARD TABLE OF t_duplicate WITH EMPTY KEY.
DATA(table) = VALUE t_lines(
( item = 121 secno = 904 drawing = 5000 )
( item = 122 secno = 904 drawing = 5000 )
( item = 123 secno = 816 drawing = 5100 )
( item = 124 secno = 813 drawing = 5200 )
( item = 125 secno = 812 drawing = 4900 )
( item = 126 secno = 812 drawing = 4900 )
( item = 127 secno = 814 drawing = 5300 ) ).
DATA(expected_duplicates) = VALUE t_duplicates(
( secno = 904 drawing = 5000 num_dup = 2 lines = VALUE #( ( REF #( table[ 1 ] ) ) ( REF #( table[ 2 ] ) ) ) )
( secno = 812 drawing = 4900 num_dup = 2 lines = VALUE #( ( REF #( table[ 5 ] ) ) ( REF #( table[ 6 ] ) ) ) ) ).
DATA(actual_duplicates) = VALUE t_duplicates( ).
LOOP AT table
ASSIGNING FIELD-SYMBOL(<line>)
GROUP BY
( secno = <line>-secno
drawing = <line>-drawing
gs = GROUP SIZE )
ASSIGNING FIELD-SYMBOL(<group_table>).
IF <group_table>-gs >= 2.
actual_duplicates = VALUE #( BASE actual_duplicates
( secno = <group_table>-secno
drawing = <group_table>-drawing
num_dup = <group_table>-gs
lines = VALUE #( FOR <line2> IN GROUP <group_table> ( REF #( <line2> ) ) ) ) ).
ENDIF.
ENDLOOP.
WRITE : / 'List of duplicates:'.
SKIP 1.
WRITE : / 'Secno Drawing List of concerned items'.
WRITE : / '---------- ---------- ---------------------------------- ...'.
LOOP AT actual_duplicates ASSIGNING FIELD-SYMBOL(<duplicate>).
WRITE : / <duplicate>-secno, <duplicate>-drawing NO-GROUPING.
LOOP AT <duplicate>-lines INTO DATA(line).
WRITE line->*-item.
ENDLOOP.
ENDLOOP.
ASSERT actual_duplicates = expected_duplicates. " short dump if not equal
Output:
List of duplicates:
Secno Drawing List of concerned items
---------- ---------- ---------------------------------- ...
904 5000 121 122
812 4900 125 126
Solution with ... VALUE type|#( FOR GROUPS ... GROUP BY ...:
DATA(actual_duplicates) = VALUE t_duplicates(
FOR GROUPS <group_table> OF <line> IN table
GROUP BY
( secno = <line>-secno
drawing = <line>-drawing
gs = GROUP SIZE )
( secno = <group_table>-secno
drawing = <group_table>-drawing
num_dup = <group_table>-gs
lines = VALUE #( FOR <line2> IN GROUP <group_table> ( REF #( <line2> ) ) ) ) ).
DELETE actual_duplicates WHERE num_dup = 1.
Note: for deleting non-duplicates, instead of using an additional DELETE statement, it can be done inside the VALUE construct by adding a LINES OF COND construct which adds 1 line if group size >= 2, or none otherwise (if group size = 1):
...
gs = GROUP SIZE )
( LINES OF COND #( WHEN <group_table>-gs >= 2 THEN VALUE #( "<== new line
( secno = <group_table>-secno
...
... REF #( <line2> ) ) ) ) ) ) ) ). "<== 3 extra right parentheses

Answer 2

You can use AT...ENDAT for this, provided that you arrange the fields correctly:
TYPES: BEGIN OF t_my_line,
secno TYPE foo,
drawing TYPE bar,
item TYPE baz, " this field has to appear AFTER the other ones in the table
END OF t_my_line.
DATA: lt_my_table TYPE TABLE OF t_my_line,
lt_duplicates TYPE TABLE OF t_my_line.
FIELD-SYMBOLS: <ls_line> TYPE t_my_line.
START-OF-WHATEVER.
* ... fill the table ...
SORT lt_my_table BY secno drawing.
LOOP AT lt_my_table ASSIGNING <ls_line>.
AT NEW drawing. " whenever drawing or any field left of it changes...
FREE lt_duplicates.
ENDAT.
APPEND <ls_line> TO lt_duplicates.
AT END OF drawing.
IF lines( lt_duplicates ) > 1.
* congrats, here are your duplicates...
ENDIF.
ENDAT.
ENDLOOP.

Answer 3

I needed simply to report duplicate lines in error based on two fields so used the following.
LOOP AT gt_data INTO DATA(gs_data)
GROUP BY ( columnA = gs_data-columnA columnB = gs_data-columnB
size = GROUP SIZE index = GROUP INDEX ) ASCENDING
REFERENCE INTO DATA(group_ref).
IF group_ref->size > 1.
PERFORM insert_error USING group_ref->columnA group_ref->columnB.
ENDIF.
ENDLOOP.

Answer 4

Here is my 2p worth, you could cut some out of this depending on what you want to do, and you should consider the amount of data being processed too. This method is only really for smaller sets.
Personally I like to prevent erroneous records at the source. Catching an error during input. But if you do end up in a pickle there is definitely more than one way to solve the issue.
TYPES: BEGIN OF ty_itab,
item TYPE i,
secno TYPE i,
drawing TYPE i,
END OF ty_itab.
TYPES: itab_tt TYPE STANDARD TABLE OF ty_itab.
DATA: lt_itab TYPE itab_tt,
lt_itab2 TYPE itab_tt,
lt_itab3 TYPE itab_tt.
lt_itab = VALUE #(
( item = '121' secno = '904' drawing = '5000' )
( item = '122' secno = '904' drawing = '5000' )
( item = '123' secno = '816' drawing = '5100' )
( item = '124' secno = '813' drawing = '5200' )
( item = '125' secno = '812' drawing = '4900' )
( item = '126' secno = '812' drawing = '4900' )
( item = '127' secno = '814' drawing = '5300' )
).
APPEND LINES OF lt_itab TO lt_itab2.
APPEND LINES OF lt_itab TO lt_itab3.
SORT lt_itab2 BY secno drawing.
DELETE ADJACENT DUPLICATES FROM lt_itab2 COMPARING secno drawing.
* Loop at what is hopefully the smaller itab.
LOOP AT lt_itab2 ASSIGNING FIELD-SYMBOL(<line>).
DELETE TABLE lt_itab3 FROM <line>.
ENDLOOP.
* itab1 has all originals.
* itab2 has the unique.
* itab3 has the duplicates.