How would one split the data in a column of an unstuctured format and put it back into structured a structured form? - powerquery

Question

The issue I am dealing with basically has a field for 2 types of data (SERIAL_NO) and (EXP_DATE) and a third field for unstructured data entry (COMMENTS). Additional information is placed in the unstructured data field that I need to turn into structured data.
OPER_KEY
TIME_STAMP
SERIAL_NO
EXP_DATE
COMMENTS
35374
1/12/2021 11:30
M161001
10/31/2021
M190426, 5/31/2021
35374
1/8/2021 13:59
M161001
10/31/2021
M190426 , 5/31/2021
35374
1/25/2021 15:32
M190426
5/31/2021
M161001, EXP 10/31/21
35413
9/13/2019 16:21
M151144
11/30/2019
EXAMPLE TEXT
35413
9/12/2019 11:44
M15144
11/30/2019
35413
9/14/2019 12:15
M190426
4/30/2020
M151144 M190426
35058
1/14/2019 8:53
M180117
1/31/2019
E190426 5/31/2021 M161001 10/31/21
There are no easy delimiters. The format is, the first letter is a character, followed by 6 digits. Then sometimes followed by a date. These need to be merged back into the original table like so.
OPER_KEY
TIME_STAMP
SERIAL_NO
EXP_DATE
35374
1/12/2021 11:30
M161001
10/31/2021
35374
1/12/2021 11:30
M190426
5/31/2021
35374
1/8/2021 13:59
M161001
10/31/2021
35374
1/8/2021 13:59
M190426
5/31/2021
35374
1/25/2021 15:32
M190426
5/31/2021
35374
1/25/2021 15:32
M161001
10/31/21
35413
9/13/2019 16:21
M151144
11/30/2019
35413
9/12/2019 11:44
M15144
11/30/2019
35413
9/14/2019 12:15
M190426
4/30/2020
35413
9/14/2019 12:15
M151144
35413
9/14/2019 12:15
M190426
35058
1/14/2019 8:53
M180117
1/31/2019
35058
1/14/2019 8:53
E190426
5/31/2021
35058
1/14/2019 8:53
M161001
10/31/21
I can reference the original table and then group the section by "all rows" immediately to work with the data and later append the original table with the new data. The issue I am having is how to successfully parse out the relevant data into columns. Any recommendations on how to extract this information?

Answer 1

Here's one way that works on your data example: read the code comments and explore the Applied Steps to understand the algorithm:
let
Source = Excel.CurrentWorkbook(){[Name="Table5"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"OPER_KEY", Int64.Type}, {"TIME_STAMP", type datetime}, {"SERIAL_NO", type text}, {"EXP_DATE", type date}, {"COMMENTS", type text}}),
//Create list of Serial numbers from the Comments Column
#"Added Custom" = Table.AddColumn(#"Changed Type", "Comment SN", each if [COMMENTS] = null then {null} else
let
#"Split Text" = List.RemoveItems(Text.SplitAny([COMMENTS],", #(lf)"),{""}),
#"Serial Nos" = List.Transform(
List.Select(
List.Transform(#"Split Text", (li)=>Splitter.SplitTextByCharacterTransition({"A".."Z","a".."z"}, {"0".."9"})(li)),
each List.Count(_)=2),
each Text.Combine(_,""))
in
#"Serial Nos", type list),
//Create list of dates from the Comments column
#"Added Custom1" = Table.AddColumn(#"Added Custom", "Comments Dates", each if [COMMENTS] = null then {null} else
let
#"Split Text" = List.RemoveItems(Text.SplitAny([COMMENTS],", #(lf)"),{""}),
Dates = List.Select(#"Split Text", (li)=> Record.Field(try Date.From(li), "HasError")=false)
in
Dates, type list),
#"Removed Columns" = Table.Buffer(Table.RemoveColumns(#"Added Custom1",{"COMMENTS"})),
//create the results table
//First row is from the first four columns
//For subsequent rows we extract the corresponding SN and Date from the Comments column
#"Add Result Column" = Table.AddColumn(#"Removed Columns","Result", (tr)=>
List.Generate(
()=>[r=Record.FromList({tr[OPER_KEY],tr[TIME_STAMP],tr[SERIAL_NO],tr[EXP_DATE]},{"OPER_KEY","TIME_STAMP","SERIAL_NO","EXP_DATE"}), idx=0] ,
each [idx] <= List.Count(tr[Comment SN]),
each [r=Record.FromList({null, null, tr[Comment SN]{[idx]}, try tr[Comments Dates]{[idx]} otherwise null},{"OPER_KEY","TIME_STAMP","SERIAL_NO","EXP_DATE"}), idx=[idx]+1],
each [r]
) ),
//Remove all the columns except Results
//and expand
#"Removed Columns1" = Table.RemoveColumns(#"Add Result Column",{"OPER_KEY", "TIME_STAMP", "SERIAL_NO", "EXP_DATE", "Comment SN", "Comments Dates"}),
#"Expanded Result" = Table.ExpandListColumn(#"Removed Columns1", "Result"),
#"Expanded Result1" = Table.ExpandRecordColumn(#"Expanded Result", "Result", {"OPER_KEY", "TIME_STAMP", "SERIAL_NO", "EXP_DATE"}, {"OPER_KEY", "TIME_STAMP", "SERIAL_NO", "EXP_DATE"}),
//set the data types
#"Changed Type1" = Table.TransformColumnTypes(#"Expanded Result1",{{"OPER_KEY", Int64.Type}, {"TIME_STAMP", type datetime}, {"SERIAL_NO", type text}, {"EXP_DATE", type date}}),
#"Removed Blank Rows" = Table.SelectRows(#"Changed Type1", each not List.IsEmpty(List.RemoveMatchingItems(Record.FieldValues(_), {"", null}))),
#"Filled Down" = Table.FillDown(#"Removed Blank Rows",{"OPER_KEY", "TIME_STAMP"})
in
#"Filled Down"