Single and Double Quote Issue using GAWK - quotes

Question

I have been helped with the following GAWK command line in Windows
gawk "{a = gensub(/([0-9]{6} [0-9]{6} [0-9]{4}).*itemId=, "\\2 \\1", "g", $0); b = split(a, c, " "); if (c[1] in result) result[c[1]] = gensub(/(.+),(.+)/, "\\1," c[2] " " c[3] " " c[4], "g", result[c[1]]); else result[c[1]] = c[2] " " c[3] " " c[4] "," c[2] " " c[3] " " c[4]} END { for (i in result) print i ": " result[i]}" Item.log
I have thousands of lines of text, like below which contain information for an item in ITEMID, these can be spread throughout the file, I want to get the first date and time and the last date and time and create a line like :
ITEMID 170209 003122 170209 003621
The date format about is YYMMDD HHMMSS
170209 003122 0893 DE(N) ItemHandler.ItemLog event=<VOLUME_RESULT>, *************************, itemId=<12551677>, globalId=<12551677>, cmcIndex=<973>, sorter=<0: MS01>, state=<CSC: WaitForData>, volumeId=<4104: MS01.IU05-VMS04>, volumeStatus=<0>, volScanResult=<0>, length=<665 mm>, width=<610 mm>, height=<470 mm>, volume=<190656000 mm3>, position=<sorter#0.volume#4104: MS01.IU05-VMS04>, volSystemId=<4>, volReferenceRecord=<2542>, volLevelOffset=<0>, volDataCrc=<48107>, itemRevisionNumber=<3> ##[
170209 003126 0210 DE(N) ItemHandler.ItemLog event=<SCANNER_RESULT>, ************************, itemId=<12551674>, globalId=<12551674>, cmcIndex=<970>, sorter=<0: MS01>, state=<CSC: WaitForData>, barcodeCount=<1>, barcodes=<[ProxyWrapperBarcode(barcode=<T000042357>, type=<C0>, result=<OK>, ccType=<>)]>, codeSource=<ohscan>, scannerId=<4000: SCAN01>, scannerStatus=<0>, position=<sorter#0.scanner#4000: SCAN01>, itemRevisionNumber=<5> ##[
170209 003126 0210 DE(N) ItemHandler.ItemLog event=<DESTINATION_REQUEST>, *******************, itemId=<12551674>, globalId=<12551674>, cmcIndex=<970>, sorter=<0: MS01>, state=<CSC: WaitForData>, barcodes=<[ProxyWrapperBarcode(barcode=<T000042357>, type=<C0>, result=<OK>, ccType=<>)]>, ccReason=<SCANNER_DATA_ADDED>, PreviousccResult=<>, sortSchemeId=<-1>, sortSchemeName=<>, logicalDestination=<>, BatchCountItem=<true>, collectionId=<-1>, goodsId=<>, position=<sorter#0.scanner#4000: SCAN01>, dynamicDataCount=<0>, dynamicData=<{}>, carrierId=<202>, carrierCount=<1>, itemRevisionNumber=<5> ##[
170209 003126 0225 DE(N) ItemHandler.ItemLog event=<DESTINATION_REPLY>, *********************, itemId=<12551674>, globalId=<12551674>, cmcIndex=<970>, sorter=<0: MS01>, state=<CSC: WaitForDestination>, ccReason=<SCANNER_DATA_ADDED>, PendingccResult=<OK>, Pendingstrategy=<notSpecified>, PendingchuteGroup=<[3000]: Parked0>, PendingNotChutedestinationId=<-1>, PendingsortSchemeId=<-1>, PendingsortSchemeName=<>, PendinglogicalDestination=<>, PendinggoodsId=<>, PendingBatchCountItem=<true>, PendingcollectionId=<-1>, position=<sorter#0.scanner#4000: SCAN01>, dynamicDataCount=<0>, dynamicData=<{}>, itemRevisionNumber=<7> ##[
170209 003126 0225 DE(N) ItemHandler.ItemLog event=<REDIRECT_ITEM>, *************************, itemId=<12551674>, globalId=<12551674>, cmcIndex=<970>, sorter=<0: MS01>, state=<CSC: WaitForDestination>, PendingchuteGroup=<[3000]: Parked0>, Pendingstrategy=<notSpecified>, CscdestinationId=<-1: UnDef>, CmcdestinationId=<4099: All Scanners>, position=<sorter#0.scanner#4000: SCAN01>, itemRevisionNumber=<7> ##[
170209 003126 0454 DE(N) ItemHandler.ItemLog event=<REDIRECT_ITEM_REPLY>, *******************, itemId=<12551674>, globalId=<12551674>, cmcIndex=<970>, sorter=<0: MS01>, state=<CSC: WaitForDestination>, CscdestinationId=<3000: Parked0>, chuteGroup=<[3000]: Parked0>, CmcdestinationId=<3000: Parked0>, position=<sorter#0.scanner#4000: SCAN01>, chuteListStartPoint=<-1>, itemRevisionNumber=<9> ##[
170209 003126 0456 DE(N) ItemHandler.ItemLog event=<RECONVERT>, *****************************, itemId=<12551674>, globalId=<12551674>, cmcIndex=<970>, sorter=<0: MS01>, state=<CSC: WaitForData>, CscdestinationId=<3000: Parked0>, chuteGroup=<[3000]: Parked0>, CmcdestinationId=<3000: Parked0>, position=<sorter#0.scanner#4000: SCAN01>, chuteListStartPoint=<-1>, itemRevisionNumber=<10> ##[
170209 003126 0456 DE(N) ItemHandler.ItemLog event=<DESTINATION_REQUEST>, *******************, itemId=<12551674>, globalId=<12551674>, cmcIndex=<970>, sorter=<0: MS01>, state=<CSC: WaitForData>, barcodes=<[ProxyWrapperBarcode(barcode=<T000042357>, type=<C0>, result=<OK>, ccType=<>)]>, ccReason=<SCANNER_DATA_ADDED>, PreviousccResult=<OK>, sortSchemeId=<-1>, sortSchemeName=<>, logicalDestination=<>, BatchCountItem=<true>, collectionId=<-1>, goodsId=<>, position=<sorter#0.scanner#4000: SCAN01>, dynamicDataCount=<0>, dynamicData=<{}>, carrierId=<202>, carrierCount=<1>, itemRevisionNumber=<10> ##[
170209 003126 0458 DE(N) ItemHandler.ItemLog event=<DESTINATION_REPLY>, *********************, itemId=<12551674>, globalId=<12551674>, cmcIndex=<970>, sorter=<0: MS01>, state=<CSC: WaitForDestination>, ccReason=<SCANNER_DATA_ADDED>, PendingccResult=<OK>, Pendingstrategy=<priority>, PendingchuteGroup=<[186, 188, 195, 201, 184]: [XRA502, XRA503, XRA605, XRA607, XRA501]>, PendingNotChutedestinationId=<-1>, PendingsortSchemeId=<-1>, PendingsortSchemeName=<>, PendinglogicalDestination=<XRA502>, PendinggoodsId=<>, PendingBatchCountItem=<true>, PendingcollectionId=<-1>, position=<sorter#0.scanner#4000: SCAN01>, dynamicDataCount=<0>, dynamicData=<{}>, itemRevisionNumber=<12> ##[
170209 003126 0458 DE(N) ItemHandler.ItemLog event=<REDIRECT_ITEM>, *************************, itemId=<12551674>, globalId=<12551674>, cmcIndex=<970>, sorter=<0: MS01>, state=<CSC: WaitForDestination>, PendingchuteGroup=<[186, 188, 195, 201, 184]: [XRA502, XRA503, XRA605, XRA607, XRA501]>, Pendingstrategy=<priority>, CscdestinationId=<3000: Parked0>, CmcdestinationId=<3000: Parked0>, position=<sorter#0.scanner#4000: SCAN01>, itemRevisionNumber=<12> ##[
170209 003126 0681 DE(N) ItemHandler.ItemLog event=<REDIRECT_ITEM_REPLY>, *******************, itemId=<12551674>, globalId=<12551674>, cmcIndex=<970>, sorter=<0: MS01>, state=<CSC: WaitForDestination>, CscdestinationId=<186: XRA502>, chuteGroup=<[186, 188, 195, 201, 184]: [XRA502, XRA503, XRA605, XRA607, XRA501]>, CmcdestinationId=<186: XRA502>, position=<sorter#0.scanner#4000: SCAN01>, chuteListStartPoint=<186>, itemRevisionNumber=<14> ##[
170209 003126 0981 DE(N) ItemHandler.ItemLog event=<ITEM_INDUCTED>, *************************, itemId=<12551677>, globalId=<12551677>, cmcIndex=<973>, sorter=<0: MS01>, state=<CSC: WaitForData>, inductionId=<4: IU05>, inductionMode=<SCANNER>, inductStatus=<NORMAL_ITEM>, carrierId=<343>, carrierCount=<1>, CmcdestinationId=<4099: All Scanners>, position=<sorter#0: MS01>, itemRevisionNumber=<4> ##[
170209 003129 0845 DE(N) ItemHandler.ItemLog event=<ITEM_AT_INDUCTION>, *********************, itemId=<12551678>, globalId=<12551678>, cmcIndex=<974>, sorter=<0: MS01>, state=<CSC: ProjectIdle>, inductionId=<4: IU05>, position=<sorter#0.induction#4: IU05>, itemRevisionNumber=<0> ##[
170209 003129 0845 DE(N) ItemHandler.ItemLog event=<SET_ITEM_ID>, ***************************, itemId=<12551678>, globalId=<12551678>, cmcIndex=<974>, sorter=<0: MS01>, state=<CSC: ProjectIdle>, itemRevisionNumber=<0> ##[
170209 003130 0027 DE(N) ItemHandler.ItemLog event=<ITEM_PASSED_SPS>, ***********************, itemId=<12551665>, globalId=<12551665>, cmcIndex=<961>, sorter=<0: MS01>, state=<CSC: ProjectHeadingForChute>, carrierId=<363>, carrierCount=<1>, CmcdestinationId=<198: EXP507>, CscdestinationId=<198: EXP507>, itemRevisionNumber=<15> ##[
170209 003130 0350 DE(N) ItemHandler.ItemLog event=<DISCHARGE_ATTEMPTED>, *******************, itemId=<12551665>, globalId=<12551665>, cmcIndex=<961>, sorter=<0: MS01>, state=<CSC: ProjectHeadingForChute>, CscdestinationId=<198: EXP507>, chuteGroup=<[198]: EXP507>, CmcdestinationId=<198: EXP507>, position=<sorter#0: MS01>, itemRevisionNumber=<15> ##[
170209 003130 0352 DE(N) ItemHandler.ItemLog event=<RECONVERT>, *****************************, itemId=<12551665>, globalId=<12551665>, cmcIndex=<961>, sorter=<0: MS01>, state=<CSC: ProjectHeadingForChute>, CscdestinationId=<198: EXP507>, chuteGroup=<[198]: EXP507>, CmcdestinationId=<198: EXP507>, position=<sorter#0: MS01>, chuteListStartPoint=<198>, itemRevisionNumber=<16> ##[
170209 003130 0352 DE(N) ItemHandler.ItemLog event=<DESTINATION_REQUEST>, *******************, itemId=<12551665>, globalId=<12551665>, cmcIndex=<961>, sorter=<0: MS01>, state=<CSC: ProjectHeadingForChute>, barcodes=<[ProxyWrapperBarcode(barcode=<3709160626>, type=<A0>, result=<OK>, ccType=<>), ProxyWrapperBarcode(barcode=<2LIN400023+48000001>, type=<C0>, result=<OK>, ccType=<>)]>, ccReason=<SCANNER_DATA_ADDED>, PreviousccResult=<OK>, sortSchemeId=<-1>, sortSchemeName=<>, logicalDestination=<EXP507>, BatchCountItem=<true>, collectionId=<-1>, goodsId=<>, position=<sorter#0: MS01>, dynamicDataCount=<1>, dynamicData=<{VID=2017020905551665}>, carrierId=<363>, carrierCount=<1>, itemRevisionNumber=<16> ##[
170209 003130 0354 DE(N) ItemHandler.ItemLog event=<DESTINATION_REPLY>, *********************, itemId=<12551665>, globalId=<12551665>, cmcIndex=<961>, sorter=<0: MS01>, state=<CSC: WaitForDestination>, ccReason=<SCANNER_DATA_ADDED>, PendingccResult=<OK>, Pendingstrategy=<notSpecified>, PendingchuteGroup=<[3000]: Parked0>, PendingNotChutedestinationId=<-1>, PendingsortSchemeId=<-1>, PendingsortSchemeName=<>, PendinglogicalDestination=<>, PendinggoodsId=<>, PendingBatchCountItem=<true>, PendingcollectionId=<-1>, position=<sorter#0: MS01>, dynamicDataCount=<1>, dynamicData=<{VID=2017020905551665}>, itemRevisionNumber=<18> ##[
170209 003130 0354 DE(N) ItemHandler.ItemLog event=<REDIRECT_ITEM>, *************************, itemId=<12551665>, globalId=<12551665>, cmcIndex=<961>, sorter=<0: MS01>, state=<CSC: WaitForDestination>, PendingchuteGroup=<[3000]: Parked0>, Pendingstrategy=<notSpecified>, CscdestinationId=<198: EXP507>, CmcdestinationId=<198: EXP507>, position=<sorter#0: MS01>, itemRevisionNumber=<18> ##[
170209 003130 0517 DE(N) ItemHandler.ItemLog event=<REDIRECT_ITEM_REPLY>, *******************, itemId=<12551665>, globalId=<12551665>, cmcIndex=<961>, sorter=<0: MS01>, state=<CSC: WaitForDestination>, CscdestinationId=<3000: Parked0>, chuteGroup=<[3000]: Parked0>, CmcdestinationId=<3000: Parked0>, position=<sorter#0: MS01>, chuteListStartPoint=<198>, itemRevisionNumber=<20> ##[
170209 003130 0520 DE(N) ItemHandler.ItemLog event=<RECONVERT>, *****************************, itemId=<12551665>, globalId=<12551665>, cmcIndex=<961>, sorter=<0: MS01>, state=<CSC: WaitForData>, CscdestinationId=<3000: Parked0>, chuteGroup=<[3000]: Parked0>, CmcdestinationId=<3000: Parked0>, position=<sorter#0: MS01>, chuteListStartPoint=<198>, itemRevisionNumber=<21> ##[
The issue I have is that I can't get the above command line to work because of the issues around single and double quotes when I run it in a windows batch file in DOS, I have other gawk commands working but they use " double quotes can anyone help
Thanks
Phil

Answer 1

Getting quotes and sometimes other special characters correctly through Windows commandline is frequently hard, especially with programs designed for Unix like gawk because there are several different methods of porting Unix-based or otherwise non-Windows C programs to the multiple subtly different implementations of C (or C++) on Windows.
A usually better approach for awk (and perl) in particular is to put the script in a file, something like this, which also allows you to use linebreaks to format it better:
myscript.awk:
{a = gensub(/([0-9]{6} [0-9]{6} [0-9]{4}).*itemId=, "\\2 \\1", "g", $0);
b = split(a, c, " ");
if (c[1] in result) result[c[1]] = gensub(/(.+),(.+)/, "\\1," c[2] " " c[3] " " c[4], "g", result[c[1]]);
else result[c[1]] = c[2] " " c[3] " " c[4] "," c[2] " " c[3] " " c[4] }
END { for (i in result) print i ": " result[i] }
then
gawk -f myscript.awk item.log
PS: your first gensub is clearly missing its closing / presumably after itemId= but even with that change it makes little sense because you have only one capture group but you specify a replacement string that uses two. Conversely your second gensub (the one in the if branch) has two capture groups but uses only one. Please check back with whoever 'helped' you.