I am using Splunk to extract a number of fields from xml data this is contained in a log file. So to limit the search to be MOSTLY the xml file I start the search with this:
sourcetype="name of type here" "RULE"
This returns:
0123459 TripMessage.createMessage MsgSource <?xml version="1.0" encoding="UTF-8"?>
<tmsTrip xmlns="http://ground.fedex.com/schemas/linehaul/trip" xmlns:ns2="http://ground.fedex.com/schemas/linehaul/TMSCommon">
...
The file is very large. This is part of it.
<?xml version="1.0" encoding="UTF-8"?>
<tmsTrip xmlns="http://ground.fedex.com/schemas/linehaul/trip" xmlns:ns2="http://ground.fedex.com/schemas/linehaul/TMSCommon">
<recordType>PURCHASEDLINEHAUL</recordType>
<eventType>APPROVE</eventType>
<tripId>116029927</tripId>
<legId>104257037</legId>
<tripNumber>104257037</tripNumber>
<tripLegNumber>1</tripLegNumber>
<updatedDateGMT>2020-02-20T21:53:39.000Z</updatedDateGMT>
.... more lines here that are not important
<purchasedCost>
<purchasedCostTripSegment>
<purchCostReference>1587040</purchCostReference>
<carrier>FXTR</carrier>
<vendorType>DRAY</vendorType>
<billingMethod>RULE</billingMethod>
<carrierTrailerType>PZ1</carrierTrailerType>
<origin>
<ns2:numberCode>923</ns2:numberCode>
<ns2:locAbbr>RLTO</ns2:locAbbr>
<ns2:address1>330 RESOURCE DRIVE</ns2:address1>
<ns2:address2>LH PHONE 877-851-3543</ns2:address2>
<ns2:daylightSavingsFlag>true</ns2:daylightSavingsFlag>
</origin>
This query selects the xml part text in the logging file and some of the fields are extracted and I can add to a table. (not including the source and sourcetype..)
| xmlkv | table purchCostReference, eventType, carrier, billingMethod
But need more fields that are child elements within the xml data. One of them is the numberCode. I am trying to use xpath to extract these additional fields.
| xmlkv | xpath
"//tmsTrip/purchasedCost/purchasedCostTripSegment/origin/ns2:numberCode" outfield=Origin | table purchCostReference, eventType, carrier, billingMethod, Origin
But no Origin data is returned when I add the field to the table. There is no error. The Origin column is empty.
UPDATE
I think the problem is that I need to add the field parameter. The xml file is within a log text file. I limit the search to get the xml file but not only the xml. So I think xpath is struggling with the other text that is not xml.
UPDATE
I tried creating an extracted field using the wizard of the xml file that is within the logging statement. The xml is huge and I can only select about 30% of it. If anyone is good at regex, maybe they can give me some pointers as to how to complete the regex command to get all of the xml. (I tried updating the props.conf file but do not have permission to add TRUNCATE = 0).
This is the xml file sample:
<?xml version="1.0" encoding="UTF-8"?>
<tmsTrip xmlns="http://ground.fedex.com/schemas/linehaul/trip" xmlns:ns2="http://ground.fedex.com/schemas/linehaul/TMSCommon">
<recordType>PURCHASEDLINEHAUL</recordType>
<eventType>APPROVE</eventType>
<tripId>143642990</tripId>
<legId>129014817</legId>
<tripNumber>129014817</tripNumber>
<tripLegNumber>1</tripLegNumber>
<updatedDateGMT>2020-05-22T00:53:21.000Z</updatedDateGMT>
<origin>
<ns2:numberCode>928</ns2:numberCode>
<ns2:locAbbr>ANAH</ns2:locAbbr>
<ns2:address1>590 E ORANGE THORPE AVENUE</ns2:address1>
<ns2:city>ANAHEIM</ns2:city>
<ns2:stateProvince>CA</ns2:stateProvince>
<ns2:postalCode>92801</ns2:postalCode>
<ns2:locType>FDEG</ns2:locType>
<ns2:numberType>1</ns2:numberType>
<ns2:timeZoneAbbr>PST</ns2:timeZoneAbbr>
<ns2:daylightSavingsFlag>true</ns2:daylightSavingsFlag>
</origin>
<destination>
<ns2:numberCode>89</ns2:numberCode>
<ns2:locAbbr>WOOD</ns2:locAbbr>
<ns2:address1>6000 RIVERSIDE DR</ns2:address1>
<ns2:address2>LH PHONE 732-512-5579</ns2:address2>
<ns2:city>KEASBEY</ns2:city>
<ns2:stateProvince>NJ</ns2:stateProvince>
<ns2:postalCode>08832</ns2:postalCode>
<ns2:locType>FDEG</ns2:locType>
<ns2:numberType>2</ns2:numberType>
<ns2:timeZoneAbbr>EST</ns2:timeZoneAbbr>
<ns2:daylightSavingsFlag>true</ns2:daylightSavingsFlag>
</destination>
<schedDispatchDateGMT>2020-05-22T13:00:00.000Z</schedDispatchDateGMT>
<estimatedArrivalDateGMT>2020-05-26T06:00:00.000Z</estimatedArrivalDateGMT>
<drop/>
<hook/>
<actualRoute>
<routeNumber>308229</routeNumber>
<routeOrderNumber>0</routeOrderNumber>
<totalMiles>2787</totalMiles>
<runTime>54.6</runTime>
</actualRoute>
<standardRoute>
<routeNumber>308229</routeNumber>
<routeOrderNumber>0</routeOrderNumber>
<totalMiles>2787</totalMiles>
<runTime>54.6</runTime>
</standardRoute>
<paidRoute>
<routeNumber>308229</routeNumber>
<routeOrderNumber>0</routeOrderNumber>
<totalMiles>2787</totalMiles>
<runTime>54.6</runTime>
</paidRoute>
<settlement>
<dispatchSettlementEligibility>false</dispatchSettlementEligibility>
</settlement>
<livePkgCount>0.0</livePkgCount>
<tripTollAmount>0.0</tripTollAmount>
<trailers>
<ns2:trailer>
<ns2:trailerNbr>531823</ns2:trailerNbr>
<ns2:trailerPrefix>FDXU</ns2:trailerPrefix>
<ns2:configOrderNbr>1</ns2:configOrderNbr>
<ns2:sealNbr>60606220</ns2:sealNbr>
<ns2:packageWeight>9931.59</ns2:packageWeight>
<ns2:unladenWeight>13870.0</ns2:unladenWeight>
<ns2:totalWeight>23801.59</ns2:totalWeight>
<ns2:packageNumber>703</ns2:packageNumber>
<ns2:percentCube>1</ns2:percentCube>
<ns2:hazmatFlag>false</ns2:hazmatFlag>
<ns2:originPlanned>
<ns2:numberCode>928</ns2:numberCode>
<ns2:locAbbr>ANAH</ns2:locAbbr>
<ns2:address1>590 E ORANGE THORPE AVENUE</ns2:address1>
<ns2:city>ANAHEIM</ns2:city>
<ns2:stateProvince>CA</ns2:stateProvince>
<ns2:postalCode>92801</ns2:postalCode>
<ns2:locType>FDEG</ns2:locType>
<ns2:numberType>1</ns2:numberType>
<ns2:timeZoneAbbr>PST</ns2:timeZoneAbbr>
<ns2:daylightSavingsFlag>true</ns2:daylightSavingsFlag>
</ns2:originPlanned>
<ns2:nextSortLocation>
<ns2:numberCode>89</ns2:numberCode>
<ns2:locAbbr>WOOD</ns2:locAbbr>
<ns2:address1>6000 RIVERSIDE DR</ns2:address1>
<ns2:address2>LH PHONE 732-512-5579</ns2:address2>
<ns2:city>KEASBEY</ns2:city>
<ns2:stateProvince>NJ</ns2:stateProvince>
<ns2:postalCode>08832</ns2:postalCode>
<ns2:locType>FDEG</ns2:locType>
<ns2:numberType>2</ns2:numberType>
<ns2:timeZoneAbbr>EST</ns2:timeZoneAbbr>
<ns2:daylightSavingsFlag>true</ns2:daylightSavingsFlag>
</ns2:nextSortLocation>
<ns2:destinationPlanned>
<ns2:numberCode>89</ns2:numberCode>
<ns2:locAbbr>WOOD</ns2:locAbbr>
<ns2:address1>6000 RIVERSIDE DR</ns2:address1>
<ns2:address2>LH PHONE 732-512-5579</ns2:address2>
<ns2:city>KEASBEY</ns2:city>
<ns2:stateProvince>NJ</ns2:stateProvince>
<ns2:postalCode>08832</ns2:postalCode>
<ns2:locType>FDEG</ns2:locType>
<ns2:numberType>2</ns2:numberType>
<ns2:timeZoneAbbr>EST</ns2:timeZoneAbbr>
<ns2:daylightSavingsFlag>true</ns2:daylightSavingsFlag>
</ns2:destinationPlanned>
<ns2:loads>
<ns2:load>
<ns2:loadId>103718801</ns2:loadId>
<ns2:loadNumber>1</ns2:loadNumber>
<ns2:origin>
<ns2:numberCode>928</ns2:numberCode>
<ns2:locAbbr>ANAH</ns2:locAbbr>
<ns2:numberType>1</ns2:numberType>
</ns2:origin>
<ns2:destination>
<ns2:numberCode>89</ns2:numberCode>
<ns2:locAbbr>WOOD</ns2:locAbbr>
<ns2:address2>LH PHONE 732-512-5579</ns2:address2>
<ns2:numberType>2</ns2:numberType>
</ns2:destination>
<ns2:openDateGMT>2020-05-21T19:53:46.000Z</ns2:openDateGMT>
<ns2:dueOverrideFlag>false</ns2:dueOverrideFlag>
<ns2:hazmatFlag>false</ns2:hazmatFlag>
</ns2:load>
</ns2:loads>
</ns2:trailer>
</trailers>
<dollys/>
<purchasedCost>
<purchasedCostTripSegment>
<purchCostReference>2625998</purchCostReference>
<carrier>BNSF</carrier>
<vendorType>RAIL</vendorType>
<carrierTrailerType>53PC</carrierTrailerType>
<origin>
<ns2:numberCode>4022</ns2:numberCode>
<ns2:locAbbr>BNSF</ns2:locAbbr>
<ns2:address1>3770 EAST WASHINGTON AVENUE</ns2:address1>
<ns2:city>LOS ANGELES</ns2:city>
<ns2:stateProvince>CA</ns2:stateProvince>
<ns2:postalCode>90040</ns2:postalCode>
<ns2:locType>FDEG</ns2:locType>
<ns2:numberType>8</ns2:numberType>
<ns2:timeZoneAbbr>PST</ns2:timeZoneAbbr>
<ns2:daylightSavingsFlag>true</ns2:daylightSavingsFlag>
</origin>
<destination>
<ns2:numberCode>4040</ns2:numberCode>
<ns2:locAbbr>CROX</ns2:locAbbr>
<ns2:address1>NORFOLK SOUTHERN RAILROAD</ns2:address1>
<ns2:address2>125 COUNTY ROAD</ns2:address2>
<ns2:city>CROXTON</ns2:city>
<ns2:stateProvince>NJ</ns2:stateProvince>
<ns2:postalCode>07307</ns2:postalCode>
<ns2:locType>FDEG</ns2:locType>
<ns2:numberType>8</ns2:numberType>
<ns2:timeZoneAbbr>EST</ns2:timeZoneAbbr>
<ns2:daylightSavingsFlag>true</ns2:daylightSavingsFlag>
</destination>
<stopOff>
<ns2:stopOffLocation>
<ns2:numberCode>9996</ns2:numberCode>
<ns2:stateProvince>DU</ns2:stateProvince>
<ns2:postalCode>00000</ns2:postalCode>
<ns2:locType>FDEG</ns2:locType>
<ns2:numberType>1</ns2:numberType>
</ns2:stopOffLocation>
</stopOff>
<schedDispatchDate>2020-05-22T05:00:00.000Z</schedDispatchDate>
<estimatedArrivalDate>2020-05-26T00:59:00.000Z</estimatedArrivalDate>
<billingMethod>RULE</billingMethod>
<STCCCode>4711110</STCCCode>
<planNumber>065</planNumber>
<powerType>1X</powerType>
<powerOnlyFlag>false</powerOnlyFlag>
</purchasedCostTripSegment>
<purchasedCostTripSegment>
<purchCostReference>2625998</purchCostReference>
<carrier>NS</carrier>
<vendorType>RAIL</vendorType>
<carrierTrailerType>53PC</carrierTrailerType>
<origin>
<ns2:numberCode>4061</ns2:numberCode>
<ns2:locAbbr>NSAU</ns2:locAbbr>
<ns2:address1>6300 SOUTH INDIANA AVENUE</ns2:address1>
<ns2:city>CHICAGO</ns2:city>
<ns2:stateProvince>IL</ns2:stateProvince>
<ns2:postalCode>60637</ns2:postalCode>
<ns2:locType>FDEG</ns2:locType>
<ns2:numberType>8</ns2:numberType>
<ns2:timeZoneAbbr>CST</ns2:timeZoneAbbr>
<ns2:daylightSavingsFlag>true</ns2:daylightSavingsFlag>
</origin>
<destination>
<ns2:numberCode>4040</ns2:numberCode>
<ns2:locAbbr>CROX</ns2:locAbbr>
<ns2:address1>NORFOLK SOUTHERN RAILROAD</ns2:address1>
<ns2:address2>125 COUNTY ROAD</ns2:address2>
<ns2:city>CROXTON</ns2:city>
<ns2:stateProvince>NJ</ns2:stateProvince>
<ns2:postalCode>07307</ns2:postalCode>
<ns2:locType>FDEG</ns2:locType>
<ns2:numberType>8</ns2:numberType>
<ns2:timeZoneAbbr>EST</ns2:timeZoneAbbr>
<ns2:daylightSavingsFlag>true</ns2:daylightSavingsFlag>
</destination>
<stopOff>
<ns2:stopOffLocation>
<ns2:numberCode>4040</ns2:numberCode>
<ns2:locAbbr>CROX</ns2:locAbbr>
<ns2:address1>NORFOLK SOUTHERN RAILROAD</ns2:address1>
<ns2:address2>125 COUNTY ROAD</ns2:address2>
<ns2:city>CROXTON</ns2:city>
<ns2:stateProvince>NJ</ns2:stateProvince>
<ns2:postalCode>07307</ns2:postalCode>
<ns2:locType>FDEG</ns2:locType>
<ns2:numberType>8</ns2:numberType>
<ns2:timeZoneAbbr>EST</ns2:timeZoneAbbr>
<ns2:daylightSavingsFlag>true</ns2:daylightSavingsFlag>
</ns2:stopOffLocation>
</stopOff>
<schedDispatchDate>2020-05-22T05:00:00.000Z</schedDispatchDate>
<estimatedArrivalDate>2020-05-26T01:00:00.000Z</estimatedArrivalDate>
<billingMethod>LOCAL</billingMethod>
<STCCCode>4711110</STCCCode>
<planNumber>045</planNumber>
<powerType>1X</powerType>
<powerOnlyFlag>false</powerOnlyFlag>
</purchasedCostTripSegment>
</purchasedCost>
<drivers/>
</tmsTrip>
This is how much the extracted field I can select:
http://ground.fedex.com/schemas/linehaul/trip\" xmlns:ns2=\"http://ground.fedex.com/schemas/linehaul/TMSCommon\">
PURCHASEDLINEHAUL
APPROVE
143642990
129014817
129014817
1
2020-05-22T00:53:21.000Z
928
ANAH
590 E ORANGE THORPE AVENUE
ANAHEIM
CA
92801
FDEG
1
PST
true
This is the regex that Splunk creates to select the above xml
^[^\$\n]*\$\d+\.\w+\s+\w+\s+(?P<xmlMessage><\?\w+\s+\w+="\d+\.\d+"\s+\w+="\w+\-\d+"\?>\s+<\w+\s+\w+="\w+://\w+\.\w+\.\w+/\w+/\w+/\w+"\s+\w+:\w+="\w+://\w+\.\w+\.\w+/\w+/\w+/\w+">\s+<\w+>\w+</\w+>\s+<\w+>\w+</\w+>\s+<\w+>\d+</\w+>\s+<\w+>\d+</\w+>\s+<\w+>\d+</\w+>\s+<\w+>\d+</\w+>\s+<\w+>\d+\-\d+\-\d+\w+:\d+:\d+\.\d+\w+</\w+>\s+<\w+>\s+<\w+:\w+>\d+</\w+:\w+>\s+<\w+:\w+>\w+</\w+:\w+>\s+<\w+:\w+>\d+\s+\w+\s+\w+\s+\w+\s+\w+</\w+:\w+>\s+<\w+:\w+>\w+</\w+:\w+>\s+<\w+:\w+>\w+</\w+:\w+>\s+<\w+:\w+>\d+</\w+:\w+>\s+<\w+:\w+>\w+</\w+:\w+>\s+<\w+:\w+>\d+</\w+:\w+>\s+<\w+:\w+>\w+</\w+:\w+>\s+<\w+:\w+>\w+</\w+:\w+>)
So can I change the above regex to include the entire xml?
UPDATE
I tried extracting a field from the xmlMessage extracted field. The xmlMessage field is above. I used the xpath command to extract recordType. Put the result in a table. This is the command
| xmlkv | xpath field=xmlMessage
"//tmsTrip/recordType" outfield=Origin | table Origin
It returned no results. This xpath command does not work for the simplest of queries.
What am I doing wrong?
Without seeing the rest of the event data, I can't say why the xpath command isn't working.
However, as a workaround, try the following instead of the xmlkv and xpath commands
| rex field=_raw "numberCode>(?<Origin>\d+)</"
This should work fine with mixed text and xml events
| xmlkv | spath path="tmsTrip.purchasedCost.purchasedCostTripSegment.origin.ns2:numberCode" output=Origin
Try spath
I was able to extract the data from the xml using rex and identify each instance of the numberCode using max_match and mvindex
Here is an example of anyone who has this problem:
rex max_match=0 "\<ns2\:numberCode\>(?P<location>[^\<]+)"| eval Segment1_Origin = mvindex(location, 7)
The xml element is ns2:numberCode. It is renamed to location. max_match=0 means unlimited number of instances. The mvindex is zero based. So the 8th instance of the location is set to the variable Segment1_origin
I want to read the data from passage_para tag, after passage_para I have 2 spaces before the expression tag and after the expression tag I have one more space, etc. When I use extract function to get the passage_para tag from the XMLTYPE column it is eliminating all the spaces.
<?xml version="1.0" encoding="UTF-8"?> <item> <information number="sdjsadh" > <response_direction delivery_mode="xcs"> <dparagraph>test</dparagraph> </response_direction> </information> <i_content> <stimulus_reference> <passage> <prose style="1"> <passage_para> <expression> <math xmlns="Math" xmlns:xlink="xlink" display="inline" overflow="scroll"> <mr> <mi>z</mi> <mo>></mo> <mn>0</mn> </mr> </math> </expression> </passage_para> </prose> </passage> </stimulus_reference> </i_content> </item>
which I don't want because it is taking out the spaces. The desired output I need is " z > 0 ".
Note: Between the passage_para tag the child nodes may change, they are not going to be the same.
I have an CSV file like:
1234|abc|val1=12;val2=13;val3=14
1235|xyz|val1=15;val2=16;val3=18
I need to convert it into XML using mfl file.
This is my approach:
<StructFormat name='player' delimOptional='n' repeat='*'>
<FieldFormat name='FieldID' type='String' delimRef='' delim='|' dataDelim='"' delimOptional='n' optional='n' codepage='UTF-8'/>
<FieldFormat name='playerName' type='String' delimRef='' delim='|' dataDelim='"' delimOptional='n' optional='n' codepage='UTF-8'/>
<StructFormat name='extraList' delim='|' delimOptional='n' optional='y'>
<FieldFormat name='extra' type='String' delimRef='' delim='|' delimOptional='n' optional='y' codepage='UTF-8' repeat='*'/>
</StructFormat>
</StructFormat>
I don't know how to implement unlimited amount of couples: val1=12 assigned to each player.
Any help? Thank you!
Appears to be generated by tool, check used wizard for
Group Occurrence -> Repeat Delimiter -> Select this option to indicate that the group will repeat until the specified delimiter is encountered.
as per http://docs.oracle.com/cd/E13214_01/wli/docs70/diuser/fmtdef.htm - dated but still ok
I do have a text file as below:
Employee details.txt
Raja Palit 77489 24 84 12/12/2011
Mathew bargur 77559 25 88 01/12/2011
harin Roy 77787 24 80 12/12/2012
Soumi paul 77251 24 88 11/11/2012
I want the file as below:
Expected file:
Raja,Palit,77489,24,84,12/12/2011
Mathew,bargur,77559,25,88,01/12/2011
harin,Roy,77787,24,80,12/12/2012
Soumi,paul,77251,24,88,11/11/2012
What I tried below:
IO.foreach('D://docs//details.txt') do |line|
splits = line.split("\t")
col1, col2, col3, col4, col5, col6 = splits
splits[6..-1].join(',')
end
Though it seems like a quick way to deal with this sort of data by splitting on whitespace, that will fail if any field contains embedded whitespace. For instance, if the name of the person in the record is something like "Maria Von Trapp" or "Smokey the Bear", the resulting comma-delimited fields will be wrong.
The correct way to deal with this is to parse based on column-field widths, then squeeze and strip whitespace inside those fields, then turn the record into a CSV record.
require 'csv'
require 'scanf' if (RUBY_VERSION >= '1.9.3')
FORMAT = '%15c %d %d %d %10c'
data = <<EOT
Raja Palit 77489 24 84 12/12/2011
Mathew bargur 77559 25 88 01/12/2011
harin Roy 77787 24 80 12/12/2012
Soumi paul 77251 24 88 11/11/2012
Maria Von Trapp 99999 99 99 12/31/2012
Smokey the Bear 99999 99 99 12/31/2012
EOT
data.split("\n").each do |li|
fields = li.scanf(FORMAT)
puts [fields.first.strip, *fields[1 .. -1]].to_csv
end
Which outputs:
Raja Palit,77489,24,84,12/12/2011
Mathew bargur,77559,25,88,01/12/2011
harin Roy,77787,24,80,12/12/2012
Soumi paul,77251,24,88,11/11/2012
Maria Von Trapp,99999,99,99,12/31/2012
Smokey the Bear,99999,99,99,12/31/2012
Note, Ruby 1.9.3 split scanf into its own module, which explains the conditional require.
Strings come with a squeeze method, it squeezes runs of the char(s) in the argument into one char. In this case it reduces the multiple spaces into one space, which is then replaced by a comma:
File.open("test.txt") do |in_file|
File.open("test.csv", 'w') do |out_file| #the 'w' opens the file for writing
in_file.each {|line| out_file << line.squeeze(' ').gsub(' ', ',') }
end # closes test.csv
end # closes test.txt
You could use a regular expression to replace any whitespace characters with a comma:
my_string.sub! /\s/g, ','
If you want to discard empty fields, you could use this:
my_string.sub! /\s+/g, ','
An alternative would be to split it on spaces and join on commas. This will also discard empty fields:
my_string = my_string.split(' ').join(',')
File.open("details.txt", "r+"){|io| io.write(io.read.gsub(/[ \t]+/, ","))}
I am passing a ZPL code to a Zebra printer. And in this ZPL code, I have a portion where it will generate a QR Code. The QR Code's value is from string that I got from using a barcode scanning function in the mobile app im developing for WM6.5.
The string result from the scan looks like this:
Name:John Smith
Gender:Male
Position:Developer
I need to pass this string to a datagrid. So I parse it line by line via "\n" or environment new line. passing it to the datagrid is ok. But when I print out the string result to a QR Code on a Zebra printer. It seems like the new lines are not being included in the QR Code.
If you want to pass in non-printable characters, use the ^FH command to pass it in as a hex value
^XA
^FO100,100
^AD^FH
^FDTilde _7e used for hex^FS
^XZ
This will print out a ~ instead of _7e
ISO-8859-1 encoding characters
HTML OCTL HEX CMP CHR MEANING
------ + ---- + --- + --- + --- + ------------------------------
| \012 | =0A | | | Line feed (ASCII NL, newline)
Use 0x0A instead of \n ?