Golang, parsing xml to struct? - go

I have such xml document and I need to get an array of DATA. I cannot solve this simple task for 4 hours.... want back to node.js :-)
<?xml version="1.0" standalone="no"?>
<RETS ReplyCode="0" ReplyText="Operation Successful" >
<COUNT Records="58951" />
<DELIMITER value="09"/>
<COLUMNS> LN </COLUMNS>
<DATA> 09361303 </DATA>
<DATA> 09333085 </DATA>
<MAXROWS/>
type DATA struct {
DATA string `xml:"DATA"`
}
type Rets struct {
DATA []DATA `xml:"RETS>DATA"`
}
data := &Rets{}
decoder := xml.Unmarshal(body,&data)
fmt.Println(decoder)

There are a couple of tools for turning XML into Go structs. One that is suitable for reading is zek (try it online), and although it is experimental, it can deal with a couple of common XML constructs already.
You can go from a XML file (raw) to a struct with a simple command:
$ curl -sL https://git.io/fN4Pq | zek -e -p -c
This will create a struct plus an example program for testing out the marshalling (the example does very little, it read XML from stdin and turns it into JSON).
Here is an example struct (for some XML take from this repo):
// RETS was generated 2018-09-07 12:11:10 by tir on apollo.local.
type RETS struct {
XMLName xml.Name `xml:"RETS"`
Text string `xml:",chardata"`
ReplyCode string `xml:"ReplyCode,attr"`
ReplyText string `xml:"ReplyText,attr"`
METADATATABLE struct {
Text string `xml:",chardata"`
Resource string `xml:"Resource,attr"`
Class string `xml:"Class,attr"`
Version string `xml:"Version,attr"`
Date string `xml:"Date,attr"`
COLUMNS string `xml:"COLUMNS"` // MetadataEntryID SystemNam...
DATA []string `xml:"DATA"` // 7 City City Ci ...
} `xml:"METADATA-TABLE"`
}
Disclaimer: I wrote zek.

From the xml.Unmarshal docs
If the XML element contains character data, that data is
accumulated in the first struct field that has tag ",chardata".
The struct field may have type []byte or string.
If there is no such field, the character data is discarded.
Use
type DATA struct {
DATA string `xml:",chardata"`
}
type Rets struct {
DATA []DATA `xml:"DATA"`
}
Or in this simple case, you can just use
type Rets struct {
DATA []string `xml:"DATA"`
}
Which collects the charadata from a list of elements by default

Related

Dynamically Create Structs in Golang

So I am working with an external API, whose responses I wanted to parse. The incoming responses are of a fixed format i.e.
type APIResponse struct {
Items []interface{} `json:"items"`
QuotaMax int `json:"quota_max"`
QuotaRemaining int `json:"quota_remaining"`
}
So for each response I am parsing the items. Now the items can be of diff types as per the request. It can be a slice of sites, articles, etc. Which have their individual models. like:
type ArticleInfo struct {
ArticleId uint64 `json:"article_id"`
ArticleType string `json:"article_type"`
Link string `json:"link"`
Title string `json:"title"`
}
type SiteInfo struct {
Name string `json:"name"`
Slug string `json:"slug"`
SiteURL string `json:"site_url"`
}
Is there any way, when parsing the input define the type of Items in APIResponse. I don't want to create separate types for individual responses.
Basically want to Unmarshall any incoming response into the APIResponse struct.
Change type of the Items field to interface{}:
type APIResponse struct {
Items interface{} `json:"items"`
...
}
Set the response Items field to pointer of the desired type. Unmarshal to the response:
var articles []ArticleInfo
response := APIResponse{Items: &articles}
err := json.Unmarshal(data, &response)
Access the articles using variable articles.
Run an example on the playground.

Going through multiple XML files to find a specific value

I'm trying to find a solution where I can search through 1189 XML files to find a specific value that a user provides.
As an example, the user is looking for a postal code, 8913CK, which can be found in any of the 1189 XML files. What would be the best way to approach this? I've tried looping through the files, which results in slow and inefficient responses and have tried setting up channels and workers which made the results a lot quicker, but still not desirable and sometimes no results showed up at all (this might be because I'm not as familiar with GoLang).
The struct of the XML file is as follows:
type NumPostal struct {
BagObject struct {
Nummeraanduiding struct {
Identificatie struct {
Text string `xml:",chardata"`
Domein string `xml:"domein,attr"`
} `xml:"identificatie"`
Huisnummer struct {
Text string `xml:",chardata"`
} `xml:"huisnummer"`
Huisletter struct {
Text string `xml:",chardata"`
} `xml:"huisletter"`
Postcode struct {
Text string `xml:",chardata"`
} `xml:"postcode"`
LigtAan struct {
OpenbareRuimteRef struct {
Text string `xml:",chardata"`
} `xml:"OpenbareRuimteRef"`
} `xml:"ligtAan"`
Huisnummertoevoeging struct {
Text string `xml:",chardata"`
} `xml:"huisnummertoevoeging"`
} `xml:"Nummeraanduiding"`
} `xml:"bagObject"`
}
Effectively querying the BAG files directly isn't doable in a reasonable time due to the amount of unindexed data. I recommend setting up BAG-Extract, an existing project which allows you to import the BAG dataset into a postgres database which you can then query.

Using base of embedded types in Go

I'm new to Go and working with a set of types generated by gowsdl based on the NetSuite SuiteTalk web service definition. It has created the following types:
type BaseRef struct {
XMLName xml.Name `xml:"urn:core_2018_2.platform.webservices.netsuite.com BaseRef"`
Name string `xml:"name,omitempty"`
}
type RecordRef struct {
XMLName xml.Name `xml:"urn:core_2018_2.platform.webservices.netsuite.com RecordRef"`
*BaseRef
InternalId string `xml:"internalId,attr,omitempty"`
ExternalId string `xml:"externalId,attr,omitempty"`
Type *RecordType `xml:"type,attr,omitempty"`
}
type GetRequest struct {
XMLName xml.Name `xml:"urn:messages_2018_2.platform.webservices.netsuite.com GetRequest"`
BaseRef *BaseRef `xml:"baseRef,omitempty"`
}
As I try to use these types it is unhappy in my ability to use the specific type of reference record in a GetRequest structure which is looking for a BaseRef which is what RecordRef is based on.
var partnerRecordType RecordType
partnerRecordType = RecordTypePartner
recordRef := RecordRef{
Type:&partnerRecordType,
InternalId:internalIdString,
}
var getRequest GetRequest
getRequest.BaseRef = &recordRef
The error I get is on the last line is:
cannot use &recordRef (type *RecordRef) as type *BaseRef in assignment
Any thoughts on how to proceed?
Go does not support polymorphism in this way, neither does it support inheritance in way that say C# or Java does. Embedded structs are quite literally just embedded, they do not create a classical inheritance hierarchy. They simply give the wrapping struct all of the exposed methods and fields of the embedded struct (with some subtle caveats - check out the spec)
That said, in your example RecordRef is not related to BaseRef in terms of its type, instead it could be considered to "contain" a pointer to a BaseRef. In order for your program to compile, you will have explicitly assign the embedded BaseRef like so:
getRequest.BaseRef = &recordRef.BaseRef
As this code you are referencing has been auto generated from a WSDL, it may be a little cumbersome to update the GetRequest to provide a BaseRef like data structure in a more polymorphic, flexible fashion, but in order to do so you will need to use Go interfaces.
You could update the GetRequest to have a method with accepts in interface type, say XmlRef which would expose getters that can derive the data you need to assign to the GetRequest
For example
type XmlRef interface {
Name() string
InternalID() string
ExternalID() string
}
func (r *GetRequest) SetRef(ref XmlRef) {
r.BaseRef.Name = ref.Name()
// etc...
}
Then simply implement the interface for RecordRef and any other structs that would need to be used in this context.
If I understand you right, you are looking for a method to access a struct's embbed field. You can simply use recordRef.BaseRef for that purpose.
Further reading: https://golang.org/ref/spec#Struct_types
It is possible to connect to netsuite with the output of go2wsdl but many tweaks are required. A full proof of concept is available here:
https://github.com/mk-j/go-netsuite-soap
But here are some examples of some of the tweaks required:
sed -i 's/urn:.*1.lists.webservices.netsuite.com //' netsuite.go
Tweaking BaseRef and GetRequest:
type BaseRef struct {
Name string `xml:"name,omitempty" json:"name,omitempty"`
InternalId string `xml:"internalId,attr,omitempty" json:"internalId,omitempty"`
ExternalId string `xml:"externalId,attr,omitempty" json:"externalId,omitempty"`
Type *RecordType `xml:"type,attr,omitempty" json:"type,omitempty"`
XsiType string `xml:"xsi:type,attr,omitempty"`
}
type GetRequest struct {
XMLName xml.Name `xml:"get"`
XmlNSXSI string `xml:"xmlns:xsi,attr"`
XmlNSPC string `xml:"xmlns:platformCore,attr"`
BaseRef *BaseRef `xml:"baseRef,omitempty" json:"baseRef,omitempty"`
}
Then this code produced an successful xml response from the soap server:
recordType := netsuite.RecordTypePartner
baseRef :=netsuite.BaseRef{
InternalId:"1234",
Type:&recordType,
XsiType:"platformCore:RecordRef",
}
request := netsuite.GetRequest{
BaseRef:&baseRef,
XmlNSXSI: "http://www.w3.org/2001/XMLSchema-instance",
XmlNSPC:"urn:core_2022_1.platform.webservices.netsuite.com",
}
getResponse, err := service.Get(&request)
Which made the raw request:
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Header><tokenPassport>[redacted]</tokenPassport></soap:Header>
<soap:Body>
<get xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:platformCore="urn:core_2022_1.platform.webservices.netsuite.com">
<baseRef internalId="1234" type="partner" xsi:type="platformCore:RecordRef"></baseRef>
</get>
</soap:Body>
</soap:Envelope>
Sources used: https://www.zuar.com/blog/netsuite-api-exploring-soap/

How to write dynamically typed data to field within a struct

I have the following response struct that I want to use as a base wrapper for responding to API calls users send to me.
type Response struct {
Data ??? `json:"data,omitempty"`
Time int64 `json:"time,omitempty"`
Message string `json:"message,omitempty"`
}
The type of the Data field is varying and could be a map[string]*CustomStruct1 map[string*CustomStruct2 or an []CustomStruct3.
What is the best way to attack this kind of problem?
One option is to simply treat "Data" as the interface{} (any) type, instead of using your custom structs, and handle the resulting values based on inspection of what actually got unmarshaled. Of course, once you've inspected the data to determine what type it should be you could convert it into the appropriate strong type after the fact.
type Response struct {
Data interface{} `json:"data,omitempty"`
Time int64 `json:"time,omitempty"`
Message string `json:"message,omitempty"`
}
Another option is to embed the "Response" struct into specialized structs that look for your custom types and unmarshal into the appropriate one, assuming you know which one you've got ahead of time:
type BaseResponse struct {
Time int64 `json:"time,omitempty"`
Message string `json:"message,omitempty"`
}
type Response1 struct {
BaseResponse
Data map[string]*CustomStruct1 `json:"data"`
}
type Response2 struct {
BaseResponse
Data map[string]*CustomStruct2 `json:"data"`
}
// etc...
Ultimately, the unmarshaler cannot pick a varying type based on the document that gets unmarshaled, it only deserializes JSON values into structures either defined explicitly by you or into generic ones.
You could try to use reflection, but it wouldn't be very idiomatic.

What is the use of tag in golang struct?

I don't understand the significance of struct tags. I have been looking them, and noticed that they can be used with reflect package. But I don't know any practical uses of them.
type TagType struct { // tags
field1 bool “An important answer”
field2 string “The name of the thing”
field3 int “How much there are”
}
The use of tags strongly depends on how your struct is used.
A typical use is to add specifications or constraints for persistence or serialisation.
For example, when using the JSON parser/encoder, tags are used to specify how the struct will be read from JSON or written in JSON, when the default encoding scheme (i.e. the name of the field) isn't to be used.
Here are a few examples from the json package documentation :
// Field is ignored by this package.
Field int `json:"-"`
// Field appears in JSON as key "myName".
Field int `json:"myName"`
// Field appears in JSON as key "myName" and
// the field is omitted from the object if its value is empty,
// as defined above.
Field int `json:"myName,omitempty"`
// Field appears in JSON as key "Field" (the default), but
// the field is skipped if empty.
// Note the leading comma.
Field int `json:",omitempty"`
Example of use is json encoding/decoding in encoding/json:
type TagType struct {
field1 bool `json:"fieldName"`
field2 string `json:",omitempty"`
}
More details in documentation: encoding/json
You can also use XML struct tags as shown below
type SearchResult struct {
Title string `xml:"title,attr"`
Author string `xml:"author,attr"`
Year string `xml:"hyr,attr"`
ID string `xml:"owi,attr"`
}

Resources