I'm trying to find a solution where I can search through 1189 XML files to find a specific value that a user provides.
As an example, the user is looking for a postal code, 8913CK, which can be found in any of the 1189 XML files. What would be the best way to approach this? I've tried looping through the files, which results in slow and inefficient responses and have tried setting up channels and workers which made the results a lot quicker, but still not desirable and sometimes no results showed up at all (this might be because I'm not as familiar with GoLang).
The struct of the XML file is as follows:
type NumPostal struct {
BagObject struct {
Nummeraanduiding struct {
Identificatie struct {
Text string `xml:",chardata"`
Domein string `xml:"domein,attr"`
} `xml:"identificatie"`
Huisnummer struct {
Text string `xml:",chardata"`
} `xml:"huisnummer"`
Huisletter struct {
Text string `xml:",chardata"`
} `xml:"huisletter"`
Postcode struct {
Text string `xml:",chardata"`
} `xml:"postcode"`
LigtAan struct {
OpenbareRuimteRef struct {
Text string `xml:",chardata"`
} `xml:"OpenbareRuimteRef"`
} `xml:"ligtAan"`
Huisnummertoevoeging struct {
Text string `xml:",chardata"`
} `xml:"huisnummertoevoeging"`
} `xml:"Nummeraanduiding"`
} `xml:"bagObject"`
}
Effectively querying the BAG files directly isn't doable in a reasonable time due to the amount of unindexed data. I recommend setting up BAG-Extract, an existing project which allows you to import the BAG dataset into a postgres database which you can then query.
Related
This is API response data, looks like this.
{
"result":1,
"message":"",
"pds": [
{
"state":"Y",
"code":13,
"name":"AAA",
"price":39900,
},
{
"state":"Y",
"code":12,
"name":"BBB",
"price":38000,
}
],
"request":
{
"op":"new",
}
}
How can I get this data in Go lang?
I tried json.Unmarshall and get with map[string]interface{} but it looks like I used the wrong type to get the data.
Should I use structure??
You need to create a struct to handle this properly if you don't want the json.Unmarshall output to be a map[string]interface{}.
If you map this JSON object to a Go struct you find the following structure:
type APIResponse struct {
Result int `json:"result"`
Message string `json:"message"`
Pds []struct {
State string `json:"state"`
Code int `json:"code"`
Name string `json:"name"`
Price float64 `json:"price"`
} `json:"pds"`
Request struct {
Op string `json:"op"`
} `json:"request"`
}
You also can find a great tool to convert JSON objects to Go structs here
If you don't want to use the native json.Unmarshall, here's a good option to use the package go-simplejson.
Documentation
I have a yaml file of a certain configuration which is read by the go program to build a struct object.
The struct itself looks like this
type YamlConfig struct {
Attributes map[string]struct {
Label string `yaml:"label"`
Type string `yaml:"type"`
Presence bool `yaml:"presence"`
Uniqueness bool `yaml:"uniqueness"`
Strip bool `yaml:"strip"`
Values []string `yaml:"values"`
Default string `yaml:"default"`
Multi bool `yaml:"multi"`
Searchable bool `yaml:"searchable"`
Pattern struct {
Value string `yaml:"value"`
Message string `yaml:"message"`
} `yaml:"pattern"`
Length struct {
Min int `yaml:"min"`
Max int `yaml:"max"`
} `yaml:"length"`
} `yaml:"attributes"`
}
I have that map of Attributes that can be anything from "name" to "whatever", that should represent and db table columns with their types.
My question is - can I somehow take that object, which is quite dynamic and may not include the data for all the attributes' properties and convert it somehow into a usable ORM model with Gorm or something?
Should I always define a model struct or is it possible to build structs dynamically?
I think you are already using yaml.Unmarshal() for parsing your yaml config, right?
When you unmarshall yaml into a struct, you can use empty interface instead of complete yaml struct
var config map[string]interface{}
yaml.Unrmarshal(configFile, &config)
fmt.Println(config["label"])
Let's say I have the following types.
type Contract struct {
Id string `json:"id" gorm:"column:uuid"`
Name string `json:"name" gorm:"column:name"`
Description string `json:"descr" gorm:"column:descr"`
ContractTypeId int `json:"contract_type_id" gorm:"column:contract_type_id"`
}
type ContractModel struct {
Id string `json:"id" gorm:"column:uuid"`
Name string `json:"name" gorm:"column:name"`
Description string `json:"descr" gorm:"column:descr"`
}
I have a SQL query using gorm to scan results into a contract object.
How can I map the values from the contract object into contractModel object?
I tried using the package go-automapper as such:
automapper.Map(contract, ContractModel{})
I want to drop the ContractTypeId.
Can I do this for multiple types in a list?
var contractModels []ContractModel
automapper.Map(contracts, &contractModels)
You can do either:
models := []ContractModel{}
automapper.Map(contracts, &models)
Or call automapper.Map in a loop:
models := make([]ContractModel, len(contracts))
for i := range contracts {
automapper.Map(contracts[i], &models[i])
}
You should be aware that automapper uses reflection behind the scenes and thus is much slower than straight forward non-polymorphic copying like #ThinkGoodly suggests. It's a totally fine solution if performance isn't top priority though.
Assume a JSON object with the general format
"accounts": [
{
"id": "<ACCOUNT>",
"tags": []
}
]
}
I can create a struct with corresponding json tags to decode it like so
type AccountProperties struct {
ID AccountID `json:"id"`
MT4AccountID int `json:"mt4AccountID,omitempty"`
Tags []string `json:"tags"`
}
type Accounts struct {
Accounts []AccountProperties `json:"accounts"`
}
But the last struct with just one element seems incorrect to me. Is there a way I could simply say type Accounts []AccountProperties `json:"accounts"` instead of creating an entire new struct just to decode this object?
You need somewhere to store the json string accounts. Using a:
var m map[string][]AccountProperties
suffices, though of course you then need to know to use the string literal accounts to access the (single) map entry thus created:
type AccountProperties struct {
ID string `json:"id"`
MT4AccountID int `json:"mt4AccountID,omitempty"`
Tags []string `json:"tags"`
}
func main() {
var m map[string][]AccountProperties
err := json.Unmarshal([]byte(data), &m)
fmt.Println(err, m["accounts"])
}
See complete Go Playground example (I had to change the type of ID to string and fix the missing { in the json).
As Dave C points out in comments, this is no shorter than just using an anonymous struct type:
var a struct{ Accounts []AccountProperties }
in terms of the Unmarshall call (and when done this way it's more convenient to use). Should you want to use an anonymous struct like this in a json.Marshall call, you'll need to tag its single element to get a lowercase encoding: without a tag it will be called "Accounts" rather than "accounts".
(I don't claim the map method to be better, just an alternative.)
I have such xml document and I need to get an array of DATA. I cannot solve this simple task for 4 hours.... want back to node.js :-)
<?xml version="1.0" standalone="no"?>
<RETS ReplyCode="0" ReplyText="Operation Successful" >
<COUNT Records="58951" />
<DELIMITER value="09"/>
<COLUMNS> LN </COLUMNS>
<DATA> 09361303 </DATA>
<DATA> 09333085 </DATA>
<MAXROWS/>
type DATA struct {
DATA string `xml:"DATA"`
}
type Rets struct {
DATA []DATA `xml:"RETS>DATA"`
}
data := &Rets{}
decoder := xml.Unmarshal(body,&data)
fmt.Println(decoder)
There are a couple of tools for turning XML into Go structs. One that is suitable for reading is zek (try it online), and although it is experimental, it can deal with a couple of common XML constructs already.
You can go from a XML file (raw) to a struct with a simple command:
$ curl -sL https://git.io/fN4Pq | zek -e -p -c
This will create a struct plus an example program for testing out the marshalling (the example does very little, it read XML from stdin and turns it into JSON).
Here is an example struct (for some XML take from this repo):
// RETS was generated 2018-09-07 12:11:10 by tir on apollo.local.
type RETS struct {
XMLName xml.Name `xml:"RETS"`
Text string `xml:",chardata"`
ReplyCode string `xml:"ReplyCode,attr"`
ReplyText string `xml:"ReplyText,attr"`
METADATATABLE struct {
Text string `xml:",chardata"`
Resource string `xml:"Resource,attr"`
Class string `xml:"Class,attr"`
Version string `xml:"Version,attr"`
Date string `xml:"Date,attr"`
COLUMNS string `xml:"COLUMNS"` // MetadataEntryID SystemNam...
DATA []string `xml:"DATA"` // 7 City City Ci ...
} `xml:"METADATA-TABLE"`
}
Disclaimer: I wrote zek.
From the xml.Unmarshal docs
If the XML element contains character data, that data is
accumulated in the first struct field that has tag ",chardata".
The struct field may have type []byte or string.
If there is no such field, the character data is discarded.
Use
type DATA struct {
DATA string `xml:",chardata"`
}
type Rets struct {
DATA []DATA `xml:"DATA"`
}
Or in this simple case, you can just use
type Rets struct {
DATA []string `xml:"DATA"`
}
Which collects the charadata from a list of elements by default