How to read config.json in Go lang? - go

I have used the following code in filLib.go:
func LoadConfiguration(filename string) (Configuration, error) {
bytes, err := ioutil.ReadFile(filename)
if err != nil {
return Configuration{}, err
}
var c Configuration
err = json.Unmarshal(bytes, &c)
if err != nil {
return Configuration{}, err
}
return c, nil
}
But ioutil.ReadFile(filename) return *os.PathError.
Both the files config.json and filLib.go are in same folder.

The path of *.go file is not directly relevant to the working directory of the executing compiled code. Verify where your code thinks it actually is (compare to where you think it should be :).
import(
"os"
"fmt"
"log"
)
func main() {
dir, err := os.Getwd()
if err != nil {
log.Fatal(err)
}
fmt.Println(dir)
}

The issue might be with the filename you're providing. Below is the code sample that working fine for me.
func loadConfig() {
var AppConfig Conf
raw, err := ioutil.ReadFile("conf/conf.json")
if err != nil {
log.Println("Error occured while reading config")
return
}
json.Unmarshal(raw, &AppConfig)
}

I found this library enter link description here
It is a very simple and easy to use configuration library, allowing Json based config files for your Go application. Configuration provider reads configuration data from config.json file. You can get the string value of a configuration, or bind an interface to a valid JSON section by related section name convention parameter.
Consider the following config.json file:
{
"ConnectionStrings": {
"DbConnection": "Server=.;User Id=app;Password=123;Database=Db",
"LogDbConnection": "Server=.;User Id=app;Password=123;Database=Log"
},
"Caching": {
"ApplicationKey": "key",
"Host": "127.0.01"
},
"Website": {
"ActivityLogEnable": "true",
"ErrorMessages": {
"InvalidTelephoneNumber": "Invalid Telephone Number",
"RequestNotFound": "Request Not Found",
"InvalidConfirmationCode": "Invalid Confirmation Code"
}
},
"Services": {
"List": [
{
"Id": 1,
"Name": "Service1"
},
{
"Id": 2,
"Name": "Service2"
},
{
"Id": 3,
"Name": "Service3"
}
]
}
}
The following code displays how to access some of the preceding configuration settings. You can get config value via GetSection function with specifying Json sections as string parameter split by ":"
c, err := jsonconfig.GetSection("ConnectionStrings:DbConnection")
Any valid Json is a valid configuration type. You can also bind a struct via jsonconfig. For example, Caching configuration can be bind to valid struct:
type Caching struct {
ApplicationKey string
Host string
}
var c Caching
err = jsonconfig.Bind(&c, "Caching")

Related

Golang - Access to struct property dynamically by name

I have struct of configuration like this(in short version):
type Config struct {
Environment string
Service1 Service
Service2 Service
}
type Service struct {
CRC string
Cards Cards
}
type Cards struct {
GBP CardCfg
USD CardCfg
}
type CardCfg struct {
CRC string
}
func Cfg() *Config {
return &Config{
Environment: os.Getenv("ENVIRONMENT"),
Service1: Service{
CRC: os.Getenv("Service1_CRC"),
Cards: Cards{
GBP: CardCfg{
CRC: os.Getenv("Service1_CARD_GBP_CRC"),
},
USD: CardCfg{
CRC: os.Getenv("Service1_CARD_USD_CRC"),
},
},
},
Service2: Service{
CRC: os.Getenv("Service2_CRC"),
Cards: Cards{
GBP: CardCfg{
CRC: os.Getenv("Service2_CARD_GBP_CRC"),
},
USD: CardCfg{
CRC: os.Getenv("Service2_CARD_USD_CRC"),
},
},
},
}
}
I try to get access to service crc or service card crc by variable like this:
variable := "Service1"
currency := "EUR"
cfg := config.Cfg()
crc := cfg[variable].cards[currency] // DOESN'T WORK
I always tried with map, like this:
package main
import "fmt"
type Config map[string]interface{}
func main() {
config := Config{
"field": "value",
"service1": Config{
"crc": "secret1",
"cards": Config{
"crc": "secret2",
},
},
}
fmt.Println(config["WT"].(Config)["cards"].(Config)["crc"]) //WORK
}
but it looks wierd for me. Do you know better way to write config? It's possible to use struct? I come form Ruby planet, Golang is new for me.
edit:
I receive messages from rabbit queue, based on them I create a payment. Unfortunately, various payment methods require "own" authorization (crc and merchantId). Call looks like this:
trn, err := p24Client.RegisterTrn(context.Background(), &p24.RegisterTrnReq{
CRC: cfg[payinRequested.Service].cards[payinRequested.Currency].CRC,
MerchantId: cfg[payinRequested.Service].cards[payinRequested.Currency].MerchantId,
PosId: cfg[payinRequested.Service].cards[payinRequested.Currency].MerchantId,
SessionId: payinRequested.PaymentId,
Amount: payinRequested.Amount,
Currency: payinRequested.Currency,
Description: payinRequested.Desc,
Email: payinRequested.Email,
Method: payinRequested.BankId,
UrlReturn: payinRequested.ReturnUrl,
UrlStatus: cfg.StatusUri,
UrlCardPaymentNotification: cfg.CardStatusUri,
})
Any ideas on how to do it right?
Ignoring the reflect package, the simple answer is: you can't. You cannot access struct fields dynamically (using string variables). You can, use variables on a map, because accessing data in a map is a hashtable lookup. A struct isn't.
I will reiterate the main point of my comments though: What you're seemingly trying to do is using environment variables to set values on a config struct. This is very much a solved problem. We've been doing this for years at this point. I did a quick google search and found this repo which does exactly what you seem to want to do (and more): called configure
With this package, you can declare your config struct like this:
package config
type Config struct {
Environment string `env:"ENVIRONMENT" cli:"env" yaml:"environment"`
Services []*Service `env:"SERVICE" cli:"service" yaml:"service"`
serviceByName map[string]*Service
}
Then, to load from environment variables:
func LoadEnv() (*Config, err) {
c := Config{
serviceByName: map[string]*Service{},
} // set default values if needed
if err := configure.ParseEnv(&c); err != nil {
return nil, err
}
// initialise convenience fields like serviceByName:
for _, svc := range c.Services {
c.serviceByName[svc.Name] = svc
}
return &c, nil
}
// ServiceByName returns a COPY of the config for a given service
func (c Config) ServiceByName(n string) (Service, error) {
s, ok := c.serviceByName[n]
if !ok {
return nil, errrors.New("service with given name does not exist")
}
return *s, nil
}
You can also define a single Load function that will prioritise one type of config over the other. With these tags, we're supporting environment variables, a Yaml file, and command line arguments. Generally command line arguments override any of the other formats. As for Yaml vs environment variables, you could argue both ways: an environment variable like ENVIRONMENT isn't very specific, and could easily be used by multiple processes by mistake. Then again, if you deploy things properly, that shouldn't be an issue, so for that reason, I'd prioritise environment variables over the Yaml file:
func Load(args []string) (*Config, error) {
c := &Config{
Environment: "devel", // default
serviceByName: map[string]*Service{},
}
if err := configure.ParseYaml(c); err != nil {
return nil, err
}
if err := configure.ParseEnv(c); err != nil {
return nil, err
}
if len(args) > 0 {
if err := configure.ParseCommanLine(c, args); err != nil {
return nil, err
}
}
// initialise convenience fields like serviceByName:
for _, svc := range c.Services {
c.serviceByName[svc.Name] = svc
}
return &c, nil
}
Then in your main package:
func main() {
cfg, err := config.Load(os.Args[1:])
if err != nil {
fmt.Printf("Failed to load config: %v\n", err)
os.Exit(1)
}
wtCfg, err := config.ServiceByName("WT")
if err != nil {
fmt.Printf("WT service not found: %v\n", err)
return
}
fmt.Printf("%#v\n", wtCfg)
}

Unmarshal nested GRPC structure in go

We want to unmarshal (in golang) a GRPC message and transform it into a map[string]interface{} to further process it. After using this code:
err := ptypes.UnmarshalAny(resource, config)
configMarshal, err := json.Marshal(config)
var configInterface map[string]interface{}
err = json.Unmarshal(configMarshal, &configInterface)
we get the following structure:
{
"name": "envoy.filters.network.tcp_proxy",
"ConfigType": {
"TypedConfig": {
"type_url": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
"value": "ChBCbGFja0hvbGVDbHVzdGVyEhBCbGFja0hvbGVDbHVzdGVy"
}
}
}
Where the TypedConfig field remains encoded. How can we decode the TypedConfig field? We know the type_url and we know the value, but to unmarshal the field, it needs to be of the pbany.Any type. But because the TypedConfig structure is a map[string] interface {}, our program either fails to compile, or it crashes, complaining that it is expecting a pbany.Any type, but instead it is getting a map[string] interface {}.
We have the following questions:
Is there a way to turn the structure under TypedConfig into a pbany.Any type that can be subsequently unmarshalled?
Is there a way to recursively unmarshal the entire GRPC message?
Edit (provide more information about the code, schemas/packages used)
We are looking at the code of xds_proxy.go here: https://github.com/istio/istio/blob/master/pkg/istio-agent/xds_proxy.go
This code uses a *discovery.DiscoveryResponse structure in this function:
func forwardToEnvoy(con *ProxyConnection, resp *discovery.DiscoveryResponse) {
The protobuf schema for discovery.DiscoveryResponse (and every other structure used in the code) is in the https://github.com/envoyproxy/go-control-plane/ repository in this file: https://github.com/envoyproxy/go-control-plane/blob/main/envoy/service/discovery/v3/discovery.pb.go
We added code to the forwardToEnvoy function to see the entire unmarshalled contents of the *discovery.DiscoveryResponse structure:
var config proto.Message
switch resp.TypeUrl {
case "type.googleapis.com/envoy.config.route.v3.RouteConfiguration":
config = &route.RouteConfiguration{}
case "type.googleapis.com/envoy.config.listener.v3.Listener":
config = &listener.Listener{}
// Six more cases here, truncated to save space
}
for _, resource := range resp.Resources {
err := ptypes.UnmarshalAny(resource, config)
if err != nil {
proxyLog.Infof("UnmarshalAny err %v", err)
return false
}
configMarshal, err := json.Marshal(config)
if err != nil {
proxyLog.Infof("Marshal err %v", err)
return false
}
var configInterface map[string]interface{}
err = json.Unmarshal(configMarshal, &configInterface)
if err != nil {
proxyLog.Infof("Unmarshal err %v", err)
return false
}
}
And this works well, except that now we have these TypedConfig fields that are still encoded:
{
"name": "virtualOutbound",
"address": {
"Address": {
"SocketAddress": {
"address": "0.0.0.0",
"PortSpecifier": {
"PortValue": 15001
}
}
}
},
"filter_chains": [
{
"filter_chain_match": {
"destination_port": {
"value": 15001
}
},
"filters": [
{
"name": "istio.stats",
"ConfigType": {
"TypedConfig": {
"type_url": "type.googleapis.com/udpa.type.v1.TypedStruct",
"value": "CkF0eXBlLmdvb2dsZWFwaXMuY29tL2Vudm95LmV4dGVuc2lvbnMuZmlsdG"
}
}
},
One way to visualize the contents of the TypedConfig fields is to use this code:
for index1, filterChain := range listenerConfig.FilterChains {
for index2, filter := range filterChain.Filters {
proxyLog.Infof("Listener %d: Handling filter chain %d, filter %d", i, index1, index2)
switch filter.ConfigType.(type) {
case *listener.Filter_TypedConfig:
proxyLog.Infof("Found TypedConfig")
typedConfig := filter.GetTypedConfig()
proxyLog.Infof("typedConfig.TypeUrl = %s", typedConfig.TypeUrl)
switch typedConfig.TypeUrl {
case "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy":
tcpProxyConfig := &tcp_proxy.TcpProxy{}
err := typedConfig.UnmarshalTo(tcpProxyConfig)
if err != nil {
proxyLog.Errorf("Failed to unmarshal TCP proxy configuration")
} else {
proxyLog.Infof("TcpProxy Config for filter chain %d filter %d: %s", index1, index2, prettyPrint(tcpProxyConfig))
}
But then the code becomes very complex, as we have a large number of structures, and these structures can occur in different order in the messages.
So we wanted to get a generic way of unmarshalling these TypedConfig message by using pbAny, and hence our questions.

Unable to read terraform variables.tf files into may go program

I am attempting to write a go program that reads in a terraform variables.tf and populates a struct for later manipulation. However, I am getting errors when attempting to "parse" the file. I Am hoping someone can tell me what I am doing wrong:
Code:
package main
import (
"fmt"
"io/ioutil"
"log"
"os"
"github.com/hashicorp/hcl/v2"
"github.com/hashicorp/hcl/v2/gohcl"
"github.com/hashicorp/hcl/v2/hclsyntax"
)
type Config struct {
Upstreams []*TfVariable `hcl:"variable,block"`
}
type TfVariable struct {
Name string `hcl:",label"`
// Default string `hcl:"default,optional"`
Type string `hcl:"type"`
Description string `hcl:"description,attr"`
// validation block
Sensitive bool `hcl:"sensitive,optional"`
}
func main() {
readHCLFile("examples/string.tf")
}
// Exits program by sending error message to standard error and specified error code.
func abort(errorMessage string, exitcode int) {
fmt.Fprintln(os.Stderr, errorMessage)
os.Exit(exitcode)
}
func readHCLFile(filePath string) {
content, err := ioutil.ReadFile(filePath)
if err != nil {
log.Fatal(err)
}
fmt.Printf("File contents: %s", content) // TODO: Remove me
file, diags := hclsyntax.ParseConfig(content, filePath, hcl.Pos{Line: 1, Column: 1})
if diags.HasErrors() {
log.Fatal(fmt.Errorf("ParseConfig: %w", diags))
}
c := &Config{}
diags = gohcl.DecodeBody(file.Body, nil, c)
if diags.HasErrors() {
log.Fatal(fmt.Errorf("DecodeBody: %w", diags))
}
fmt.Println(c) // TODO: Remove me
}
ERROR
File contents: variable "image_id" {
type = string
description = "The id of the machine image (AMI) to use for the server."
sensitive = false
}
variable "other_id" {
type = string
description = "The id of the machine image (AMI) to use for the server."
sensitive = true
}
2021/03/13 19:55:49 DecodeBody: examples/string.tf:2,17-23: Variables not allowed; Variables may not be used here., and 3 other diagnostic(s)
exit status 1
Stack driver question is sadly for hcl1
Blog post I am referencing.
It looks like it's a bug/feature of the library, since as soon as you change string to "string", e.g.,
variable "image_id" {
type = string
...
to
variable "image_id" {
type = "string"
...
gohcl.DecodeBody succeeds.
--- UPDATE ---
So, they do use this package in Terraform, BUT they custom-parse configs, i.e., they don't use gohcl.DecodeBody. They also custom-treat type attributes by using hcl.ExprAsKeyword (compare with description). As you assumed, they do use a custom type for type, but with custom parsing you don't have to.
Below is a working example:
package main
import (
"fmt"
"log"
"os"
"github.com/hashicorp/hcl/v2"
"github.com/hashicorp/hcl/v2/gohcl"
"github.com/hashicorp/hcl/v2/hclsyntax"
)
var (
configFileSchema = &hcl.BodySchema{
Blocks: []hcl.BlockHeaderSchema{
{
Type: "variable",
LabelNames: []string{"name"},
},
},
}
variableBlockSchema = &hcl.BodySchema{
Attributes: []hcl.AttributeSchema{
{
Name: "description",
},
{
Name: "type",
},
{
Name: "sensitive",
},
},
}
)
type Config struct {
Variables []*Variable
}
type Variable struct {
Name string
Description string
Type string
Sensitive bool
}
func main() {
config := configFromFile("examples/string.tf")
for _, v := range config.Variables {
fmt.Printf("%+v\n", v)
}
}
func configFromFile(filePath string) *Config {
content, err := os.ReadFile(filePath) // go 1.16
if err != nil {
log.Fatal(err)
}
file, diags := hclsyntax.ParseConfig(content, filePath, hcl.Pos{Line: 1, Column: 1})
if diags.HasErrors() {
log.Fatal("ParseConfig", diags)
}
bodyCont, diags := file.Body.Content(configFileSchema)
if diags.HasErrors() {
log.Fatal("file content", diags)
}
res := &Config{}
for _, block := range bodyCont.Blocks {
v := &Variable{
Name: block.Labels[0],
}
blockCont, diags := block.Body.Content(variableBlockSchema)
if diags.HasErrors() {
log.Fatal("block content", diags)
}
if attr, exists := blockCont.Attributes["description"]; exists {
diags := gohcl.DecodeExpression(attr.Expr, nil, &v.Description)
if diags.HasErrors() {
log.Fatal("description attr", diags)
}
}
if attr, exists := blockCont.Attributes["sensitive"]; exists {
diags := gohcl.DecodeExpression(attr.Expr, nil, &v.Sensitive)
if diags.HasErrors() {
log.Fatal("sensitive attr", diags)
}
}
if attr, exists := blockCont.Attributes["type"]; exists {
v.Type = hcl.ExprAsKeyword(attr.Expr)
if v.Type == "" {
log.Fatal("type attr", "invalid value")
}
}
res.Variables = append(res.Variables, v)
}
return res
}
Add for completeness, example/string.tf:
variable "image_id" {
type = string
description = "The id of the machine image (AMI) to use for the server."
sensitive = false
}
variable "other_id" {
type = string
description = "The id of the machine image (AMI) to use for the server."
sensitive = true
}
Since the Terraform language makes extensive use of various HCL features that require custom programming with the low-level HCL API, the Terraform team maintains a Go library terraform-config-inspect which understands the Terraform language enough to extract static metadata about top-level objects, including variables. It also deals with the fact that Terraform allows variable definitions in any .tf or .tf.json file interleaved with other declarations; putting them in variables.tf is only a convention.
For example:
mod, diags := tfconfig.LoadModule("examples")
if diags.HasErrors() {
log.Fatalf(diags.Error())
}
for _, variable := range mod.Variables {
fmt.Printf("%#v\n", variable)
}
This library is the same code used by Terraform Registry to produce the documentation about module input variables, so it supports all Terraform language versions that the Terraform Registry does (at the time of writing, going back to the Terraform v0.10 language, since that's the first version that can install modules from a registry) and supports both the HCL native syntax and JSON representations of the Terraform language.

Stream data from a request response into a CSV in go

I have something like a data pipeline.
API response (10k) rows as JSON.
=> Sanitize some of the data into a new structure
=> Create a CSV File
I can currently do that by getting the full response and doing that step by step.
I was wondering if there's a simpler way to stream the response reading into CSV right away and also writing in the file as it goes over the request-response.
Current code:
I will have a JSON like { "name": "Full Name", ...( 20 columns)} and that data repeats about 10-20k times with different values.
For request
var res *http.Response
if res, err = client.Do(request); err != nil {
return errors.Wrap(err, "failed to perform request")
}
For Unmarshal
var record []RecordStruct
if err = json.NewDecoder(res.Body).Decode(&record); err != nil {
return err
}
For CSV
var row []byte
if row, err = csvutil.Marshal(record); err != nil {
return err
}
To stream an array of JSON objects you have to decode nested objects instead of root object. To do this you need read data using tokens (check out Token method). According to the documentation:
Token returns the next JSON token in the input stream. At the end of the input stream, Token returns nil, io.EOF.
Token guarantees that the delimiters [ ] { } it returns are properly nested and matched: if Token encounters an unexpected delimiter in the input, it will return an error.
The input stream consists of basic JSON values—bool, string, number, and null—along with delimiters [ ] { } of type Delim to mark the start and end of arrays and objects. Commas and colons are elided.
That mean you can decode document part by part. Find an official example how to do it here
I will post a code snippet that shows how you can combine json stream technic with writing result to the CSV:
package main
import (
"encoding/csv"
"encoding/json"
"log"
"os"
"strings"
)
type RecordStruct struct {
Name string `json:"name"`
Info string `json:"info"`
// ... any field you want
}
func (rs *RecordStruct) CSVRecord() []string {
// Here we form data for CSV writer
return []string{rs.Name, rs.Info}
}
const jsonData =
`[
{ "name": "Full Name", "info": "..."},
{ "name": "Full Name", "info": "..."},
{ "name": "Full Name", "info": "..."},
{ "name": "Full Name", "info": "..."},
{ "name": "Full Name", "info": "..."}
]`
func main() {
// Create file for storing our result
file, err := os.Create("result.csv")
if err != nil {
log.Fatalln(err)
}
defer file.Close()
// Create CSV writer using standard "encoding/csv" package
var w = csv.NewWriter(file)
// Put your reader here. In this case I use strings.Reader
// If you are getting data through http it will be resp.Body
var jsonReader = strings.NewReader(jsonData)
// Create JSON decoder using "encoding/json" package
decoder := json.NewDecoder(jsonReader)
// Token returns the next JSON token in the input stream.
// At the end of the input stream, Token returns nil, io.EOF.
// In this case our first token is '[', i.e. array start
_, err = decoder.Token()
if err != nil {
log.Fatalln(err)
}
// More reports whether there is another element in the
// current array or object being parsed.
for decoder.More() {
var record RecordStruct
// Decode only the one item from our array
if err := decoder.Decode(&record); err != nil {
log.Fatalln(err)
}
// Convert and put out record to the csv file
if err := writeToCSV(w, record.CSVRecord()); err != nil {
log.Fatalln(err)
}
}
// Our last token is ']', i.e. array end
_, err = decoder.Token()
if err != nil {
log.Fatalln(err)
}
}
func writeToCSV(w *csv.Writer, record []string) error {
if err := w.Write(record); err != nil {
return err
}
w.Flush()
return nil
}
You can also use 3d party packages like github.com/bcicen/jstream

how I can mapping complex JSON to other JSON

I am trying to build aggregation services, to all third party APIs that's I used,
this aggregation services taking json values coming from my main system and it will put this value to key equivalent to third party api key then, aggregation services it will send request to third party api with new json format.
example-1:
package main
import (
"encoding/json"
"fmt"
"log"
"github.com/tidwall/gjson"
)
func main() {
// mapping JSON
mapB := []byte(`
{
"date": "createdAt",
"clientName": "data.user.name"
}
`)
// from my main system
dataB := []byte(`
{
"createdAt": "2017-05-17T08:52:36.024Z",
"data": {
"user": {
"name": "xxx"
}
}
}
`)
mapJSON := make(map[string]interface{})
dataJSON := make(map[string]interface{})
newJSON := make(map[string]interface{})
err := json.Unmarshal(mapB, &mapJSON)
if err != nil {
log.Panic(err)
}
err = json.Unmarshal(dataB, &dataJSON)
if err != nil {
log.Panic(err)
}
for i := range mapJSON {
r := gjson.GetBytes(dataB, mapJSON[i].(string))
newJSON[i] = r.Value()
}
newB, err := json.MarshalIndent(newJSON, "", " ")
if err != nil {
log.Println(err)
}
fmt.Println(string(newB))
}
output:
{
"clientName": "xxx",
"date": "2017-05-17T08:52:36.024Z"
}
I use gjson package to get values form my main system request in simple way from a json document.
example-2:
import (
"encoding/json"
"fmt"
"log"
"github.com/tidwall/gjson"
)
func main() {
// mapping JSON
mapB := []byte(`
{
"date": "createdAt",
"clientName": "data.user.name",
"server":{
"google":{
"date" :"createdAt"
}
}
}
`)
// from my main system
dataB := []byte(`
{
"createdAt": "2017-05-17T08:52:36.024Z",
"data": {
"user": {
"name": "xxx"
}
}
}
`)
mapJSON := make(map[string]interface{})
dataJSON := make(map[string]interface{})
newJSON := make(map[string]interface{})
err := json.Unmarshal(mapB, &mapJSON)
if err != nil {
log.Panic(err)
}
err = json.Unmarshal(dataB, &dataJSON)
if err != nil {
log.Panic(err)
}
for i := range mapJSON {
r := gjson.GetBytes(dataB, mapJSON[i].(string))
newJSON[i] = r.Value()
}
newB, err := json.MarshalIndent(newJSON, "", " ")
if err != nil {
log.Println(err)
}
fmt.Println(string(newB))
}
output:
panic: interface conversion: interface {} is map[string]interface {}, not string
I can handle this error by using https://golang.org/ref/spec#Type_assertions, but what if this json object have array and inside this array have json object ....
my problem is I have different apis, every api have own json schema, and my way for mapping json only work if
third party api have json key value only, without nested json or array inside this array json object.
is there a way to mapping complex json schema, or golang package to help me to do that?
EDIT:
After comment interaction and with updated question. Before we move forward, I would like to mention.
I just looked at your example-2 Remember one thing. Mapping is from one form to another form. Basically one known format to targeted format. Each data type have to handled. You cannot do generic to generic mapping logically (technically feasible though, would take more time & efforts, you can play around on this).
I have created sample working program of one approach; it does a mapping of source to targeted format. Refer this program as a start point and use your creativity to implement yours.
Playground link: https://play.golang.org/p/MEk_nGcPjZ
Explanation: Sample program achieves two different source format to one target format. The program consist of -
Targeted Mapping definition of Provider 1
Targeted Mapping definition of Provider 2
Provider 1 JSON
Provider 2 JSON
Mapping function
Targeted JSON marshal
Key elements from program: refer play link for complete program.
type MappingInfo struct {
TargetKey string
SourceKeyPath string
DataType string
}
Map function:
func mapIt(mapping []*MappingInfo, parsedResult gjson.Result) map[string]interface{} {
mappedData := make(map[string]interface{})
for _, m := range mapping {
switch m.DataType {
case "time":
mappedData[m.TargetKey] = parsedResult.Get(m.SourceKeyPath).Time()
case "string":
mappedData[m.TargetKey] = parsedResult.Get(m.SourceKeyPath).String()
}
}
return mappedData
}
Output:
Provider 1 Result: map[date:2017-05-17 08:52:36.024 +0000 UTC clientName:provider1 username]
Provider 1 JSON: {
"clientName": "provider1 username",
"date": "2017-05-17T08:52:36.024Z"
}
Provider 2 Result: map[date:2017-05-12 06:32:46.014 +0000 UTC clientName:provider2 username]
Provider 2 JSON: {
"clientName": "provider2 username",
"date": "2017-05-12T06:32:46.014Z"
}
Good luck, happy coding!
Typically Converting/Transforming one structure to another structure, you will have to handle this with application logic.
As you mentioned in the question:
my problem is I have different apis, every api have own json schema
This is true for every aggregation system.
One approach to handle this requirement effectively; is to keep mapping of keys for each provider JSON structure and targeted JSON structure.
For example: This is an approach, please go with your design as you see fit.
JSON structures from various provider:
// Provider 1 : JSON structrure
{
"createdAt": "2017-05-17T08:52:36.024Z",
"data": {
"user": {
"name": "xxx"
}
}
}
// Provider 2 : JSON structrure
{
"username": "yyy"
"since": "2017-05-17T08:52:36.024Z",
}
Mapping for target JSON structure:
jsonMappingByProvider := make(map[string]string)
// Targeted Mapping for Provider 1
jsonMappingByProvider["provider1"] = `
{
"date": "createdAt",
"clientName": "data.user.name"
}
`
// Targeted Mapping for Provider 2
jsonMappingByProvider["provider2"] = `
{
"date": "since",
"clientName": "username"
}
`
Now, based the on the provider you're handling, get the mapping and map the response JSON into targeted structure.
// get the mapping info by provider
mapping := jsonMappingByProvider["provider1"]
// Parse the response JSON
// Do the mapping
This way you can control each provider and it's mapping effectively.

Resources