What happen when field value string is unmarshalled into []byte - go

What happens to my original data in the variable str here?
After converting to the struct the length of bytes is 6
And the value of byte don't match ascii code of 1, 2, 3, 4, 5, 6, 7, 8 at all
package main
import (
"encoding/json"
"fmt"
)
type Encrypt struct {
Bytes []byte `json:"bytes"`
}
func main(){
str := "12345678"
raw := fmt.Sprintf("{\"bytes\":%s}", str)
var encrypt = Encrypt{}
fmt.Println([]byte(raw)[0])
err := json.Unmarshal([]byte(raw), &encrypt)
if err != nil{
fmt.Println(err)
return
}
fmt.Println("final result")
fmt.Println(len(encrypt.Bytes))
fmt.Println(string(encrypt.Bytes))
for _, b := range encrypt.Bytes{
fmt.Print(b)
fmt.Print(" ")
}
fmt.Print("\n")
}

As per the documentation https://pkg.go.dev/encoding/json#Unmarshal:
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
As you can see there is no place for []byte. Also, in you code raw := fmt.Sprintf("{\"bytes\":%s}", str) you are sending str as a number. You can send it as string as in the following code.
package main
import (
"encoding/json"
"fmt"
)
type Encrypt struct {
ByteString string `json:"bytes"`
}
func main() {
str := "12345678"
raw := fmt.Sprintf("{\"bytes\":\"%s\"}", str)
fmt.Println(raw)
var encrypt = Encrypt{}
fmt.Println([]byte(str))
err := json.Unmarshal([]byte(raw), &encrypt)
if err != nil {
panic(err)
}
fmt.Println("final result ", []byte(encrypt.ByteString))
}

Related

How to parse a JSON string returned from scanner.Text() [duplicate]

Objects like the below can be parsed quite easily using the encoding/json package.
[
{"something":"foo"},
{"something-else":"bar"}
]
The trouble I am facing is when there are multiple dicts returned from the server like this :
{"something":"foo"}
{"something-else":"bar"}
This can't be parsed using the code below.
correct_format := strings.Replace(string(resp_body), "}{", "},{", -1)
json_output := "[" + correct_format + "]"
I am trying to parse Common Crawl data (see example).
How can I do this?
Assuming your input is really a series of valid JSON documents, use a json.Decoder to decode them:
package main
import (
"encoding/json"
"fmt"
"io"
"log"
"strings"
)
var input = `
{"foo": "bar"}
{"foo": "baz"}
`
type Doc struct {
Foo string
}
func main() {
dec := json.NewDecoder(strings.NewReader(input))
for {
var doc Doc
err := dec.Decode(&doc)
if err == io.EOF {
// all done
break
}
if err != nil {
log.Fatal(err)
}
fmt.Printf("%+v\n", doc)
}
}
Playground: https://play.golang.org/p/ANx8MoMC0yq
If your input really is what you've shown in the question, that's not JSON and you have to write your own parser.
Seems like each line is its own json object.
You may get away with the following code which will structure this output into correct json:
package main
import (
"fmt"
"strings"
)
func main() {
base := `{"trolo":"lolo"}
{"trolo2":"lolo2"}`
delimited := strings.Replace(base, "\n", ",", -1)
final := "[" + delimited + "]"
fmt.Println(final)
}
You should be able to use encoding/json library on final now.
Another option would be to parse each incoming line, line by line, and then add each one to a collection in code (ie a slice) Go provides a line scanner for this.
yourCollection := []yourObject{}
scanner := bufio.NewScanner(YOUR_SOURCE)
for scanner.Scan() {
obj, err := PARSE_JSON_INTO_yourObject(scanner.Text())
if err != nil {
// something
}
yourCollection = append(yourCollection, obj)
}
if err := scanner.Err(); err != nil {
fmt.Fprintln(os.Stderr, "reading standard input:", err)
}
You can read the ndjson from the file row by row and parse it then apply the logical operations on it. In the below sample instead of reading from the file, I have used an Array of JSON string.
import (
"encoding/json"
"fmt"
)
type NestedObject struct {
D string
E string
}
type OuterObject struct {
A string
B string
C []NestedObject
}
func main() {
myJsonString := []string{`{"A":"1","B":"2","C":[{"D":"100","E":"10"}]}`, `{"A":"11","B":"21","C":[{"D":"1001","E":"101"}]}`}
for index, each := range myJsonString {
fmt.Printf("Index value [%d] is [%v]\n", index, each)
var obj OuterObject
json.Unmarshal([]byte(each), &obj)
fmt.Printf("a: %v, b: %v, c: %v", obj.A, obj.B, obj.C)
fmt.Println()
}
}
Output:
Index value [0] is [{"A":"1","B":"2","C":[{"D":"100","E":"10"}]}]
a: 1, b: 2, c: [{100 10}]
Index value [1] is [{"A":"11","B":"21","C":[{"D":"1001","E":"101"}]}]
a: 11, b: 21, c: [{1001 101}]
Try it on golang play

Convert binary value as string to uint32 in Golang

Hello i am trying to convert 00000000000000000000000000001011 to uint32 in golang using
var v = "00000000000000000000000000001011"
fmt.Printf("%T\n", v)
c := []byte(v)
u := binary.LittleEndian.Uint32(c)
However it is not working.
You can't use encoding/binary for this, as that is to serialize and deserialize the (memory) bytes of different values (e.g. numbers). What you have is the base 2 string representation of the number.
To get its integer value you have to parse it. For that, use strconv.ParseUint():
s := "00000000000000000000000000001011"
u, err := strconv.ParseUint(s, 2, 32)
if err != nil {
panic(err)
}
fmt.Println(u)
This outputs (try it on the Go Playground):
11
Note that strconv.ParseUint() returns a value of type uint64, so if you need uint32, you have to manually convert it, e.g.:
u32 := uint32(u)
There are more options for parsing numbers from strings, for an overview, check Convert string to integer type in Go?
For example,
package main
import (
"fmt"
"strconv"
)
func main() {
s := "00000000000000000000000000001011"
fmt.Println(s)
u64, err := strconv.ParseUint(s, 2, 32)
u32 := uint32(u64)
if err == nil {
fmt.Println(u32)
}
}
Playground: https://play.golang.org/p/yiicgWsb7B_M
Output:
00000000000000000000000000001011
11

Unmarshal GO YAML to either a Map or a String

I'm trying to unmarshal YAML entries that can be either a string or a list of key: value strings (a map as per Go). I cannot figure out how to get this done sadly. I know I can write my own unmarshaller but that seems to only work with structs.
I have the first part working:
package main
import (
"log"
"gopkg.in/yaml.v2"
)
type Data struct {
Entry []Entry `yaml:"entries"`
}
type Entry map[string]string
var dat string = `
entries:
- keya1: val1
keya2: val2
- keyb1: val1
keyb2: val2
- val3`
func main() {
out := Data{}
if err := yaml.Unmarshal([]byte(dat), &out); err != nil {
log.Fatal(err)
}
log.Printf("%+v", out)
}
But the - val3 entry causes an error now, obviously. How can I get it to recognise both lists and single string entries?
Thank you
This is just a followup to the excellent #Benjamin Kadish answer above, but here's a somewhat more complete version and this uses yaml.v3, which makes it just a bit more obvious. Note that the type of the unmarshalled items is map[string]interface{} instead of map[interface{}]interface{} in yaml v3.
package main
import (
"gopkg.in/yaml.v3"
"log"
)
type Data struct {
Entry []Entry `yaml:"entries"`
}
type Entry interface {}
var dat string = `
entries:
- keya1: val1
keya2: val2
- keyb1: val1
keyb2: val2
- val3`
func main() {
out := Data{}
if err := yaml.Unmarshal([]byte(dat), &out); err != nil {
log.Fatal(err)
}
for _, entry := range out.Entry {
switch i := entry.(type) {
case string:
log.Printf("i is a string: %+v\n", i)
case map[string]interface{}:
log.Printf("i is a map.")
for k,v := range i {
log.Printf("%s=%v\n",k,v)
}
default:
log.Printf("Type i=%s", i)
}
}
}
This has been answered in various ways before, but long story short it is easy unmarshall into an interface and then deal with both cases
type Entry interface{}
for _, entry := range out.Entry {
switch i := entry.(type) {
case string:
log.Printf("i is a string %+v\n", i)
case map[interface{}]interface{}:
log.Printf("i is a map %+v\n", i)
}
}
As, go is static type language, you can't leave val3 only a list item if you convert it to defined struct. It should be key value pair. E.g. (val3: "") (if you want it empty.)
Here is modified code
package main
import (
"log"
"gopkg.in/yaml.v2"
)
type Data struct {
Entry []Entry `yaml:"entries"`
}
type Entry map[string]string
var dat string = `
entries:
- keya1: val1
keya2: val2
- keyb1: val1
keyb2: val2
- val3: ""`
func main() {
out := Data{}
if err := yaml.Unmarshal([]byte(dat), &out); err != nil {
log.Fatal(err)
}
log.Printf("%+v", out)
}
And output:
2018/02/02 01:07:36 {Entry:[map[keya1:val1 keya2:val2] map[keyb2:val2 keyb1:val1] map[val3:]]}

How to convert string from interface to []string in golang?

I'm parsing a JSON object which contains an array of strings :
var ii interface{}
json := "{\"aString\": [\"aaa_111\", \"bbb_222\"], \"whatever\":\"ccc\"}"
err := json.Unmarshal([]byte(json), &ii)
if err != nil {
log.Fatal(err)
}
data := ii.(map[string]interface{})
fmt.Println(data["aString"]) // outputs: ["aaa_111" "bbb_222"]
I tried to convert data["aString"] to []string to be able to loop over it, but it fails :
test := []string(data["aString"]).([]string)
fmt.Println(test) // panic -> interface conversion:
// interface is string, not []string
How can I convert data["aString"] ?
edit:
I didn't express myself properly. If I print data, I have such map :
map[aString:["BBB-222","AAA-111"] whatever:ccc]
I want to loop over aString (to manipule each array entry). But I can't find how, because aString is type interface {} :
for i, v := range aString { // <-- fails
// ...
fmt.Println(i, v)
}
That's why I want to convert aString. I don't want to convert a string which looks like an array to an array.
I recommend you move away from this implementation in general. Your json may vary but you can easily use objects and avoid all this type unsafe nonsense.
Anyway, that conversion doesn't work because the types inside the slice are not string, they're also interface{}. You have to iterate the collection then do a type assertion on each item like so:
aInterface := data["aString"].([]interface{})
aString := make([]string, len(aInterface))
for i, v := range aInterface {
aString[i] = v.(string)
}
Is it what you need?
package main
import (
"fmt"
"encoding/json"
)
func main() {
js := "{\"aString\": [\"aaa_111\", \"bbb_222\"], \"whatever\":\"ccc\"}"
a := make(map[string]interface{})
json.Unmarshal([]byte(js), &a)
for _, v := range a["aString"].([]interface{}) {
str := v.(string)
fmt.Println(str)
}
}
Check on Go Playground
For another approach, you can use a struct instead:
package main
import (
"encoding/json"
"fmt"
)
func main() {
s := []byte(`{"aString": ["aaa_111", "bbb_222"], "whatever":"ccc"}`)
var t struct {
Astring []string
Whatever string
}
json.Unmarshal(s, &t)
fmt.Printf("%+v\n", t) // {Astring:[aaa_111 bbb_222] Whatever:ccc}
}

How to decompress a []byte content in gzip format that gives an error when unmarshaling

I'm making a request to an API, which with I get a []byte out of the response (ioutil.ReadAll(resp.Body)). I'm trying to unmarshal this content, but seems to be not encoded on utf-8 format, as unmarshal returns an error. I'm trying this to do so:
package main
import (
"encoding/json"
"fmt"
"some/api"
)
func main() {
content := api.SomeAPI.SomeRequest() // []byte variable
var data interface{}
err := json.Unmarshal(content, &data)
if err != nil {
panic(err.Error())
}
fmt.Println("Data from response", data)
}
I get as an error that invalid character '\x1f' looking for beginning of value. For the record, the response includes in the header that Content-Type:[application/json; charset=utf-8].
How can I decode content to avoid these invalid characters when unmarshaling?
Edit
This is the hexdump of content: play.golang.org/p/oJ5mqERAmj
Judging by your hex dump you are receiving gzip encoded data so you'll need to use compress/gzip to decode it first.
Try something like this
package main
import (
"bytes"
"compress/gzip"
"encoding/json"
"fmt"
"io"
"some/api"
)
func main() {
content := api.SomeAPI.SomeRequest() // []byte variable
// decompress the content into an io.Reader
buf := bytes.NewBuffer(content)
reader, err := gzip.NewReader(buf)
if err != nil {
panic(err)
}
// Use the stream interface to decode json from the io.Reader
var data interface{}
dec := json.NewDecoder(reader)
err = dec.Decode(&data)
if err != nil && err != io.EOF {
panic(err)
}
fmt.Println("Data from response", data)
}
Previous
Character \x1f is the unit separator character in ASCII and UTF-8. It is never part of an UTF-8 encoding, however can be used to mark off different bits of text. A string with an \x1f can valid UTF-8 but not valid json as far as I know.
I think you need to read the API specification closely to find out what they are using the \x1f markers for, but in the meantime you could try removing them and see what happens, eg
import (
"bytes"
"fmt"
)
func main() {
b := []byte("hello\x1fGoodbye")
fmt.Printf("b was %q\n", b)
b = bytes.Replace(b, []byte{0x1f}, []byte{' '}, -1)
fmt.Printf("b is now %q\n", b)
}
Prints
b was "hello\x1fGoodbye"
b is now "hello Goodbye"
Playground link

Resources