Output json with os.stdout.write and a new line? - go

new to golang here. I wanted to simply output some json to stdout for debug purposes. I think I'm going about this all wrong. Here is what I have:
type SomeObject struct {
Thing1 string
Thing2 string
Thing3 otherStruct
}
...
someObject = &SomeObject{
Thing1: "hello",
Thing2: "world",
...
}
someObjectLogEntry, err := json.Marshal(someObject)
if err != nil {
fmt.Println("error:", err)
}
os.Stdout.Write(someObjectLogEntry)
When I run this, it outputs the json as one line, however my service also has a heartbeat going and so the two coincide and output both things in the same line, something like:
{/// All that json content }[GIN] 2022/03/16 - 02:07:16 | 200 | 1.16µs | 127.0.0.1 | GET "/heartbeat"
What's the correct way to do what I'm doing (simply constructing a json object and outputting it)? If i do fmt.println it will then print out the byte code. Thanks!

You need to output a line feed or newline (\n) after the JSON. The simplest approach is probably using fmt.Printf e.g. (playground)
type SomeObject struct {
Thing1 string
Thing2 string
}
someObject := &SomeObject{
Thing1: "hello",
Thing2: "world",
}
someObjectLogEntry, err := json.Marshal(someObject)
if err != nil {
fmt.Println("error:", err)
}
fmt.Printf("%s\n", someObjectLogEntry)
os.Stdout.Write([]byte("something else\n"))

The typical way to achieve this while using the Printf function from the fmt package is by including a newline character \n in the format string. You don’t necessarily need to write the data to standard output, because Printf does that for you.
Another option is to use the Println function from the fmt package which formats using the default formats for its operands writes to standard output. It’s necessary to convert your JSON bytes to string while using Println. Note that, spaces are always added between the operands and a newline is appended while using Println.
The reason that you see individual byte values while using fmt.Println is because a byte slice is printed out as uninterpreted bytes of the string or slice — the byte slice can contain anything at all, not just printable characters. On the other hand, os.Stdout.Write writes the byte slice to standard out and your terminal renders them properly because they are printable characters.

The cleanest approach is to use json.Encoder to os.Stdout.
json.Encoder already appends a newline to each message written and is more efficient than using fmt with json bytes.
You can re-use the json encoder instead of calling json.Marshal then some type of separate write for each message.
import (
"os"
"encoding/json"
)
type SomeObject struct {
Thing1 string
Thing2 string
Thing3 otherStruct
}
...
encoder := json.NewEncoder(os.Stdout)
someObject = &SomeObject{
Thing1: "hello",
Thing2: "world",
...
}
if err := encoder.Encode(&someObject); err != nil {
// handle error
}

Related

How to discard lines with certain strings in list and output to new list

I have a list downloaded from a website in XML. I am trying to filter the list by discarding lines that contain a certain string and building the same type of list without the lines containing the string.
I have a struct type that's having another struct.
I'm trying to use regexp and replaceall, and failing at replaceall.
func (*Regexp) ReplaceAll
func (re *Regexp) ReplaceAll(src, repl []byte) []byte
There may be an entirely simpler way to filter a list to a new list that I'm missing somewhere, but I've found this as the closest possible solution so far. Please share other ways to grep and delete lines to a new list. The list is a byte at body and downloaded as a xml.
type PeopleList struct {
Peoples []Person `xml:"peoples>person"`
}
type Person struct {
ADD string `xml:"add,attr"`
Loc string `xml:"loc,attr"`
Har string `xml:"har,attr"`
Name string `xml:"name,attr"`
Country string `xml:"country,attr"`
Num string `xml:"num,attr"`
ADD2 string `xml:"add2,attr"`
Distance float64
func fetchPeopleList(userinfo Userinfo) PeopleList {
var p byte
jam, err := http.Get(string(peoplelisturl))
iferror (err)
body, err := ioutil.ReadAll(jam.Body)
peeps := body
reg := regexp.MustCompile("(?m)[\r\n]+^.*BAD:.*$")
rep := reg.ReplaceAll(peeps, p) // Here fails probably because of my syntax. Error: cannot use p (variable of type byte) as []byte value in argument to re.ReplaceAll
fmt.Println(rep)
iferror (err)
defer jam.Body.Close()
Finally, I would like a new list in the same format as the first, only without the lines containing the string.
Your question says you want to "discard lines", but Replace/ReplaceAll, as their names suggest, are for replacing matched patterns. Your regex is also a simple substring match, so the obvious solution would seem to be reading the file line by line and - as your title says - discarding lines containing the substring.
func fetchPeopleList(userinfo Userinfo) PeopleList {
jam, err := http.Get(string(peoplelisturl))
iferror (err)
br := bufio.NewReader(jam.Body)
defer jam.Body.Close()
for {
line,err := br.ReadString('\n')
if !strings.Contains(line, "BAD:") {
fmt.Println(line) // or whatever you want to do with non-discarded lines
}
if err != nil {
break
}
}

Unmarshal returns blank object due to encoding

I'm attempting to unmarshal a raw json string. There seems to be an error with encoding but I can't quite figure it out.
package main
import (
"encoding/json"
"fmt"
"log"
)
type Foo struct {
Transmission string `json:"transmission"`
Trim string `json:"trim"`
Uuid string `json:"uuid"`
Vin string `json:"vin"`
}
func main() {
var foo Foo
sample := `{
"transmission": "continuously\x20variable\x20automatic",
"trim": "SL",
"uuid" : "6993e4090a0e0ae80c59a76326e360a1",
"vin": "5N1AZ2MH6JN192059"
}`
err := json.Unmarshal([]byte(sample), &foo)
if err != nil {
log.Fatal(err)
}
fmt.Println(foo)
}
2009/11/10 23:00:00 invalid character 'x' in string escape code
It works if transmission entry is removed.
Here is a working playground.
Your input is not valid JSON. The JSON spec states that
All code points may be placed within the quotation marks except for the code points that must be escaped: quotation mark (U+0022), reverse solidus (U+005C), and the control characters U+0000 to U+001F.
Additionally, although there are two-character escape sequences, \x is not one of them, and thus it is being correctly interpreted as an invalid escape sequence by the Go parser. If you want to have a backslash literal in your JSON, it needs to be represented by \\ in the JSON input itself. See a modified version of your example: https://play.golang.org/p/JZdPJGpPR5q
(note that this is not an issue with your Go string literal since you're already using a raw (``) string literal—the JSON itself needs to have two backslashes.)
You can replace \x with \\x using string.Replace() function. Then, Unmarshal the replaced string. Here is a working example.

How can I convert a JSON string to a byte array?

I need some help with unmarshaling. I have this example code:
package main
import (
"encoding/json"
"fmt"
)
type Obj struct {
Id string `json:"id"`
Data []byte `json:"data"`
}
func main() {
byt := []byte(`{"id":"someID","data":["str1","str2"]}`)
var obj Obj
if err := json.Unmarshal(byt, &obj); err != nil {
panic(err)
}
fmt.Println(obj)
}
What I try to do here - convert bytes to the struct, where type of one field is []byte. The error I get:
panic: json: cannot unmarshal string into Go struct field Obj.data of
type uint8
That's probably because parser already sees that "data" field is already a slice and tries to represent "str1" as some char bytecode (type uint8?).
How do I store the whole data value as one bytes array? Because I want to unmarshal the value to the slice of strings later. I don't include a slice of strings into struct because this type can change (array of strings, int, string, etc), I wish this to be universal.
My first recommendation would be for you to just use []string instead of []byte if you know the input type is going to be an array of strings.
If data is going to be a JSON array with various types, then your best option is to use []interface{} instead - Go will happily unmarshal the JSON for you and you can perform checks at runtime to cast those into more specific typed variables on an as-needed basis.
If []byte really is what you want, use json.RawMessage, which is of type []byte, but also implements the methods for JSON parsing. I believe this may be what you want, as it will accept whatever ends up in data. Of course, you then have to manually parse Data to figure out just what actually IS in there.
One possible bonus is that this skips any heavy parsing because it just copies the bytes over. When you want to use this data for something, you use a []interface{}, then use a type switch to use individual values.
https://play.golang.org/p/og88qb_qtpSGJ
package main
import (
"encoding/json"
"fmt"
)
type Obj struct {
Id string `json:"id"`
Data json.RawMessage `json:"data"`
}
func main() {
byt := []byte(`{"id":"someID","data":["str1","str2", 1337, {"my": "obj", "id": 42}]}`)
var obj Obj
if err := json.Unmarshal(byt, &obj); err != nil {
panic(err)
}
fmt.Printf("%+v\n", obj)
fmt.Printf("Data: %s\n", obj.Data)
// use it
var d []interface{}
if err := json.Unmarshal(obj.Data, &d); err != nil {
panic(err)
}
fmt.Printf("%+v\n", d)
for _, v := range d {
// you need a type switch to deterine the type and be able to use most of these
switch real := v.(type) {
case string:
fmt.Println("I'm a string!", real)
case float64:
fmt.Println("I'm a number!", real)
default:
fmt.Printf("Unaccounted for: %+v\n", v)
}
}
}
Your question is:
convert bytes array to struct with a field of type []byte
But you do not have a bytearray but a string array. Your question is not the same as your example. So let answer your question, there are more solutions possible depending in how far you want to diverge from your original requirements.
One string can be converted to one byte-slice, two strings need first to be transformed to one string. So that is problem one. The second problem are the square-brackets in your json-string
This works fine, it implicitly converts the string in the json-string to a byte-slice:
byt := []byte(`{"id":"someID","data":"str1str2"}`)
var obj Obj
if err := json.Unmarshal(byt, &obj); err != nil {
panic(err)
}
fmt.Println(obj)

Golang types in net package

I'm coding in golang some tools to make my life easier and I'm not understanding at all how the types in the net package works. This is part of my code:
import (
"bufio"
"fmt"
"log"
"net"
"os"
)
type configFile struct {
gateway, net net.IP
mask, port int
telnetUser, telnetPasswd string
}
var dataConfig configFile
func createConfigFile() {
reader := bufio.NewReader(os.Stdin)
fmt.Println("Let's fill te config file for the application.")
fmt.Println("Which is your gateway IP?")
readGateway, err := reader.ReadString('\n')
if err != nil {
log.Fatal(err)
}
dataConfig.gateway = net.ParseIP(readGateway)
if dataConfig.gateway == nil {
log.Fatal("Problem here")
} else {
fmt.Println("Your gateway is: ", dataConfig.gateway.String())
}
}
My problem is the next:
I want to read an IP address from the command line and storage it in the configFile object, which I will user later to create a .json file with all the configuration of my program.
When I read from the command line the IP address the readGateway variable storages it ok, that's expected, but when I try to make
dataConfig.gateway = net.ParseIP(readGateway)
and I try to cast the string object to a net.IP object I'm always getting a nill in the dataConfig.gateway field, so I'm not able to work with that parameter neither convert it to a string.
Could somebody help me?
Thanks in advance.
bufio.Reader.ReadString's docs explain (emphasis mine)
ReadString reads until the first occurrence of delim in the input,
returning a string containing the data up to and including the
delimiter
So readGateway ends up looking like "192.168.1.1\n". Your newline delimiter would not exist in a properly formatted IP address, which means when you parse it with net.ParseIP it's kicking it out as nil.
You can use strings.TrimRight to trim out the newline:
readGateway, err := reader.ReadString('\n')
if err != nil {
log.Fatal(err)
}
readGateway = strings.TrimRight(readGateway, "\n")
The documentation of net.ParseIP(s string) states that if the string s provided is not a valid textual representation of an IP address the function will return a nil value, so that must be the case.
Please log the string s before calling net.ParseIP so you can check if you are passing and reading it properly in the program.

How to stop json.Marshal from escaping < and >?

package main
import "fmt"
import "encoding/json"
type Track struct {
XmlRequest string `json:"xmlRequest"`
}
func main() {
message := new(Track)
message.XmlRequest = "<car><mirror>XML</mirror></car>"
fmt.Println("Before Marshal", message)
messageJSON, _ := json.Marshal(message)
fmt.Println("After marshal", string(messageJSON))
}
Is it possible to make json.Marshal not escape < and >? I currently get:
{"xmlRequest":"\u003ccar\u003e\u003cmirror\u003eXML\u003c/mirror\u003e\u003c/car\u003e"}
but I am looking for something like this:
{"xmlRequest":"<car><mirror>XML</mirror></car>"}
As of Go 1.7, you still cannot do this with json.Marshal(). The source code for json.Marshal shows:
> err := e.marshal(v, encOpts{escapeHTML: true})
The reason json.Marshal always does this is:
String values encode as JSON strings coerced to valid UTF-8,
replacing invalid bytes with the Unicode replacement rune.
The angle brackets "<" and ">" are escaped to "\u003c" and "\u003e"
to keep some browsers from misinterpreting JSON output as HTML.
Ampersand "&" is also escaped to "\u0026" for the same reason.
This means you cannot even do it by writing a custom func (t *Track) MarshalJSON(), you have to use something that does not satisfy the json.Marshaler interface.
So, the workaround, is to write your own function:
func (t *Track) JSON() ([]byte, error) {
buffer := &bytes.Buffer{}
encoder := json.NewEncoder(buffer)
encoder.SetEscapeHTML(false)
err := encoder.Encode(t)
return buffer.Bytes(), err
}
https://play.golang.org/p/FAH-XS-QMC
If you want a generic solution for any struct, you could do:
func JSONMarshal(t interface{}) ([]byte, error) {
buffer := &bytes.Buffer{}
encoder := json.NewEncoder(buffer)
encoder.SetEscapeHTML(false)
err := encoder.Encode(t)
return buffer.Bytes(), err
}
https://play.golang.org/p/bdqv3TUGr3
In Go1.7 the have added a new option to fix this:
encoding/json:
add Encoder.DisableHTMLEscaping This provides a way to disable the escaping of <, >, and & in JSON strings.
The relevant function is
func (*Encoder) SetEscapeHTML
That should be applied to a Encoder.
enc := json.NewEncoder(os.Stdout)
enc.SetEscapeHTML(false)
Simple example: https://play.golang.org/p/SJM3KLkYW-
This doesn't answer the question directly but it could be an answer if you're looking for a way how to deal with json.Marshal escaping < and >...
Another way to solve the problem is to replace those escaped characters in json.RawMessage into just valid UTF-8 characters, after the json.Marshal() call.
It will work as well for any letters other than < and >. (I used to do this to make non-English letters to be human readable in JSON :D)
func _UnescapeUnicodeCharactersInJSON(_jsonRaw json.RawMessage) (json.RawMessage, error) {
str, err := strconv.Unquote(strings.Replace(strconv.Quote(string(_jsonRaw)), `\\u`, `\u`, -1))
if err != nil {
return nil, err
}
return []byte(str), nil
}
func main() {
// Both are valid JSON.
var jsonRawEscaped json.RawMessage // json raw with escaped unicode chars
var jsonRawUnescaped json.RawMessage // json raw with unescaped unicode chars
// '\u263a' == '☺'
jsonRawEscaped = []byte(`{"HelloWorld": "\uC548\uB155, \uC138\uC0C1(\u4E16\u4E0A). \u263a"}`) // "\\u263a"
jsonRawUnescaped, _ = _UnescapeUnicodeCharactersInJSON(jsonRawEscaped) // "☺"
fmt.Println(string(jsonRawEscaped)) // {"HelloWorld": "\uC548\uB155, \uC138\uC0C1(\u4E16\u4E0A). \u263a"}
fmt.Println(string(jsonRawUnescaped)) // {"HelloWorld": "안녕, 세상(世上). ☺"}
}
https://play.golang.org/p/pUsrzrrcDG-
I hope this helps someone.
Here's my workaround:
// Marshal is a UTF-8 friendly marshaler. Go's json.Marshal is not UTF-8
// friendly because it replaces the valid UTF-8 and JSON characters "&". "<",
// ">" with the "slash u" unicode escaped forms (e.g. \u0026). It preemptively
// escapes for HTML friendliness. Where text may include any of these
// characters, json.Marshal should not be used. Playground of Go breaking a
// title: https://play.golang.org/p/o2hiX0c62oN
func Marshal(i interface{}) ([]byte, error) {
buffer := &bytes.Buffer{}
encoder := json.NewEncoder(buffer)
encoder.SetEscapeHTML(false)
err := encoder.Encode(i)
return bytes.TrimRight(buffer.Bytes(), "\n"), err
}
No, you can't.
A third-party json package might be the choice rather than the std json lib.
More detail:https://github.com/golang/go/issues/8592
I had a requirement to store xml inside json :puke:
At first I was having significant difficulty unmarshalling that xml after passing it via json, but my issue was actually due to trying to unmarshall the xml string as a json.RawMessage. I actually needed to unmarshall it as a string and then coerce it into []byte for the xml.Unmarshal.
type xmlInJson struct {
Data string `json:"data"`
}
var response xmlInJson
err := json.Unmarshall(xmlJsonData, &response)
var xmlData someOtherStructThatMatchesTheXmlFormat
err = xml.Unmarshall([]byte(response.Data), &xmlData)
Custom function is not kind of the best solution.
How about another library to solve this.
I use gabs
import
go get "github.com/Jeffail/gabs"
use
message := new(Track)
resultJson,_:=gabs.Consume(message)
fmt.Println(string(resultJson.EncodeJSON()))
I solve that problem like this.

Resources