I have a file with serialized array in PHP.
The content of the file locks like this
a:2:{i:250;s:7:"my_catz";s:7:"abcd.jp";a:2:{s:11:"category_id";i:250;s:13:"category_name";s:7:"my_catz";}}
The array unserialized is this
(
[250] => my_catz
[abcd.jp] => Array
(
[category_id] => 250
[category_name] => my_catz
)
)
Now, i want to get the content of the file in GO, unserialize it convert it to an array.
In GO i can get the content of the file using
dat, err := os.ReadFile("/etc/squid3/compiled-categories.db")
if err != nil {
if e.Debug {
log.Printf("error reading /etc/squid3/compiled-categories.db: ", err)
}
}
And unserialized it using github.com/techoner/gophp library
package categorization
import (
"fmt"
"os"
"github.com/techoner/gophp"
"log"
"errors"
)
type Data struct {
Website string
Debug bool
}
func (e Data) CheckPersonalCategories() (int,string) {
if e.Debug {
log.Printf("Checking Personal Categories")
}
if _, err := os.Stat("/etc/squid3/compiled-categories.db"); errors.Is(err, os.ErrNotExist) {
if e.Debug {
log.Printf("/etc/squid3/compiled-categories.db not exit: ", err)
}
return 0,""
}
dat, err := os.ReadFile("/etc/squid3/compiled-categories.db")
if err != nil {
if e.Debug {
log.Printf("error reading /etc/squid3/compiled-categories.db: ", err)
}
}
out, _ := gophp.Unserialize(dat)
fmt.Println(out["abcd.jp"])
return 0,""
}
But I can't access to the array, for example, when I try access to array key using out["abcd.jp"] i get this error message
invalid operation: out["abcd.jp"] (type interface {} does not support indexing)
The result of out is
map[250:my_catz abcd.jp:map[category_id:250 category_name:my_catz]]
Seams that is unserializing
Don't make assumptions about what is and isn't succeeding in your code. Error responses are the only reliable way to know whether a function succeeded. In this case the assumption may hold, but ignoring errors is always a mistake. Invest time in catching errors and at least panic them - don't instead waste your time ignoring errors and then trying to debug unreliable code.
invalid operation: out["abcd.jp"] (type interface {} does not support indexing)
The package you're using unfortunately doesn't provide any documentation so you have to read the source to understand that gophp.Unserialize returns (interface{}, error). This makes sense; php can serialize any value, so Unserialize must be able to return any value.
out is therefore an interface{} whose underlying value depends on the data. To turn an interface{} into a particular value requires a type assertion. In this case, we think the underlying data should be map[string]interface{}. So we need to do a type assertion:
mout, ok := out.(map[string]interface{})
Before we get to the working code, one more point I'd like you to think about. Look at the code below: I started it from your code, but the resemblance is very slight. I took out almost all the code because it was completely irrelevant to your question. I added the input data to the code to make a minimal reproduction of your code (as I asked you to do and you declined to do). This is a very good use of your time for 2 reasons: first, it makes it a lot easier to get answers (both because it shows sufficient effort on your part and because it simplifies the description of the problem), and second, because it's excellent practice for debugging. I make minimal reproductions of code flows all the time to better understand how to do things.
You'll notice you can run this code now without any additional effort. That's the right way to provide a minimal reproducible example - not with a chunk of mostly irrelevant code which still can't be executed by anybody.
The Go Plaground is a great way to demonstrate go-specific code that others can execute and investigate. You can also see the code below at https://go.dev/play/p/QfCl08Gx53e
package main
import (
"fmt"
"github.com/techoner/gophp"
)
type Data struct {
Website string
Debug bool
}
func main() {
var dat = []byte(`a:2:{i:250;s:7:"my_catz";s:7:"abcd.jp";a:2:{s:11:"category_id";i:250;s:13:"category_name";s:7:"my_catz";}}`)
out, err := gophp.Unserialize(dat)
if err != nil {
panic(err)
}
if mout, ok := out.(map[string]interface{}); ok {
fmt.Println(mout["abcd.jp"])
}
}
I'm facing with a weird golang issue. The following code will clarify:
package main
import (
"os/exec"
"io"
"fmt"
"os"
)
var (
pw io.WriteCloser
pr io.ReadCloser
)
func main() {
term := exec.Command("/bin/sh")
// Get stdin writer pipe
pw, _ = term.StdinPipe()
pr, _ = term.StdoutPipe()
term.Start()
run("cd ~")
pwd := run("pwd");
// Do something with pwd output
...
term.Wait()
}
func run(c string) string {
io.WriteString(pw, fmt.Sprintln(c))
buf := make([]byte, 32 * 1024)
pr.Read(buf)
return string(buf)
}
I'd like to run some commands in a shell env and read their output. There's no problem on write/run command but it seems that there're some limitations while reading:
you can't know if a command doesn't output anything or not;
there's no way to check if stdout is ready to be read or not.
The pr.Read(dest) method will block the code flow until something is read from stdout. As said, the goal is to read sequentially (without using a go routine and/or an infinite loop). This means that if we send a cd command the func end is never reached.
Setting the non-block flag through unix.SetNonblock on stdout file descriptor seems to solve the above issue but you can't know prior if it's ready or not and an error saying "resource temporary not available" is returned from .Read call.
As Cerise Limón mentioned go functions whould be the way to go here, since these sorts of interactive scripting exercises are traditionally done with expect.
You can wrap the the parrellel execution into a library to it might still look like sequencial code, so this might be helpful: https://github.com/ThomasRooney/gexpect
From the readme:
child, err := gexpect.Spawn("python")
if err != nil { panic(err) }
child.Expect(">>>")
child.SendLine("print 'Hello World'")
child.Interact()
child.Close()
I know this has been asked a few times, but I don't see one that matches my need to support NULL values. I have fields that are optional in a DB. I need to output data in JSON format that includes these potentially NULL fields, which I would like to omit from the JSON if they are still NULL. I can change to some other default value other than NULL, but I haven't found one that works. I'm storing JSON arrays in JSON format (longtext) in MariaDB. Here's my code that fails (Playground link):
package main
import (
"encoding/json"
"fmt"
)
var respBytes = []byte("")
// Example data [12345, 23456, 34567]
func main() {
var jsonData []interface{}
err := json.Unmarshal(respBytes, &jsonData)
if err != nil {
fmt.Println(err)
}
fmt.Println(jsonData)
}
Replace respBytes empty string with example data and it works. Example data is simple JSON number array such as: "[12345, 23456, 34567]". How can I get it to work with data and without?
How can I get it to work with data and without?
Unfortunately, you can't, with the standard library json package. It validates the JSON input before (while) decoding it, and an empty string is not valid JSON, so it will err before you even have a chance to decode with a custom unmarshaler.
In this particular case, the only real option is to check for valid input first. Of course, you can do this with a wrapping function if you want:
func myUnmarshal(input []byte, target interface{}) error {
if len(input) == 0 {
return nil
}
return json.Unmarshal(input, target)
}
So there may be questions like this but its not a super easy thing to google. Basically I have a file thats a set of protobufs encoded and sequenced as they normally are from the protobuf spec.
So think of the bytes values being chunked something like this throughout the file:
[EncodeVarInt(size of protobuf struct)] [protobuf stuct bytes]
So you have a few bytes read one at a time that are used for large jump of a read on our protof structure.
My implementation using the os ReadAt method on a file currently looks something like this.
// getting the next value in a file context feature
func (geobuf *Geobuf_Reader) Next() bool {
if geobuf.EndPos <= geobuf.Pos {
return false
} else {
startpos := int64(geobuf.Pos)
for int(geobuf.Get_Byte(geobuf.Pos)) > 127 {
geobuf.Pos += 1
}
geobuf.Pos += 1
sizebytes := make([]byte,geobuf.Pos-int(startpos))
geobuf.File.ReadAt(sizebytes,startpos)
size,_ := DecodeVarint(sizebytes)
geobuf.Feat_Pos = [2]int{int(size),geobuf.Pos}
geobuf.Pos = geobuf.Pos+int(size)
return true
}
return false
}
// reads a geobuf feature as geojson
func (geobuf *Geobuf_Reader) Feature() *geojson.Feature {
// getting raw bytes
a := make([]byte,geobuf.Feat_Pos[0])
geobuf.File.ReadAt(a,int64(geobuf.Feat_Pos[1]))
return Read_Feature(a)
}
How can I implement something like bufio or other chunked reading mechanisms to speed up so many file ReadAt's? Most bufio implementations I've seen are for having a specific delimitter. Thanks in advance hopefully this wasn't a horrible question.
Package bufio
import "bufio"
type SplitFunc
SplitFunc is the signature of the split function used to tokenize the
input. The arguments are an initial substring of the remaining
unprocessed data and a flag, atEOF, that reports whether the Reader
has no more data to give. The return values are the number of bytes to
advance the input and the next token to return to the user, plus an
error, if any. If the data does not yet hold a complete token, for
instance if it has no newline while scanning lines, SplitFunc can
return (0, nil, nil) to signal the Scanner to read more data into the
slice and try again with a longer slice starting at the same point in
the input.
If the returned error is non-nil, scanning stops and the error is
returned to the client.
The function is never called with an empty data slice unless atEOF is
true. If atEOF is true, however, data may be non-empty and, as always,
holds unprocessed text.
type SplitFunc func(data []byte, atEOF bool) (advance int, token []byte, err error)
Use bufio.Scanner and write a custom protobuf struct SplitFunc.
I have a short Go program that runs the go list -json command for several packages, stores the output of each run of the command in a json.RawMessage, appends each json.RawMessage into a slice of json.RawMessages, and then returns the result to the server after concatenating each of the json.RawMessages together and compacting the json. However, there is an error message that gets produced when I run json.Compact that I can't locate the source of. Googling this error message reveals that most people who seem to encounter it--whether it's for an invalid , or some other character--have a hard time finding the source of it.
invalid character ',' looking for beginning of value
The code with comments is available to view here on play.golang.org (although it won't run there) and also below.
Question: can you explain the source of this error and how to prevent it?
(Note, some of the packages were included just for testing purposes)
package main
import (
"expvar"
"encoding/json"
"bytes"
"fmt"
"github.com/go-martini/martini"
"github.com/zenazn/goji"
"github.com/zenazn/goji/web"
"go/build"
"log"
"math/rand"
"net/http"
_ "net/http/pprof"
"os/exec"
)
type myType struct {
J []json.RawMessage
}
var pack map[string]string
type GoList struct {
Imports []string
}
type Import struct {
Dir string
ImportPath string
Name string
Target string
Standard bool
Root string
GoFiles []string
Imports []string
Deps []string
}
const contentTypeJSON = "application/json"
func main() {
http.HandleFunc("/importgraph", func(w http.ResponseWriter, r *http.Request) { importGraph(w, r) })
http.HandleFunc("/", handler)
http.ListenAndServe(":8080", nil)
}
func handler(w http.ResponseWriter, r *http.Request) {
fmt.Println("Inside handler")
fmt.Fprintf(w, "Hello world from my Go program!")
}
func importGraph(w http.ResponseWriter, r *http.Request) {
pack = make(map[string]string)
var t myType
cmd := exec.Command("go", "list", "-json")
stdout, err := cmd.Output()
if err != nil {
println(err.Error())
return
}
var list GoList
err = json.Unmarshal(stdout, &list)
for _, d := range list.Imports {
//get the imports for each of the packages listed by go list -json
t.imports(d)
}
var buff bytes.Buffer
//concatenate the separate json.RawMessages together into json
buff.WriteByte('[')
for i, j := range t.J {
if i != 0 {
buff.WriteByte(',')
}
buff.Write([]byte(j))
}
buff.WriteByte(']')
var buffer bytes.Buffer
if err := json.Compact(&buffer, buff.Bytes()); err != nil {
println(err.Error()) //error message: invalid character ',' looking for beginning of value
return
}
w.Header().Set("Content-Type", contentTypeJSON)
w.Write(buffer.Bytes())
}
func (myObj *myType) imports(pk string) error {
cmd := exec.Command("go", "list", "-json", pk)
stdout, _ := cmd.Output()
pack[pk] = pk
var deplist Import
json.Unmarshal(stdout, &deplist)
var newj json.RawMessage
json.Unmarshal(stdout, &newj)
myObj.J = append(myObj.J, newj)
for _, imp := range deplist.Imports {
if _, ok := pack[imp]; !ok {
myObj.imports(imp) //recursive call to get the imports of the imports etc
}
}
return nil
}
First, as has been commented, are you sure you can't use
the go/build package directly rather than running go list?
I Wouldn't use println (or fmt.Println) inside HTTP handlers. It's much better to use log.Println and/or get the error into the ResponseWriter. Also, it's a good idea to wrap your ListenAndServe call with log.Fatal.
When printing/logging error values you can just use err, no need to have err.Error().
Further, when you actually want to do something more detailed than just reporting/logging the error message you can look at it's type and other info. For example, log.Printf("verbose error info: %#v", err) gives:
&json.SyntaxError{msg:"invalid character ',' looking for beginning of value", Offset:0}
I tried this because I know the json package returns various error types with additional info and I was hoping the offset value would be of help. If it had been then something like this might have been helpful:
if err := json.Compact(…) {
if err != nil {
log.Println("json.Compact:", err)
if serr, ok := err.(*json.SyntaxError); ok {
log.Println("Occurred at offset:", serr.Offset)
// … something to show the data in buff around that offset …
}
}
}
But offset zero isn't helpful :(
So although this doesn't identify you problem hopefully
it can be of some help to your further investigation.
Edit:
So after adding:
log.Println("Write file:", ioutil.WriteFile("data.json", buff.Bytes(), 0600))
to the above error handling block I then ran a JSON validator on the resultant file and it identified this piece:
"XTestImports": [
"io",
"log",
"net"
]
},,{
"Dir": "/usr/local/go/src/mime",
"ImportPath": "mime",
"Name": "mime",
Note the double ,,.
That should tell you whete the error in your code is.
But if not, you need to skip empty entries, either when processing t.J or when you build it. The later is better and just involves:
if len(newj) > 0 {
myObj.J = append(myObj.J, newj)
}
(where btw you don't check for errors from json.Unmarshal so it's not clear if that is supposed to ever be empty or if it's empty due to a preceeding error. Never ignore error returns!)
I also encountered the same error message in a Go program, but the error message was within the HTTP response error, in HTML format when my HTTP response parser expected JSON.
For me, the solution was to change my request to include setting the Content-Type header to application/json. How you do this depends on which http client library you happen to be using; if you have access to the http.Header core type, you can set the header with .Set(...).
I realize the scope of this fix for me may not apply to the original question, but I came here first after googling and thought this would help others, since the message was not particularly obvious at first glance. The hint is that the invalid < character is the first HTML character in the error/response, which is likely the result of the request type not being set to application/json, thus the server responds with a non JSON response.
For me the issue was I was trying to parse the already parsed JSON.
I was also facing this error "invalid character 'N' looking for beginning of value".
This error was coming while "unmarshalling the non-json response into a json". I was expecting a json response, so wrote go code to unmarshal it into a json. But, due to URL change, the response that I was getting was a text ie. "404 Not found" error, which cannot be unmarshalled into a json.
"invalid character 'N' looking for beginning of value"
So, to summarise, this error appears when we are trying to unmarshal a non-json response (text/html/xml) into json.
Reason for this eerie error message is :
// When unmarshaling quoted strings, invalid UTF-8 or
// invalid UTF-16 surrogate pairs are not treated as an error.
// Instead, they are replaced by the Unicode replacement
// character U+FFFD.
https://golang.org/src/encoding/json/decode.go
In my case I saved my json as string then parsed it by :
stringData = JSON.parse(myJsonString)
I also had the same error another time using gin-context-ShouldBind() (https://godoc.org/github.com/gin-gonic/gin#Context.ShouldBind) and mapping my json to go object:
error was because it needs a json as string, so I used : JSON.stringify(jsonObject) when sending my request from front-end part.
And in case someone has the same problem as me, I needed to call JSON.stringify on my post data.
I encountered a similar problem with my error message being:
invalid character 'I' looking for beginning of value
In my case, i was trying to decode BSON using json.Unmarshal. Json doesn't recognize the ISODate type, which caused this error.
I had a similar issue. For me I omitted the first letter of my authorization token. So instead of
"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6InJhcGhhZWxuZ0BlbWFpbC5jb20iLCJleHAiOjE2MTM5NTQzMjB9.yPGC937VNAF8Qg05Z1x3fZ3zu_MUs-cA_Iag5-4RcJE"
I used this
"yJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6InJhcGhhZWxuZ0BlbWFpbC5jb20iLCJleHAiOjE2MTM5NTQzMjB9.yPGC937VNAF8Qg05Z1x3fZ3zu_MUs-cA_Iag5-4RcJE"