How to parse non standard time format from json - go

lets say i have the following json
{
name: "John",
birth_date: "1996-10-07"
}
and i want to decode it into the following structure
type Person struct {
Name string `json:"name"`
BirthDate time.Time `json:"birth_date"`
}
like this
person := Person{}
decoder := json.NewDecoder(req.Body);
if err := decoder.Decode(&person); err != nil {
log.Println(err)
}
which gives me the error parsing time ""1996-10-07"" as ""2006-01-02T15:04:05Z07:00"": cannot parse """ as "T"
if i were to parse it manually i would do it like this
t, err := time.Parse("2006-01-02", "1996-10-07")
but when the time value is from a json string how do i get the decoder to parse it in the above format?

That's a case when you need to implement custom marshal and unmarshal functions.
UnmarshalJSON(b []byte) error { ... }
MarshalJSON() ([]byte, error) { ... }
By following the example in the Golang documentation of json package you get something like:
// First create a type alias
type JsonBirthDate time.Time
// Add that to your struct
type Person struct {
Name string `json:"name"`
BirthDate JsonBirthDate `json:"birth_date"`
}
// Implement Marshaler and Unmarshaler interface
func (j *JsonBirthDate) UnmarshalJSON(b []byte) error {
s := strings.Trim(string(b), "\"")
t, err := time.Parse("2006-01-02", s)
if err != nil {
return err
}
*j = JsonBirthDate(t)
return nil
}
func (j JsonBirthDate) MarshalJSON() ([]byte, error) {
return json.Marshal(time.Time(j))
}
// Maybe a Format function for printing your date
func (j JsonBirthDate) Format(s string) string {
t := time.Time(j)
return t.Format(s)
}

If there are lots of struct and you just implement custom marshal und unmarshal functions, that's a lot of work to do so. You can use another lib instead,such as a json-iterator extension jsontime:
import "github.com/liamylian/jsontime"
var json = jsontime.ConfigWithCustomTimeFormat
type Book struct {
Id int `json:"id"`
UpdatedAt *time.Time `json:"updated_at" time_format:"sql_date" time_utc:"true"`
CreatedAt time.Time `json:"created_at" time_format:"sql_datetime" time_location:"UTC"`
}

I wrote a package for handling yyyy-MM-dd and yyyy-MM-ddThh:mm:ss dates at https://github.com/a-h/date
It uses the type alias approach in the answer above, then implements the MarshalJSON and UnmarshalJSON functions with a few alterations.
// MarshalJSON outputs JSON.
func (d YYYYMMDD) MarshalJSON() ([]byte, error) {
return []byte("\"" + time.Time(d).Format(formatStringYYYYMMDD) + "\""), nil
}
// UnmarshalJSON handles incoming JSON.
func (d *YYYYMMDD) UnmarshalJSON(b []byte) (err error) {
if err = checkJSONYYYYMMDD(string(b)); err != nil {
return
}
t, err := time.ParseInLocation(parseJSONYYYYMMDD, string(b), time.UTC)
if err != nil {
return
}
*d = YYYYMMDD(t)
return
}
It's important to parse in the correct timezone. My code assumes UTC, but you may wish to use the computer's timezone for some reason.
I also found that solutions which involved using the time.Parse function leaked Go's internal mechanisms as an error message which clients didn't find helpful, for example: cannot parse "sdfdf-01-01" as "2006". That's only useful if you know that the server is written in Go, and that 2006 is the example date format, so I put in more readable error messages.
I also implemented the Stringer interface so that it gets pretty printed in log or debug messages.

Related

How do I bind a date string to a struct?

type TestModel struct {
Date time.Time `json:"date" form:"date" gorm:"index"`
gorm.Model
}
i'm using echo framwork, and
I have a struct like the one above, and I get string data like '2021-09-27' , how can I bind it to the struct?
func CreateDiary(c echo.Context) error {
var getData model.TestModel
if err := (&echo.DefaultBinder{}).BindBody(c, &getData); err != nil {
fmt.Print(err.Error())
}
return c.JSON(200, getData)
}
When I code like this, I get the following error:
code=400, message=parsing time "2021-09-27" as "2006-01-02T15:04:05Z07:00": cannot parse "" as "T", internal=parsing time "2021-09-27" as "2006-01-02T15:04:05Z07:00": cannot parse "" as "T"
I'm a golang beginner, can you show me a simple example??, please.
i'm using echo framwork
Type CustomTime time.Time
func (ct *CustomTime) UnmarshalParam(param string) error {
t, err := time.Parse(`2006-01-02`, param)
if err != nil {
return err
}
*ct = CustomTime(t)
return nil
}
ref: https://github.com/labstack/echo/issues/1571
Here is list of available tags used in echo. If you want to parse from body, then use json
query - source is request query parameters.
param - source is route path parameter.
header - source is header parameter.
form - source is form. Values are taken from query and request body. Uses Go standard library form parsing.
json - source is request body. Uses Go json package for unmarshalling.
xml - source is request body. Uses Go xml package for unmarshalling.
You need to wrap time.Time into custom struct and then implement json.Marshaler and json.Unmarshaler interfaces
Example
package main
import (
"fmt"
"strings"
"time"
"github.com/labstack/echo/v4"
)
type CustomTime struct {
time.Time
}
type TestModel struct {
Date CustomTime `json:"date"`
}
func (t CustomTime) MarshalJSON() ([]byte, error) {
date := t.Time.Format("2006-01-02")
date = fmt.Sprintf(`"%s"`, date)
return []byte(date), nil
}
func (t *CustomTime) UnmarshalJSON(b []byte) (err error) {
s := strings.Trim(string(b), "\"")
date, err := time.Parse("2006-01-02", s)
if err != nil {
return err
}
t.Time = date
return
}
func main() {
e := echo.New()
e.POST("/test", CreateDiary)
e.Logger.Fatal(e.Start(":1323"))
}
func CreateDiary(c echo.Context) error {
var getData TestModel
if err := (&echo.DefaultBinder{}).BindBody(c, &getData); err != nil {
fmt.Print(err.Error())
}
return c.JSON(200, getData)
}
test
curl -X POST http://localhost:1323/test -H 'Content-Type: application/json' -d '{"date":"2021-09-27"}'

write custom Marshall and Unmarshaller for proto buf type in golang

I have custom type that I wrote my own Marshall and Unmarshaller
And the problem is I Want to do the same using protobuf
I just want to implement the same using protobuf, so that I can implement my own Marshall and Unmarshaller
syntax="proto3";
package main;
message NullInt64{
bool Valid = 1;
int64 Int64 = 2;
}
In the way that if the Valid value is false in return the null string
type NullInt64 struct {
Int64 int64
Valid bool
}
// MarshalJSON try to marshaling to json
func (nt NullInt64) MarshalJSON() ([]byte, error) {
if nt.Valid {
return []byte(fmt.Sprintf(`%d`, nt.Int64)), nil
}
return []byte("null"), nil
}
// UnmarshalJSON try to unmarshal dae from input
func (nt *NullInt64) UnmarshalJSON(b []byte) error {
text := strings.ToLower(string(b))
if text == "null" {
nt.Valid = false
return nil
}
err := json.Unmarshal(b, &nt.Int64)
if err != nil {
return err
}
nt.Valid = true
return nil
}
Protoc will not generate MarshalJSON and UnmarshalJSON functions.
You can:
Use a different protobuf generator (see gogo/protobuf and it's many extensions or fork golang/protobuf to change their generator)
Insert your own functions into the proto package by adding a file to that folder. You can either hand-write or code gen these functions.

How do I Override UnmarshalJSON Correctly?

I am trying to write a simple custom marshaler and failing. Notice I have an interface that has three functions. Both Happy and Sad structs implement this interface by embedding the emotion struct which implements all the three required functions.
The problem is UnmarshalJSON does not get invoked when I call json.Unmarshal() on the pointer to either Happy or Sad and I can't understand why. You can reproduce the exact codebase in Go Playground or just look below. You will notice that while MarshalJSON is correctly called, UnmarshalJSON isn't.
type Emotion interface {
String() string
MarshalJSON() ([]byte, error)
UnmarshalJSON(data []byte) error
}
type emotion struct {
status string
}
func (s emotion) String() string {
return s.status
}
func (s emotion) MarshalJSON() ([]byte, error) {
fmt.Println("MarshalJSON is overriden: I am called fine")
x := struct {
Status string
}{
Status: s.String(),
}
return json.Marshal(x)
}
func (s *emotion) UnmarshalJSON(data []byte) error {
fmt.Println("MarshalJSON is overriden: I am never called")
y := struct {
Status string
}{
Status: "",
}
err := json.Unmarshal(data, &y)
if err != nil {
return err
}
s.status = y.Status
return nil
}
type Happy struct {
*emotion
}
// Job is not in any detention
type Sad struct {
*emotion
}
func main() {
x := Happy{&emotion{status: "happy"}}
jsonX, _ := json.Marshal(x)
var y Emotion
err := json.Unmarshal(jsonX, &y)
fmt.Printf("%v", err)
}
You cannot unmarshal into an abstract interface type.
An interface value is just a pointer to a type (associating that types methods) - it has no storage behind it - because an abstract type cannot know the exact size of any concrete value it may have in the future.
Using a concrete value type (that also implements that interface) will work:
y2 := emotion{}
err = json.Unmarshal(jsonX, &y2)
Playground: https://play.golang.org/p/8aCEjLgfKVQ
MarshalJSON is overriden: I am called fine
EXPECTED ERROR, Can't unmarshal into a non-concrete value: json: cannot unmarshal object into Go value of type main.Emotion
MarshalJSON is overriden: I am (fixed) and now called
SHOULD NOT ERROR: <nil>
VALUE: happy

Redigo ScanStruct error with time.Time

I am trying to read a struct that has a field of type time.Time using redigo's ScanStruct, which gives me the following error: cannot convert from Redis bulk string to time.Time.
Is the only way of fixing this to create my own time type that extends time.Time and implements RedisScan? That sounds bad as well...
Since Redis has no concept of time values it would make no sense for a generic driver such as redigo to perform some automatic conversion between the builin time.Time type and an arbitrary byte array. As such, it's up to the programmer to decide how to perform that conversion.
For example, supposing you have a "Person" type defined as such, including a created_at timestamp formatted as RFC3339 (a form of ISO 8601), you could define a custom "Timestamp" type with a "RedisScan" method as follows:
type Timestamp time.Time
type Person struct {
Id int `redis:"id"`
Name string `redis:"name"`
CreatedAt Timestamp `redis:"created_at"`
}
func (t *Timestamp) RedisScan(x interface{}) error {
bs, ok := x.([]byte)
if !ok {
return fmt.Errorf("expected []byte, got %T", x)
}
tt, err := time.Parse(time.RFC3339, string(bs))
if err != nil {
return err
}
*t = Timestamp(tt)
return nil
}
// ...
response, err := redis.Values(conn.Do("HGETALL", "person:1"))
if err != nil {
panic(err)
}
var p Person
err = redis.ScanStruct(response, &p)
if err != nil {
panic(err)
}
log.Printf("OK: p=%v", p)

composite type and Pointer Methods

i have some problems creating a own type Date based on type time.Time
i tried to create the Date type as follows:
type Date time.Time
func (d Date) UnmarshalJSON(buf []byte) error {
var s string
json.Unmarshal(buf, &s)
t, err := time.Parse(timeLayout,s)
d= Date(t)
return err
}
func (d Date) MarshalJSON() ([]byte, error) {
b,_ := json.Marshal(d.Format(timeLayout))
return b,nil
}
this itself works, i can store this Date as a time.Time in AppEngine Datastore.
the marshaling itself also works,
but is not working is: then when unmarshal from json, the Date d is filled with the value.
this is, as i understand right, because the unmarshalJson function create a copy of Date.
so when i change the unmarshalJson function to use a Pointer to Date
then i cant use:
d=Date(t)
so first question, is there a solution how todo this ?
what i have done now is to rewrite the Code as follows:
type Date struct {
t time.Time
}
func (d *Date) UnmarshalJSON(buf []byte) error {
var s string
json.Unmarshal(buf, &s)
t, err := time.Parse(timeLayout,s)
d.t = t
return err
}
func (d Date) MarshalJSON() ([]byte, error) {
b,_ := json.Marshal(d.t.Format(timeLayout))
return b,nil
}
this works, but in this case Date is not an extending type of time.Time its just a wrapper to a time.Time type.
is there a better solution todo this ? im still new to go.
i need this Date type, to have a Date only json formated type, because Chrome only supports the html5 type: date and not datetime.
and method overriding is not possible in go (means to override the un/marshalJson methods of type time.Time ) ?
thanks
Totally untested code:
type Date time.Time
func (d *Date) UnmarshalJSON(buf []byte) error {
var s string
json.Unmarshal(buf, &s)
t, err := time.Parse(timeLayout, s)
*d = *(*Date)(&t)
return err
}
func (d *Date) MarshalJSON() ([]byte, error) {
b, _ := json.Marshal(d.Format(timeLayout))
return b, nil
}

Resources