What's the format of timestamp to write parquet file in go - go

I am trying to write a Go struct in a Parquet file and upload to S3. What format and type do I specify for timestamp parameter in the struct so that athena displays correct timestamp when reading from the parquet file.
type example struct {
ID int64 `parquet:"name=id, type=INT64"`
CreatedAt int64 `parquet:"name=created_at,type=TIMESTAMP_MILLIS"`
}
ex := example{}
ex.ID = int64(10)
ex.CreatedAt = time.Now().Unix()
fw, err := ParquetFile.NewLocalFileWriter("new.parquet")
pw, err := ParquetWriter.NewParquetWriter(fw, new(example), 1)
pw.Write(ex)
Upload the file new.parquet to S3
Reference - https://github.com/xitongsys/parquet-go. I created a table in Athena with int and timestamp field for the same and trying querying the table. The date is showing something like - 1970-01-18 21:54:23.751.
which no where matches the current timestamp.

For example,
package main
import (
"fmt"
"time"
)
func main() {
type example struct {
CreatedAt int64 `parquet:"name=created_at,type=TIMESTAMP_MILLIS"`
}
ex := example{}
ex.CreatedAt = time.Now().UnixNano() / int64(time.Millisecond)
fmt.Println(ex.CreatedAt)
}
Playground: https://play.golang.org/p/ePOlUKiT6fD
Output:
1257894000000

Related

db.First() not using primary key name

I am writing some sample code to understand gorm but appear to be having an issue with setting up the primary_key value. As a result, the resulting SQL query is broken.
Please see the sample code:
package main
import (
"os"
"fmt"
"time"
"gorm.io/driver/postgres"
"gorm.io/gorm"
)
type Post struct {
id int `json:"id" gorm:"primary_key:id"`
url string `gorm:"url"`
content string `gorm:"content"`
created_at time.Time `gorm:"created_at"`
normalized string `gorm:"normalized"`
account_id int `gorm:"account_id"`
posthash []byte `gorm:"posthash"`
received_at time.Time `gorm:"received_at"`
}
func main() {
dsn := "user=postgres dbname=localtest"
db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{})
if err != nil {
fmt.Println("An error occurred", err)
os.Exit(1)
}
db.AutoMigrate(&Post{})
var post Post
db.First(&post, 1)
fmt.Println(post)
}
When I run this code, I receive the following error:
$ go run gorm_test.go
2020/12/01 00:34:06 /home/farhan/gorm_test.go:32 ERROR: syntax error at or near "=" (SQLSTATE 42601)
[0.128ms] [rows:0] SELECT * FROM "posts" WHERE "posts". = 1 ORDER BY "posts". LIMIT 1
{0 {0 0 <nil>} 0 [] {0 0 <nil>}}
The nature of this error suggests to me that the primary_key value is not set. I tried primaryKey and primarykey, but none appeared to work. Any ideas?
#Shubham Srivastava - Your structure declaration is bad; you need to export fields so gorm can use them Declaring Models
❗️Use exported ID field
In Go, when a field name starts with a lowercase letter that means the field is private (called unexported in Go parlance). Gorm, being a third party package, cannot see inside your struct to know there is an id field, or any other unexported field for that matter.
The solution is to make sure all the fields that need to come from the DB are exported:
type Post struct {
ID uint `json:"id" gorm:"primaryKey"`
...
}

sql.Rows to slice of strings

I'm using the standard go sql package to interface with AWS Athena.
My query returns for each record a uuid (string) and an array of emails.
Here is the code:
package main
import (
"fmt"
"database/sql"
_ "github.com/segmentio/go-athena"
_"encoding/json"
)
type Contact struct {
userid string
emails []string
}
func main() {
fmt.Println("hello")
db, err := sql.Open("athena", "db=example")
if err != nil {
panic(err)
}
rows, err := db.Query("SELECT userid, transform(value.emails, x -> x.value) from database LIMIT 10")
// Returns
// Row 1: "abc-123", ["email1#gmail.com", "email2#gmail.com"]
// Row 2: "def-456", ["email3#gmail.com"]
if err != nil {
panic(err)
}
for rows.Next() {
var contact Contact
rows.Scan(&contact.userid, &contact.emails)
fmt.Println(contact)
}
}
However, I get this error in the for loop:
panic: unknown type `array` with value [email1#gmail.com]
I'm confused about the array type mentioned and I can't make sense of the error.
How can I map the list of emails returned to a slice of strings in the Contact struct ?
Athena supports structural data types. For example the structural data type array:
Structural types
ARRAY < data_type >
From the message you get I assume the column email is of type ARRAY<VARCHAR>. In addition segmentio/go-athena panics on unsupported operations like Begin for a transaction (which are not supported in Athena). To read data in a Go array you have to put in some logic. See read "SELECT *" columns into []string in go or Read a Postgresql array directly into a Golang Slice for a starter. As you can see with the pq driver, reading an array might be implemented differently than just scanning a row

How to get time without timezone? [duplicate]

I'm trying to add some values from my database to a []string in Go. Some of these are timestamps.
I get the error:
cannot use U.Created_date (type time.Time) as type string in array element
Can I convert time.Time to string?
type UsersSession struct {
Userid int
Timestamp time.Time
Created_date time.Time
}
type Users struct {
Name string
Email string
Country string
Created_date time.Time
Id int
Hash string
IP string
}
-
var usersArray = [][]string{}
rows, err := db.Query("SELECT u.id, u.hash, u.name, u.email, u.country, u.IP, u.created_date, us.timestamp, us.created_date FROM usersSession AS us LEFT JOIN users AS u ON u.id = us.userid WHERE us.timestamp + interval 30 minute >= now()")
U := Users{}
US := UsersSession{}
for rows.Next() {
err = rows.Scan(&U.Id, &U.Hash, &U.Name, &U.Email, &U.Country, &U.IP, &U.Created_date, &US.Timestamp, &US.Created_date)
checkErr(err)
userid_string := strconv.Itoa(U.Id)
user := []string{userid_string, U.Hash, U.Name, U.Email, U.Country, U.IP, U.Created_date, US.Timestamp, US.Created_date}
// -------------
// ^ this is where the error occurs
// cannot use U.Created_date (type time.Time) as type string in array element (for US.Created_date and US.Timestamp aswell)
// -------------
usersArray = append(usersArray, user)
log.Print("usersArray: ", usersArray)
}
EDIT
I added the following. It works now, thanks.
userCreatedDate := U.Created_date.Format("2006-01-02 15:04:05")
userSessionCreatedDate := US.Created_date.Format("2006-01-02 15:04:05")
userSessionTimestamp := US.Timestamp.Format("2006-01-02 15:04:05")
You can use the Time.String() method to convert a time.Time to a string. This uses the format string "2006-01-02 15:04:05.999999999 -0700 MST".
If you need other custom format, you can use Time.Format(). For example to get the timestamp in the format of yyyy-MM-dd HH:mm:ss use the format string "2006-01-02 15:04:05".
Example:
t := time.Now()
fmt.Println(t.String())
fmt.Println(t.Format("2006-01-02 15:04:05"))
Output (try it on the Go Playground):
2009-11-10 23:00:00 +0000 UTC
2009-11-10 23:00:00
Note: time on the Go Playground is always set to the value seen above. Run it locally to see current date/time.
Also note that using Time.Format(), as the layout string you always have to pass the same time –called the reference time– formatted in a way you want the result to be formatted. This is documented at Time.Format():
Format returns a textual representation of the time value formatted according to layout, which defines the format by showing how the reference time, defined to be
Mon Jan 2 15:04:05 -0700 MST 2006
would be displayed if it were the value; it serves as an example of the desired output. The same display rules will then be applied to the time value.
package main
import (
"fmt"
"time"
)
// #link https://golang.org/pkg/time/
func main() {
//caution : format string is `2006-01-02 15:04:05.000000000`
current := time.Now()
fmt.Println("origin : ", current.String())
// origin : 2016-09-02 15:53:07.159994437 +0800 CST
fmt.Println("mm-dd-yyyy : ", current.Format("01-02-2006"))
// mm-dd-yyyy : 09-02-2016
fmt.Println("yyyy-mm-dd : ", current.Format("2006-01-02"))
// yyyy-mm-dd : 2016-09-02
// separated by .
fmt.Println("yyyy.mm.dd : ", current.Format("2006.01.02"))
// yyyy.mm.dd : 2016.09.02
fmt.Println("yyyy-mm-dd HH:mm:ss : ", current.Format("2006-01-02 15:04:05"))
// yyyy-mm-dd HH:mm:ss : 2016-09-02 15:53:07
// StampMicro
fmt.Println("yyyy-mm-dd HH:mm:ss: ", current.Format("2006-01-02 15:04:05.000000"))
// yyyy-mm-dd HH:mm:ss: 2016-09-02 15:53:07.159994
//StampNano
fmt.Println("yyyy-mm-dd HH:mm:ss: ", current.Format("2006-01-02 15:04:05.000000000"))
// yyyy-mm-dd HH:mm:ss: 2016-09-02 15:53:07.159994437
}
package main
import (
"fmt"
"time"
)
func main() {
v , _ := time.Now().UTC().MarshalText()
fmt.Println(string(v))
}
Output : 2009-11-10T23:00:00Z
Go Playground
Please find the simple solution to convete Date & Time Format in Go Lang. Please find the example below.
Package Link: https://github.com/vigneshuvi/GoDateFormat.
Please find the plackholders:https://medium.com/#Martynas/formatting-date-and-time-in-golang-5816112bf098
package main
// Import Package
import (
"fmt"
"time"
"github.com/vigneshuvi/GoDateFormat"
)
func main() {
fmt.Println("Go Date Format(Today - 'yyyy-MM-dd HH:mm:ss Z'): ", GetToday(GoDateFormat.ConvertFormat("yyyy-MM-dd HH:mm:ss Z")))
fmt.Println("Go Date Format(Today - 'yyyy-MMM-dd'): ", GetToday(GoDateFormat.ConvertFormat("yyyy-MMM-dd")))
fmt.Println("Go Time Format(NOW - 'HH:MM:SS'): ", GetToday(GoDateFormat.ConvertFormat("HH:MM:SS")))
fmt.Println("Go Time Format(NOW - 'HH:MM:SS tt'): ", GetToday(GoDateFormat.ConvertFormat("HH:MM:SS tt")))
}
func GetToday(format string) (todayString string){
today := time.Now()
todayString = today.Format(format);
return
}
strconv.Itoa(int(time.Now().Unix()))
Go Playground
http://play.golang.org/p/DN5Py5MxaB
package main
import (
"fmt"
"time"
)
func main() {
t := time.Now()
// The Time type implements the Stringer interface -- it
// has a String() method which gets called automatically by
// functions like Printf().
fmt.Printf("%s\n", t)
// See the Constants section for more formats
// http://golang.org/pkg/time/#Time.Format
formatedTime := t.Format(time.RFC1123)
fmt.Println(formatedTime)
}

GET blob from sql database Golang

import (
"database/sql"
"encoding/json"
"fmt"
_ "github.com/go-sql-driver/mysql"
)
type User struct {
Name string `json:name`
Picture []uint8 `json:picture`
}
func main(){
//straight to the query
rows, err := 'SELECT name, picture FROM ms_users' // picture is longblob type in database
checkErr(err)
var usr User
for rows.Next(){
err = rows.Scan(&usr.Name, &usr.Picture)
checkErr(err)
}
jsn, err := json.Marshal(usr)
fmt.Printf("%v, "string(jsn))
}
With above code, I only get name value but the picture is empty.
How do I store blob value from databse to struct ?
Any answer will be appreciated! thank you!
I'm relatively new to GO I encountered this question while searching a solution for a similar problem I was able to find a solution.
When you get BLOB data from the database you get it as type []byte your struct can look like this below
type User struct {
Name string `json:name`
Picture []byte`json:picture`
}
I guess you can process the byte array according to you need later. In my case I needed a JSON object so I unmarshalled it to a type interface{} variable.

struct Time property doesn't load from Go sqlx library

I have a struct with a property time:
type Basket struct {
...
Created_at time.Time `db:"created_at"`
}
with the time saved as:
basket.Created_at = time.Now().UTC()
If I save it using Insert sql statement, it saves the time nicely in the SQLite3 but when I select the desired record using:
ret_basket := Basket{}
err := database.DB.Get(&ret_basket, "SELECT id, ..., created_at FROM baskets WHERE user_id = ?", some_user_id)
It returns the record with other properties loaded properly except the time property which is ret_basket.Created_at as 0001-01-01 00:00:00 +0000 UTC
Any suggestions?
There is no official package for Sqlite so I assume you are using https://github.com/mattn/go-sqlite3 Probably your issue is result of wrong created_at field declaration in database which should be DATETIME because the next code works perfectly on my machine (I've removed all error checks):
package main
import (
"github.com/jmoiron/sqlx"
_ "github.com/mattn/go-sqlite3"
"log"
"time"
)
type Post struct {
Id int64 `db:"post_id"`
Created time.Time `db:"created"`
}
func main() {
db, _ := sqlx.Connect("sqlite3", "post_db.db")
db.MustExec("DROP TABLE IF EXISTS posts; CREATE TABLE posts (post_id INT, created DATETIME);")
p1 := Post{Id: 1, Created: time.Now().UTC()}
p2 := Post{}
tx := db.MustBegin()
tx.NamedExec("INSERT INTO posts (post_id, created) VALUES (:post_id, :created)", &p1)
tx.Commit()
db.Get(&p2, "SELECT post_id, created FROM posts WHERE post_id = $1", p1.Id)
log.Println(p2.Created.Format("2006-01-02"))
}

Resources