Using Go, when fetching sheet data, the data is arriving with its applied cell formatting
i.e. "$123,456" while I need the original 123456.
is there something in the api that can remove formatting? like formatting: false
code:
package main
import (
"log"
"golang.org/x/net/context"
"golang.org/x/oauth2/google"
"gopkg.in/Iwark/spreadsheet.v2"
)
func main() {
service := authenticate()
spreadsheet, err := service.FetchSpreadsheet(spreadsheetID)
checkError(err)
sheet, err := spreadsheet.SheetByIndex(1)
checkError(err)
for _, row := range sheet.Rows {
var csvRow []string
for _, cell := range row {
csvRow = append(csvRow, cell.Value)
}
log.Println(csvRow)
}
}
// function to authenticate on Google
func authenticate() *spreadsheet.Service {
data, err := ioutil.ReadFile("secret.json")
checkError(err)
conf, err := google.JWTConfigFromJSON(data, spreadsheet.Scope)
checkError(err)
client := conf.Client(context.TODO())
service := spreadsheet.NewServiceWithClient(client)
return service
}
func checkError(err error) {
if err != nil {
panic(err.Error())
}
}
Yes, there is a way. In the Method: spreadsheets.values.get endpoint, there is a request parameter called valueRenderOption, which one of its values is UNFORMATTED_VALUE, as its name suggests, it will give you back all the data without format.
Try this Request with the range of values you want to play around with the API and see in that way the unformatted values it will return.
You want to retrieve $123,456 as 123456 from Google Spreadsheet.
$123,456 is shown by the cell format. It's actually the number.
You want to achieve this using gopkg.in/Iwark/spreadsheet.v2 with golang.
You have already been able to get and put values for Google Spreadsheet using the service account with Sheets API.
If my understanding is correct, how about this answer? Please think of this as just one of several possible answers.
Modification points:
When I saw the script of the library of gopkg.in/Iwark/spreadsheet.v2, I noticed that the values are retrieved by the method of spreadsheets.get in Sheets API.
Furthermore, it was found that spreadsheetId,properties.title,sheets(properties,data.rowData.values(formattedValue,note)) was used as the fields. It seems that the fields is constant.
The reason of your issue is that the values are retrieved with formattedValue. In your case, the values are required to be retrieved with userEnteredValue.
When you want to achieve your goal using the library of gopkg.in/Iwark/spreadsheet.v2, in order to reflect above to the library, it is required to modify the script of library.
Modified script:
Please modify the files of gopkg.in/Iwark/spreadsheet.v2 as follows. Of course, please backup the original files in order to back to the original library.
1. service.go
Modify the line 116 as follows.
From:
fields := "spreadsheetId,properties.title,sheets(properties,data.rowData.values(formattedValue,note))"
To:
fields := "spreadsheetId,properties.title,sheets(properties,data.rowData.values(formattedValue,note,userEnteredValue))"
2. sheet.go
Modify the line 52 as follows.
From:
Value: cellData.FormattedValue,
To:
Value: strconv.FormatFloat(cellData.UserEnteredValue.NumberValue, 'f', 4, 64),
And add "strconv" to import section like below.
import (
"encoding/json"
"strings"
"strconv"
)
3. cell_data.go
Modify the line 8 as follows.
From:
// UserEnteredFormat *CellFormat `json:"userEnteredFormat"`
To:
UserEnteredFormat struct {
NumberValue float64 `json:"numberValue"`
} `json:"userEnteredFormat"`
Result:
In this case, your script is not required to be modified. After above modification, when you run your script, you can see [123456.0000] at the console. As an important point, it seems that this library uses the values as the string type. In this modification, I used this. But if you want to use it as other type, please modify the library.
Other pattern:
As the other pattern for achieving your goal, how about using google-api-go-client? About this, you can see it at Go Quickstart. When google-api-go-client is used, the sample script becomes as follows. In this case, as a test case, the method of spreadsheets.get was used.
Sample script 1:
In this sample script, authenticate() and checkError() in your script are used by modifying.
package main
import (
"fmt"
"io/ioutil"
"net/http"
"golang.org/x/net/context"
"golang.org/x/oauth2/google"
"google.golang.org/api/sheets/v4"
)
func main() {
c := authenticate()
sheetsService, err := sheets.New(c)
checkError(err)
spreadsheetId := "###" // Please set the Spreadsheet ID.
ranges := []string{"Sheet1"} // Please set the sheet name.
resp, err := sheetsService.Spreadsheets.Get(spreadsheetId).Ranges(ranges...).Fields("sheets.data.rowData.values.userEnteredValue").Do()
checkError(err)
for _, row := range resp.Sheets[0].Data[0].RowData {
for _, col := range row.Values {
fmt.Println(col.UserEnteredValue)
}
}
}
func authenticate() *http.Client {
data, err := ioutil.ReadFile("serviceAccount_20190511.json")
checkError(err)
conf, err := google.JWTConfigFromJSON(data, sheets.SpreadsheetsScope)
checkError(err)
client := conf.Client(context.TODO())
return client
}
func checkError(err error) {
if err != nil {
panic(err.Error())
}
}
Sample script 2:
When spreadsheets.values.get is used, the script of main() is as follows.
func main() {
c := authenticate()
sheetsService, err := sheets.New(c)
checkError(err)
spreadsheetId := "###" // Please set the Spreadsheet ID.
sheetName := "Sheet1" // Please set the sheet name.
resp, err := sheetsService.Spreadsheets.Values.Get(spreadsheetId, sheetName).ValueRenderOption("UNFORMATTED_VALUE").Do()
checkError(err)
fmt.Println(resp.Values)
}
Here, UNFORMATTED_VALUE is used for retrieving the values without the cell format. This has already been answered by alberto vielma
References:
Method: spreadsheets.get
google-api-go-client
Go Quickstart
Method: spreadsheets.values.get
If I misunderstood your question and this was not the direction you want, I apologize.
Related
I have a Gorm delete with the returning result:
expirationDate := time.Now().UTC().Add(-(48 * time.hour))
var deletedUsers Users
res := gormDB.WithContext(ctx).
Table("my_users").
Clauses(clause.Returning{Columns: []clause.Column{{Name: "email"}}}).
Where("created_at < ?", expirationDate).
Delete(&deletedUsers)
Now the test with clauses always fails. e.g. :
sqlMock.ExpectExec(`DELETE`)
.WithArgs(expirationDate)
.WillReturnResult(sqlmock.NewResult(1, 1))
Receiving error:
"call to Query 'DELETE FROM "my_users" WHERE created_at < $1 RETURNING "email"' with args [{Name: Ordinal:1 Value:2023-01-18 06:15:34.694274 +0000 UTC}], was not expected, next expectation is: ExpectedExec => expecting Exec or ExecContext which:\n - matches sql: 'DELETE'\n - is with arguments:\n 0 - 2023-01-18 06:15:34.694274 +0000 UTC\n - should return Result having:\n LastInsertId: 1\n RowsAffected: 1"
I tried many other sqlMock expectations, but they have a similar issue.
Also, we don't have a return value in ExpectExec, only in ExpectQuery...
Any chance someone has to test the Gorm query with the Clauses?
I was able to successfully manage what you need. First, let me share the files I wrote, and then I'll walk you through all of the relevant changes. The files are repo.go for production and repo_test.go for the test code.
repo.go
package gormdelete
import (
"context"
"time"
"gorm.io/gorm"
"gorm.io/gorm/clause"
)
type Users struct {
Email string
}
func Delete(ctx context.Context, gormDB *gorm.DB) error {
expirationDate := time.Now().UTC().Add(-(48 * time.Hour))
var deletedUsers Users
res := gormDB.WithContext(ctx).Table("my_users").Clauses(clause.Returning{Columns: []clause.Column{{Name: "email"}}}).Where("created_at < ?", expirationDate).Delete(&deletedUsers)
if res.Error != nil {
return res.Error
}
return nil
}
As you didn't provide the full file I tried to guess what was missing.
repo_test.go
package gormdelete
import (
"context"
"database/sql/driver"
"testing"
"time"
"github.com/DATA-DOG/go-sqlmock"
"github.com/stretchr/testify/assert"
"gorm.io/driver/postgres"
"gorm.io/gorm"
)
// this is taken directly from the docs
// https://github.com/DATA-DOG/go-sqlmock#matching-arguments-like-timetime
type AnyTime struct{}
// Match satisfies sqlmock.Argument interface
func (a AnyTime) Match(v driver.Value) bool {
_, ok := v.(time.Time)
return ok
}
func TestDelete(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatalf("an error was not expected: %v", err)
}
conn, _ := db.Conn(context.Background())
gormDb, err := gorm.Open(postgres.New(postgres.Config{
Conn: conn,
}))
row := sqlmock.NewRows([]string{"email"}).AddRow("test#example.com")
mock.ExpectBegin()
mock.ExpectQuery("DELETE FROM \"my_users\" WHERE created_at < ?").WithArgs(AnyTime{}).WillReturnRows(row)
mock.ExpectCommit()
err = Delete(context.Background(), gormDb)
assert.Nil(t, err)
if err = mock.ExpectationsWereMet(); err != nil {
t.Errorf("not all expectations were met: %v", err)
}
}
Here, there are more changes that it's worth mentioning:
I instantiated the AnyTime as per the documentation (you can see the link in the comment).
Again, I guessed the setup of the db, mock, and gormDb but I think it should be more or less the same.
I switch the usage of ExpectExec to ExpectQuery as we'll have back a result set as specified by the Clauses method in your repo.go file.
You've to wrap the ExpectQuery within an ExpectBegin and an ExpectCommit.
Finally, pay attention to the difference in how the driver expects the parameters in the SQL statement. In the production code, you can choose to use ? or $1. However, in the test code, you can only use the ? otherwise it won't match the expectations.
Hope to help a little bit, otherwise, let me know!
I am a newbie to Go. Was starting to write my first code in which I have to download a bunch of CSV's from AWS. I don't understand why it is giving me the below error with O_APPEND mode. If I remove os.O_APPEND, I only get the last file data which is not the objective.
The objective is to download all CSV files into one file locally. I'd like to understand what I'm doing incorrectly.
package main
import (
"fmt"
"os"
"path/filepath"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
"github.com/aws/aws-sdk-go/service/s3/s3manager"
)
const (
AccessKeyId = "xxxxxxxxx"
SecretAccessKey = "xxxxxxxxxxxxxxxxxxxx"
Region = "eu-central-1"
Bucket = "dexter-reports"
bucketKey = "Jenkins/pluginVersions/"
)
func main() {
// Load the Shared AWS Configuration
os.Setenv("AWS_ACCESS_KEY_ID", AccessKeyId)
os.Setenv("AWS_SECRET_ACCESS_KEY", SecretAccessKey)
filename := "JenkinsPluginDetais.txt"
cred := credentials.NewStaticCredentials(AccessKeyId, SecretAccessKey, "")
config := aws.Config{Credentials: cred, Region: aws.String(Region), Endpoint: aws.String("s3.amazonaws.com")}
file, err := os.OpenFile(filename, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0666)
if err != nil {
panic(err)
}
defer file.Close()
sess, err := session.NewSession(&config)
if err != nil {
fmt.Println(err)
}
//list Buckets
ObjectList := listBucketObjects(sess)
//loop over the obectlist. First initialize the s3 downloader via s3manager
downloader := s3manager.NewDownloader(sess)
for _, item := range ObjectList.Contents {
csvFile := filepath.Base(*item.Key)
if csvFile != "pluginVersions" {
downloadBucketObjects(downloader, file, csvFile)
}
}
}
func listBucketObjects(sess *session.Session) *s3.ListObjectsV2Output {
//create a new s3 client
svc := s3.New(sess)
resp, err := svc.ListObjectsV2(&s3.ListObjectsV2Input{
Bucket: aws.String(Bucket),
Prefix: aws.String(bucketKey),
})
if err != nil {
panic(err)
}
return resp
}
func downloadBucketObjects(downloader *s3manager.Downloader, file *os.File, keyobj string) {
fileToDownload := bucketKey + keyobj
numBytes, err := downloader.Download(file,
&s3.GetObjectInput{
Bucket: aws.String(Bucket),
Key: aws.String(fileToDownload),
})
if err != nil {
panic(err)
}
fmt.Println("Downloaded", file.Name(), numBytes, "bytes")
}
Firstly, I don't get it why do you even need os.O_APPEND flag in the first place. As per my understanding, you can omit os.O_APPEND.
Now, let's come to the actual problem of why it's happening:
Doc for O_APPEND (Ref: https://man7.org/linux/man-pages/man2/open.2.html):
O_APPEND
The file is opened in append mode. Before each write(2),
the file offset is positioned at the end of the file, as
if with lseek(2). The modification of the file offset and
the write operation are performed as a single atomic step.
So for every call to write the file offset is positioned at the end of the file.
But (*s3Manager.Download).Download supposedly be using WriteAt method, i.e.,
Doc for WriteAt:
$ go doc os WriteAt
package os // import "os"
func (f *File) WriteAt(b []byte, off int64) (n int, err error)
WriteAt writes len(b) bytes to the File starting at byte offset off. It
returns the number of bytes written and an error, if any. WriteAt returns a
non-nil error when n != len(b).
If file was opened with the O_APPEND flag, WriteAt returns an error.
Notice the last line, that if the file's opened with O_APPEND flag it will result in an error and it's even right because WriteAt's second argument is an offset but mixing O_APPEND's behaviour and WriteAt offset seeking might create problem resulting in unexpected results and it errors out.
Consider the definition of s3manager.Downloader:
func (d Downloader) Download(w io.WriterAt, input *s3.GetObjectInput, options ...func(*Downloader)) (n int64, err error)
The first argument is an io.WriterAt; this interface is:
type WriterAt interface {
WriteAt(p []byte, off int64) (n int, err error)
}
This means that the Download function is going to call the WriteAt method in the File you are passing it. As per the documentation for File.WriteAt
If file was opened with the O_APPEND flag, WriteAt returns an error.
So this explains why you are getting the error but raises the question "why is Download using WriteAt and not accepting an io.Writer (and calling Write)?"; the answer can be found in the documentation:
The w io.WriterAt can be satisfied by an os.File to do multipart concurrent downloads, or in memory []byte wrapper using aws.WriteAtBuffer
So, to increase performance, Downloader might make multiple simultaneous requests for parts of the file and then write these out as they are received (meaning it may not write the data in order). This also explains why calling the function multiple times with the same File results in overwritten data (when Downloader retrieves the each chunk of the file it writes it out at the appropriate position in the output file; this overwrites any data already there).
The above quote from the documentation also points to a possible solution; use an aws.WriteAtBuffer and, once the download is finished, write the data to your file (which could then be opened with O_APPEND) - something like this:
buf := aws.NewWriteAtBuffer([]byte{})
numBytes, err := downloader.Download(buf,
&s3.GetObjectInput{
Bucket: aws.String(Bucket),
Key: aws.String(fileToDownload),
})
if err != nil {
panic(err)
}
_, err = file.Write(buf.Bytes())
if err != nil {
panic(err)
}
An alternative would be to download into a temporary file and then append that to your output file (you may need to do this if the files are large).
Here's my code ( part of it ) :
type SitemapIndex struct {
// Locations []Location `xml:"sitemap"`
Locations []string `xml:"sitemap>loc"`
}
~~~ SNIP ~~~
func main(){
var s SitemapIndex
resp, _ := http.Get("https://www.washingtonpost.com/news-sitemaps/index.xml")
bytes, _ := ioutil.ReadAll(resp.Body)
xml.Unmarshal(bytes, &s)
for _, Location := range s.Locations {
fmt.Printf("%s\n", Location)
resp, err := http.Get(Location)
if err != nil {
log.Fatal(err)
} else {
bytes, _ := ioutil.ReadAll(resp.Body)
xml.Unmarshal(bytes, &n)
for idx := range n.Titles {
newsMap[n.Titles[idx]] = NewsMap{n.Keywords[idx], n.Locations[idx]}
}
}
for idx, data := range newsMap {
fmt.Println("\n\n\n", idx)
fmt.Println("\n", data.Keyword)
fmt.Println("\n", data.Location)
}
}
Now, when I run this code I get this output :
https://www.washingtonpost.com/news-sitemaps/politics.xml
2019/01/28 02:37:13 parse
https://www.washingtonpost.com/news-sitemaps/politics.xml
: first path segment in URL cannot contain colon
exit status 1
I read a few posts and did some experiment myself, like I made another file with the following code
package main
import ("fmt"
"net/url")
func main(){
fmt.Println(url.Parse("https://www.washingtonpost.com/news-sitemaps/politics.xml"))
}
And it didn't throw any error, so I understand the error is not with the url .
Now, I just started learning Go using sentdex's tutorials , a few hours ago and so don't have much idea as of now. Here's the video link
Thanks and regards.
Temporarya
The problem here is that Location has whitespace prefix and suffix so string is not valid URL. Unfortunately, error message does not help to see that.
How to detect:
I typically use %q fmt helper that wraps string into parentheses:
fmt.Printf("%q", Location)
Will be printed as "\nhttps://www.washingtonpost.com/news-sitemaps/politics.xml\n"
How to fix:
add this line before using Location in code:
Location = strings.TrimSpace(Location)
Another reason to get this error is when you use an IP address without specifying the protocol before it.
Example for cases you'll get this error:
parsedUrl, err := url.Parse("127.0.0.1:3213")
How to fix it:
parsedUrl, err := url.Parse("http://127.0.0.1:3213")
Poor documentation, unfortunately.
I have found many many code examples of writing to a CSV by passing in a [][]string. (like the following):
package main
import (
"os"
"log"
"encoding/csv"
)
var data = [][]string{
{"Row 1", "30"},
{"Row 2", "60"},
{"Row 3", "90"}}
func main() {
file, err := os.Create("tutorials_technology.csv")
if err != nil {
log.Fatal(err)
}
defer file.Close()
w := csv.NewWriter(file)
for _, value := range data {
if err := w.Write(value); err != nil {
log.Fatalln("Error writing record to csv: ", err)
}
}
w.Flush()
}
However, I haven't found any code examples that show how to use the gota dataframe.WriteCSV() function to write to a CSV. In the gota dataframe documentation, there isn't an example for writing to a csv, but there is an example for reading from a csv.
The dataframe function WriteCSV() requires an input of the io.Writer{} interface. I wasn't sure how to satsify that.
The following didn't work
writer := csv.NewWriter(f)
df.WriteCSV(writer) // TODO This writer needs to be a []byte writer
I've been working on this for quite a while. Does anyone have any clues?
I have looked into turning my gota dataframe into a [][]string type, but that's a little inconvenient because I put my data into a gota dataframe with the package's LoadStructs() function and I had read in some CSV in a semi-custom way before putting them into structs.
So I could write a function to turn my structs into a [][]string format, but I feel like that is pretty tedious and I'm sure there has got to be a better way. In fact, I'm sure there is because the dataframe type has the WriteCSV() method but I just haven't figured out how to use it.
Here are my structs
type CsvLine struct {
Index int
Date string
Symbol string
Open float64
High float64
Low float64
Close float64
// Volume float64
// Market_Cap float64
}
type File struct {
Rows []CsvLine
}
Disclaimer: I am a little bit of a golang newbie. I've only been using Go for a couple months, and this is the first time I've tried to write to a file. I haven't interacted much with the io.Writer interface, but I hear that it's very useful.
And yes, I frequently look at the Golang.org blog and I've read "Effective Go" and I keep referencing it.
So it turns out I misunderstood the io.Writer interface and I didn't understand what the os.Create() function returns.
It turns out the code is even simpler and easier than I thought it would be.
Here is the working code example:
df := dataframe.LoadStructs(file.Rows)
f, err := os.Create(outFileName)
if err != nil {
log.Fatal(err)
}
df.WriteCSV(f)
Currently on what I've seen so far is that, converting database rows to JSON or to []map[string]interface{} is not simple. I have to create two slices and then loop through columns and create keys every time.
...Some code
tableData := make([]map[string]interface{}, 0)
values := make([]interface{}, count)
valuePtrs := make([]interface{}, count)
for rows.Next() {
for i := 0; i < count; i++ {
valuePtrs[i] = &values[i]
}
rows.Scan(valuePtrs...)
entry := make(map[string]interface{})
for i, col := range columns {
var v interface{}
val := values[i]
b, ok := val.([]byte)
if ok {
v = string(b)
} else {
v = val
}
entry[col] = v
}
tableData = append(tableData, entry)
}
Is there any package for this ? Or I am missing some basics here
I'm dealing with the same issue, as far as my investigation goes it looks that there is no other way.
All the packages that I have seen use basically the same method
Few things you should know, hopefully will save you time:
database/sql package converts all the data to the appropriate types
if you are using the mysql driver(go-sql-driver/mysql) you need to add
params to your db string for it to return type time instead of a string
(use ?parseTime=true, default is false)
You can use tools that were written by the community, to offload the overhead:
A minimalistic wrapper around database/sql, sqlx, uses similar way internally with reflection.
If you need more functionality, try using an "orm": gorp, gorm.
If you interested in diving deeper check out:
Using reflection in sqlx package, sqlx.go line 560
Data type conversion in database/sql package, convert.go line 86
One thing you could do is create a struct that models your data.
**Note: I am using MS SQLServer
So lets say you want to get a user
type User struct {
ID int `json:"id,omitempty"`
UserName string `json:"user_name,omitempty"`
...
}
then you can do this
func GetUser(w http.ResponseWriter, req *http.Request) {
var r Role
params := mux.Vars(req)
db, err := sql.Open("mssql", "server=ServerName")
if err != nil {
log.Fatal(err)
}
err1 := db.QueryRow("select Id, UserName from [Your Datavse].dbo.Users where Id = ?", params["id"]).Scan(&r.ID, &r.Name)
if err1 != nil {
log.Fatal(err1)
}
json.NewEncoder(w).Encode(&r)
if err != nil {
log.Fatal(err)
}
}
Here are the imports I used
import (
"database/sql"
"net/http"
"log"
"encoding/json"
_ "github.com/denisenkom/go-mssqldb"
"github.com/gorilla/mux"
)
This allowed me to get data from the database and get it into JSON.
This takes a while to code, but it works really well.
Not in the Go distribution itself, but there is the wonderful jmoiron/sqlx:
import "github.com/jmoiron/sqlx"
tableData := make([]map[string]interface{}, 0)
for rows.Next() {
entry := make(map[string]interface{})
err := rows.MapScan(entry)
if err != nil {
log.Fatal("SQL error: " + err.Error())
}
tableData = append(tableData, entry)
}
If you know the data type that you are reading, then you can read into the data type without using generic interface.
Otherwise, there is no solution regardless of the language used due to nature of JSON itself.
JSON does not have description of composite data structures. In other words, JSON is a generic key-value structure. When parser encounters what is supposed to be a specific structure there is no identification of that structure in JSON itself. For example, if you have a structure User the parser would not know how a set of key-value pairs maps to your structure User.
The problem of type recognition is usually addressed with document schema (a.k.a. XSD in XML world) or explicitly through passed expected data type.
One quick way to go about being able to get an arbirtrary and generic []map[string]interface{} from these query libraries is to populate an array of interface pointers with the same size of the amount of columns on the query, and then pass that as a parameter on the scan function:
// For example, for the go-mssqldb lib:
queryResponse, err := d.pool.Query(query)
if err != nil {
return nil, err
}
defer queryResponse.Close()
// Holds all the end-results
results := []map[string]interface{}{}
// Getting details about all the fields from the query
fieldNames, err := queryResponse.Columns()
if err != nil {
return nil, err
}
// Creating interface-type pointers within an array of the same
// size of the number of columns we have, so that we can properly
// pass this to the "Scan" function and get all the query parameters back :)
var scanResults []interface{}
for range fieldNames {
var v interface{}
scanResults = append(scanResults, &v)
}
// Parsing the query results into the result map
for queryResponse.Next() {
// This variable will hold the value for all the columns, named by the column name
rowValues := map[string]interface{}{}
// Cleaning up old values just in case
for _, column := range scanResults {
*(column.(*interface{})) = nil
}
// Scan into the array of pointers
err := queryResponse.Scan(scanResults...)
if err != nil {
return nil, err
}
// Map the pointers back to their value and the associated column name
for index, column := range scanResults {
rowValues[fieldNames[index]] = *(column.(*interface{}))
}
results = append(results, rowValues)
}
return results, nil