I'm new in go lang. I'm trying to read csv file and collecting data.
But after run it I got this error :
panic: assignment to entry in nil map
goroutine 1 [running]:
panic(0x4dedc0, 0xc082002440)
C:/Go/src/runtime/panic.go:464 +0x3f4
main.(*stateInformation).setColumns(0xc08202bd40, 0xc082060000, 0x11, 0x20)
F:/Works/Go/src/examples/state-info/main.go:25 +0xda
main.main()
F:/Works/Go/src/examples/state-info/main.go:69 +0xaea
My code :
package main
import (
"encoding/csv"
"fmt"
"io"
"log"
"os"
"strconv"
)
type stateInformation struct {
columns map[string]int
}
type state struct {
id int
name string
abbreviation string
censusRegionName string
}
func (info *stateInformation) setColumns(record []string) {
for idx, column := range record {
info.columns[column] = idx
}
}
func (info *stateInformation) parseState(record []string) (*state, error) {
column := info.columns["id"]
id, err := strconv.Atoi(record[column])
if err != nil {
return nil, err
}
name := record[info.columns["name"]]
abbreviation := record[info.columns["abbreviation"]]
censusRegionName := record[info.columns["census_region_name"]]
return &state{
id: id,
name: name,
abbreviation: abbreviation,
censusRegionName: censusRegionName,
}, nil
}
func main() {
// #1 open a file
f, err := os.Open("state_table.csv")
if err != nil {
log.Fatalln(err)
}
defer f.Close()
stateLookup := map[string]*state{}
info := &stateInformation{}
// #2 parse a csv file
csvReader := csv.NewReader(f)
for rowCount := 0; ; rowCount++ {
record, err := csvReader.Read()
if err == io.EOF {
break
} else if err != nil {
log.Fatalln(err)
}
if rowCount == 0 {
info.setColumns(record)
} else {
state, err := info.parseState(record)
if err != nil {
log.Fatalln(err)
}
stateLookup[state.abbreviation] = state
}
}
// state-information AL
if len(os.Args) < 2 {
log.Fatalln("expected state abbreviation")
}
abbreviation := os.Args[1]
state, ok := stateLookup[abbreviation]
if !ok {
log.Fatalln("invalid state abbreviation")
}
fmt.Println(`
<html>
<head></head>
<body>
<table>
<tr>
<th>Abbreviation</th>
<th>Name</th>
</tr>`)
fmt.Println(`
<tr>
<td>` + state.abbreviation + `</td>
<td>` + state.name + `</td>
</tr>
`)
fmt.Println(`
</table>
</body>
</html>
`)
}
What's wrong in my code?
I don't know what you are trying to obtain, but the error tells, that columns map does not have a column index on the moment of assignment and for this reason is throwing a panic.
panic: assignment to entry in nil map
To make it work you have to initialize the map itself before to start to populate with indexes.
state := &stateInformation{
columns: make(map[string]int),
}
Or another way to initialize:
func (info *stateInformation) setColumns(record []string) {
info.columns = make(map[string]int)
for idx, column := range record {
info.columns[column] = idx
}
}
Related
For the below type of inputs in golang coding interviews, what is the best way to get the input?
Input:
3
hello elloh
test estt
tier riet
I found two methods:
Method 1:
reader := bufio.NewReader(os.Stdin)
var lines []string
for {
line,err := reader.ReadString('\n') //this reads only one read
if err != nil {
log.Fatal(err)
}
if len(strings.TrimSpace(line)) == 0 {
break
}
line_s := strings.Split(line, " ")
lines = append(lines, line_s...)
}
Method 2:
bytes, err := ioutil.ReadAll(os.Stdin)
fmt.Println(len(bytes))
if err == nil {
input := strings.Split(string(bytes), "\n")
count, _ := strconv.Atoi(input[0])
fmt.Println(input)
var lines []string
for i := 1; i < count; i++ {
line := strings.Split(input[i], " ")
lines = append(lines, line...)
}
fmt.Println(lines)
}
But not sure how to end getting input from stdin in Method2.
Please suggest the best method to get input.
Use bufio.Scanner to read input. Use a function to encapsulate complexity and implementation details. For example,
package main
import (
"bufio"
"fmt"
"os"
"strconv"
"strings"
)
func readData(s *bufio.Scanner) ([][]string, error) {
var data [][]string
if !s.Scan() {
return nil, s.Err()
}
nLine, err := strconv.Atoi(strings.TrimSpace(s.Text()))
if err != nil {
return nil, err
}
for ; nLine > 0 && s.Scan(); nLine-- {
data = append(data, strings.Fields(s.Text()))
}
if err := s.Err(); err != nil {
return nil, err
}
if nLine != 0 {
err := fmt.Errorf("missing %d lines of data", nLine)
return nil, err
}
return data, nil
}
func main() {
s := bufio.NewScanner(os.Stdin)
data, err := readData(s)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Println(len(data))
for _, datum := range data {
fmt.Println(datum)
}
}
https://go.dev/play/p/0Xwp3-hwGyK
3
hello elloh
test estt
tier riet
3
[hello elloh]
[test estt]
[tier riet]
I have a struct and now want to instantiate it from received http data. But now the code I write is cumbersome and has a lot of lines of code. Is there any way to simplify the code? All fields except the field id can correspond
model
type ALiNotifyLog struct {
ID *int `json:"id"`
APPId *string `json:"app_id"`
AuthAppId *string `json:"auth_app_id"`
BuyerId *string `json:"buyer_id"`
BuyerPayAmount *string `json:"buyer_pay_amount"`
GmtCreate *string `json:"gmt_create"`
GmtPayment *string `json:"gmt_payment"`
InvoiceAmount *string `json:"invoice_amount"`
NotifyId *string `json:"notify_id"`
NotifyTime *string `json:"notify_time"`
OutTradeNo *string `json:"out_trade_no"`
PointAmount *string `json:"point_amount"`
ReceiptAmount *string `json:"receipt_amount"`
Sign *string `json:"sign"`
TotalAmount *string `json:"total_amount"`
TradeNo *string `json:"trade_no"`
TradeStatus *string `json:"trade_status"`
}
func
func SaveData(data map[string]interface{}) {
app_id := data["app_id"].(string)
auth_app_id := data["auth_app_id"].(string)
buyer_id := data["buyer_id"].(string)
buyer_pay_amount := data["buyer_pay_amount"].(string)
gmt_create := data["gmt_create"].(string)
gmt_payment := data["gmt_payment"].(string)
invoice_amount := data["invoice_amount"].(string)
notify_id := data["notify_id"].(string)
notify_time := data["notify_time"].(string)
out_trade_no := data["out_trade_no"].(string)
point_amount := data["point_amount"].(string)
receipt_amount := data["receipt_amount"].(string)
sign := data["sign"].(string)
total_amount := data["total_amount"].(string)
trade_no := data["trade_no"].(string)
trade_status := data["trade_status"].(string)
model := payment.ALiNotifyLog{
APPId: &app_id,
AuthAppId: &auth_app_id,
BuyerId: &buyer_id,
BuyerPayAmount: &buyer_pay_amount,
GmtCreate: &gmt_create,
GmtPayment: &gmt_payment,
InvoiceAmount: &invoice_amount,
NotifyId: ¬ify_id,
NotifyTime: ¬ify_time,
OutTradeNo: &out_trade_no,
PointAmount: &point_amount,
ReceiptAmount: &receipt_amount,
Sign: &sign,
TotalAmount: &total_amount,
TradeNo: &trade_no,
TradeStatus: &trade_status}
res := global.Orm.Table(paynotifylog).Create(&model)
fmt.Println(res)
}
I see that you are JSON. May be decode the JSON directly into a struct instance.
I will structure the code something like the below snippet:
type ALiNotifyLog struct {
// your fields here
}
func parseRequest(r *http.Request) {
var notifyLog ALiNotifyLog
err := json.NewDecoder(r.Body).Decode(¬ifyLog)
if err != nil {
// do something
}
// ............ more code
}
func SaveData(data ALiNotifyLog) {
res := global.Orm.Table(paynotifylog).Create(&data)
fmt.Println(res)
// ........... more code
}
Use the reflect package to programmatically iterate over the fields:
func setStringpFields(pmodel any, data map[string]any) {
v := reflect.ValueOf(pmodel).Elem()
t := v.Type()
for i := 0; i < t.NumField(); i++ {
sf := t.Field(i)
if sf.Type != stringpType {
continue
}
name, _, _ := strings.Cut(sf.Tag.Get("json"), ",")
if s, ok := data[name].(string); ok {
v.Field(i).Set(reflect.ValueOf(&s))
}
}
}
var stringpType = reflect.PtrTo(reflect.TypeOf(""))
Use it like this:
var model ALiNotifyLog
setStringpFields(&model, data)
Run an example on the Go Playground.
I took the liberty of skipping fields that are missing from data. The code in the question panics on a missing value.
A simpler approach is to create a function with the repeated functionality:
func stringp(data map[string]interface{}, name string) *string {
if s, ok := data[name].(string); ok {
return &s
}
return nil
}
Use that function to initialize the fields:
model := payment.ALiNotifyLog{
APPId: stringp("app_id"),
AuthAppId: stringp("auth_app_id"),
...
TradeStatus: stringp("trade_status")}
res := global.Orm.Table(paynotifylog).Create(&model)
fmt.Println(res)
1: ScanMapToStruct
func scanMapToStruct(dest interface{}, vals map[string]interface{}) error {
srcValue := indirect(reflect.ValueOf(dest))
srcType := indirectType(reflect.TypeOf(dest))
for m := 0; m < srcType.NumField(); m++ {
field := srcType.Field(m)
fieldName, _ := getFieldName(field)
jsonTypeName := getJsonDataType(field.Type)
if jsonTypeName == "" {
continue
}
v, ok := vals[fieldName].(string)
if !ok {
continue
}
dec := decoders[field.Type.Kind()]
if dec == nil {
continue
}
err := dec(srcValue.Field(m), v)
if err != nil {
fmt.Printf("set field(%s)=%s err:%v\n", fieldName, v, err)
continue
}
}
return nil
}
2: copier.Copy()
Use package mapstructure from GitHub.
go get https://github.com/mitchellh/mapstructure
package main
import (
"log"
"os"
"github.com/mitchellh/mapstructure"
)
type MyStruct struct {
This int
That string `json:"thaaaaat"`
}
func main() {
var result map[string]interface{}
cfg := &mapstructure.DecoderConfig{
TagName: "json",
Result: &result,
}
decoder, err := mapstructure.NewDecoder(cfg)
if err != nil {
log.Printf("Could not create decoder: %v", err)
os.Exit(1)
}
myData := &MyStruct{
This: 42,
That: "foobar",
}
err = decoder.Decode(myData)
if err != nil {
log.Printf("Decoding failed: %v", err)
os.Exit(1)
}
log.Print(cfg.Result)
}
Output:
&map[This:42 thaaaaat:foobar]
https://go.dev/play/p/mPK_9fEevyC
For example, we have 3 CSV files and common for all is Email column. In first file are Name and Email, in another are Email (plus different info) and no Name field. So, if I need to fill in 2 and 3 files field Name based on the correspondence of the Name and Еmail from the first file than... I wrote code like this:
package main
import (
"fmt"
"io/ioutil"
"log"
"path/filepath"
"strings"
"github.com/jszwec/csvutil"
)
type User struct {
Name string `csv:"name"`
Email string `csv:"email"`
}
type Good struct {
User
Dt string `csv:"details"`
}
type Strange struct {
User
St string `csv:"status"`
Dt string `csv:"details"`
}
var lst map[string]string
func readCSV(fn string, dat interface{}) error {
raw, err := ioutil.ReadFile(fn)
if err != nil {
return fmt.Errorf("Cannot read CSV: %w", err)
}
if err := csvutil.Unmarshal(raw, dat); err != nil {
return fmt.Errorf("Cannot unmarshal CSV: %w", err)
}
return nil
}
func fixNames(fl string, in interface{}) error {
if err := readCSV(fl, in); err != nil {
return fmt.Errorf("CSV: %w", err)
}
switch in.(type) {
case *[]Good:
var vals []Good
for _, v := range *in.(*[]Good) {
v.Name = lst[strings.TrimSpace(strings.ToLower(v.Email))]
vals = append(vals, v)
}
in = vals
case *[]Strange:
var vals []Strange
for _, v := range *in.(*[]Strange) {
v.Name = lst[strings.TrimSpace(strings.ToLower(v.Email))]
vals = append(vals, v)
}
in = vals
}
b, err := csvutil.Marshal(in)
if err != nil {
return fmt.Errorf("Cannot marshal CSV: %w", err)
}
ext := filepath.Ext(fl)
bas := filepath.Base(fl)
err = ioutil.WriteFile(bas[:len(bas)-len(ext)]+"-XIAOSE"+ext, b, 0644)
if err != nil {
return fmt.Errorf("Cannot save CSV: %w", err)
}
return nil
}
func main() {
var users []User
if err := readCSV("./Guitar_Contacts.csv", &users); err != nil {
log.Fatalf("CSV: %s", err)
}
lst = make(map[string]string)
for _, v := range users {
lst[strings.TrimSpace(strings.ToLower(v.Email))] = v.Name
}
var usersGood []Good
if err := fixNames("./Guitar-Good.csv", &usersGood); err != nil {
log.Fatalf("fix: %s", err)
}
var usersStrange []Strange
if err := fixNames("./Guitar-Uknown.csv", &usersStrange); err != nil {
log.Fatalf("fix: %s", err)
}
fmt.Println("OK")
}
in this code I don't like part in func fixNames where is switch:
switch in.(type) {
case *[]Good:
var vals []Good
for _, v := range *in.(*[]Good) {
v.Name = lst[strings.TrimSpace(strings.ToLower(v.Email))]
vals = append(vals, v)
}
in = vals
case *[]Strange:
var vals []Strange
for _, v := range *in.(*[]Strange) {
v.Name = lst[strings.TrimSpace(strings.ToLower(v.Email))]
vals = append(vals, v)
}
in = vals
}
because I just repeat code in part where *in.(SOME_TYPE). I want one loop and one action for different types, structs where are Name and Email fields...
Also was idea to do it with reflection smth. like this:
v := reflect.ValueOf(in)
v = v.Elem()
for i := 0; i < v.Len(); i++ {
fmt.Println(v.Index(i))
}
but I do not know what to do next, how to add in that v value for Name
You don't need reflection for this particular case. You can clean the code up by realizing that you are only working on the User part of the structs, and that you can simplify the type switch:
fix:=func(in *User) {
in.Name = lst[strings.TrimSpace(strings.ToLower(in.Email))]
}
switch k:=in.(type) {
case *[]Good:
for i := range *k {
fix( &(*k)[i].User )
}
case *[]Strange:
for i := range *k {
fix( &(*k)[i].User )
}
}
You have to repeat the for loop, but above code does the correction in place.
You can clean up a bit more by not passing a reference to the slice.
With reflect package, you can do that like this.
func fixNames(fl string, in interface{}) error {
//other code
v := reflect.ValueOf(in)
if v.Kind() == reflect.Ptr {
arr := v.Elem()
fmt.Println(arr.Len())
if arr.Kind() == reflect.Slice || arr.Kind() == reflect.Array {
for i := 0; i < arr.Len(); i++ {
elem := arr.Index(i)
f := elem.FieldByName("Name")
f.SetString("NameOfUser")
}
}
}
// other code
}
Also playground example: https://play.golang.org/p/KrGvLVprslH
I am get leveldb's all key-val to a map[string][]byte, but it is not running as my expection.
code is as below
package main
import (
"fmt"
"strconv"
"github.com/syndtr/goleveldb/leveldb"
)
func main() {
db, err := leveldb.OpenFile("db", nil)
if err != nil {
panic(err)
}
defer db.Close()
for i := 0; i < 10; i++ {
err := db.Put([]byte("key"+strconv.Itoa(i)), []byte("value"+strconv.Itoa(i)), nil)
if err != nil {
panic(err)
}
}
snap, err := db.GetSnapshot()
if err != nil {
panic(err)
}
if snap == nil {
panic("snap shot is nil")
}
data := make(map[string][]byte)
iter := snap.NewIterator(nil, nil)
for iter.Next() {
Key := iter.Key()
Value := iter.Value()
data[string(Key)] = Value
}
iter.Release()
if iter.Error() != nil {
panic(iter.Error())
}
for k, v := range data {
fmt.Println(string(k) + ":" + string(v))
}
}
but the result is below
key3:value9
key6:value9
key7:value9
key8:value9
key1:value9
key2:value9
key4:value9
key5:value9
key9:value9
key0:value9
rather not key0:value0
Problem is with casting around types (byte[] to string, etc.).
You are trying to print string values. To avoid unnecessary casting apply the following modifications:
Change data initialization into data := make(map[string]string)
Assign values into data with `data[string(Key)] = string(Value) (by the way, don't use capitalization for variables you aren't intend to export)
Print data's values with fmt.Println(k + ":" + v))
This should produce the following result:
key0:value0
key1:value1
key7:value7
key2:value2
key3:value3
key4:value4
key5:value5
key6:value6
key8:value8
key9:value9
Here is a code snippet that reads CSV file:
func parseLocation(file string) (map[string]Point, error) {
f, err := os.Open(file)
defer f.Close()
if err != nil {
return nil, err
}
lines, err := csv.NewReader(f).ReadAll()
if err != nil {
return nil, err
}
locations := make(map[string]Point)
for _, line := range lines {
name := line[0]
lat, laterr := strconv.ParseFloat(line[1], 64)
if laterr != nil {
return nil, laterr
}
lon, lonerr := strconv.ParseFloat(line[2], 64)
if lonerr != nil {
return nil, lonerr
}
locations[name] = Point{lat, lon}
}
return locations, nil
}
Is there a way to improve readability of this code? if and nil noise.
Go now has a csv package for this. Its is encoding/csv. You can find the docs here: https://golang.org/pkg/encoding/csv/
There are a couple of good examples in the docs. Here is a helper method I created to read a csv file and returns its records.
package main
import (
"encoding/csv"
"fmt"
"log"
"os"
)
func readCsvFile(filePath string) [][]string {
f, err := os.Open(filePath)
if err != nil {
log.Fatal("Unable to read input file " + filePath, err)
}
defer f.Close()
csvReader := csv.NewReader(f)
records, err := csvReader.ReadAll()
if err != nil {
log.Fatal("Unable to parse file as CSV for " + filePath, err)
}
return records
}
func main() {
records := readCsvFile("../tasks.csv")
fmt.Println(records)
}
Go is a very verbose language, however you could use something like this:
// predeclare err
func parseLocation(file string) (locations map[string]*Point, err error) {
f, err := os.Open(file)
if err != nil {
return nil, err
}
defer f.Close() // this needs to be after the err check
lines, err := csv.NewReader(f).ReadAll()
if err != nil {
return nil, err
}
//already defined in declaration, no need for :=
locations = make(map[string]*Point, len(lines))
var lat, lon float64 //predeclare lat, lon
for _, line := range lines {
// shorter, cleaner and since we already have lat and err declared, we can do this.
if lat, err = strconv.ParseFloat(line[1], 64); err != nil {
return nil, err
}
if lon, err = strconv.ParseFloat(line[2], 64); err != nil {
return nil, err
}
locations[line[0]] = &Point{lat, lon}
}
return locations, nil
}
//edit
A more efficient and proper version was posted by #Dustin in the comments, I'm adding it here for completeness sake:
func parseLocation(file string) (map[string]*Point, error) {
f, err := os.Open(file)
if err != nil {
return nil, err
}
defer f.Close()
csvr := csv.NewReader(f)
locations := map[string]*Point{}
for {
row, err := csvr.Read()
if err != nil {
if err == io.EOF {
err = nil
}
return locations, err
}
p := &Point{}
if p.lat, err = strconv.ParseFloat(row[1], 64); err != nil {
return nil, err
}
if p.lon, err = strconv.ParseFloat(row[2], 64); err != nil {
return nil, err
}
locations[row[0]] = p
}
}
playground
I basically copied my answer from here: https://www.dotnetperls.com/csv-go. For me, this was a better answer than what I found on stackoverflow.
import (
"bufio"
"encoding/csv"
"os"
"fmt"
"io"
)
func ReadCsvFile(filePath string) {
// Load a csv file.
f, _ := os.Open(filePath)
// Create a new reader.
r := csv.NewReader(f)
for {
record, err := r.Read()
// Stop at EOF.
if err == io.EOF {
break
}
if err != nil {
panic(err)
}
// Display record.
// ... Display record length.
// ... Display all individual elements of the slice.
fmt.Println(record)
fmt.Println(len(record))
for value := range record {
fmt.Printf(" %v\n", record[value])
}
}
}
I also dislike the verbosity of the default Reader, so I made a new type that is
similar to bufio#Scanner:
package main
import "encoding/csv"
import "io"
type Scanner struct {
Reader *csv.Reader
Head map[string]int
Row []string
}
func NewScanner(o io.Reader) Scanner {
csv_o := csv.NewReader(o)
a, e := csv_o.Read()
if e != nil {
return Scanner{}
}
m := map[string]int{}
for n, s := range a {
m[s] = n
}
return Scanner{Reader: csv_o, Head: m}
}
func (o *Scanner) Scan() bool {
a, e := o.Reader.Read()
o.Row = a
return e == nil
}
func (o Scanner) Text(s string) string {
return o.Row[o.Head[s]]
}
Example:
package main
import "strings"
func main() {
s := `Month,Day
January,Sunday
February,Monday`
o := NewScanner(strings.NewReader(s))
for o.Scan() {
println(o.Text("Month"), o.Text("Day"))
}
}
https://golang.org/pkg/encoding/csv
You can also read contents of a directory to load all the CSV files. And then read all those CSV files 1 by 1 with goroutines
csv file:
101,300.00,11000901,1155686400
102,250.99,11000902,1432339200
main.go file:
const sourcePath string = "./source"
func main() {
dir, _ := os.Open(sourcePath)
files, _ := dir.Readdir(-1)
for _, file := range files {
fmt.Println("SINGLE FILE: ")
fmt.Println(file.Name())
filePath := sourcePath + "/" + file.Name()
f, _ := os.Open(filePath)
defer f.Close()
// os.Remove(filePath)
//func
go func(file io.Reader) {
records, _ := csv.NewReader(file).ReadAll()
for _, row := range records {
fmt.Println(row)
}
}(f)
time.Sleep(10 * time.Millisecond)// give some time to GO routines for execute
}
}
And the OUTPUT will be:
$ go run main.go
SINGLE FILE:
batch01.csv
[101 300.00 11000901 1155686400]
[102 250.99 11000902 1432339200]
----------------- -------------- ---------------------- -------
---------------- ------------------- ----------- --------------
Below example with the Invoice struct
func main() {
dir, _ := os.Open(sourcePath)
files, _ := dir.Readdir(-1)
for _, file := range files {
fmt.Println("SINGLE FILE: ")
fmt.Println(file.Name())
filePath := sourcePath + "/" + file.Name()
f, _ := os.Open(filePath)
defer f.Close()
go func(file io.Reader) {
records, _ := csv.NewReader(file).ReadAll()
for _, row := range records {
invoice := new(Invoice)
invoice.InvoiceNumber = row[0]
invoice.Amount, _ = strconv.ParseFloat(row[1], 64)
invoice.OrderID, _ = strconv.Atoi(row[2])
unixTime, _ := strconv.ParseInt(row[3], 10, 64)
invoice.Date = time.Unix(unixTime, 0)
fmt.Printf("Received invoice `%v` for $ %.2f \n", invoice.InvoiceNumber, invoice.Amount)
}
}(f)
time.Sleep(10 * time.Millisecond)
}
}
type Invoice struct {
InvoiceNumber string
Amount float64
OrderID int
Date time.Time
}