Add a index column to dataframe in gota - go

with the following sample, I can add a new column that is diverted from the row values.
it's working well.
package main
import (
"fmt"
"strings"
"github.com/go-gota/gota/dataframe"
"github.com/go-gota/gota/series"
)
func main() {
csvStr := `accountId,deposit,Withdrawals
anil0001,50,10
vikas0002,10,10
ravi0003,20,10
user1111,NaN,20`
df := dataframe.ReadCSV(strings.NewReader(csvStr))
// Within a row, elements are indexed by their column index.
indexDeposit := 1
indexWithdrawals := 2
// Rapply reads the data by rows.
// You can access each element of the row using
// s.Elem(index) or s.Val(index).
// To browse by columns use Capply.
s := df.Rapply(func(s series.Series) series.Series {
deposit, err := s.Elem(indexDeposit).Int()
if err != nil {
return series.Ints("NAN")
}
withdrawal, err := s.Elem(indexWithdrawals).Int()
if err != nil {
return series.Ints("NAN")
}
return series.Ints(deposit - withdrawal)
})
// The new series is appended to
// the data source via a call to Mutate.
// You can print s to read its content.
df = df.Mutate(s.Col("X0")).
Rename("deposit_Withdrawals_diff", "X0")
fmt.Println(df)
}
but the question is that, I want to add an index ( row counter) to each row. ( later on I want to join it with a subset of that) So I need an index.
something like
index,accountId,deposit,Withdrawals
1,anil0001,50,10
2,vikas0002,10,10
3,ravi0003,20,10
4,user1111,NaN,20
I see there are no GetIndex or Index methods on series. How can I add this index?

I did it with a global variable ( but I'm not sure it's the best solution for gota, maybe for pure go developer is a good solution :) )
index := 0
s := df.Rapply(func(s series.Series) series.Series {
index++
return series.Ints(index)
})
df = df.Mutate(s.Col("X0")).
Rename("index", "X0")
fmt.Println(df)

Related

read csv into [][]byte

I experienced some weird behavior when I was reading a csv file into 2 dimensional byte slice. The first 42 rows are fine and after that it seems like extra line ending are put into the data which messes up things:
first row in the first 42 times:
row 0: 504921600000000000,truck_0,South,Trish,H-2,v2.3,1500,150,12,52.31854,4.72037,124,0,221,0,25
first row after I appended 43 rows:
row 0: 504921600000000000,truck_49,South,Andy,F-150,v2.0,2000,200,15,38.9349,179.94282,289,0,269,0
row 1: 25
minimal code to reproduce the problem:
package main
import (
"bufio"
"log"
"os"
)
type fileDataSource struct {
scanner *bufio.Scanner
}
type batch struct {
rows [][]byte
}
func (b *batch) Len() uint {
return uint(len(b.rows))
}
func (b *batch) Append(row []byte) {
b.rows = append(b.rows, row)
for index, row := range b.rows {
log.Printf("row %d: %s\n", index, string(row))
}
if len(b.rows) > 43 {
log.Fatalf("asdf")
}
}
type factory struct{}
func (f *factory) New() *batch {
return &batch{rows: make([][]byte, 0)}
}
func main() {
file, _ := os.Open("/tmp/data1.csv")
scanner := bufio.NewScanner(bufio.NewReaderSize(file, 4<<20))
b := batch{}
for scanner.Scan() {
b.Append(scanner.Bytes())
}
}
csv I used:
504921600000000000,truck_0,South,Trish,H-2,v2.3,1500,150,12,52.31854,4.72037,124,0,221,0,25
504921600000000000,truck_1,South,Albert,F-150,v1.5,2000,200,15,72.45258,68.83761,255,0,181,0,25
504921600000000000,truck_2,North,Derek,F-150,v1.5,2000,200,15,24.5208,28.09377,428,0,304,0,25
504921600000000000,truck_3,East,Albert,F-150,v2.0,2000,200,15,18.11037,98.65573,387,0,192,0,25
504921600000000000,truck_4,West,Andy,G-2000,v1.5,5000,300,19,81.93919,56.12266,236,0,335,0,25
504921600000000000,truck_5,East,Seth,F-150,v2.0,2000,200,15,5.00552,114.50557,89,0,187,0,25
504921600000000000,truck_6,East,Trish,G-2000,v1.0,5000,300,19,41.59689,57.90174,395,0,150,0,25
504921600000000000,truck_8,South,Seth,G-2000,v1.0,5000,300,19,21.89157,44.58919,411,0,232,0,25
504921600000000000,truck_9,South,Andy,H-2,v2.3,1500,150,12,15.67271,112.4023,402,0,75,0,25
504921600000000000,truck_10,North,Albert,F-150,v2.3,2000,200,15,35.05682,36.20513,359,0,68,0,25
504921600000000000,truck_7,East,Andy,H-2,v2.0,1500,150,12,7.74826,14.96075,105,0,323,0,25
504921600000000000,truck_11,South,Derek,F-150,v1.0,2000,200,15,87.9924,134.71544,293,0,133,0,25
504921600000000000,truck_14,North,Albert,H-2,v1.0,1500,150,12,66.68217,105.76965,222,0,252,0,25
504921600000000000,truck_18,West,Trish,F-150,v2.0,2000,200,15,67.15164,153.56165,252,0,240,0,25
504921600000000000,truck_20,North,Rodney,G-2000,v2.0,5000,300,19,38.88807,65.86698,104,0,44,0,25
504921600000000000,truck_21,East,Derek,G-2000,v2.0,5000,300,19,81.87812,167.8083,345,0,327,0,25
504921600000000000,truck_22,West,Albert,G-2000,v1.5,5000,300,19,39.9433,16.0241,449,0,42,0,25
504921600000000000,truck_23,South,Andy,F-150,v2.0,2000,200,15,73.28358,98.05159,198,0,276,0,25
504921600000000000,truck_24,West,Rodney,G-2000,v2.3,5000,300,19,22.19262,0.27462,223,0,318,0,25
504921600000000000,truck_25,North,Trish,F-150,v2.0,2000,200,15,17.26704,16.91226,461,0,183,0,25
504921600000000000,truck_26,South,Seth,F-150,v1.5,2000,200,15,45.65327,144.60354,58,0,182,0,25
504921600000000000,truck_12,East,Trish,G-2000,v1.0,5000,300,19,36.03928,113.87118,39,0,294,0,25
504921600000000000,truck_13,West,Derek,H-2,v1.0,1500,150,12,14.07479,110.77267,152,0,69,0,25
504921600000000000,truck_27,West,Seth,G-2000,v1.5,5000,300,19,79.55971,97.86182,252,0,345,0,25
504921600000000000,truck_28,West,Rodney,G-2000,v1.5,5000,300,19,60.33457,4.62029,74,0,199,0,25
504921600000000000,truck_16,South,Albert,G-2000,v1.5,5000,300,19,51.16438,121.32451,455,0,290,0,25
504921600000000000,truck_19,West,Derek,G-2000,v1.5,5000,300,19,19.69355,139.493,451,0,300,0,25
504921600000000000,truck_31,North,Albert,G-2000,v1.0,5000,300,19,0.75251,116.83474,455,0,49,0,25
504921600000000000,truck_32,West,Seth,F-150,v2.0,2000,200,15,4.07566,164.43909,297,0,277,0,25
504921600000000000,truck_33,West,Rodney,G-2000,v1.5,5000,300,19,89.19448,10.47499,407,0,169,0,25
504921600000000000,truck_34,West,Rodney,G-2000,v2.0,5000,300,19,73.7383,10.79582,488,0,170,0,25
504921600000000000,truck_35,West,Seth,G-2000,v2.3,5000,300,19,60.02428,2.51011,480,0,307,0,25
504921600000000000,truck_36,North,Andy,G-2000,v1.0,5000,300,19,87.52877,45.07308,161,0,128,0,25
504921600000000000,truck_38,West,Andy,H-2,v2.3,,150,12,63.54604,119.82031,282,0,325,0,25
504921600000000000,truck_39,East,Derek,G-2000,v1.5,5000,300,19,33.83548,3.90996,294,0,123,0,25
504921600000000000,truck_40,West,Albert,H-2,v2.0,1500,150,12,32.32773,118.43138,276,0,316,0,25
504921600000000000,truck_41,East,Rodney,F-150,v1.0,2000,200,15,68.85572,173.23123,478,0,207,0,25
504921600000000000,truck_42,West,Trish,F-150,v2.0,2000,200,15,38.45195,171.2884,113,0,180,0,25
504921600000000000,truck_43,East,Derek,H-2,v2.0,1500,150,12,52.90189,49.76966,295,0,195,0,25
504921600000000000,truck_44,South,Seth,H-2,v1.0,1500,150,12,32.33297,3.89306,396,0,320,0,25
504921600000000000,truck_30,East,Andy,G-2000,v1.5,5000,300,19,29.62198,83.73482,291,0,267,0,25
504921600000000000,truck_46,West,Seth,H-2,v2.3,1500,150,12,26.07966,118.49629,321,,267,0,25
504921600000000000,truck_37,South,Andy,G-2000,v2.0,5000,300,19,57.90077,77.20136,77,0,179,0,25
504921600000000000,truck_49,South,Andy,F-150,v2.0,2000,200,15,38.9349,179.94282,289,0,269,0,25
504921600000000000,truck_53,West,Seth,G-2000,v2.3,5000,300,19,25.02,157.45082,272,0,5,0,25
504921600000000000,truck_54,North,Andy,H-2,v2.0,1500,150,12,87.62736,106.0376,360,0,66,0,25
504921600000000000,truck_55,East,Albert,G-2000,v1.0,5000,300,19,78.56605,71.16225,295,0,150,0,25
504921600000000000,truck_56,North,Derek,F-150,v2.0,2000,200,15,23.51619,123.22682,71,0,209,0,25
504921600000000000,truck_57,South,Rodney,F-150,v2.3,2000,200,15,26.07996,159.92716,454,0,22,0,25
504921600000000000,truck_58,South,Derek,F-150,v2.0,2000,200,15,84.79333,79.23813,175,0,246,0,25
504921600000000000,truck_59,East,Andy,H-2,v2.0,1500,150,12,8.7621,82.48318,82,0,55,0,25
504921600000000000,truck_45,East,Trish,G-2000,v1.0,5000,300,19,17.48624,100.78121,306,0,193,0,25
504921600000000000,truck_47,South,Derek,G-2000,v1.5,5000,300,19,41.62173,110.80422,111,0,78,0,25
504921600000000000,truck_48,East,Trish,G-2000,v1.5,5000,300,19,63.90773,141.50555,53,0,,0,25
504921600000000000,truck_50,East,Andy,H-2,v2.3,1500,150,12,45.44111,172.39833,219,0,88,0,25
504921600000000000,truck_51,East,Rodney,F-150,v2.3,2000,200,15,89.03645,91.57675,457,0,337,0,25
504921600000000000,truck_52,West,Derek,G-2000,v1.0,5000,300,19,89.0133,97.8037,23,0,168,0,25
504921600000000000,truck_61,East,Albert,G-2000,v2.3,5000,300,19,75.91676,167.78366,462,0,60,0,25
504921600000000000,truck_62,East,Derek,H-2,v1.5,1500,150,12,54.61668,103.21398,231,0,143,0,25
504921600000000000,truck_63,South,Rodney,H-2,v2.0,1500,150,12,37.13702,149.25546,46,0,118,0,25
504921600000000000,truck_64,South,Albert,G-2000,v2.0,5000,300,19,45.04214,10.73002,447,0,253,0,25
504921600000000000,truck_60,South,Derek,H-2,v1.5,1500,150,12,57.99184,33.45994,310,0,93,0,25
504921600000000000,truck_67,South,Seth,H-2,v1.0,1500,150,12,4.62985,155.01707,308,0,22,0,25
504921600000000000,truck_68,West,Rodney,F-150,v1.5,2000,200,15,16.90741,123.03863,303,0,43,0,25
504921600000000000,truck_69,East,Derek,H-2,v2.3,1500,150,12,79.88424,120.79121,407,0,138,0,25
504921600000000000,truck_70,North,Albert,H-2,v2.0,1500,150,12,77.87592,164.70924,270,0,21,0,25
504921600000000000,truck_71,West,Seth,G-2000,v2.3,5000,300,19,72.75635,78.0365,391,0,32,0,25
504921600000000000,truck_73,North,Seth,F-150,v1.5,2000,200,15,37.67468,91.09732,489,0,103,0,25
504921600000000000,truck_74,North,Trish,H-2,v1.0,1500,150,12,41.4456,158.13897,206,0,79,0,25
504921600000000000,truck_75,South,Andy,F-150,v1.5,2000,200,15,4.11709,175.65994,378,0,176,0,25
504921600000000000,truck_66,South,Seth,G-2000,v2.0,5000,300,19,42.24286,151.8978,227,0,67,0,25
504921600000000000,truck_72,South,Andy,G-2000,v2.3,5000,300,19,82.46228,2.44504,487,0,39,0,25
504921600000000000,truck_76,South,Rodney,F-150,v2.3,2000,200,15,71.62798,121.89842,283,0,164,0,25
504921600000000000,truck_78,South,Seth,F-150,v2.0,2000,200,15,13.96218,39.04615,433,0,326,0,25
504921600000000000,truck_79,South,Andy,G-2000,v2.0,5000,300,19,56.54137,,46,0,127,0,25
504921600000000000,truck_81,West,Rodney,G-2000,v2.3,5000,300,19,59.42624,115.59744,68,0,296,0,25
504921600000000000,truck_83,South,Albert,F-150,v2.0,2000,200,15,49.20261,115.98262,449,0,132,0,25
504921600000000000,truck_84,West,Derek,H-2,v1.0,1500,150,12,70.16476,59.05399,301,0,134,0,25
504921600000000000,truck_85,West,Derek,G-2000,v1.0,5000,300,19,11.75251,142.86513,358,0,339,0,25
504921600000000000,truck_86,West,Rodney,G-2000,v1.0,5000,300,19,30.92821,127.53274,367,0,162,0,25
504921600000000000,truck_87,West,Rodney,H-2,v2.0,1500,150,12,32.86913,155.7666,122,0,337,0,25
504921600000000000,truck_88,West,Andy,G-2000,v1.5,5000,300,19,60.03367,9.5707,204,0,333,0,25
504921600000000000,truck_80,East,Andy,G-2000,v2.3,5000,300,,46.13937,137.42962,295,0,290,0,25
504921600000000000,truck_91,East,Derek,F-150,v2.0,2000,200,15,7.13401,52.78885,100,0,147,0,25
504921600000000000,truck_93,North,Derek,G-2000,v2.0,5000,300,19,11.46065,20.57173,242,0,148,0,25
504921600000000000,truck_94,North,Derek,F-150,v1.0,2000,200,15,59.53287,26.98247,427,0,341,0,25
504921600000000000,truck_95,East,Albert,G-2000,v2.0,5000,300,19,37.31513,134.40078,383,0,121,0,25
504921600000000000,truck_96,East,Albert,G-2000,v1.5,5000,300,19,15.78803,146.68255,348,0,189,0,25
504921600000000000,truck_97,South,Seth,F-150,v1.0,2000,200,15,14.08559,18.49763,369,0,34,0,25
504921600000000000,truck_98,South,Albert,G-2000,v1.5,5000,300,19,15.1474,71.85194,89,0,238,0,25
504921600000000000,truck_77,East,Trish,F-150,v2.0,2000,200,15,80.5734,17.68311,389,0,218,0,25
504921600000000000,truck_82,West,Derek,H-2,v2.0,1500,150,12,57.00976,90.13642,102,0,296,0,25
504921600000000000,truck_92,North,Derek,H-2,v1.0,1500,150,12,54.40335,153.5809,123,0,150,0,25
504921600000000000,truck_99,West,Trish,G-2000,v1.5,5000,300,19,62.73061,26.1884,309,0,202,0,25
504921610000000000,truck_1,South,Albert,F-150,v1.5,2000,200,15,72.45157,68.83919,259,0,180,2,27.5
504921610000000000,truck_2,North,Derek,F-150,v1.5,2000,200,15,24.5195,28.09369,434,6,302,0,22.1
504921610000000000,truck_3,East,Albert,F-150,v2.0,2000,200,15,18.107,98.66002,390,,190,0,21.2
504921610000000000,truck_4,West,Andy,G-2000,v1.5,5000,300,19,81.9438,56.12717,244,8,334,2,27.6
504921610000000000,truck_5,East,Seth,F-150,v2.0,2000,200,15,5.00695,114.50676,92,7,183,2,28.5
504921610000000000,truck_6,East,Trish,G-2000,v1.0,5000,300,19,41.59389,57.90166,403,0,149,0,22.7
504921610000000000,truck_7,East,Andy,H-2,v2.0,1500,150,12,7.74392,14.95756,,0,320,0,28.2
504921610000000000,truck_12,East,Trish,G-2000,v1.0,5000,300,19,36.03979,113.8752,34,0,293,1,26.3
504921610000000000,truck_13,West,Derek,H-2,v1.0,1500,150,12,14.07315,110.77235,150,0,72,,21.9
504921610000000000,truck_14,North,Albert,H-2,v1.0,1500,150,12,,105.76727,218,5,253,1,21.9
504921610000000000,truck_15,South,Albert,H-2,v1.5,1500,150,12,6.78254,166.86685,5,0,110,0,26.3
504921610000000000,truck_16,South,Albert,G-2000,v1.5,5000,300,19,51.16405,121.32556,445,0,294,3,29.9
504921610000000000,truck_17,West,Derek,H-2,v1.5,1500,150,12,8.12913,56.57343,9,0,6,4,29
504921610000000000,truck_18,West,Trish,F-150,v2.0,2000,200,15,67.15167,153.56094,260,1,239,1,23.3
504921610000000000,truck_19,West,Derek,G-2000,v1.5,5000,300,19,19.69456,139.49545,448,4,298,0,29.9
504921610000000000,truck_20,North,Rodney,G-2000,v2.0,5000,300,19,38.88968,65.86504,103,0,41,1,23.6
504921610000000000,truck_21,East,Derek,G-2000,v2.0,5000,300,19,81.88232,167.81287,345,0,326,0,20.8
504921610000000000,truck_0,South,Trish,H-2,v2.3,1500,150,12,52.32335,4.71786,128,9,225,0,25.8
504921610000000000,truck_22,West,Albert,G-2000,v1.5,5000,300,19,39.94345,16.02353,440,1,45,0,27.8
504921610000000000,truck_8,South,Seth,G-2000,v1.0,5000,300,19,21.89464,44.58628,402,0,234,0,20.3
504921610000000000,truck_23,South,Andy,F-150,v2.0,2000,200,15,73.28131,98.05635,201,7,277,0,25.3
504921610000000000,truck_24,West,Rodney,G-2000,v2.3,5000,300,19,22.19506,0.27702,217,0,321,2,29.5
504921610000000000,truck_9,South,Andy,H-2,v2.3,1500,150,12,,112.40429,402,9,75,4,29.5
504921610000000000,truck_26,South,Seth,F-150,v1.5,2000,200,15,45.65798,144.60844,59,1,183,0,21.7
504921610000000000,truck_27,West,Seth,G-2000,v1.5,5000,300,19,79.55699,97.86561,255,7,348,2,20.2
504921610000000000,truck_25,North,Trish,F-150,v2.0,2000,200,15,17.26506,16.91691,453,8,186,0,24.3
504921610000000000,truck_28,West,Rodney,G-2000,v1.5,5000,300,19,60.33272,4.61578,84,3,198,0,23.1
504921610000000000,truck_29,East,Rodney,G-2000,v2.0,5000,300,19,80.30331,146.54254,340,5,118,0,25.6
504921610000000000,truck_30,East,Andy,G-2000,v1.5,5000,300,19,29.62434,83.73246,300,0,270,4,22.3
504921610000000000,truck_33,West,Rodney,G-2000,v1.5,5000,300,19,89.19593,10.47733,403,8,170,0,29.6
504921610000000000,truck_36,North,Andy,G-2000,v1.0,5000,300,19,87.53087,45.07276,163,0,132,1,27.6
I expected the rows [][]byte contain the csv data row by row
As already suggested, you really should look to use encoding/csv.
That said, the reason for your issue is explained in the godoc above the Bytes() function:
// Bytes returns the most recent token generated by a call to Scan.
// The underlying array may point to data that will be overwritten
// by a subsequent call to Scan. It does no allocation.
func (s *Scanner) Bytes() []byte {
return s.token
}
So the returned byte slice may be modified by subsequent calls to Scan(). To avoid this, you'd need to make a copy of the byte slice, e.g.
for scanner.Scan() {
row := scanner.Bytes()
bs := make([]byte, len(row))
copy(bs, row)
b.Append(bs)
}
You need to create a copy of the data returned by Bytes.
https://pkg.go.dev/bufio#go1.19.3#Scanner.Bytes
Bytes returns the most recent token generated by a call to Scan. The underlying array may point to data that will be overwritten by a subsequent call to Scan. It does no allocation.
for scanner.Scan() {
row := make([]byte, len(scanner.Bytes()))
copy(row, scanner.Bytes())
b.Append(row)
}
https://go.dev/play/p/Lqot-wOXiwh

How to get columns data from golang apache-arrow?

I am using apache-arrow/go to read parquet data.
I can parse the data to table by using apach-arrow.
reader, err := ipc.NewReader(buf, ipc.WithAllocator(alloc))
if err != nil {
log.Println(err.Error())
return nil
}
defer reader.Release()
records := make([]array.Record, 0)
for reader.Next() {
rec := reader.Record()
rec.Retain()
defer rec.Release()
records = append(records, rec)
}
table := array.NewTableFromRecords(reader.Schema(), records)
Here, i can get the column info from table.Colunmn(index), such as:
for i, _ := range table.Schema().Fields() {
a := table.Column(i)
log.Println(a)
}
But the Column struct is defined as
type Column struct {
field arrow.Field
data *Chunked
}
and the println result is like
["WARN" "WARN" "WARN" "WARN" "WARN" "WARN" "WARN" "WARN" "WARN" "WARN"]
However, this is not a string or slice. Is there anyway that i can get the data of each column with string type or []interface{} ?
Update:
I find that i can use reflect to get the element from col.
log.Println(col.(*array.Int64).Value(0))
But i am not sure if this is the recommended way to use it.
When working with Arrow data, there's a couple concepts to understand:
Array: Metadata + contiguous buffers of data
Record Batch: A schema + a collection of Arrays that are all the same length.
Chunked Array: A group of Arrays of varying lengths but all the same data type. This allows you to treat multiple Arrays as one single column of data without having to copy them all into a contiguous buffer.
Column: Is just a Field + a Chunked Array
Table: A collection of Columns allowing you to treat multiple non-contiguous arrays as a single large table without having to copy them all into contiguous buffers.
In your case, you're reading multiple record batches (groups of contiguous Arrays) and treating them as a single large table. There's a few different ways you can work with the data:
One way is to use a TableReader:
tr := array.NewTableReader(tbl, 5)
defer tr.Release()
for tr.Next() {
rec := tr.Record()
for i, col := range rec.Columns() {
// do something with the Array
}
}
Another way would be to interact with the columns directly as you were in your example:
for i := 0; i < table.NumCols(); i++ {
col := table.Column(i)
for _, chunk := range col.Data().Chunks() {
// do something with chunk (an arrow.Array)
}
}
Either way, you eventually have an arrow.Array to deal with, which is an interface containing one of the typed Array types. At this point you are going to have to switch on something, you could type switch on the type of the Array itself:
switch arr := col.(type) {
case *array.Int64:
// do stuff with arr
case *array.Int32:
// do stuff with arr
case *array.String:
// do stuff with arr
...
}
Alternately, you could type switch on the data type:
switch col.DataType().ID() {
case arrow.INT64:
// type assertion needed col.(*array.Int64)
case arrow.INT32:
// type assertion needed col.(*array.Int32)
...
}
For getting the data out of the array, primitive types which are stored contiguously tend to have a *Values method which will return a slice of the type. For example array.Int64 has Int64Values() which returns []int64. Otherwise, all of the types have .Value(int) methods which return the value at a particular index as you showed in your example.
Hope this helps!
Make sure you use v9
(import "github.com/apache/arrow/go/v9/arrow") because it have implemented json.Marshaller (from go-json)
Use "github.com/goccy/go-json" for Marshaler (because of this)
Then you can use TableReader to Marshal it then Unmarshal with type []any
In your example maybe look like this:
import (
"github.com/apache/arrow/go/v9/arrow"
"github.com/apache/arrow/go/v9/arrow/array"
"github.com/apache/arrow/go/v9/arrow/memory"
"github.com/goccy/go-json"
)
...
tr := array.NewTableReader(tabel, 6)
defer tr.Release()
// fmt.Printf("tbl.NumRows() = %+v\n", tbl.NumRows())
// fmt.Printf("tbl.NumColumn = %+v\n", tbl.NumCols())
// keySlice is for sorting same as data source
keySlice := make([]string, 0, tabel.NumCols())
res := make(map[string][]any, 0)
var key string
for tr.Next() {
rec := tr.Record()
for i, col := range rec.Columns() {
key = rec.ColumnName(i)
if res[key] == nil {
res[key] = make([]any, 0)
keySlice = append(keySlice, key)
}
var tmp []any
b2, err := json.Marshal(col)
if err != nil {
panic(err)
}
err = json.Unmarshal(b2, &tmp)
if err != nil {
panic(err)
}
// fmt.Printf("key = %s\n", key)
// fmt.Printf("tmp = %+v\n", tmp)
res[key] = append(res[key], tmp...)
}
}
fmt.Println("res", res)

Failed to update rows in "jinzhu/gorm" pkg

I need to update value of fields in multiple rows.
I'm querying to get some of the database rows, but it doesn't work.
DB.Where("is_send = ?", "0").Find(&artists)
for _, artist := range artists {
if condition {
artist.IsSend = 1
... (more updatee)
DB.Save(&artist)
}
}
Change how you range it, by referring the below example:
for _, elem := range elems {
elem = new_val // Won't work, because elem is a copy of
// the value from elems
}
for i := range elems {
elems[i] = new_val // Works, because elems[i] deferences
// the pointer to the actual value in elems
}
Read: Gotchas
Also, if you're not modifying all fields, rather than using Save you can use Update as well. Refer: GORM CRUD's Interface UPDATE

Map seems to drop values in recursion

I've been working on a problem and I figured I would demonstrate it using a pokemon setup. I am reading from a file, parsing the file and creating objects/structs from them. This normally isn't a problem except now I need to implement interface like inheriting of traits. I don't want there to be duplicate skills in there so I figured I could use a map to replicate a set data structure. However it seems that in the transitive phase of my recursive parsePokemonFile function (see the implementsComponent case), I appear to be losing values in my map.
I am using the inputs like such:
4 files
Ratatta:
name=Ratatta
skills=Tackle:normal,Scratch:normal
Bulbosaur:
name=Bulbosaur
implements=Ratatta
skills=VineWhip:leaf
Oddish:
name=Oddish
implements=Ratatatt
skills=Acid:poison
Venosaur:
name=Venosaur
implements=bulbosaur,oddish
I'm expecting the output for the following code to be something like
Begin!
{Venosaur [{VineWhip leaf} {Acid poison} {Tackle normal} {Scratch normal}]}
but instead I get
Begin!
{Venosaur [{VineWhip leaf} {Acid poison}]}
What am I doing wrong? Could it be a logic error? Or am I making an assumption about the map holding values that I shouldn't?
package main
import (
"bufio"
"fmt"
"os"
"strings"
)
// In order to create a set of pokemon abilities and for ease of creation and lack of space being taken up
// We create an interfacer capability that imports the skills and attacks from pokemon of their previous evolution
// This reduces the amount of typing of skills we have to do.
// Algorithm is simple. Look for the name "implements=x" and then add x into set.
// Unfortunately it appears that the set is dropping values on transitive implements interfaces
func main() {
fmt.Println("Begin!")
dex, err := parsePokemonFile("Venosaur")
if err != nil {
fmt.Printf("Got error: %v\n", err)
}
fmt.Printf("%v\n", dex)
}
type pokemon struct {
Name string
Skills []skill
}
type skill struct {
SkillName string
Type string
}
func parsePokemonFile(filename string) (pokemon, error) {
file, err := os.Open(filename)
if err != nil {
return pokemon{}, err
}
defer file.Close()
scanner := bufio.NewScanner(file)
var builtPokemon pokemon
for scanner.Scan() {
component, returned := parseLine(scanner.Text())
switch component {
case nameComponent:
builtPokemon.Name = returned
case skillsComponent:
skillsStrings := strings.Split(returned, ",")
var skillsArr []skill
// split skills and add them into pokemon skillset
for _, skillStr := range skillsStrings {
skillPair := strings.Split(skillStr, ":")
skillsArr = append(skillsArr, skill{SkillName: skillPair[0], Type: skillPair[1]})
}
builtPokemon.Skills = append(builtPokemon.Skills, skillsArr...)
case implementsComponent:
implementsArr := strings.Split(returned, ",")
// create set to remove duplicates
skillsSet := make(map[*skill]bool)
for _, val := range implementsArr {
// recursively call the pokemon files and get full pokemon
implementedPokemon, err := parsePokemonFile(val)
if err != nil {
return pokemon{}, err
}
// sieve out the skills into a set
for _, skill := range implementedPokemon.Skills {
skillsSet[&skill] = true
}
}
// append final set into the currently being built pokemon
for x := range skillsSet {
builtPokemon.Skills = append(builtPokemon.Skills, *x)
}
}
}
return builtPokemon, nil
}
type component int
// components to denote where to put our strings when it comes time to assemble what we've parsed
const (
nameComponent component = iota
implementsComponent
skillsComponent
)
func parseLine(line string) (component, string) {
arr := strings.Split(line, "=")
switch arr[0] {
case "name":
return nameComponent, arr[1]
case "implements":
return implementsComponent, arr[1]
case "skills":
return skillsComponent, arr[1]
default:
panic("Invalid field found")
}
}
This has nothing to do with Golang maps dropping any values.
The problem is that you are using a map of skill pointers and not skills. Two pointers to the same skill content can be different.
skillsSet := make(map[*skill]bool)
If you change this to map[skill]bool, this should work. You may try it out!

Delete first item in slice if length > 100

When the RSS feeds updates (it doesn't right now, just dummy data) the new items are appended to the "feed" slice. Over time this could mean that it contains millions of items, I don't want that.
So when there are more than 100 items in the slice it should delete items starting from the top (item 0). In this example I'm using an RSS file with ust 100 items so the sample code below should delete from the top after 50 items:
package main
import (
"fmt"
"github.com/SlyMarbo/rss"
"time"
)
var feed *rss.Feed
var directory = "./dump"
func main() {
for {
checkRSS()
// Check every minute if feed.Refresh has passed so it run's update()
time.Sleep(1 * time.Minute)
}
}
func checkRSS() (*rss.Feed, error) {
var err error
// If feed is still empty fetch it first so we can run update()
if feed == nil {
feed, err = rss.Fetch("http://cloud.dgier.nl/api.xml")
} else {
err = feed.Update()
}
length := len(feed.Items)
for key, value := range feed.Items {
fmt.Println(key, value.Title, value.Read)
if key >= 50 {
fmt.Println("Item key is > 50")
}
}
fmt.Printf("Current length: %d\n", length)
fmt.Printf("Refreshing at %s\n", feed.Refresh)
return feed, err
}
If the number of items in the feed grows over the limit, slice it:
length := len(feed.Items)
if length > limit {
feed.Items = feed.Items[length - limit:]
}
When the length is over the limit, the new length will be exactly limit.
You don't need a for loop there.
To achieve this you probably want to use subslicing. Say you want to remove x items from feed, you can simply do feed = feed[x:] which will yield all items after index x-1 and assign it back to the feed slice. If in your actual code you just want to remove the first item then it would be feed = feed[1:]

Resources