I have data that is created rows by rows, 6 columns, I don't know the final number of rows in advance.
Currently i'm creating a 2D slice of 200x6 with all zeros and then i replace these zeros gradually with my data, row by row. The data comes from another dataframe df
It works but i don't like to end up with the last rows of my slice full of zeros. I see 2 solutions:
- I delete all the last rows with only zeros when I'm done
- I create an empty slice and append my data progressively to it
I tried various things but could not figure out how to code any of these 2 solutions.
Currently my code looks like this:
var orders [200][6]float64 // create my 2d slice with zeros
order_line := 0
for i := start_line; i <= end_line; i++ {
if sell_signal == "1" {
//record line number and sold price in orders slice
orders[order_line][1] = float64(i+1)
orders[order_line][2],err = strconv.ParseFloat(df[i][11], 64)
order_line = order_line + 1
}
}
I looked at the Append command, but I tried all sorts of combinations to make it work on a 2d slice, could not find one that works.
edit: from the comments below I understand that i'm actually creating an array, not a slice, and there is no way to append data to an array.
In Golang slices are preferred in place of arrays.
Creating so many rows in prior is not required, just create a slice every time you are looping over your data to add a new row in the parent slice. That will help you to have only required number of rows and you need to worry about the length Since you are appending a slice at an index of parent slice.
package main
import (
"fmt"
"math/rand"
)
func main() {
orders := make([][]float64, 0) // create my 2d slice with zeros
for i := 0; i <= 6; i++ {
value := rand.Float64()
temp := make([]float64, 0)
temp = append(temp, value)
orders = append(orders, [][]float64{temp}...)
}
fmt.Println(orders)
}
Please check working code on Playground
If you notice I am creating a new temp slice in loop which contains the float64 value and then appending value to the temp slice which I have passed to the parent slice.
So everytime I append the temp slice to the parent slice a new row will be created.
Note:
Arrays have their place, but they're a bit inflexible, so you don't
see them too often in Go code. Slices, though, are everywhere. They
build on arrays to provide great power and convenience.
Edited:
To work on first 3 columns and then manipulate the values for next 3 columns which will be added to the temp slice and appended to the main slice. Use below code logic:
package main
import (
"fmt"
"math/rand"
"strconv"
)
func main() {
orders := make([][]float64, 0) // create my 2d slice with zeros
for i := 0; i <= 6; i++ {
value := rand.Float64()
// logic to create first 3 columns
temp := make([]float64, 0)
temp = append(temp, value)
temp2 := make([]float64, 3)
// logic to create next 3 columns on basis of previous 3 columns
for j, value := range temp {
addcounter, _ := strconv.ParseFloat("1", 64)
temp2[j] = value + addcounter
}
temp = append(temp, temp2...)
orders = append(orders, [][]float64{temp}...)
}
fmt.Println(orders)
}
Working Example
Given that the outer container has an unknown number of elements and the inner container has exactly six elements, use a slice of arrays.
var orders [][6]float64
for i := start_line; i <= end_line; i++ {
if sell_signal == "1" {
n, err = strconv.ParseFloat(df[i][11], 64)
if err != nil {
// handle error
}
orders = append(orders, [6]float64{1: float64(i + 1), 2: n})
}
}
This code uses a composite literal [6]float64 value instead of assigning element by element as in the question.
You can come back and access elements of the [6]float64 at a later time. For example:
orders[i][3] = orders[i][1] + orders[i][2]
For better readability, and easier slice handling, just create a struct type and fill a slice with them. This allows you to properly name each field instead of magic index numbers and makes it easier to fill in the rows as well as utilize later. Unless there is some specific reason to use arrays/slices for the columns, this is more idiomatic Go. The following example would fill a slice with however many results you have, and no more.
Full example here: https://play.golang.org/p/mLtabqO8MNj
type Row struct {
Thing float64
Data float64
More float64
Stuff float64
Another float64
Number float64
}
var rows []*Row
numResults := 15
for i := 0; i <= numResults; i++ {
row := &Row{}
row.Thing = 2.5
// ... fill values
rows = append(rows, row)
}
Related
This is super trivial question but I could not find it asked here. Sorry if I missed it.
Can I count that the order in slice is always the order of insertion of elements in that slice?
I tested it with the following code:
func main() {
for i := 0; i < 10000; i++ {
testOrder()
}
}
func testOrder() {
sl := []int{}
for i := 0; i < 50; i++ {
sl = append(sl, i)
}
for i, el := range sl {
if el != i {
panic("Order not quaranteed")
}
}
}
Fill a slice with numbers and then check if the order of elements in the slice is as they were populated in it. I do this check 10000 times.
You're not inserting but appending. append() always appends the elements to the end of the slice, and it does not leave "holes". So yes, you can count on that your example will never panic with "Order not guaranteed".
Read more about the append() function in its doc: https://golang.org/pkg/builtin/#append
Also in the Spec: Appending to and copying slices
Also read official blog post: The Go Blog: Arrays, slices (and strings): The mechanics of 'append'
Slices and arrays are sequentially ordered structures. The order of elements will never change if you append to them, even if they are copied in memory.
I need to create a 2 dimensional string array as shown below -
matrix = [['cat,'cat','cat'],['dog','dog']]
Code:-
package main
import (
"fmt"
)
func main() {
{ // using append
var matrix [][]string
matrix[0] = append(matrix[0],'cat')
fmt.Println(matrix)
}
}
Error:-
panic: runtime error: index out of range
goroutine 1 [running]:
main.main()
/tmp/sandbox863026592/main.go:11 +0x20
You have a slice of slices, and the outer slice is nil until it's initialized:
matrix := make([][]string, 1)
matrix[0] = append(matrix[0],'cat')
fmt.Println(matrix)
Or:
var matrix [][]string
matrix = append(matrix, []string{"cat"})
fmt.Println(matrix)
Or:
var matrix [][]string
var row []string
row = append(row, "cat")
matrix = append(matrix, row)
The problem with doing two-dimensional arrays with Go is that you have to initialise each part individually, e.g., if you have a [][]bool, you gotta allocate []([]bool) first, and then allocate the individual []bool afterwards; this is the same logic regardless of whether you're using make() or append() to perform the allocations.
In your example, the matrix[0] doesn't exist yet after a mere var matrix [][]string, hence you're getting the index out of range error.
For example, the code below would create another slice based on the size of an existing slice of a different type:
func solve(board [][]rune, …) {
x := len(board)
y := len(board[0])
visited := make([][]bool, x)
for i := range visited {
visited[i] = make([]bool, y)
}
…
If you simply want to initialise the slice based on a static array you have, you can do it directly like this, without even having to use append() or make():
package main
import (
"fmt"
)
func main() {
matrix := [][]string{{"cat", "cat", "cat"}, {"dog", "dog"}}
fmt.Println(matrix)
}
https://play.golang.org/p/iWgts-m7c4u
consider this piece of code:
package main
import (
"fmt"
)
func main() {
fmt.Println(Part(11))
}
func Part(n int) string {
enumResult := [][]int{}
enum(n, n, []int{}, &enumResult)
fmt.Println(enumResult)
fmt.Println(40, enumResult[40])
return ""
}
var abc int = 0
func enum(n int, top int, pre []int, result *[][]int) {
var i int
if n > top {
i = top
} else {
i = n
}
for ; i > 0; i-- {
tempResult := append(pre, i)
if n-i == 0 {
/* if tempResult[0] == 3 && tempResult[1] == 3 && tempResult[2] == 3 && tempResult[3] == 2 {
tempResult = append(tempResult, 12345)
}*/
fmt.Println(abc, tempResult)
abc++
*result = append(*result, tempResult)
} else {
enum(n-i, i, tempResult, result)
}
}
}
When I run this code
I append value '[3,3,3,2]' to 'enumResult'
but If I check the value of 'enumResult' then '[3,3,3,1]' is appear
it`s index is 40 =>enumResult[40]
(other value is correct)
I don`t know why this is happening
Can you explain to me why?
The problem is indeed due to append.
There are two thing about append. First is, that append doe not necessarily copy memory. As the spec specifies:
If the capacity of s is not large enough to fit the additional values,
append allocates a new, sufficiently large underlying array that fits
both the existing slice elements and the additional values. Otherwise,
append re-uses the underlying array.
This may cause unexpected behavior if you are not clear. A playground example: https://play.golang.org/p/7A3JR-5IX8o
The second part is, that when append does copy memory, it grows the capacity of the slice. However, it does not grow it just by 1. A playground example: https://play.golang.org/p/STr9jMqORUz
How much append grows a slice is undocumented and considered an implentation details. But till Go 1.10, it follows this rule:
Go slices grow by doubling until size 1024, after which they grow by
25% each time.
Note that when enabling race-detector, this may change. The code for growing slice is located in $GOROOT/src/runtime/slice.go in growslice function.
Now back to the question. It should be clear now that your code did append from a same slice with sufficient capacity due to growth of the slice from append before. To solve it, make a new slice and copy the memory.
tempResult := make([]int,len(pre)+1)
copy(tempResult,pre)
tempResult[len(pre)] = i
Suppose I have a slice slice of type int. While declaring, I set the third argument to size, which I believe reserves memory for at least size ints by setting the cap parameter of the slice.
slice:=make([]int,0,size)
Now, suppose I have an integer variable value. To add it to the slice at the end, I use
slice=append(slice,value)
If the number of elements currently in the slice is less than size, then there will be no need to copy the entire underlying array to a new location in order to add the new element.
Further, if I want to prepend value to slice, as suggested here and here, I use
slice=append([]int{value},slice...)
My question is, what happens in this case? If the number of elements is still less than size, how are the elements stored in the memory? Assuming a contiguous allocation when the make() function was invoked, are all existing elements right shifted to free the first space for value? Or is memory reallocated and all elements copied?
The reason for asking is that I would like my program to be as fast as possible, and would like to know if this is a possible cause for slowing it down. If it is so, is there any alternative way of prepending that would be more time efficient?
With reslicing and copying
The builtin append() always appends elements to a slice. You cannot use it (alone) to prepend elements.
Having said that, if you have a slice capacity bigger than length (has "free" space after its elements) to which you want to prepend an element, you may reslice the original slice, copy all elements to an index one higher to make room for the new element, then set the element to the 0th index. This will require no new allocation. This is how it could look like:
func prepend(dest []int, value int) []int {
if cap(dest) > len(dest) {
dest = dest[:len(dest)+1]
copy(dest[1:], dest)
dest[0] = value
return dest
}
// No room, new slice need to be allocated:
// Use some extra space for future:
res := make([]int, len(dest)+1, len(dest)+5)
res[0] = value
copy(res[1:], dest)
return res
}
Testing it:
s := make([]int, 0, 5)
s = append(s, 1, 2, 3, 4)
fmt.Println(s)
s = prepend(s, 9)
fmt.Println(s)
s = prepend(s, 8)
fmt.Println(s)
Output (try it on the Go Playground):
[1 2 3 4]
[9 1 2 3 4]
[8 9 1 2 3 4]
Note: if no room for the new element, since performance does matter now, we didn't just do:
res := append([]int{value}, dest...)
Because it does more allocations and copying than needed: allocates a slice for the literal ([]int{value}), then append() allocates a new when appending dest to it.
Instead our solution allocates just one new array (by make(), even reserving some space for future growth), then just set value as the first element and copy dest (the previous elements).
With linked list
If you need to prepend many times, a normal slice may not be the right choice. A faster alternative would be to use a linked list, to which prepending an element requires no allocations of slices / arrays and copying, you just create a new node element, and you designate it to be the root by pointing it to the old root (first element).
The standard library provides a general implementation in the container/list package.
With manually managing a larger backing array
Sticking to normal slices and arrays, there is another solution.
If you're willing to manage a larger backing array (or slice) yourself, you can do so by leaving free space before the slice you use. When prepending, you can create a new slice value from the backing larger array or slice which starts at an index that leaves room for 1 element to be prepended.
Without completeness, just for demonstration:
var backing = make([]int, 15) // 15 elements
var start int
func prepend(dest []int, value int) []int {
if start == 0 {
// No more room for new value, must allocate bigger backing array:
newbacking := make([]int, len(backing)+5)
start = 5
copy(newbacking[5:], backing)
backing = newbacking
}
start--
dest = backing[start : start+len(dest)+1]
dest[0] = value
return dest
}
Testing / using it:
start = 5
s := backing[start:start] // empty slice, starting at idx=5
s = append(s, 1, 2, 3, 4)
fmt.Println(s)
s = prepend(s, 9)
fmt.Println(s)
s = prepend(s, 8)
fmt.Println(s)
// Prepend more to test reallocation:
for i := 10; i < 15; i++ {
s = prepend(s, i)
}
fmt.Println(s)
Output (try it on the Go Playground):
[1 2 3 4]
[9 1 2 3 4]
[8 9 1 2 3 4]
[14 13 12 11 10 8 9 1 2 3 4]
Analysis: this solution makes no allocations and no copying when there is room in the backing slice to prepend the value! All that happens is it creates a new slice from the backing slice that covers the destination +1 space for the value to be prepended, sets it and returns this slice value. You can't really get better than this.
If there is no room, then it allocates a larger backing slice, copies over the content of the old, and then does the "normal" prepending.
With tricky slice usage
Idea: imagine that you always store elements in a slice in backward order.
Storing your elements in backward order in a slice means a prepand becomes append!
So to "prepand" an element, you can simply use append(s, value). And that's all.
Yes, this has its limited uses (e.g. append to a slice stored in reverse order has the same issues and complexity as a "normal" slice and prepand operation), and you lose many conveniences (ability to list elements using for range just to name one), but performance wise nothing beats prepanding a value just by using append().
Note: iterating over the elements that stores elements in backward order has to use a downward loop, e.g.:
for i := len(s) - 1; i >= 0; i-- {
// do something with s[i]
}
Final note: all these solutions can easily be extended to prepend a slice instead of just a value. Generally the additional space when reslicing is not +1 but +len(values), and not simply setting dst[0] = value but instead a call to copy(dst, values).
The "prepend" call will need to allocate an array and copy all elements because a slice in Go is defined as a starting point, a size and an allocation (with the allocation being counted from the starting point).
There is no way a slice can know that the element before the first one can be used to extend the slice.
What will happen with
slice = append([]int{value}, slice...)
is
a new array of a single element value is allocated (probably on stack)
a slice is created to map this element (start=0, size=1, alloc=1)
the append call is done
append sees that there is not enough room to extend the single-element slice so allocates a new array and copies all the elements
a new slice object is created to refer to this array
If appending/removing at both ends of a large container is the common use case for your application then you need a deque container. It is unfortunately unavailable in Go and impossible to write efficiently for generic contained types while maintaining usability (because Go still lacks generics).
You can however implement a deque for your specific case and this is easy (for example if you have a large container with a known upper bound may be a circular buffer is all you need and that is just a couple of lines of code away).
I'm very new to Go, so may be the following is very bad Go code... but it's an attempt to implement a deque using a growing circular buffer (depending on the use case this may be or may be not a good solution)
type Deque struct {
buffer []interface{}
f, b, n int
}
func (d *Deque) resize() {
new_buffer := make([]interface{}, 2*(1+d.n))
j := d.f
for i := 0; i < d.n; i++ {
new_buffer[i] = d.buffer[j]
d.buffer[j] = nil
j++
if j == len(d.buffer) {
j = 0
}
}
d.f = 0
d.b = d.n
d.buffer = new_buffer
}
func (d *Deque) push_back(x interface{}) {
if d.n == len(d.buffer) {
d.resize()
}
d.buffer[d.b] = x
d.b++
if d.b == len(d.buffer) {
d.b = 0
}
d.n++
}
func (d *Deque) push_front(x interface{}) {
if d.n == len(d.buffer) {
d.resize()
}
if d.f == 0 {
d.f = len(d.buffer)
}
d.f--
d.buffer[d.f] = x
d.n++
}
func (d *Deque) pop_back() interface{} {
if d.n == 0 {
panic("Cannot pop from an empty deque")
}
if d.b == 0 {
d.b = len(d.buffer)
}
d.b--
x := d.buffer[d.b]
d.buffer[d.b] = nil
d.n--
return x
}
func (d *Deque) pop_front() interface{} {
if d.n == 0 {
panic("Cannot pop from an empty deque")
}
x := d.buffer[d.f]
d.buffer[d.f] = nil
d.f++
if d.f == len(d.buffer) {
d.f = 0
}
d.n--
return x
}
I wanted to implement time based slots for holding data using golang slices. I managed to come up with a go program like this and it also works. But I have few questions regarding garbage collection and the general performance of this program. Does this program guarantee garbage collection of items once slice is equated to nil? And while shuffling slices, I hope this program does not do any deep copying.
type DataSlots struct {
slotDuration int //in milliseconds
slots [][]interface{}
totalDuration int //in milliseconds
}
func New(slotDur int, totalDur int) *DataSlots {
dat := &DataSlots{slotDuration: slotDur,
totalDuration: totalDur}
n := totalDur / slotDur
dat.slots = make([][]interface{}, n)
for i := 0; i < n; i++ {
dat.slots[i] = make([]interface{}, 0)
}
go dat.manageSlots()
return dat
}
func (self *DataSlots) addData(data interface{}) {
self.slots[0] = append(self.slots[0], data)
}
// This should be a go routine
func (self *DataSlots) manageSlots() {
n := self.totalDuration / self.slotDuration
for {
time.Sleep(time.Duration(self.slotDuration) * time.Millisecond)
for i := n - 1; i > 0; i-- {
self.slots[i] = self.slots[i-1]
}
self.slots[0] = nil
}
}
I removed critical section handling in this snippet to make it concise.
Once your slice is set too nil, any values contained in the slice are available for garbage collection, provided that the underlying array isn't shared with another slice.
Since there are no slice operations in your program, you never have multiple references to the same array, nor are you leaving data in any inaccessible portions of the underlying array.
What you need to be careful of, is when you're using slice operations:
a := []int{1, 2, 3, 4}
b := a[1:3]
a = nil
// the values 1 and 4 can't be collected, because they are
// still contained in b's underlying array
c := []int{1, 2, 3, 4}
c = append(c[1:2], 5)
// c is now []int{2, 5}, but again the values 1 and 4 are
// still in the underlying array. The 4 may be overwritten
// by a later append, but the 1 is inaccessible and won't
// be collected until the underlying array is copied.
While append does copy values when the capacity of the slice in insufficient, only the values contained in the slice are copied. There is no deep copy of any of the values.