I was able to use the CollectionGroup to access all the subCollection documents (records) that satisfy a condition. Then I was able to iterate in a loop to delete the subCollection documents. The problem, although it works perfectly but is an hack. Is there a better way to delete subcollections in Firestore using Golang?
it := clientdb.CollectionGroup("mychildSubcollection").Where(...mycondition).OrderBy("myfield", firestore.Desc).Documents(context.Background())
for {
doc, err := it.Next()
if err == iterator.Done {
break
}
if err != nil {
// return err
}
// Strucure of the doc.Ref is --> &{0xc0000d6788 projects/myproj/databases/(default)/documents/myparentCollection/Ki8sr65sKIoZaCviCp/mychildSubcollection/JwvKbuyRTGx5wZaCviCp myparentCollection/Ki8sr65sKIoZaCviCp/mychildSubcollection/JwvKbuyRTGx5wZaCviCp JwvKbuyRTGx5wZaCviCp}
// fmt.Println(doc.Ref.ID)
// fmt.Println(doc.Ref.Path)
// fmt.Println(doc.Ref.Parent.Path)
// fmt.Println(doc.Ref.Parent.ID)
path1 := doc.Ref.Parent.Path
path2 := path1[0: strings.LastIndex(path1, "myparentCollection")]
path3 := strings.Replace(path1, path2, "", -1)
clientdb.Collection(path3).Doc(doc.Ref.ID).Delete(context.Background()) // Regular collection command, to delete the subcollection
}
At least to make it a less hack, this may also help -> As you see the doc.Ref provides three fields showing the complete path [doc.Ref.Path or doc.Ref.Parent.Path] the subcollection document id (doc.Ref.ID), how to access the middle field in the structure: "myparentCollection/Ki8sr65sKIoZaCviCp/mychildSubcollection/JwvKbuyRTGx5wZaCviCp"
Thanks!
The above is accepted as solution. The only comment I am adding is that if there is an better way to extract the path for the collection, it will be more generic. As you see I am using the actual collection name [myparentCollection] to extract the path. Works great, but something generic would be better.
now := time.Now() // start deleting old ones, even ones that still have one second left to expire.
it := clientdb.CollectionGroup("mychildSubcollection").Where("expireafter", "<", now.Add(-1*time.Second)).OrderBy("myfield", firestore.Desc).Documents(context.Background())
for {
doc, err := it.Next()
if err == iterator.Done {
break
}
if err != nil {
// return err
}
// Strucure of the doc.Ref is --> &{0xc0000d6788 projects/myproj/databases/(default)/documents/myparentCollection/Ki8sr65sKIoZaCviCp/mychildSubcollection/JwvKbuyRTGx5wZaCviCp myparentCollection/Ki8sr65sKIoZaCviCp/mychildSubcollection/JwvKbuyRTGx5wZaCviCp JwvKbuyRTGx5wZaCviCp}
// fmt.Println(doc.Ref.ID)
// fmt.Println(doc.Ref.Path)
// fmt.Println(doc.Ref.Parent.Path)
// fmt.Println(doc.Ref.Parent.ID)
// using some logic with strings below to get (extract) the collection path example: myparentCollection/Ki8sr65sKIoZaCviCp/mychildSubcollection, so the delete can use it and delete the particular document under the sub collection.
path1 := doc.Ref.Parent.Path
path2 := path1[0: strings.LastIndex(path1, "myparentCollection")]
path3 := strings.Replace(path1, path2, "", -1)
clientdb.Collection(path3).Doc(doc.Ref.ID).Delete(context.Background()) // Regular collection command, to delete the subcollection
}
Related
I am using apache-arrow/go to read parquet data.
I can parse the data to table by using apach-arrow.
reader, err := ipc.NewReader(buf, ipc.WithAllocator(alloc))
if err != nil {
log.Println(err.Error())
return nil
}
defer reader.Release()
records := make([]array.Record, 0)
for reader.Next() {
rec := reader.Record()
rec.Retain()
defer rec.Release()
records = append(records, rec)
}
table := array.NewTableFromRecords(reader.Schema(), records)
Here, i can get the column info from table.Colunmn(index), such as:
for i, _ := range table.Schema().Fields() {
a := table.Column(i)
log.Println(a)
}
But the Column struct is defined as
type Column struct {
field arrow.Field
data *Chunked
}
and the println result is like
["WARN" "WARN" "WARN" "WARN" "WARN" "WARN" "WARN" "WARN" "WARN" "WARN"]
However, this is not a string or slice. Is there anyway that i can get the data of each column with string type or []interface{} ?
Update:
I find that i can use reflect to get the element from col.
log.Println(col.(*array.Int64).Value(0))
But i am not sure if this is the recommended way to use it.
When working with Arrow data, there's a couple concepts to understand:
Array: Metadata + contiguous buffers of data
Record Batch: A schema + a collection of Arrays that are all the same length.
Chunked Array: A group of Arrays of varying lengths but all the same data type. This allows you to treat multiple Arrays as one single column of data without having to copy them all into a contiguous buffer.
Column: Is just a Field + a Chunked Array
Table: A collection of Columns allowing you to treat multiple non-contiguous arrays as a single large table without having to copy them all into contiguous buffers.
In your case, you're reading multiple record batches (groups of contiguous Arrays) and treating them as a single large table. There's a few different ways you can work with the data:
One way is to use a TableReader:
tr := array.NewTableReader(tbl, 5)
defer tr.Release()
for tr.Next() {
rec := tr.Record()
for i, col := range rec.Columns() {
// do something with the Array
}
}
Another way would be to interact with the columns directly as you were in your example:
for i := 0; i < table.NumCols(); i++ {
col := table.Column(i)
for _, chunk := range col.Data().Chunks() {
// do something with chunk (an arrow.Array)
}
}
Either way, you eventually have an arrow.Array to deal with, which is an interface containing one of the typed Array types. At this point you are going to have to switch on something, you could type switch on the type of the Array itself:
switch arr := col.(type) {
case *array.Int64:
// do stuff with arr
case *array.Int32:
// do stuff with arr
case *array.String:
// do stuff with arr
...
}
Alternately, you could type switch on the data type:
switch col.DataType().ID() {
case arrow.INT64:
// type assertion needed col.(*array.Int64)
case arrow.INT32:
// type assertion needed col.(*array.Int32)
...
}
For getting the data out of the array, primitive types which are stored contiguously tend to have a *Values method which will return a slice of the type. For example array.Int64 has Int64Values() which returns []int64. Otherwise, all of the types have .Value(int) methods which return the value at a particular index as you showed in your example.
Hope this helps!
Make sure you use v9
(import "github.com/apache/arrow/go/v9/arrow") because it have implemented json.Marshaller (from go-json)
Use "github.com/goccy/go-json" for Marshaler (because of this)
Then you can use TableReader to Marshal it then Unmarshal with type []any
In your example maybe look like this:
import (
"github.com/apache/arrow/go/v9/arrow"
"github.com/apache/arrow/go/v9/arrow/array"
"github.com/apache/arrow/go/v9/arrow/memory"
"github.com/goccy/go-json"
)
...
tr := array.NewTableReader(tabel, 6)
defer tr.Release()
// fmt.Printf("tbl.NumRows() = %+v\n", tbl.NumRows())
// fmt.Printf("tbl.NumColumn = %+v\n", tbl.NumCols())
// keySlice is for sorting same as data source
keySlice := make([]string, 0, tabel.NumCols())
res := make(map[string][]any, 0)
var key string
for tr.Next() {
rec := tr.Record()
for i, col := range rec.Columns() {
key = rec.ColumnName(i)
if res[key] == nil {
res[key] = make([]any, 0)
keySlice = append(keySlice, key)
}
var tmp []any
b2, err := json.Marshal(col)
if err != nil {
panic(err)
}
err = json.Unmarshal(b2, &tmp)
if err != nil {
panic(err)
}
// fmt.Printf("key = %s\n", key)
// fmt.Printf("tmp = %+v\n", tmp)
res[key] = append(res[key], tmp...)
}
}
fmt.Println("res", res)
How do I append output from a twitter search to the field Data in the SearchTwitterOutput{} struct.
Thanks!
I am using a twitter library to search twitter base on a query input. The search returns an array of strings(I believe), I am able to fmt.println the data but I need the data as a struct.
type SearchTwitterOutput struct {
Data string
}
func (SearchTwitter) execute(input SearchTwitterInput) (*SearchTwitterOutput, error) {
credentials := Credentials{
AccessToken: input.AccessToken,
AccessTokenSecret: input.AccessTokenSecret,
ConsumerKey: input.ConsumerKey,
ConsumerSecret: input.ConsumerSecret,
}
client, err := GetUserClient(&credentials)
if err != nil {
return nil, err
}
// search through the tweet and returns a
search, _ , err := client.Search.Tweets(&twitter.SearchTweetParams{
Query: input.Text,
})
if err != nil {
println("PANIC")
panic(err.Error())
return &SearchTwitterOutput{}, err
}
for k, v := range search.Statuses {
fmt.Printf("Tweet %d - %s\n", k, v.Text)
}
return &SearchTwitterOutput{
Data: "test", //data is a string for now it can be anything
}, nil
}
//Data field is a string type for now it can be anything
//I use "test" as a placeholder, bc IDK...
Result from fmt.Printf("Tweet %d - %s\n", k, v.Text):
Tweet 0 - You know I had to do it to them! #JennaJulien #Jenna_Marbles #juliensolomita #notjulen Got my first hydroflask ever…
Tweet 1 - RT #brenna_hinshaw: I was in J2 today and watched someone fill their hydroflask with vanilla soft serve... what starts here changes the wor…
Tweet 2 - I miss my hydroflask :(
This is my second week working with go and new to development. Any help would be great.
It doesn't look like the client is just returning you a slice of strings. The range syntax you're using (for k, v := range search.Statuses) returns two values for each iteration, the index in the slice (in this case k), and the object from the slice (in this case v). I don't know the type of search.Statuses - but I know that strings don't have a .Text field or method, which is how you're printing v currently.
To your question:
Is there any particular reason to return just a single struct with a Data field rather than directly returning the output of the twitter client?
Your function signature could look like this instead:
func (SearchTwitter) execute(input SearchTwitterInput) ([]<client response struct>, error)
And then you could operate on the text in those objects in wherever this function was called.
If you're dead-set on placing the data in your own struct, you could return a slice of them ([]*SearchTwitterOutput), in which case you could build a single SearchTwitterOutput in the for loop you're currently printing the tweets in and append it to the output list. That might look like this:
var output []*SearchTwitterOutput
for k, v := range search.Statuses {
fmt.Printf("Tweet %d - %s\n", k, v.Text)
output = append(output, &SearchTwitterOutput{
Data: v.Text,
})
}
return output, nil
But if your goal really is to return all of the results concatenated together and placed inside a single struct, I would suggest building a slice of strings (containing the text you want), and then joining them with the delimiter of your choosing. Then you could place the single output string in your return object, which might look something like this:
var outputStrings []string
for k, v := range search.Statuses {
fmt.Printf("Tweet %d - %s\n", k, v.Text)
outputStrings = append(outputStrings, v.Text)
}
output = strings.Join(outputStrings, ",")
return &SearchTwitterOutput{
Data: output,
}, nil
Though I would caution, it might be tricky to find a delimiter that will never show up in a tweet..
I've been working on a problem and I figured I would demonstrate it using a pokemon setup. I am reading from a file, parsing the file and creating objects/structs from them. This normally isn't a problem except now I need to implement interface like inheriting of traits. I don't want there to be duplicate skills in there so I figured I could use a map to replicate a set data structure. However it seems that in the transitive phase of my recursive parsePokemonFile function (see the implementsComponent case), I appear to be losing values in my map.
I am using the inputs like such:
4 files
Ratatta:
name=Ratatta
skills=Tackle:normal,Scratch:normal
Bulbosaur:
name=Bulbosaur
implements=Ratatta
skills=VineWhip:leaf
Oddish:
name=Oddish
implements=Ratatatt
skills=Acid:poison
Venosaur:
name=Venosaur
implements=bulbosaur,oddish
I'm expecting the output for the following code to be something like
Begin!
{Venosaur [{VineWhip leaf} {Acid poison} {Tackle normal} {Scratch normal}]}
but instead I get
Begin!
{Venosaur [{VineWhip leaf} {Acid poison}]}
What am I doing wrong? Could it be a logic error? Or am I making an assumption about the map holding values that I shouldn't?
package main
import (
"bufio"
"fmt"
"os"
"strings"
)
// In order to create a set of pokemon abilities and for ease of creation and lack of space being taken up
// We create an interfacer capability that imports the skills and attacks from pokemon of their previous evolution
// This reduces the amount of typing of skills we have to do.
// Algorithm is simple. Look for the name "implements=x" and then add x into set.
// Unfortunately it appears that the set is dropping values on transitive implements interfaces
func main() {
fmt.Println("Begin!")
dex, err := parsePokemonFile("Venosaur")
if err != nil {
fmt.Printf("Got error: %v\n", err)
}
fmt.Printf("%v\n", dex)
}
type pokemon struct {
Name string
Skills []skill
}
type skill struct {
SkillName string
Type string
}
func parsePokemonFile(filename string) (pokemon, error) {
file, err := os.Open(filename)
if err != nil {
return pokemon{}, err
}
defer file.Close()
scanner := bufio.NewScanner(file)
var builtPokemon pokemon
for scanner.Scan() {
component, returned := parseLine(scanner.Text())
switch component {
case nameComponent:
builtPokemon.Name = returned
case skillsComponent:
skillsStrings := strings.Split(returned, ",")
var skillsArr []skill
// split skills and add them into pokemon skillset
for _, skillStr := range skillsStrings {
skillPair := strings.Split(skillStr, ":")
skillsArr = append(skillsArr, skill{SkillName: skillPair[0], Type: skillPair[1]})
}
builtPokemon.Skills = append(builtPokemon.Skills, skillsArr...)
case implementsComponent:
implementsArr := strings.Split(returned, ",")
// create set to remove duplicates
skillsSet := make(map[*skill]bool)
for _, val := range implementsArr {
// recursively call the pokemon files and get full pokemon
implementedPokemon, err := parsePokemonFile(val)
if err != nil {
return pokemon{}, err
}
// sieve out the skills into a set
for _, skill := range implementedPokemon.Skills {
skillsSet[&skill] = true
}
}
// append final set into the currently being built pokemon
for x := range skillsSet {
builtPokemon.Skills = append(builtPokemon.Skills, *x)
}
}
}
return builtPokemon, nil
}
type component int
// components to denote where to put our strings when it comes time to assemble what we've parsed
const (
nameComponent component = iota
implementsComponent
skillsComponent
)
func parseLine(line string) (component, string) {
arr := strings.Split(line, "=")
switch arr[0] {
case "name":
return nameComponent, arr[1]
case "implements":
return implementsComponent, arr[1]
case "skills":
return skillsComponent, arr[1]
default:
panic("Invalid field found")
}
}
This has nothing to do with Golang maps dropping any values.
The problem is that you are using a map of skill pointers and not skills. Two pointers to the same skill content can be different.
skillsSet := make(map[*skill]bool)
If you change this to map[skill]bool, this should work. You may try it out!
In my code I want to do or not to do some actions depending on document with given key existence. But can't avoid additional network overhead retrieving all document content.
Now I'm using
cas, err := bucket.Get(key, &value)
And looking for err == gocb.ErrKeyNotFound to determine document missed case.
Is there some more efficient approach?
You can use the sub-document API and check for the existence of a field.
Example from Using the Sub-Document API to get (only) what you want
:
rv = bucket.lookup_in('customer123', SD.exists('purchases.pending[-1]'))
rv.exists(0) # (check if path for first command exists): =>; False
Edit: Add go example
You can use the sub-document API to check for document existence like this:
frag, err := bucket.LookupIn("document-key").
Exists("any-path").Execute()
if err != nil && err == gocb.ErrKeyNotFound {
fmt.Printf("Key does not exist\n")
} else {
if frag.Exists("any-path") {
fmt.Printf("Path exists\n")
} else {
fmt.Printf("Path does not exist\n")
}
}
Problem:
I have created a simple form, where there's an input field "num". After submission I want to show the value of num in the same input field, in other words want to retain the input in that field. If the value was set to 0 then I want to ignore that.
I can do it in several languages but I'm not sure about how to do it in Golang. My current template file has,
<input type="text" placeholder="foo" name="bar" value="{{if gt .N 0 }} {{.N}} {{end}} "/>
Server file contains:
data := &listOfReport {
R: r,
I: i,
N: n
}
listTmpl := template.Must(template.New("list_tmpl").Parse(string(report.Template["xxx.tmpl"])))
if err := listTmpl.Execute(w, data); err != nil {
http.Error(w, fmt.Sprintf("Error rendering template %v", err), 500)
}
Another thought is to make N a string so make it '' or value in the server file. But that actually spoils the variable's name/purpose.
Is there any better way to do it? Is ther any better way to access GET parameters directly from template? Please note that the value of N is originally got from a GET variable.
*This code is not tested
There is no standard/builtin way to get any request parameters from within a template, you'll have to put it into your data. (You could write a function which does this for you, but that will result in an ugly hack.)
I don't see what's wrong with your solution.
I take a similar approach, but use structs.
type SignupForm struct {
Name string
Email string
Etcera bool
}
// Type alias
type M map[string]interface{}
...
// In the handler that accepts your form
err := r.ParseForm()
if err != nil {
// handle error
}
signup := SignupForm{}
err := decoder.Decode(signup, r.PostForm)
if err != nil {
// handle error
}
// Store the 'saved' form contents somewhere temporary -
// e.g.
// - cookies (keep in mind the 4K browser limit)
// - server side sessions (Redis; how I do it)
// - db
// In the handler that renders your form
err := template.ExecuteTemplate(w, "form.html", M{
"form": signup,
"csrfToken": csrfToken,
// and so on...
})
Note that wherever you store the form data, make sure it is temporary. Server side sessions are ideal as you can have them expire (if you don't want to delete them manually).