I have some issue with a gRPC server-stream on golang.
I had no problem with one row, I just used the simple gRPC response for it.
But now I need to send a number of rows from my database and I can't finish my server-stream application.
I just learn Golang and gRPC and this task a little bit difficult to me to solve this task now. And I will be very grateful if someone could help with this because not too many examples of this material on the web.
Or maybe you now where I can find an example, how to stream data from database using gRPC + golang. Thank you
I have this code:
....
type record struct {
id int
lastname string
}
type server struct {
}
func (s *server) QueryData(city *pb.City, stream pb.GetDataStream_QueryDataServer) error {
db, err := sql.Open("postgres", "postgres://adresspass")
if err != nil {
log.Fatal(err)
}
defer db.Close()
rows, err := db.Query(`SELECT id, last_name FROM person WHERE city=$1`, city.City)
if err != nil {
log.Fatal(err)
}
var rs = make([]record, 0)
var rec record
for rows.Next() {
err = rows.Scan(&rec.id, &rec.lastname)
if err != nil {
return err
}
rs = append(rs, rec)
}
for _, onedata := range rs {
if err := stream.Send(onedata); err != nil {
return err
}
}
return nil
}
...
You should scan your values from database as protocol buffer types and not native types.
You should have defined a message format in proto file
message DbVal{
uint32 id = 1;
message string = 2;
}
And pass that struct to the stream.
Related
I'm trying to list all the items in a DynamoDB table with pagination, and here below is my attempt:
const tableName = "RecordingTable"
type Recording struct {
ID string `dynamodbav:"id"`
CreatedAt string `dynamodbav:"createdAt"`
UpdatedAt string `dynamodbav:"updatedAt"`
Duration int `dynamodbav:"duration"`
}
type RecordingRepository struct {
ctx context.Context
svc *dynamodb.Client
}
func NewRecordingRepository(ctx context.Context) (*RecordingRepository, error) {
cfg, err := config.LoadDefaultConfig(ctx)
if err != nil {
return nil, err
}
return &RecordingRepository{ctx, dynamodb.NewFromConfig(cfg)}, nil
}
func (r *RecordingRepository) List(page int, size int) ([]Recording, error) {
size32 := int32(size)
queryInput := &dynamodb.QueryInput{
TableName: aws.String(tableName),
Limit: &size32,
}
recordings := []Recording{}
queryPaginator := dynamodb.NewQueryPaginator(r.svc, queryInput)
for i := 0; queryPaginator.HasMorePages(); i++ {
result, err := queryPaginator.NextPage(r.ctx)
if err != nil {
return nil, err
}
if i == page {
if result.Count > 0 {
for _, v := range result.Items {
recording := Recording{}
if err := attributevalue.UnmarshalMap(v, &recording); err != nil {
return nil, err
}
recordings = append(recordings, recording)
}
}
break
}
}
return recordings, nil
}
When I run the code above, I get the following error message:
api error ValidationException: Either the KeyConditions or KeyConditionExpression parameter must be specified in the request.
But why should I specify a KeyConditionExpression when I want to get all the items? Is there another way to go or a workaround this?
Query does need your keys. It is meant to find specific items in your DynamoDB. To get all items in your DynamoDB, you need to use the Scan operation.
This should be easily fixed in your code.
Instead of QueryInput use ScanInput and instead of NewQueryPaginator use NewScanPaginator.
Just replaced QueryInput with ScanInput and QueryPaginator with ScanPaginator.
When I loaded my data from Cloud Storage into Bigquery, the click_url field turned into Record type when get created even though it's a string (maybe because of the noindex not sure tho). When I try to insert the data into BigQuery using Inserter. I got this error message:
Cannot convert std::string to a record field:optional .Msg_0_CLOUD_QUERY_TABLE.Msg_1_CLOUD_QUERY_TABLE_click_url click_url = 1
Table in bigquery:
Schema:
Here's the code:
type Product struct {
Name string `datastore:"name" bigquery:"name"`
ClickUrl string `datastore:"click_url,noindex" bigquery:"click_url"`
DateAdded time.Time `datastore:"date_added" bigquery:"date_added"`
}
func insertRows(data interface{}) error {
projectID := "my-project-id"
datasetID := "mydataset"
tableID := "mytable"
ctx := context.Background()
client, err := bigquery.NewClient(ctx, projectID)
if err != nil {
return fmt.Errorf("bigquery.NewClient: %v", err)
}
defer client.Close()
inserter := client.Dataset(datasetID).Table(tableID).Inserter()
if err := inserter.Put(ctx, data); err != nil {
return err
}
return nil
}
func main() {
product := Product{"product_name", "click_url", "date_added_value"} // Example data from datastore
if err := insertRows(product); err != nil {
fmt.Println(err)
}
}
What should I put on the entity tag "bigquery:click_url" to make this work?
Because the ClickUrl in BigQuery is a STRUCT type, which has key-value pairs.
Maybe try this?
type Product struct {
Name string `datastore:"name" bigquery:"name"`
ClickUrl struct {
String string
Text string
Provided string
} `datastore:"click_url,noindex" bigquery:"click_url"`
DateAdded time.Time `datastore:"date_added" bigquery:"date_added"`
}
I'm fairly new to handling sql databases through GO. I have an instance where I am scanning rows from my DB connection, into an slices of a nested struct in an instantiated slice. But can't seem to properly preform it. Is there some looping techniques or references that would be of some use in Golang. I have provided example code and can provide anything else. The connection pool is established and only when I go to scan the rows is where my program craps out. So my question is, if there are multiple rows (4 rows & 2 columns) that I want to insert into the (tiger & lion) objects (columns) how would i loop over and do that with the rows.Scan??
rows, err := db.Query(`Query`)
if err != nil {
//error
return
} else {
// logging
}
}
for rows.Next() {
ref := &structurre{}
err := rows.Scan(&ref.number, &ref.animal[0].tiger, &ref.animal[o].lion)
if err != nil {
logEntry.Info(err)
return details, err
}
details = append(details, ref)
}
type structure struct {
number string
animal []*zoo
}
type zoo struct {
tiger string
lion string
}
Maybe you're looking for something like this:
type zoo struct {
tiger string
lion string
}
type structure struct {
number string
animal []*zoo
}
var ref []structure
rows, err := db.QueryContext(ctx, `query`, `args...`)
if err != nil {
//error
return err
}
// logging
for rows.Next() {
var scans structure
err = rows.Scan(&scans.number, &scans.animal[0].tiger, &scans.animal[0].lion)
if err != nil {
fmt.Println(err)
if err == sql.ErrNoRows {
fmt.Println("No Rows found")
}
return err
}
ref = append(ref, scans)
}
I'm creating a function which receives an interface{}, which will always be a struct, but it could be any struct.
I need to fill one slice with pointers of all the fields of the struct received. The place in the code below where I need to pick the field pointer is flagged with **FIELD POINTER**
My final goal is create a function to receive a struct equivalent to the return of the query sent in the parameter sqlQuery. I want to create a dynamic function to perform any type of query to select, always using the struct received for .Scan.
It may be that I'm thinking the wrong way, I'm still starting Golang.
func QuerySelect(entity interface{}, sqlQuery string) {
val := reflect.Indirect(reflect.ValueOf(entity))
psqlInfo := fmt.Sprintf("host=%s port=%d user=%s "+
"password=%s dbname=%s sslmode=disable",
host, port, user, password, dbname)
// Validate database params
db, err := sql.Open(driver, psqlInfo)
// If returned any error
if err != nil {
panic(err)
}
// Validate connection with database
err = db.Ping()
if err != nil {
panic(err)
}
// Execute query
rows, err := db.Query(sqlQuery)
countColumns := val.Type().NumField()
var allRows []interface{}
for rows.Next() {
columnsPointers := make([]interface{}, countColumns)
for i := 0; i < countColumns; i++ {
columnsPointers[i] = **FIELD POINTER (entity struct)**
}
if err := rows.Scan(columnsPointers...); err != nil {
log.Fatal(err)
}
allRows = append(allRows, entity)
}
}
Per mkopriva's commentary on the question, use the following to get the address of the field:
for i := 0; i < countColumns; i++ {
columnsPointers[i] = val.Field(i).Addr().Interface()
}
I am attempting to map a response from an API to a struct using Golang.
The JSON that comes back when I view the link in the browser is below:
{
"GBP": 657.54
}
And I just want to map it to a simple struct like so:
type Price struct {
Name string
Value float64
}
Here is my current code.
func FetchCoinPrice(fsym string, tsyms string) Price {
url := fmt.Sprintf("https://min-api.cryptocompare.com/data/price?fsym=" + fsym + "&tsyms=" + tsyms)
fmt.Println("Requesting data from " + url)
price := Price{}
// getting the data using http
request, err := http.Get(url)
if err != nil {
log.Fatal(err.Error())
}
// Read the response body using ioutil
body, err := ioutil.ReadAll(request.Body)
if err != nil {
log.Fatal(err.Error())
}
defer request.Body.Close()
if request.StatusCode == http.StatusOK {
json.Unmarshal(body, &price)
}
return price
}
At the moment all I receive is an empty struct, I know the link is bringing back the correct data and I've tested it in my browser.
The mapping doesn't work that way. Instead, you should use a map:
data := []byte(`{
"GBP": 657.54
}`)
priceMap := map[string]float64{}
err := json.Unmarshal(data, &priceMap)
// Check your errors!
if err != nil {
log.Fatal(err.Error())
}
fmt.Println(priceMap)
This will print:
map[GBP:657.54]
You can then iterate over the map and build the struct you mentioned above, or just access the entry directly if you know the currency. eg: priceMap["GBP"]
You should really check your errors, especially if you're not getting the output you expect from Unmarshal.
The problem is that the unmarshaler cannot guess that keys in a JSON object should correspond to some value in a struct. Golang JSON mapping simply doesn't work that way.
However, you can make your "Price" type implement json.Unmarshaler to deserialize a message into a map of floats (map[string]float64) then ensure the shape is right and populate the struct accordingly:
func (p *Price) UnmarshalJSON(bs []byte) error {
kvs := map[string]float64{}
err := json.Unmarshal(bs, &kvs)
if err != nil {
return err
}
if len(kvs) != 1 {
return fmt.Errorf("expected 1 key, got %d", len(kvs))
}
for name, value := range kvs {
p.Name, p.Value = name, value
}
return nil
}
func main() {
jsonstr := `[{"GBP":657.54},{"USD":123.45}]`
ps := []Price{}
err := json.Unmarshal([]byte(jsonstr), &ps)
if err != nil {
panic(err)
}
// ps=[]main.Price{
// main.Price{Name:"GBP", Value:657.54},
// main.Price{Name:"USD", Value:123.45}
// }
}