I was trying to implement GetHistoryFromKey() in my Golang Chaincode, but I just get the current token state all the time (and not historic).
My Hyperledger Fabric network (latest version) is very basic for this issue. I am using the fabcar chaincode from fabric-samples in Golang and SDK Node.
After calling InitLedger, I was creating a new car with CreateCar(CarNumber: "121", ...). CarNumber is key in this situation.
Next step is a transaction with ChangeCarOwner("121", "NewOwner")
Now I want to get the history of this car. Therefore I was implementing following function.
func (s *SmartContract) GetHistoryForKeyAccIDUUIDAMO(ctx contractapi.TransactionContextInterface, carNumber string) (string, error) {
historyIter, err := ctx.GetStub().GetHistoryForKey(carNumber)
if err != nil {
return "0", fmt.Errorf("Error in loading History by Key", carNumber)
}
if historyIter.HasNext() {
modification, err := historyIter.Next()
if err != nil {
return "0", fmt.Errorf("Error in getting History by Key in Iteration", carNumber)
}
return string(modification.Value), nil
} else {
return "0", fmt.Errorf("Error in getting HistorybyKey", carNumber)
}
}
Result: {"make":"Toyota","model":"Rav4","colour":"Grey","owner":"NewOwner"}
This is not a historic car token, but the car after changing the owner of Car121 to PersonX
How can I get historic transaction data from the creation of car?
EDIT:
Here is my new function, but it is still the same behaviour.
func (s *SmartContract) GetAssetHistory(ctx contractapi.TransactionContextInterface, account string, carNumber uint64) ([]HistoryQueryResult, error) {
historyKey, err := ctx.GetStub().CreateCompositeKey(Prefix, []string{owner, strconv.FormatUint(carNumber, 10)})
resultsIterator, err := ctx.GetStub().GetHistoryForKey(historyKey)
defer resultsIterator.Close()
var records []HistoryQueryResult
for resultsIterator.HasNext() {
response, err := resultsIterator.Next()
if err != nil {
return nil, err
}
var asset Asset
if len(response.Value) > 0 {
err = json.Unmarshal(response.Value, &asset)
if err != nil {
return nil, err
}
} else {
asset = Asset{Account: account}
}
timestamp, err := ptypes.Timestamp(response.Timestamp)
if err != nil {
return nil, err
}
record := HistoryQueryResult{
TxId: response.TxId,
Timestamp: timestamp,
Record: &asset,
IsDelete: response.IsDelete,
}
records = append(records, record)
}
return records, nil
}
You are just reading the first value from the history iterator so you are seeing just one of the values that has been associated with that key. To see all the historic values you need to iterate over all of them, perhaps collecting them into a slice. See here for an example:
https://github.com/hyperledger/fabric-samples/blob/8ca50df4ffec311e59451c2a7ebe210d9e6f0004/asset-transfer-ledger-queries/chaincode-go/asset_transfer_ledger_chaincode.go#L389-L433
Related
Im newbie in Golang, so it may be simple for professionals but I got stuck with no idea what to do next.
I'm making some migration app that extract some data from oracle DB and after some conversion insert it to Postges one-by-one.
The result of native Query in DB console returns about 400k of rows and takes about 13 sec to end.
The data from Oracle extracts with rows.Next() with some strange behavior:
First 25 rows extracted fast enough, then about few sec paused, then new 25 rows until it pauses "forever".
Here is the function:
func GetHrTicketsFromOra() (*sql.Rows, error) {
rows, err := oraDB.Query("select id,STATE_ID,REMEDY_ID,HEADER,CREATE_DATE,TEXT,SOLUTION,SOLUTION_USER_LOGIN,LAST_SOLUTION_DATE from TICKET where SOLUTION_GROUP_ID = 5549")
if err != nil {
println("Error while getting rows from Ora")
return nil, err
}
log.Println("Finished legacy tickets export")
return rows, err
}
And here I export data:
func ConvertRows(rows *sql.Rows, c chan util.ArchTicket, m chan int) error {
log.Println("Conversion start")
defer func(rows *sql.Rows) {
err := rows.Close()
if err != nil {
log.Println("ORA connection closed", err)
return
}
}(rows)
for rows.Next() {
log.Println("Reading the ticket")
ot := util.OraTicket{}
at := util.ArchTicket{}
err := rows.Scan(&ot.ID, &ot.StateId, &ot.RemedyId, &ot.Header, &ot.CreateDate, &ot.Text, &ot.Solution, &ot.SolutionUserLogin, &ot.LastSolutionDate)
if err != nil {
log.Println("Error while reading row", err)
return err
}
at = convertLegTOArch(ot)
c <- at
}
if err := rows.Err(); err != nil {
log.Println("Error while reading row", err)
return err
}
m <- 1
return nil
}
UPD. I use "github.com/sijms/go-ora/v2" driver
UPD2. Seems like the root cause of the problem is in TEXT and SOLUTION fields of the result rows. They are varchar and can be big enough. Deleting them from the direct query changes the time of execution from 13sec to 258ms. But I still have no idea what to do with that.
UPD3.
Minimal reproducible example
package main
import (
"database/sql"
_ "github.com/sijms/go-ora/v2"
"log"
)
var oraDB *sql.DB
var con = "oracle://login:password#ora_db:1521/database"
func InitOraDB(dataSourceName string) error {
var err error
oraDB, err = sql.Open("oracle", dataSourceName)
if err != nil {
return err
}
return oraDB.Ping()
}
func GetHrTicketsFromOra() {
var ot string
rows, err := oraDB.Query("select TEXT from TICKET where SOLUTION_GROUP_ID = 5549")
if err != nil {
println("Error while getting rows from Ora")
}
for rows.Next() {
log.Println("Reading the ticket")
err := rows.Scan(&ot)
if err != nil {
log.Println("Reading failed", err)
}
log.Println("Read:")
}
log.Println("Finished legacy tickets export")
}
func main() {
err := InitOraDB(con)
if err != nil {
log.Println("Error connection Ora")
}
GetHrTicketsFromOra()
}
I'm trying to bulk remove objects in minio as described here:
objectsCh := make(chan minio.ObjectInfo)
// Send object names that are needed to be removed to objectsCh
go func() {
defer close(objectsCh)
// List all objects from a bucket-name with a matching prefix.
for object := range minioClient.ListObjects(context.Background(), "my-bucketname", "my-prefixname", true, nil) {
if object.Err != nil {
log.Fatalln(object.Err)
}
objectsCh <- object
}
}()
opts := minio.RemoveObjectsOptions{
GovernanceBypass: true,
}
for rErr := range minioClient.RemoveObjects(context.Background(), "my-bucketname", objectsCh, opts) {
fmt.Println("Error detected during deletion: ", rErr)
}
Where I can ListObjects by bucketname and prefixname. However I'm struggling to find an approach where I can ListObjects by for example a slice of object names which I want to remove or any other way. So my question is: how can I properly generate a ListObjects for arbitrary objectNames in a given bucket? Or is there any other way to do remove objects by their names? Thanks.
func DeleteItemInMinio(ctx context.Context, item []string) (string, error) {
minioClient, err := minio.New("test.com", os.Getenv("MINIO_ACCESS_KEY"), os.Getenv("MINIO_SECRET_KEY"), true)
if err != nil {
log.Println(err)
}
for _, val := range item {
err = minioClient.RemoveObject("my-bucketname", val)
if err != nil {
panic(err)
}
}
return "success", nil
}
and call it with :
r.POST("/test/delete", func(c *gin.Context) {
item := []string{"golang.png", "phplogo.jpg"}
execute.DeleteItemInMinio(context.Background(), item)
})
i tried it and it works, in case you still need it
CentOS 7, Github boltdb/bolt version 1.3.1, go version go1.17.7 linux/amd64
This issue may go to a misunderstanding of how BoltDB works, or maybe I have a bug, or maybe there is an issue. I've used BoltDB before, and have had very good results. Though, I didn't explicly look for this issue. What I'm seeing is that I try to delete a key from a bucket, and the key and its value are deleted in the active db.Update, but it's still there after that db.Update is ended. Looking for any explanation of what might be going on. Seems like this functionality couldn't possibly be broken.
I am using a BoltDB bucket for storing a temporary token associated with an email address for creating a new account. Want to be tidy and clean up old data right away (expired tokens, misused tokens, etc). Pretty standard stuff. The structure for the temporary token is (the key is the temporary token, a 10 digit random character string):
(Temporary Token is the Bucket key)
type tempTokenStruct struct {
EmailAddress string `json:"emailaddress"` // Email Address to be changed
TokenExpiryTime int64 `json:"tokenexpirytime"` // Expiry Time for token in Epoch time
}
The user enters an email address in a web form and hits 'submit'. That creates a call to the REST service that creates an entry in the temporary token table, like:
"BpLnfgDsc2" => foo#bar.com, 1645650084
The service emails a URL that has the temporary token embedded, and that link takes the user to a form that allows them to put in their email address (again to verify) and new password (twice). Hitting Submit then results in the following code being called from within a web handler:
func checkTokenValid(emailAddress string, tempToken string) error {
var tempTokenData tempTokenStruct
var tempTokenBytes []byte
tempTokenBytes = []byte(tempToken)
db, err := bolt.Open(USER_DB_NAME, 0644, nil)
if err != nil {
return err
}
defer db.Close()
err = db.Update(func(tx *bolt.Tx) error {
tempTokenBucket := tx.Bucket([]byte("temptokens"))
// The bucket hasn't been created, so there are no stored tokens
if tempTokenBucket == nil {
return errors.New("Not Authorized (1): Please request a new password new/change email from the login page.")
}
// There is no matching token stored in the bucket, so this is an invalid request
tempTokenJSON := tempTokenBucket.Get(tempTokenBytes)
//[I've put a printf here: A]
if tempTokenJSON == nil {
return errors.New("Not Authorized (2): Please request a new password new/change email from the login page.")
}
jsonConvertErr := json.Unmarshal(tempTokenJSON, &tempTokenData)
if jsonConvertErr != nil {
tempTokenBucket.Delete(tempTokenBytes)
return errors.New("Not Authorized (3): Please request a new password new/change email from the login page.")
}
// Check to see if the time is expired, if so, remove the key and indicate error
if tempTokenData.TokenExpiryTime < time.Now().Unix() {
tempTokenBucket.Delete(tempTokenBytes)
//[I've put a printf here: B]
return errors.New("Not Authorized (4): Please request a new password new/change email from the login page.")
}
// Check to see if the email addresses match
if emailAddress != tempTokenData.EmailAddress {
tempTokenBucket.Delete(tempTokenBytes)
return errors.New("Not Authorized (5): Please request a new password new/change email from the login page.")
}
tempTokenBucket.Delete(tempTokenBytes)
return nil
})
// This is test code to see if the key was in fact deleted
db.Update(func(tx *bolt.Tx) error {
tempTokenBucket := tx.Bucket([]byte("temptokens"))
tempTokenJSON := tempTokenBucket.Get(tempTokenBytes)
// [I've put a printf here: C]
return nil
})
return err
}
I'm testing with a timed-out token (4), so the idea is that when it encounters that timed out token, it wants to delete this now invalid token from the bucket.
At the A location, it prints:
First Get call token BpLnfgDsc2 is {"emailaddress":"foo#bar.com","tokenexpirytime":1645650084}
At the B location I put code in that does a .Get, it prints out (looks to be deleted):
Before the DB Close (4), after deleting, token BpLnfgDsc2 is
At the C location, it prints (looks to be back):
After the DB Close, token BpLnfgDsc2 is {"emailaddress":"foo#bar.com","tokenexpirytime":1645650084}
There are no errors returned for anything. I've repeated this many times, putting fmt.Printfs everywhere to see what's going on. The results are the same, the key doesn't seem to be getting deleted. After this sits, I 'vi -b' the DB file, and the key, value is still there. Running after it sits, it still sees the key value there. I'm confused, and any pointers will be appreciated.
Update: The basic bolt functionality of Put/Get/Delete/Get works as per this test code (should be obvious):
package main
import "fmt"
import "encoding/json"
import "github.com/boltdb/bolt"
type tempTokenStruct struct {
EmailAddress string `json:"emailaddress"` // Email Address to be changed (Temporary Token is the DB key)
TokenExpiryTime int64 `json:"tokenexpirytime"` // Expiry Time for token in Epoch time
}
func main() {
var tempToken tempTokenStruct
tempToken.EmailAddress = "foo#bar.com"
tempToken.TokenExpiryTime = 1234567890
tempTokenDataJSON, jsonMarshalError := json.Marshal(tempToken)
if jsonMarshalError != nil {
fmt.Printf("JSON Marshal Error: %s\n", jsonMarshalError.Error())
return
}
tempTokenKey := []byte("foo")
db, err := bolt.Open("test.db", 0644, nil)
if err != nil {
fmt.Printf("Error opening Database\n")
return
}
defer db.Close()
// Put a key in the table
err = db.Update(func(tx *bolt.Tx) error {
tempTokenBucket, err := tx.CreateBucketIfNotExists([]byte("temptokens"))
if err != nil {
return err
}
dbPutError := tempTokenBucket.Put(tempTokenKey, []byte(tempTokenDataJSON))
return dbPutError
})
if err != nil {
fmt.Printf("Error putting key value pair into table: %s\n", err.Error())
}
// Check if the key/value is there after putting it in
err = db.Update(func(tx *bolt.Tx) error {
tempTokenBucket, err := tx.CreateBucketIfNotExists([]byte("temptokens"))
if err != nil {
return err
}
valueGet := tempTokenBucket.Get(tempTokenKey)
fmt.Printf("Value for Token: \"%s\" is \"%s\" just after putting it in there\n", tempTokenKey, valueGet)
return nil
})
// Delete that key from the table
err = db.Update(func(tx *bolt.Tx) error {
tempTokenBucket, err := tx.CreateBucketIfNotExists([]byte("temptokens"))
if err != nil {
return err
}
dbDeleteError := tempTokenBucket.Delete(tempTokenKey)
return dbDeleteError
})
if err != nil {
fmt.Printf("Error Deleting key from bucket: %s\n", err.Error())
}
// Check if the key/value is there after deleting it
err = db.Update(func(tx *bolt.Tx) error {
tempTokenBucket, err := tx.CreateBucketIfNotExists([]byte("temptokens"))
if err != nil {
return err
}
valueGet := tempTokenBucket.Get(tempTokenKey)
fmt.Printf("Value for Token: \"%s\" is \"%s\" after the delete\n", tempTokenKey, valueGet)
return nil
})
if err != nil {
fmt.Printf("Error getting key from table: %s\n", err.Error())
}
}
Prints out:
Value for Token: "foo" is "{"emailaddress":"foo#bar.com","tokenexpirytime":1234567890}" just after putting it in there
Value for Token: "foo" is "" after the delete
So, not sure why the other code doesn't work. Almost as if the delete is using a different key, but the key is the same across the other code.
I believe that the behaviour of db.Update with a non-nil return value is the confusion here. As per the docs
Inside the closure, you have a consistent view of the database. You commit the transaction by returning nil at the end. You can also rollback the transaction at any point by returning an error.
You are returning an error with:
return errors.New("Not Authorized (4): Please request a new password new/change email from the login page.")
This means that all operations within that db.Update( are rolled back. This can be replicated in your simple example with a small change (return fmt.Errorf("RETURNING ERROR HERE")):
package main
import "fmt"
import "encoding/json"
import "github.com/boltdb/bolt"
type tempTokenStruct struct {
EmailAddress string `json:"emailaddress"` // Email Address to be changed (Temporary Token is the DB key)
TokenExpiryTime int64 `json:"tokenexpirytime"` // Expiry Time for token in Epoch time
}
func main() {
var tempToken tempTokenStruct
tempToken.EmailAddress = "foo#bar.com"
tempToken.TokenExpiryTime = 1234567890
tempTokenDataJSON, jsonMarshalError := json.Marshal(tempToken)
if jsonMarshalError != nil {
fmt.Printf("JSON Marshal Error: %s\n", jsonMarshalError.Error())
return
}
tempTokenKey := []byte("foo")
db, err := bolt.Open("test.db", 0644, nil)
if err != nil {
fmt.Printf("Error opening Database\n")
return
}
defer db.Close()
// Put a key in the table
err = db.Update(func(tx *bolt.Tx) error {
tempTokenBucket, err := tx.CreateBucketIfNotExists([]byte("temptokens"))
if err != nil {
return err
}
dbPutError := tempTokenBucket.Put(tempTokenKey, []byte(tempTokenDataJSON))
return dbPutError
})
if err != nil {
fmt.Printf("Error putting key value pair into table: %s\n", err.Error())
}
// Check if the key/value is there after putting it in
err = db.Update(func(tx *bolt.Tx) error {
tempTokenBucket, err := tx.CreateBucketIfNotExists([]byte("temptokens"))
if err != nil {
return err
}
valueGet := tempTokenBucket.Get(tempTokenKey)
fmt.Printf("Value for Token: \"%s\" is \"%s\" just after putting it in there\n", tempTokenKey, valueGet)
return nil
})
// Delete that key from the table
err = db.Update(func(tx *bolt.Tx) error {
tempTokenBucket, err := tx.CreateBucketIfNotExists([]byte("temptokens"))
if err != nil {
return err
}
tempTokenBucket.Delete(tempTokenKey)
return fmt.Errorf("RETURNING ERROR HERE") // CHANGED HERE
})
if err != nil {
fmt.Printf("Error Deleting key from bucket: %s\n", err.Error())
}
// Check if the key/value is there after deleting it
err = db.Update(func(tx *bolt.Tx) error {
tempTokenBucket, err := tx.CreateBucketIfNotExists([]byte("temptokens"))
if err != nil {
return err
}
valueGet := tempTokenBucket.Get(tempTokenKey)
fmt.Printf("Value for Token: \"%s\" is \"%s\" after the delete\n", tempTokenKey, valueGet)
return nil
})
if err != nil {
fmt.Printf("Error getting key from table: %s\n", err.Error())
}
}
The output is now:
Value for Token: "foo" is "{"emailaddress":"foo#bar.com","tokenexpirytime":1234567890}" just after putting it in there
Error Deleting key from bucket: RETURNING ERROR HERE
Value for Token: "foo" is "{"emailaddress":"foo#bar.com","tokenexpirytime":1234567890}" after the delete
This appears to match what you are seeing in your main code. The fix is relatively simple - don't return an error if you want changes to be committed.
I used the operator-sdk to create a custom resource DatabaseService. Creation a DatabaseService CR should trigger the Reconcile function that would create a secret in the CR namespace after getting it from a third party.
I set the CR as the owner of the secret so that whenever the secret is manually deleted, the reconcile function will trigger again and recreate the secret.
Here is the code:
func (r *DatabaseServiceReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
requeueResult := ctrl.Result{Requeue: true, RequeueAfter: time.Minute}
emptyResult := ctrl.Result{Requeue: false, RequeueAfter: 0}
ds := &operatorsv1alpha1.DatabaseService{}
if err := r.Client.Get(context.Background(), req.NamespacedName, ds); err != nil {
if misc.IsNotFound(err) {
return emptyResult, nil
})
return requeueResult, err
}
secret, err := getSecretFromThirdParty(ds)
if err != nil {
return requeueResult, err
}
if err := controllerutil.SetControllerReference(ds, secret, r.Scheme); err != nil {
logger.Error("failed to set controller reference for the secret", zap.Error(err))
return requeueResult, err
}
if err := r.createOrUpdateSecret(secret, logger); err != nil {
return requeueResult, err
}
return emptyResult, nil
}
func (r *DatabaseServiceReconciler) createOrUpdateSecret(secret *corev1.Secret) error {
if err := r.createNamespaceIfNotExist(secret.Namespace, logger); err != nil {
return err
}
if err := r.Client.Create(context.TODO(), secret); err == nil {
return nil
}
if !apierrors.IsAlreadyExists(err) {
return err
}
if err := r.Client.Update(context.TODO(), secret); err != nil {
return err
}
return nil
}
I am observing that if I set the CR as the owner of the secret before calling createOrUpdateSecret - Reconcile function will be triggered again, because something in the owned object (the secret) has changed.
My Reconcile logic is idempotent so it is not a big problem. However, I've no need for Reconcile to run again after changes to the owned object that took place from inside Reconcile. Right now, every time Reconcile creates / updates a secret it would run again. This behavior seems a bit clunky and results in extra work for the operator and extra calls to the third party.
Is there a way to bypass the re-activation of Reconcile creation / update of owned object from inside Reconcile? Or is it not recommended and I should allow reconcile run repeatedly until nothing is changed?
I'm trying to list all the items in a DynamoDB table with pagination, and here below is my attempt:
const tableName = "RecordingTable"
type Recording struct {
ID string `dynamodbav:"id"`
CreatedAt string `dynamodbav:"createdAt"`
UpdatedAt string `dynamodbav:"updatedAt"`
Duration int `dynamodbav:"duration"`
}
type RecordingRepository struct {
ctx context.Context
svc *dynamodb.Client
}
func NewRecordingRepository(ctx context.Context) (*RecordingRepository, error) {
cfg, err := config.LoadDefaultConfig(ctx)
if err != nil {
return nil, err
}
return &RecordingRepository{ctx, dynamodb.NewFromConfig(cfg)}, nil
}
func (r *RecordingRepository) List(page int, size int) ([]Recording, error) {
size32 := int32(size)
queryInput := &dynamodb.QueryInput{
TableName: aws.String(tableName),
Limit: &size32,
}
recordings := []Recording{}
queryPaginator := dynamodb.NewQueryPaginator(r.svc, queryInput)
for i := 0; queryPaginator.HasMorePages(); i++ {
result, err := queryPaginator.NextPage(r.ctx)
if err != nil {
return nil, err
}
if i == page {
if result.Count > 0 {
for _, v := range result.Items {
recording := Recording{}
if err := attributevalue.UnmarshalMap(v, &recording); err != nil {
return nil, err
}
recordings = append(recordings, recording)
}
}
break
}
}
return recordings, nil
}
When I run the code above, I get the following error message:
api error ValidationException: Either the KeyConditions or KeyConditionExpression parameter must be specified in the request.
But why should I specify a KeyConditionExpression when I want to get all the items? Is there another way to go or a workaround this?
Query does need your keys. It is meant to find specific items in your DynamoDB. To get all items in your DynamoDB, you need to use the Scan operation.
This should be easily fixed in your code.
Instead of QueryInput use ScanInput and instead of NewQueryPaginator use NewScanPaginator.
Just replaced QueryInput with ScanInput and QueryPaginator with ScanPaginator.