how to override image registry urls that testcontainer uses - go

I am getting error [ERROR] when trying to pull image through test container. Reason CI machines in your organisation have access to a common registry server and is not allowed to talk to external web. Testcontainer java has something like for this use case
Private registry image name
// Referring directly to an image on a private registry - image name will vary
final MySQLContainer<?> mysql = new MySQLContainer<>(
DockerImageName.parse("registry.mycompany.com/mirror/mysql:8.0.24")
.asCompatibleSubstituteFor("mysql")
)
what's the go equivalent to override image registry urls that testcontainer-go uses?
Code
req := testcontainers.ContainerRequest{
Image: "amazon/dynamodb-local:1.15.0",
ExposedPorts: []string{"8000" + "/tcp"},
ReaperImage: artifact_path,
}
container, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
ContainerRequest: req,
// auto-start the container
Started: true,
})
[ERROR]
2021/09/28 20:21:11 Failed to pull image: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers), will retry

Have you tried to put the fully qualified image name in the container request and passing your registry credentials?
func TestFoo(t *testing.T) {
authConfig := types.AuthConfig{
Username: os.Getenv("DOCKER_USER"),
Password: os.Getenv("DOCKER_PWD"),
}
json, err := json.Marshal(authConfig)
assert.Nil(t, err)
req := testcontainers.ContainerRequest{
Image: "docker.my-company.org/my-namespace/dynamodb-local:1.15.0",
ExposedPorts: []string{"8000" + "/tcp"},
RegistryCred: base64.URLEncoding.EncodeToString(json),
}
container, err := testcontainers.GenericContainer(context.Background(), testcontainers.GenericContainerRequest{
ContainerRequest: req,
// auto-start the container
Started: true,
})
assert.Nil(t, err)
assert.NotNil(t, container)
}
As far as I see in the code, there is no replacer function as in the Java version, but I do not think it makes sense, as in Java they have MySQL containers, as opposite what the Go version does, which does not have specialised containers. Therefore, in the Java version it makes sense to replace the default image for the mysql container.

Related

Envoy WASM failing to load due to missing import (using net/http Go module)

I'm trying to run my WASM Go filter to make an external HTTP call using the net/http module. Envoy fails to load the WASM code. Why is the import failing?
Envoy/Istio version: istio/proxyv2:1.11.4
SDK version: v0.16.1-0.20220127085108-af57b89bc067
TinyGo version: tinygo version 0.22.0 darwin/amd64 (using go version go1.17.6 and LLVM version 13.0.0)
Error Logs
2022-01-31T20:34:18.513749Z error envoy wasm Failed to load Wasm module due to a missing import: env.time.resetTimer
2022-01-31T20:34:18.513794Z error envoy wasm Failed to load Wasm module due to a missing import: env.time.stopTimer
2022-01-31T20:34:18.513807Z error envoy wasm Failed to load Wasm module due to a missing import: env.time.startTimer
2022-01-31T20:34:18.513817Z error envoy wasm Failed to load Wasm module due to a missing import: env.sync/atomic.AddInt32
2022-01-31T20:34:18.513826Z error envoy wasm Failed to load Wasm module due to a missing import: wasi_snapshot_preview1.fd_filestat_get
2022-01-31T20:34:18.513833Z error envoy wasm Failed to load Wasm module due to a missing import: wasi_snapshot_preview1.fd_pread
2022-01-31T20:34:18.513840Z error envoy wasm Failed to load Wasm module due to a missing import: wasi_snapshot_preview1.fd_prestat_get
2022-01-31T20:34:18.513846Z error envoy wasm Failed to load Wasm module due to a missing import: wasi_snapshot_preview1.fd_prestat_dir_name
2022-01-31T20:34:18.513854Z error envoy wasm Failed to load Wasm module due to a missing import: wasi_snapshot_preview1.path_open
2022-01-31T20:34:18.513864Z error envoy wasm Wasm VM failed Failed to initialize Wasm code
2022-01-31T20:34:18.517062Z critical envoy wasm Plugin configured to fail closed failed to load
2022-01-31T20:34:18.517191Z warning envoy config gRPC config for type.googleapis.com/envoy.config.core.v3.TypedExtensionConfig rejected: Unable to create Wasm HTTP filter
tinygo build -o main.wasm -scheduler=asyncify -target=wasi main.go
Actual Code
package main
import (
"errors"
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm"
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm/types"
"io/ioutil"
"time"
"net/http"
)
const (
sharedDataKey = "hello_world_shared_data_key"
)
func main() {
proxywasm.SetVMContext(&vmContext{})
}
type (
vmContext struct{}
pluginContext struct {
// Embed the default plugin context here,
// so that we don't need to reimplement all the methods.
types.DefaultPluginContext
}
httpContext struct {
// Embed the default http context here,
// so that we don't need to reimplement all the methods.
types.DefaultHttpContext
}
)
// Override types.VMContext.
func (*vmContext) OnVMStart(vmConfigurationSize int) types.OnVMStartStatus {
proxywasm.LogInfo("Inside OnVMStart")
http := http.Client{Timeout: time.Duration(10) * time.Second}
resp, err := http.Get("http://SOME_URL:8001/echo?message=hello_world")
if err != nil {
proxywasm.LogWarnf("Error calling hello_world/echo on OnVMStart: %v", err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
proxywasm.LogWarnf("Error parsing hello_world/echo response on OnVMStart: %v", err)
}
proxywasm.LogInfof("Response Body : %s", body)
initialValueBuf := []byte("body")
if err := proxywasm.SetSharedData(sharedDataKey, initialValueBuf, 0); err != nil {
proxywasm.LogWarnf("Error setting shared hello_world data on OnVMStart: %v", err)
}
return types.OnVMStartStatusOK
}
// Override types.DefaultVMContext.
func (*vmContext) NewPluginContext(contextID uint32) types.PluginContext {
return &pluginContext{}
}
// Override types.DefaultPluginContext.
func (*pluginContext) NewHttpContext(contextID uint32) types.HttpContext {
return &httpContext{}
}
// Override types.DefaultHttpContext.
func (ctx *httpContext) OnHttpRequestHeaders(numHeaders int, endOfStream bool) types.Action {
for {
value, err := ctx.getSharedData()
if err == nil {
proxywasm.LogInfof("shared data value: %s", value)
} else if errors.Is(err, types.ErrorStatusCasMismatch) {
continue
}
break
}
return types.ActionContinue
}
func (ctx *httpContext) getSharedData() (string, error) {
value, cas, err := proxywasm.GetSharedData(sharedDataKey)
if err != nil {
proxywasm.LogWarnf("error getting shared data on OnHttpRequestHeaders with cas %d: %v ", cas, err)
return "error", err
}
shared_value := string(value)
return shared_value, err
}
Unfortunately, this is not so easy.
TinyGo might support the module, but you can't "just" call some arbitrary API when using a WASM module for Envoy.
To be slightly more precise, WASM modules run a in sandbox and can only make calls which are explicitly allowed by the runtime. In the case of Envoy, the wasm proxy sdk provides a simple mechanism to call those API.
proxy-wasm-go-sdk provides these API calls which you can use.
There is a function proxywasm.DispatchHttpCall. However, you have to "use the Envoy way" of making http calls.
Note that the "cluster" in that call is not a simple URL, but an Envoy Cluster. You might also try to use Istio-defined cluster like outbound|80||some-service.some-namespace.svc.cluster.local if you have any services defined with Istio Proxies.
You can look up the proxy-config, for example, for an ingress gateway, with istioctl:
istioctl proxy-config all istio-ingressgateway-YOUR-POD -o json | less
When adding ServiceEntries in Istio, you might also get such a "cluster" in your mesh. Note that Service Entries can also refer to external hosts, not only in-cluster services.
Otherwise, you might try adding a manual cluster like in an Envoy-based rate limiting, although this is also easy to get wrong.
- applyTo: CLUSTER
match:
cluster:
service: ratelimit.default.svc.cluster.local
patch:
operation: ADD
# Adds the rate limit service cluster for rate limit service defined in step 1.
value:
name: rate_limit_cluster
type: STRICT_DNS
connect_timeout: 10s
lb_policy: ROUND_ROBIN
http2_protocol_options: {}
load_assignment:
cluster_name: rate_limit_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: ratelimit.default.svc.cluster.local
port_value: 8081
In this description of Envoy Lua Filters, you see some examples. Although it is not WASM, the principle remains the same
For Go, you might try something like
headers := [][2]string{
{":method", "GET"},
{":path", "/echo?message=hello_world"},
{":authority", "SOME_HOST"},
{":scheme", "http"},
}
_, err := proxywasm.DispatchHttpCall("CLUSTER",
headers,
nil,
nil,
1000,
func(numHeaders, bodySize, numTrailers int) {
resp, _ := proxywasm.GetHttpCallResponseBody(0, 10000)
r := string(resp)
proxywasm.LogDebugf("RESPONSE %v", r)
},
)

cloudtasks.CreateTask fails: `lacks IAM permission "cloudtasks.tasks.create"` even though my account has that permission

I'm following the Creating HTTP Target tasks guide.
When I run the code posted below I get this error:
cloudtasks.CreateTask: rpc error: code = PermissionDenied
desc = The principal (user or service account)
lacks IAM permission "cloudtasks.tasks.create" for the resource
"projects/my_project/locations/europe-west1/queues/my_queue"
(or the resource may not exist).
I have signed in with gcloud auth login my#email.com.
my#email.com has the following permissions set by my custom cloud task role:
cloudtasks.locations.get
cloudtasks.locations.list
cloudtasks.queues.get
cloudtasks.queues.list
cloudtasks.tasks.create
cloudtasks.tasks.delete
cloudtasks.tasks.fullView
cloudtasks.tasks.get
cloudtasks.tasks.list
cloudtasks.tasks.run
I don't get it. What more should I check?
main.go
// Run `PROJECT_ID=my_project QUEUE_ID=my_queue go run main.go`
package main
import (
"context"
"fmt"
"os"
cloudtasks "cloud.google.com/go/cloudtasks/apiv2"
taskspb "google.golang.org/genproto/googleapis/cloud/tasks/v2"
)
var (
locationID = "europe-west1"
url = "example.com/callback"
message = "testing"
)
func main() {
projectID := os.Getenv("PROJECT_ID")
queueID := os.Getenv("QUEUE_ID")
task, err := createHTTPTask(projectID, locationID, queueID, url, message)
if err != nil {
fmt.Println(err)
}
fmt.Println(task)
}
// createHTTPTask creates a new task with a HTTP target then adds it to a Queue.
func createHTTPTask(projectID, locationID, queueID, url, message string) (*taskspb.Task, error) {
// Create a new Cloud Tasks client instance.
// See https://godoc.org/cloud.google.com/go/cloudtasks/apiv2
ctx := context.Background()
client, err := cloudtasks.NewClient(ctx)
if err != nil {
return nil, fmt.Errorf("NewClient: %v", err)
}
// Build the Task queue path.
queuePath := fmt.Sprintf("projects/%s/locations/%s/queues/%s", projectID, locationID, queueID)
// Build the Task payload.
// https://godoc.org/google.golang.org/genproto/googleapis/cloud/tasks/v2#CreateTaskRequest
req := &taskspb.CreateTaskRequest{
Parent: queuePath,
Task: &taskspb.Task{
// https://godoc.org/google.golang.org/genproto/googleapis/cloud/tasks/v2#HttpRequest
MessageType: &taskspb.Task_HttpRequest{
HttpRequest: &taskspb.HttpRequest{
HttpMethod: taskspb.HttpMethod_POST,
Url: url,
},
},
},
}
// Add a payload message if one is present.
req.Task.GetHttpRequest().Body = []byte(message)
createdTask, err := client.CreateTask(ctx, req)
if err != nil {
return nil, fmt.Errorf("cloudtasks.CreateTask: %v", err)
}
return createdTask, nil
}
The Cloud Tasks API is enabled.
I've been having the same issue for the past couple of days and figured it out. The library I was using to create the API client and create a task was using different credentials than I expected.
For those that are using "application default credentials", or at least letting the client find credentials automatically, take a look at this page: https://cloud.google.com/docs/authentication/production#finding_credentials_automatically
I had created a service account with all the right roles and was assuming the API client was using the service account. Turns out I wasn't passing in the key file and thus it was using the "application default credentials". For my use case, "application default credentials" referred to the App Engine default service account. When I supplied the API client with a key file for my custom service account, it worked.
Application Default Credentials (ADC) provide a method to get credentials used in calling Google APIs. The gcloud auth application-default command group allows you to manage active credentials on your machine that are used for local application development.
Acquire new user credentials to use for ADC with the following command:
gcloud auth application-default login

Golang HTTP uploading file to S3 using tusd only uploading metadata

I am using the tusd library to upload a file directly to S3 in Go. It seems to be functioning however tusd uploads two files a .info metadata file and a .bin actual content file. For some reason my code is only uploading the info file.
The documentation is quite tricky to navigate so perhaps I have missed a setting somewhere
Code as gist to show both the server and the client code.
There are mutiple issues here.
Your tus libary import paths are wrong they should be:
"github.com/tus/tusd/pkg/handler"
"github.com/tus/tusd/pkg/s3store"
You dont use the S3 store propely, you setup a configuration to have storage directly on your server
fStore := filestore.FileStore{
Path: "./uploads",
}
Instead it should be something like this:
// S3 acces configuration
s3Config := &aws.Config{
Region: aws.String(os.Getenv("AWS_REGION")),
Credentials: credentials.NewStaticCredentials(os.Getenv("AWS_ACCESS_KEY_ID"), os.Getenv("AWS_SECRET_ACCESS_KEY"), ""),
DisableSSL: aws.Bool(true),
S3ForcePathStyle: aws.Bool(true),
}
// Setting up the s3 storage
s3Store := s3store.New(os.Getenv("AWS_BUCKET_NAME"), s3.New(session.Must(session.NewSession()), s3Config))
// Creates a new and empty store composer
composer := handler.NewStoreComposer()
// UseIn sets this store as the core data store in the passed composer and adds all possible extension to it.
s3Store.UseIn(composer)
// Setting up handler
handler, err := handler.NewHandler(handler.Config{
BasePath: "/files/",
StoreComposer: composer,
})
if err != nil {
panic(fmt.Errorf("Unable to create handler: %s", err))
}
// Listen and serve
http.Handle("/files/", http.StripPrefix("/files/", handler))
err = http.ListenAndServe(":8080", nil)
if err != nil {
panic(fmt.Errorf("Unable to listen: %s", err))
}
It is possible that your client isnt working proprely also (I didnt test it).
I would recommend you use https://github.com/eventials/go-tus instead of trying to implement the protocol by yourself.

Getting Unimplemented desc = unknown service error gRPC

In one of my services that happens to be my loadbalancer, I am getting the following error when calling the server method in one of my deployed services:
rpc error: code = Unimplemented desc = unknown service
fooService.FooService
I have a few other services set up with gRPC and they work fine. It just seems to be this one and I am wondering if that is because it is the loadbalancer?
func GetResult(w http.ResponseWriter, r *http.Request) {
conn, errOne := grpc.Dial("redis-gateway:10006", grpc.WithInsecure())
defer conn.Close()
rClient := rs.NewRedisGatewayClient(conn)
result , errTwo := rClient.GetData(context.Background(), &rs.KeyRequest{Key: "trump", Value: "trumpVal"}, grpc.WithInsecure())
fmt.Fprintf(w, "print result: %s \n", result) //prints nil
fmt.Fprintf(w, "print error one: %v \n", errOne) // prints nil
fmt.Fprintf(w, "print error two: %s \n", errTwo) // prints error
}
The error says there is no service called fooService.FooService which is true because the dns name for the service I am calling is called foo-service. However it is the exact same setup for my other services that use gRPC and work fine. Also my proto files are correctly configured so that is not an issue.
server I am calling:
func main() {
lis, err := net.Listen("tcp", ":10006")
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
grpcServer := grpc.NewServer()
newServer := &RedisGatewayServer{}
rs.RegisterRedisGatewayServer(grpcServer, newServer)
if err := grpcServer.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}
The function I am trying to access from client:
func (s *RedisGatewayServer) GetData(ctx context.Context, in *rs.KeyRequest)(*rs.KeyRequest, error) {
return in, nil
}
My docker and yaml files are all correct also with the right naming and ports.
I had this exact problem, and it was because of a very simple mistake: I had put the call to the service registration after the server start. The code looked like this:
err = s.Serve(listener)
if err != nil {
log.Fatalf("[x] serve: %v", err)
}
primepb.RegisterPrimeServiceServer(s, &server{})
Obviously, the registration should have been called before the server was ran:
primepb.RegisterPrimeServiceServer(s, &server{})
err = s.Serve(listener)
if err != nil {
log.Fatalf("[x] serve: %v", err)
}
thanks #dolan's for the comment, it solved the problem.
Basically we have to make sure, the method value should be same in both server and client (you can even copy the method name from the pb.go file generated from server side)
func (cc *ClientConn) Invoke(ctx context.Context, method string, args, reply interface{}, opts ...CallOption) error {
this invoke function will be there inside all the methods which you have implemented in gRPC service.
for me, my client was connecting to the wrong server port.
My server listens to localhost:50052
My client connects to localhost:50051
and hence I see this error.
I had the same problem - I was able to perform rpc call from Postman to the service whereas apigateway was able to connect to the service but on method call It gave error code 12 unknown service and the reason was in my proto files client was under package X whereas server was under package Y.
Quite silly but yeah making the package for both proto under apigateway and service solved my problem.
There are many scenarios in which this can happen. The underlying issue seems to be the same - the GRPC server is reachable but cannot service client requests.
The two scenarios I faced that are not documented in previous answers are:
1. Client and server are not running the same contract version
A breaking change introduced in the contract can cause this error.
In my case, the server was running the older version of the contract while the client was running the latest one.
A breaking change meant that the server could not resolve the service my client was asking for thus returning the unimplemented error.
2. The client is connecting to the wrong GRPC server
The client reached the incorrect server that doesn't implement the contract.
Consider this scenario if you're running multiple different GRPC services. You might be mistakingly dialing the wrong one.

How to Assume Cross-Account Role?

AWS' Golang SDK says that I should use stscreds.AssumeRoleProvider to assume a cross-account role (in this case, for querying another account's DynamoDb table from a web server). This code works:
var sess *session.Session
func init() {
sess = session.Must(session.NewSession(&aws.Config{
Region: aws.String("us-west-2"),
}))
}
func getDynamoDbClient() *dynamodb.DynamoDB {
crossAccountRoleArn := "arn:...:my-cross-account-role-ARN"
creds := stscreds.NewCredentials(sess, crossAccountRoleArn, func(arp *stscreds.AssumeRoleProvider) {
arp.RoleSessionName = "my-role-session-name"
arp.Duration = 60 * time.Minute
arp.ExpiryWindow = 30 * time.Second
})
dynamoDbClient := dynamodb.New(sess, aws.NewConfig().WithCredentials(creds))
return dynamoDbClient
}
According to the documentation, the returned client is thread-safe:
DynamoDB methods are safe to use concurrently.
The question is, since the credential are auto-renewed via stscreds.AssumeRoleProvider, do I
Need to new up a new client on each request (to ensure that I've got unexpired credentials), or
Can I new up a DynamoDb client when the web server starts up, and reuse it for the life of the web server?
Edited To Note:
I dug into the source code for the Golang AWS SDK, and it looks like the credentials returned by stscreds.NewCredentials() are nothing more than a wrapper around a reference to the stscreds.AssumeRoleProvider. So it seems likely to me that the client will magically get auto-renewed credentials.
AWS' documentation leaves something to be desired.
roleArn := "arn:aws:iam::1234567890:role/my-role"
awsSession, _ := session.NewSession(&aws.Config{
Region: aws.String("us-west-2"),
})
stsClient := sts.New(awsSession)
stsRequest := sts.AssumeRoleInput{
RoleArn: aws.String(roleArn),
RoleSessionName: aws.String("my-role-test"),
DurationSeconds: aws.Int64(900), //min allowed
}
stsResponse, err := stsClient.AssumeRole(&stsRequest)
if err != nil {
log.Fatal("an exception occurred when attempting to assume the my role. error=" + err.Error())
}
os.Setenv("AWS_ACCESS_KEY_ID", *stsResponse.Credentials.AccessKeyId)
os.Setenv("AWS_SECRET_ACCESS_KEY", *stsResponse.Credentials.SecretAccessKey)
os.Setenv("AWS_SESSION_TOKEN", *stsResponse.Credentials.SessionToken)
os.Setenv("AWS_SECURITY_TOKEN", *stsResponse.Credentials.SessionToken)
os.Setenv("ASSUMED_ROLE", roleArn)

Resources