I'm trying to write an application in GO that will get all the image vulnerabilities inside a GCP project for me using the Container Analysis API.
The GO Client library for this API has the function findVulnerabilityOccurrencesForImage() to do this, however it requires you to pass the URL of the image you want to get the vulnerability report from in the form resourceURL := "https://gcr.io/my-project/my-repo/my-image" and the projectID. This means that if there are multiple images in your project, you have to list and store them first and only after that you can recursively call the findVulnerabilityOccurrencesForImage() function to get ALL of the vulnerabilities.
So I need a way to get and store all of the images' URLs inside all of the repos inside a given GCP project, but so far I couldn't find a solution. I can easily do that in the CLI by running gcloud container images list command but I don't see a way how that can be done with an API.
Thank you in advance for your help!
You can use the Cloud Storage package and the Objects method to do so. For example:
func GetURLs() ([]string, error) {
bucket := "bucket-name"
urls := []string{}
results := client.Bucket(bucket).Objects(context.Background(), nil)
for {
attrs, err := results.Next()
if err != nil {
if err == iterator.Done {
break
}
return nil, fmt.Errorf("iterating results: %w", err)
}
urls = append(urls, fmt.Sprint("https://storage.googleapis.com", "/", bucket, "/", attrs.Name))
}
return urls, nil
}
Related
A bit over my head here, and the for the life me I could not find suitable solution to my usage case scenario.
I'm currently working on a project where in some scenario, when a user visit a certain page "Using Gin Framework Templates", part of the page content is rendered via data retrieved from the DB.
The issue with this approach is that Gin dose not seam to provide the ability to allow me to render any content Without fully escaping it.
I have developed previously with "PHP Framework", and used its template engine "Balde" I was able to if the need arises-ed to have the option to render HTML directly without escaping by using the following Directive "{!! $variableName !!}".
But in Using Gin, I was not successful into finding any builtin template directive that would allow me to render retrieved HTML content directly from DB.
Hope if anyone could support or provide solution or direction on how to solve this issue.
The following is a could sample and which results I get and which results I hope to get.
The following is a quick example of my current issue:
router := gin.Default()
router.LoadHTMLGlob("templates/*")
router.GET("/", func(c *gin.Context) {
db, err := gorm.Open(mysql.Open(configs.DB_Connection()), &gorm.Config{})
if err != nil {
panic("failed to connect database")
}
c.HTML(http.StatusOK, "index.html", gin.H{
"title": "<h1>Hello World</h1>",
})
})
Using the builtin template engine in Gin, I get the following results:
<h1>Hello World</h1>
What I'm hoping to get is:
Hello World
Please note that I'm getting that HTML directly from the DB.
I have managed to find a solution to my use case scenario, based on the following Thread , and my own adjustments to make work with GORM as well.
Solution is:
create the following function:
func getStructFeild(v *migrations.TableName, field string) string {
r := reflect.ValueOf(v)
f := reflect.Indirect(r).FieldByName(field).String()
return f
}
router := gin.Default()
router.LoadHTMLGlob("templates/*")
router.GET("/", func(c *gin.Context) {
db, err := gorm.Open(mysql.Open(configs.DB_Connection()), &gorm.Config{})
if err != nil {
panic("failed to connect database")
}
var table migrations.TableName
db.Where("id = ?", 1).Select("column_name").Find(&table)
c.HTML(http.StatusOK, "index.html", gin.H{
"title": template.HTML(getStructFeild(&table, "ModalStructColumnName")),
})
})
and that's it Gin Template is now able fully render the HTML that is retrieved from the DB.
Go has a builtin library for rendering html template but it's syntax might be a little different then others like blade, mustache.
I am building a microservice app, and currently writing some tests. The function that I am testing is below where it's owned by cart service and tries to get all cart items and append the item details with other details of each item from catalog service.
func (s *Server) Grpc_GetCartItems(ctx context.Context, in *pb.GetCartItemsRequest) (*pb.ItemsResponse, error) {
// Get product ids and its quantity in cart by userId
res, err := s.Repo.GetCartItems(ctx, in.UserId)
if err != nil {
return nil, err
}
// Return empty response if there is no items in cart
if len(res) == 0 {
return &pb.ItemsResponse{}, nil
}
// Get Product ID Keys from map
ids := GetMapKeys(res)
// RPC call catalog server to get cart products' names
products, err := s.CatalogClient.Grpc_GetProductsByIds(ctx, &catalogpb.GetProductsByIdsRequest{ProductIds: ids})
if err != nil{
return nil, err
}
// Return response in format product id, product name, and qty in cart
items, err := AppendItemToResponse(products, res)
if err != nil{
return nil, err
}
return items, nil
}
The problem is for the test setup, I need to seed some test data to both of the cart and catalog repositories. I can do that with cart repo just fine, but for the catalog is it a common practice to just mock the dependency s.CatalogClient.Grpc_GetProductsByIds instead? I am still new to testing, and from what I understand you generally don't do mocking in integration tests, but I am not sure if there's a better way to tackle this kind of issue.
You're correct in that for an integration test you would not mock a service.
Usually, if it is a service you do not have control over, you would stub the service.
Integration tests can be run against staging or testing services (in an E2E capacity) or in a virtual environment (like compose, K8S, etc.).
I think for your requirement, I would stage it using docker-compose or something similar. If you intend to go for an E2E setup in the future, you may want to look into having a testing environment.
See: https://www.testenvironmentmanagement.com/types-of-testing-environments/
Obligatory "you should not be calling one microservice directly from another" comment. While you can find a way to make testing work, you've tightly coupled the architecture. This (testing) concern is only the first of what will become many since your cart service directly ties to your catalog service. If you fix you close-coupled architecture problem, your testing problem will also be resolved.
I'm new to developing web applications. I'm working with golang and would like to serve user uploaded files securely, such as allowing them to view their own files only.
Now I have saved those files to a local file system with random names. If I serve the entire directory, malicious users may view other users files. This sounds like a common use case, I wonder what's the best approach to deal with it?
This question in pretty vague and architectural decisions must be made to optimize data access and secure the files.
However, here is a simple solution that might serve your use-case.
package main
import (
"fmt"
"mime"
"net/http"
"path/filepath"
)
//UserFilesMap is the map that contains
var UserFilesMap map[string]FilePermission
type FilePermission map[string]struct{}
//FileServer is the function that serves files
func FileServer(w http.ResponseWriter, r *http.Request) {
//get the file path the user wants to access
filename := r.URL.Path[9:]
var uname, pass string
var ok bool
if uname, pass, ok = r.BasicAuth(); !ok {
w.WriteHeader(http.StatusForbidden)
return
}
if !(uname == "user" && pass == "1234") {
w.WriteHeader(http.StatusForbidden)
return
}
//Checking if user has permission to the file
if _, ok := UserFilesMap[uname][filename]; !ok {
w.WriteHeader(http.StatusForbidden)
return
}
w.Header().Set("Content-Type", mime.TypeByExtension(filepath.Ext(filename)))
http.ServeFile(w, r, "files/"+filename)
}
func main() {
UserFilesMap = make(map[string]FilePermission)
// UserFilesMap["user"] = FilePermission{"xyz.txt": struct{}{}}
UserFilesMap["user"] = FilePermission{"abc.txt": struct{}{}}
http.HandleFunc("/getFile/", FileServer)
if err := http.ListenAndServe(":8080", nil); err != nil {
fmt.Println("Error in ListenAndServe")
}
}
Here, I used a map to store the permissions of files. I would suggest you go for a SQL table instead.
If ur filenames are random and long enough and use a secure random generator this is already secure (unless directory-listing is enabled) with some limits though.
https://golang.org/pkg/crypto/rand/
One user will have access to the file only if he has the url with the random name. The limitation is though that the URL will be saved in the browser history, if someone else finds it he will also have access to it.
I'm trying to insert a simple record as in GoDoc. But this returns,
rpc error: code = 7 desc = "User can't access project: tidy-groove"
When I searched for grpc codes, it says..
PermissionDenied Code = 7
// Unauthenticated indicates the request does not have valid
// authentication credentials for the operation.
I've enabled Big table in my console and created a cluster and a service account and recieved the json. What I'm doing wrong here?
package main
import (
"fmt"
"golang.org/x/net/context"
"golang.org/x/oauth2/google"
"google.golang.org/cloud"
"google.golang.org/cloud/bigtable"
"io/ioutil"
)
func main() {
fmt.Println("Start!")
put()
}
func getClient() *bigtable.Client {
jsonKey, err := ioutil.ReadFile("TestProject-7854ea9op741.json")
if err != nil {
fmt.Println(err.Error())
}
config, err := google.JWTConfigFromJSON(
jsonKey,
bigtable.Scope,
) // or bigtable.AdminScope, etc.
if err != nil {
fmt.Println(err.Error())
}
ctx := context.Background()
client, err := bigtable.NewClient(ctx, "tidy-groove", "asia-east1-b", "test1-bigtable", cloud.WithTokenSource(config.TokenSource(ctx)))
if err != nil {
fmt.Println(err.Error())
}
return client
}
func put() {
ctx := context.Background()
client := getClient()
tbl := client.Open("table1")
mut := bigtable.NewMutation()
mut.Set("links", "maps.google.com", bigtable.Now(), []byte("1"))
mut.Set("links", "golang.org", bigtable.Now(), []byte("1"))
err := tbl.Apply(ctx, "com.google.cloud", mut)
if err != nil {
fmt.Println(err.Error())
}
}
I've solved the problem. It's nothing wrong with the code, but config json itself. So anyone who out there want to authenticate and came here by google search... This code is correct and working perfectly. What I've done wrong is follows.
First I made a service account and got the json. But google warned me that im not an owner of project hence it wont be added to accept list but anyway it let me download the json.
Then I deleted that key from console and requested project owner to create a key for me.
There he has created another key with the same name I given.. And since he's the owner no error/warning msgs displayed and successfully json file was downloaded.
When I tried with that... my question begun. That's when i posted this question.
After that with no solutions. I asked owner to delete that key and create another key but with a different name..
Then it worked! It seems if you try to create a key with non-owner account and then again create with same name ( after deleting original of course ) has no effect. Hope this helps everyone out there :)
Take a look at: helloworld.go or search.go which uses GOOGLE_APPLICATION_CREDENTIALS environment variable.
For most environments, you no longer even need to set GOOGLE_APPLICATION_CREDENTIALS. Google Cloud Platform, Managed VMs or Google App Engine all have the right thing set for you. Your desktop environment will also be correct if you've used gcloud init or it's predecessor gcloud auth login followed by gcloud config set project <projectID>.
I'm using the oauth package "code.google.com/p/goauth2/oauth" with revel and the it creates a few structures with quite a bit of information in it. I need this information to be persistent throughout the session but sessions can only be type string. Is there a better way of doing this than the following?
c.Session["AccessToken"] = t.Token.AccessToken
c.Session["RefreshToken"] = t.Token.RefreshToken
...
If not how do I reassign the strings to create another structure to call Client.Get() ?
You can use the json package to "convert" structs to string and vice versa. Just know that only exported fields are serialized this way.
Since oauth.Token has only exported fields, this will work:
if data, err := json.Marshal(t.Token); err == nil {
c.Session["Token"] = string(data)
} else {
panic(err)
}
And this is how you can reconstruct the token from the session:
if err := json.Unmarshal([]byte(c.Session["Token"]), &t.Token); err != nil {
panic(err)
}
Instead of that you can try to save some string ID to Session and the object you need to Cache:
c.Session["id"] = id
go cache.Set("token_"+id, t.Token, 30*time.Minute)
And then use it as follows:
var token oauth.Token
id := c.Session["id"]
if err := cache.Get("token_"+id, &token); err != nil {
// TODO: token not found, do something
}
// TODO: use your token here...
The advantage of this approach is you do not have to work with json package explicitly and cache has a few different back-ends out of the box: memory, redis, memcached. Moreover, you do not have a limitation of 4K as in case of Cookie based Session.
https://revel.github.io/manual/cache.html