I need to upload to an S3 bucket using SignedURLs. I also need to upload multipart files. I have a golang API that vends:
MultiPart Upload
Signed URL for each part
Complete Multipart Upload
Abort Multipart upload
I am attempting to complete a multipart upload. I have a web endpoint that takes an input from my swift 5 client:
type CompletedPart struct {
Etag string
PartNumber int
}
type CompleteMultipartUploadRequest struct {
BucketName string
Key string
UploadID string
MultipartUpload []CompletedPart
}
func putObjectMultipartComplete(w http.ResponseWriter, r *http.Request) {
var completeMultipartUploadRequest CompleteMultipartUploadRequest
err := json.NewDecoder(r.Body).Decode(&completeMultipartUploadRequest)
if err != nil {
http.Error(w, fmt.Sprintf("unable to decode request: %s", err.Error()), http.StatusBadRequest)
return
}
var completedParts []types.CompletedPart
for _, part := range completeMultipartUploadRequest.MultipartUpload {
temp := types.CompletedPart{
ETag: aws.String(part.Etag),
PartNumber: int32(part.PartNumber),
}
completedParts = append(completedParts, temp)
}
input := &s3.CompleteMultipartUploadInput{
Bucket: aws.String(completeMultipartUploadRequest.BucketName),
Key: aws.String(completeMultipartUploadRequest.Key),
UploadId: aws.String(completeMultipartUploadRequest.UploadID),
MultipartUpload: &types.CompletedMultipartUpload {
Parts: completedParts,
},
}
client := s3Client()
response, err := client.CompleteMultipartUpload(context.TODO(), input)
if err != nil {
log.Println(err.Error())
http.Error(w, fmt.Sprintf("could not complete multipart upload: %s", err.Error()), http.StatusFailedDependency)
return
}
object := CompleteMultipartUploadResponse{
BucketName: aws.ToString(response.Bucket),
Key: aws.ToString(response.Key),
Etag: aws.ToString(response.ETag),
}
json, err := json.Marshal(object)
w.Header().Set("Content-Type", "application/json")
w.Write(json)
}
I get a 424 back. When I check the error in the server logs I get:
2022/02/01 16:39:29 operation error S3: CompleteMultipartUpload, https response error StatusCode: 400, RequestID: QREDACTE9, HostID: OdE2MREDACThk=, api error MalformedXML: The XML you provided was not well-formed or did not validate against our published schema
What am I missing here?
This code ended up being almost correct. The AWS S3 API mentioned that a part number ID must be a whole number. I had to start counting at 1 and not 0 as sending a part with the number 0 was not valid.
I'm using golang chromedp as headless webdriver and ccproxy as proxy server, according to this document i'm set credentials headers but chrome also show basic authentication popup.
chromeDP (set headers):
func (c *Browser) setHeaders() chromedp.Tasks {
authData := base64.StdEncoding.EncodeToString([]byte(c.Proxy.User + ":" + c.Proxy.Password))
headers := map[string]interface{}{
"Proxy-Authorization": "Basic " + authData,
}
return chromedp.Tasks{
network.Enable(),
network.SetExtraHTTPHeaders(headers),
}
}
...
apply:
if err := chromedp.Run(c.ctx,
c.setHeaders(),
chromedp.Navigate(c.NavigationUrl),
chromedp.Reload(),
); err != nil {
log.Println("error# ", err)
return c
}
Consider the examples repo which has an example of proxy authentication: https://github.com/chromedp/examples/blob/master/proxy/main.go.
I'm starting to build a regular web app with golang and Angular2, and most importantly I'm trying to secure my login with the help of auth0.com. I download the quickstart code from here and try to run the code, it worked for a while and then next time I run it, the /tmp/session file cannot be found any more.
Here are some basic idea of the code auth0.com provides.
1. Initialize the gorilla sessions filesystemstore
2. Then start the authentification process
code is provided below
app.go
var (
Store *sessions.FilesystemStore
)
func Init() error {
Store = sessions.NewFilesystemStore("", []byte("something-very-secret"))
gob.Register(map[string]interface{}{})
return nil
}
login.go
func LoginHandler(w http.ResponseWriter, r *http.Request) {
domain := os.Getenv("AUTH0_DOMAIN")
aud := os.Getenv("AUTH0_AUDIENCE")
conf := &oauth2.Config{
ClientID: os.Getenv("AUTH0_CLIENT_ID"),
ClientSecret: os.Getenv("AUTH0_CLIENT_SECRET"),
RedirectURL: os.Getenv("AUTH0_CALLBACK_URL"),
Scopes: []string{"openid", "profile"},
Endpoint: oauth2.Endpoint{
AuthURL: "https://" + domain + "/authorize",
TokenURL: "https://" + domain + "/oauth/token",
},
}
if aud == "" {
aud = "https://" + domain + "/userinfo"
}
// Generate random state
b := make([]byte, 32)
rand.Read(b)
state := base64.StdEncoding.EncodeToString(b)
session, err := app.Store.Get(r, "state")
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
session.Values["state"] = state
err = session.Save(r, w)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
audience := oauth2.SetAuthURLParam("audience", aud)
url := conf.AuthCodeURL(state, audience)
http.Redirect(w, r, url, http.StatusTemporaryRedirect)
}
The Error log is
open /tmp/session_46CNLLHBKD... : no such file or directory
I try to understand the code and findout that error log comes from login.go line 39(session, err := app.Store.Get(r, "state")). And I started to track down the code and find out.
login.go:39 -->store.go: 180-->session.go:132-->store.go:186-->store.go:272
you can find store.go and session.go here.
The error log comes from this line: fdata, err := ioutil.ReadFile(filename)
Through the whole process I have not found any function call to save the session file.
I don't understand this error and I don't know why I can run the code at the very beginning, please help me with this problem.
Your any suggestion will be greatly appreciated.Thanks a lot.
I figure out the answer myself
It turns out that I changed my secret key while initializing the gorilla session filesystemstore, but I have not deleted my cookie file in chrome, so it cannot find the tmp sesiion file needed.
I change the key, then delete the coorsponding cookie file and everything is ok now.
I'm trying to create a signed URL for a GET object request in S3. I have this code working flawlessly for putting objects in S3 but I can't seem to get it to work for GET. I sign the URL with this code
//Create the signed url using the company id
func (user *User) signURLForUser(sess *session.Session) (*URLSign, error) {
svc := s3.New(sess)
svc.Config.Region = aws.String(os.Getenv("REGION"))
req, _ := svc.GetObjectRequest(&s3.GetObjectInput{
Bucket: aws.String("bucket"),
Key: aws.String(user.CompanyID + "/" + user.FileRequest),
})
var urlSign URLSign
//urlSign.Size = *out.ContentLength
str, err := req.Presign(time.Minute * 60 * 24 * 5) //Expire in 5 days
if err != nil {
log.Println("Error signing URL Request")
return nil, err
}
urlSign.URL = str
return &urlSign, nil
}
But when I try to use the URL it returns I get this error:
<Error>
<Code>AuthorizationQueryParametersError</Code>
<Message>X-Amz-Algorithm only supports "AWS4-HMAC-SHA256"</Message>
<RequestId>9D7CFB14B195A260</RequestId>
<HostId>
Dgh+SqrHbrdKcbkCYrAj3nObLMAwS7k5+VR1zwC/8ZMS3S4++IAAEXXh3zMZ3CpOAyxX1Kc7Opg=
</HostId>
</Error>
I've check the IAM permissions, they're set for GetObject. I can't think of what else I'm doing wrong.
EDIT: Here's an example of the URL
For sure:
https://rsmachiner-user-code.s3.amazonaws.com//CFDMP_ServoGear.gcode?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIAVEENPDKJRUDZKEVM%2F20180812%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180812T005443Z&X-Amz-Expires=432000&X-Amz-Security-Token=FQoGZXIvYXdzEBoaDIVdv9t408gWWi9vvSLjAaa0pZNA%2BXu83%2FFSyng4XvFdv5%2B7nRB%2FQydMLyi%2BBS84yXqwP6VYn7VlInw4ip1M0lkjHRXQf8OAvQLPrIl%2FQZoTe%2Fy3N6bqhLDOnFVJ3UZzYDQ4%2FbX%2Brc6mvVbkhRsQPiarKBuLYDiOD%2FNoSaItMwI9FsMDknw1qX0Pf%2BZ5La0GmanHrTt9YUI01cIUKJ40No5mKJIwcXw3%2F5QOpUc59rZ2zEzlWP9OXeEwWKp%2Bog5P0v7ABX1lRPsCx4HGEstKhw3ZWmJfQhAcAvhrjmXIMqGNKkaCI5L0ap23jf4GvPMGd4%2BcKIKKvtsF&X-Amz-SignedHeaders=host&X-Amz-Signature=82dfb9b392b5e1ef44c7140259ad884e696b48f8094bdd2d223b8650ebdf59f7
Such issues occur when you are not correctly specifying the algorithm. For me, I was using this url: https://s3_endpoint?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026
which returned to me the same error. I replaced the \u0026 with & and it worked fine.
Session Variables are not maintained across request while using gorilla sessions web toolkit.
When I start the server and type localhost:8100/ page is directed to login.html since session values do not exist.After I login I set the session variable in the store and the page is redirected to home.html. But when I open a new tab and type localhost:8100/ the page should be directed to home.html using already stored session variables, but the page is instead redirected to login.html.
Following is the code.
package main
import (
"crypto/md5"
"encoding/hex"
"fmt"
"github.com/gocql/gocql"
"github.com/gorilla/mux"
"github.com/gorilla/sessions"
"net/http"
"time"
)
var store = sessions.NewCookieStore([]byte("something-very-secret"))
var router = mux.NewRouter()
func init() {
store.Options = &sessions.Options{
Domain: "localhost",
Path: "/",
MaxAge: 3600 * 1, // 1 hour
HttpOnly: true,
}
}
func main() {
//session handling
router.HandleFunc("/", SessionHandler)
router.HandleFunc("/signIn", SignInHandler)
router.HandleFunc("/signUp", SignUpHandler)
router.HandleFunc("/logOut", LogOutHandler)
http.Handle("/", router)
http.ListenAndServe(":8100", nil)
}
//handler for signIn
func SignInHandler(res http.ResponseWriter, req *http.Request) {
email := req.FormValue("email")
password := req.FormValue("password")
//Generate hash of password
hasher := md5.New()
hasher.Write([]byte(password))
encrypted_password := hex.EncodeToString(hasher.Sum(nil))
//cassandra connection
cluster := gocql.NewCluster("localhost")
cluster.Keyspace = "gbuy"
cluster.DefaultPort = 9042
cluster.Consistency = gocql.Quorum
session, _ := cluster.CreateSession()
defer session.Close()
//select query
var firstname string
stmt := "SELECT firstname FROM USER WHERE email= '" + email + "' and password ='" + encrypted_password + "';"
err := session.Query(stmt).Scan(&firstname)
if err != nil {
fmt.Fprintf(res, "failed")
} else {
if firstname == "" {
fmt.Fprintf(res, "failed")
} else {
fmt.Fprintf(res, firstname)
}
}
//store in session variable
sessionNew, _ := store.Get(req, "loginSession")
// Set some session values.
sessionNew.Values["email"] = email
sessionNew.Values["name"] = firstname
// Save it.
sessionNew.Save(req, res)
//store.Save(req,res,sessionNew)
fmt.Println("Session after logging:")
fmt.Println(sessionNew)
}
//handler for signUp
func SignUpHandler(res http.ResponseWriter, req *http.Request) {
fName := req.FormValue("fName")
lName := req.FormValue("lName")
email := req.FormValue("email")
password := req.FormValue("passwd")
birthdate := req.FormValue("date")
city := req.FormValue("city")
gender := req.FormValue("gender")
//Get current timestamp and format it.
sysdate := time.Now().Format("2006-01-02 15:04:05-0700")
//Generate hash of password
hasher := md5.New()
hasher.Write([]byte(password))
encrypted_password := hex.EncodeToString(hasher.Sum(nil))
//cassandra connection
cluster := gocql.NewCluster("localhost")
cluster.Keyspace = "gbuy"
cluster.DefaultPort = 9042
cluster.Consistency = gocql.Quorum
session, _ := cluster.CreateSession()
defer session.Close()
//Insert the data into the Table
stmt := "INSERT INTO USER (email,firstname,lastname,birthdate,city,gender,password,creation_date) VALUES ('" + email + "','" + fName + "','" + lName + "','" + birthdate + "','" + city + "','" + gender + "','" + encrypted_password + "','" + sysdate + "');"
fmt.Println(stmt)
err := session.Query(stmt).Exec()
if err != nil {
fmt.Fprintf(res, "failed")
} else {
fmt.Fprintf(res, fName)
}
}
//handler for logOut
func LogOutHandler(res http.ResponseWriter, req *http.Request) {
sessionOld, err := store.Get(req, "loginSession")
fmt.Println("Session in logout")
fmt.Println(sessionOld)
if err = sessionOld.Save(req, res); err != nil {
fmt.Println("Error saving session: %v", err)
}
}
//handler for Session
func SessionHandler(res http.ResponseWriter, req *http.Request) {
router.PathPrefix("/").Handler(http.FileServer(http.Dir("../static/")))
session, _ := store.Get(req, "loginSession")
fmt.Println("Session in SessionHandler")
fmt.Println(session)
if val, ok := session.Values["email"].(string); ok {
// if val is a string
switch val {
case "": {
http.Redirect(res, req, "html/login.html", http.StatusFound) }
default:
http.Redirect(res, req, "html/home.html", http.StatusFound)
}
} else {
// if val is not a string type
http.Redirect(res, req, "html/login.html", http.StatusFound)
}
}
Can somebody tell me what I am doing wrong. Thanks in advance.
First up: you should never, ever, use md5 to hash passwords. Read this article on why, and then use Go's bcrypt package. You should also parameterise your SQL queries else you are open to catastrophic SQL injection attacks.
Anyway: there are a few problems you need to address here:
Your sessions aren't "sticking" is that you're setting the Path as /loginSession - so when a user visits any other path (i.e. /), the session isn't valid for that scope.
You should be setting up a session store on program initialisation and setting the options there:
var store = sessions.NewCookieStore([]byte("something-very-secret"))
func init() {
store.Options = &sessions.Options{
Domain: "localhost",
Path: "/",
MaxAge: 3600 * 8, // 8 hours
HttpOnly: true,
}
The reason you might set a more specific path is if logged in users are always within a sub-route like /accounts. In your case, that's not what's happening.
I should add that Chrome's "Resource" tab in the Web Inspector (Resources > Cookies) is incredibly useful for debugging issues like these as you can see the cookie expiry, path and other settings.
You're also checking session.Values["email"] == nil, which doesn't work. An empty string in Go is just "", and because session.Values is a map[string]interface{}, you need to type assert the value to a string:
i.e.
if val, ok := session.Values["email"].(string); ok {
// if val is a string
switch val {
case "":
http.Redirect(res, req, "html/login.html", http.StatusFound)
default:
http.Redirect(res, req, "html/home.html", http.StatusFound)
}
} else {
// if val is not a string type
http.Redirect(res, req, "html/login.html", http.StatusFound)
}
We deal with the "not a string" case so we're explicit about what the program should do if the session is not how we expected (client modified it, or an older version of our program used a different type).
You are not checking errors when saving your sessions.
sessionNew.Save(req, res)
... should be:
err := sessionNew.Save(req, res)
if err != nil {
// handle the error case
}
You should get/validate the session in SessionHandler before serving static files (you are doing it in a very roundabout way, however):
func SessionHandler(res http.ResponseWriter, req *http.Request) {
session, err := store.Get(req, "loginSession")
if err != nil {
// Handle the error
}
if session.Values["email"] == nil {
http.Redirect(res, req, "html/login.html", http.StatusFound)
} else {
http.Redirect(res, req, "html/home.html", http.StatusFound)
}
// This shouldn't be here - router isn't scoped in this function! You should set this in your main() and wrap it with a function that checks for a valid session.
router.PathPrefix("/").Handler(http.FileServer(http.Dir("../static/")))
}
The problem is you're writing to the response before calling session.Save. That prevents the headers from being written and thus your cookie from being sent to the client.
In the code after session.Query you're calling Fprintf on the response, as soon as this code executes, calling sessionNew.Save essentially does nothing. Remove any code that writes to the response and try again.
I guess gorilla toolkit's session ought to return an error when calling Save if the response has already been written to.
Following on from the comment chain, please try removing the Domain constraint from the session options, or replace it with a FQDN that resolves (using /etc/hosts for example).
This appears to be a bug in Chromium where cookies with an explicit 'localhost' domain aren't sent. The issue doesn't seem to present itself in Firefox.
I was able to get your demo working using
store.Options = &sessions.Options{
// Domain: "localhost",
MaxAge: 3600 * 1, // 1 hour
HttpOnly: true,
}
In my case the problem was the Path. I know the question is not about it, but this post appears first when you search Google. So, I was starting the session in a path like:
/usuario/login
So the path was set to /usuario, and then, when I made another requests from / the cookie was not set because / is not same as /usuario
I fixed it by specifying a Path, i know this should be obvious but took me some hours to realize it. So:
&sessions.Options{
MaxAge: 60 * 60 * 24,
HttpOnly: true,
Path: "/", // <-- This is very important
}
More info about general cookies: https://developer.mozilla.org/es/docs/Web/HTTP/Cookies
Use a server side "FilesystemStore" instead of a "CookieStore" to save the session variables. Another alternative would be to update the session as a context variable for the request i.e., store the session in the context and let the browser pass it around in every request, using the context.Set() from the gorilla/context package.
Using "CookieStore" is heavy for the client because as the amount of information stored in the cookie grows, more information is transmitted over the wire for every request and response. The advantage it serves is that there is no need to store the session information on the server side. If it is not a constraint to store session information on the server, the ideal way should be to store login and authentication related information on a server side "non-cookie" session store and just pass a token to the client. The server would maintain a map of the token and session information. The "FilesystemStore" allows you to do this.
Though both the "FilesystemStore" and "CookieStore" implement the "Store" interface, each of their "Save()" function's implementations are slightly different. The source code for both the functions, CookieStore.Save() and FilesystemStore.Save() will help us understand why "CookieStore" is not able to persist the session information. The FilesystemStore's Save() method apart from writing the session information to the response header, also saves the information on the server side session file. In a "CookieStore" implementation, if the browser is not able to send the new modified cookie from a response to the next request, the request might fail. In a "FilesystemStore" implementation, the token that is given to the browser always remains the same. The session information is updated in a file and is fetched based on the requesting token, whenever required.