slack integration: 404 Expired url https://hooks.slack.com/commands - go

Here is my golang app listening requests from slack command:
main.go:
package main
import (
"bytes"
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"time"
)
type SlackCmdResponse struct {
Text string `json:"text"`
}
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
if err := req.ParseForm(); err != nil {
panic(err)
}
responseUrl := req.PostFormValue("response_url")
go func() {
time.Sleep(time.Second)
postBack(responseUrl)
}()
rj, err := json.Marshal(SlackCmdResponse{Text: "Test started"})
if err != nil {
panic(err)
}
w.Header().Set("Content-Type", "application/json")
w.Write(rj)
})
fmt.Println("listening 8383")
if err := http.ListenAndServe(":8383", nil); err != nil {
panic(err)
}
}
func postBack(responseUrl string) {
fmt.Println("responseUrl", responseUrl)
cResp := SlackCmdResponse{
Text: "Test finished",
}
cj, err := json.Marshal(cResp)
if err != nil {
panic(err)
}
req, err := http.NewRequest("POST", responseUrl, bytes.NewReader(cj))
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
if resp != nil {
fmt.Println(resp.StatusCode)
if b, err := ioutil.ReadAll(resp.Body); err != nil {
panic(err)
} else {
fmt.Println(string(b))
}
if resp.Body != nil {
resp.Body.Close()
}
}
}
I run it:
$ go run main.go
listening 8383
I use ngrok to make it accessible from the internet:
ngrok http 8383
I created slack slash command /my-command with POST option and pasted https URL that ngrok gave:
Now when I run /my-command in slack, I get slack reply Test started. And then in a second slack prints Test finished
For now, it's all fine.
Problem
If I replace line
time.Sleep(time.Second)
by line
time.Sleep(time.Hour) // long test
I don't get slack printed "Test finished" in hour. Instead, I see in my app logs:
responseUrl https://hooks.slack.com/commands/T3GBFUZ64/86817661808/XRmDO21jYaP1Wzu7GFpNw9eW
404
Expired url
Looks like slack's response URL has an expiration time. How to extend this expiration time?
Or in a case of expiration, is there another way to send a message to the user about having the test finished? I have name and id of a user launching /my-command
req.PostFormValue("user_name")
req.PostFormValue("user_id")
So I want to run integration tests by slack which are longer than 2 hours and get a response after finishing such tests in slack.

You can not increase the expire time for the URL that is a Slack internal setting.
Using the Slack Web API
You can send unattended messages to any user through the Slack Web API which you need a token for it.
For more information check: https://api.slack.com/web.
chat.postMessage
The Slack API has a postMessage command that allows the user to post messages to a channel, to the slackbot channel and to the user through IM channel (direct message). It seems you want to do the later which is quite simple.
Post to an IM channel
chat.postMessage#channels
Method Url: https://slack.com/api/chat.postMessage
Posting to an IM channel is a little more complex when settings the value of channel depending on the value of as_user.
If as_user is false:
Pass a username (#chris) as the value of channel to post to that user's #slackbot channel as the bot.
Pass the IM channel's ID (D023BB3L2) as the value of channel to post to that IM channel as the bot. The IM channel's ID can be retrieved through the im.list API method.
If as_user is true:
Pass the IM channel's ID (D023BB3L2) as the value of channel to post to that IM channel as the authenticated user. The IM channel's ID can be retrieved through the im.list API method.
To send a direct message to the user owning the token used in the request, provide the channel field with the a conversation/IM ID value found in a method like im.list.
To send a direct message to the user owning the token used in the request, provide the channel field with the a conversation/IM ID value found in a method like im.list.
im.list
This method will return a list of direct messages channels if you don't have a channel open with the user yet you can call the im.open.
im.open
This method is used to open a direct message channel with a specified user.
Documentation about im.open can be found here.
Example URL
https://slack.com/api/chat.postMessage?token=**TOKEN**&channel=**Direct Channel ID**&text=HelloWorld&as_user=true&pretty=1
Just replace **TOKEN** and **Direct Channel ID** with your values and it should send a direct message to the specified user.

Related

Logging metrics for go grpc server

This is kindof an extension of my previous question Reuse log client in interceptor for Golang grpc server method.
Basically I have a grpc server (written in Go) that exposes three APIs:
SubmitJob
CancelJob
GetJobStatus
I am using Datadog to log metrics, so in each API, I have code like:
func (s *myServer) submitJob(ctx context.Context, request *submitJobRequest) (*submitJobResponse, error) {
s.dd_client.LogRequestCount("SubmitJob")
start_time := time.Now()
defer s.dd_client.LogRequestDuration("SubmitJob", time.Since(start_time))
sth, err:= someFunc1()
if err != nil {
s.dd_client.LogErrorCount("SubmitJob")
return nil, err
}
resp, err:= someFunc2(sth)
if err != nil {
s.dd_client.LogErrorCount("SubmitJob")
return nil, err
}
return resp, nil
}
This approach works but have several problems:
The LogRequestCount and LogRequestDuration is duplicated among all APIs
I am calling LogErrorCount in every places where errors are returned, which seems ugly
I learned that interceptor might help with logging, so I wrote an interceptor like
func (s *myServer) UnaryInterceptor(ctx context.Context,
request interface{},
info *grpc.UnaryServerInfo,
handler grpc.UnaryHandler,
) (interface{}, error) {
// Get method name e.g. SubmitJob, CancelJob, GetJobStatus
tmp := strings.Split(info.FullMethod, "/")
method := tmp[len(tmp)-1]
s.dd_client.LogRequestCount(method)
start_time := time.Now()
resp, err := handler(ctx, request)
server.dd_client.LogRequestDuration(method)
if err != nil {
s.dd_client.LogErrorCount(method)
}
return response, err
}
And set it in main() function:
server := grpc.NewServer(grpc.UnaryInterceptor(my_server.UnaryInterceptor))
This works for me, but I noticed two problems:
Here the interceptor takes myServer as a receiver, is this a good practice? I am doing this coz I want to reuse the Datadog client (dd_client) created within myServer. Other options would be create the Datadog client singleton which used by both interceptor and myServer, or create a interceptor struct and create a separate Datadog client there.
The interceptor could only handle logging for generic metrics e.g. request count, duration. But there could be metrics specific for each API, which means I still need to have logging related code in each API implementation. Then the question is, should I still use interceptor? Coz now the logging related code are splitted into two places (API implementation and interceptor).

How to reuse HTTP request instance in Go

I'm building an API that scrapes some data off a webpage.
To do so, i need to send a GET request to a home page, scrape a 'RequestVerificationToken' from the HTML, then send another POST request to the same URL with a username, password, and the RequestVerificationToken.
I've been able to do this previously with Python:
session_requests = requests.session()
result = session_requests.get(LOGIN_URL)
parser = createBS4Parser(result.text)
return parser.find('input', attrs={'name': '__RequestVerificationToken'})["value"]
pageDOM = session_requests.post(
LOGIN_URL,
data=requestPayload, //RequestVerificationToken is in here
headers=requestHeaders
)
It seems like when i reuse the session_requests variable in Python, it's reusing the previous instance of the HTTP request.
However, when i try to do this in Go, I get an error due to an invalid token. I assume that this is because for the POST request, Go is using a new instance.
Is there any way I can get the same behavior from Go as I was with Python?
package main
import (
"fmt"
"log"
"github.com/gocolly/colly"
"github.com/gocolly/colly/proxy"
)
func main() {
//initiates the configuration
c := colly.NewCollector(colly.AllowURLRevisit())
//defining the proxy chain
revpro, err := proxy.RoundRobinProxySwitcher("socks5://127.0.0.1:9050", "socks5://127.0.0.1:9050")
if err != nil {
log.Fatal(err)
}
c.SetProxyFunc(revpro)
//parsing the required field from html we are extracting the csrf_token required for the login
c.OnHTML("form[role=form] input[type=hidden][name=CSRF_TOKEN]", func(e *colly.HTMLElement) {
csrftok := e.Attr("value")
fmt.Println(csrftok)
//posting the csrf value along with password
err := c.Post("https://www.something.com/login.jsp", map[string]string{"CSRF_TOKEN": csrftok, "username": "username", "password": "password"})
if err != nil {
log.Fatal(err)
}
return
})
//The website to visit
c.Visit("https://www.something.com/login.jsp")
//maintaining the connection using clone not initiating a callback request
d := c.Clone()
d.OnHTML("a[href]", func(e *colly.HTMLElement) {
link := e.Attr("href")
fmt.Printf("Link found: %q -> %s\n", e.Text, link)
})
d.Visit("https://skkskskskk.htm")
}

In Go, what is the proper way to use context with pgx within http handlers?

Update 1: it seems that using a context tied to the HTTP request may lead to the 'context canceled' error. However, using the context.Background() as the parent seems to work fine.
// This works, no 'context canceled' errors
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Second)
// However, this creates 'context canceled' errors under mild load
// ctx, cancel := context.WithTimeout(r.Context(), 100*time.Second)
defer cancel()
app.Insert(ctx, record)
(updated code sample below to produce a self-contained example for repro)
In go, I have an http handler like the following code. On the first HTTP request to this endpoint I get a context cancelled error. However, the data is actually inserted into the database. On subsequent requests to this endpoint, no such error is given and data is also successfully inserted into the database.
Question: Am I setting up and passing the context correctly between the http handler and pgx QueryRow method? (if not is there a better way?)
If you copy this code into main.go and run go run main.go, go to localhost:4444/create and hold ctrl-R to produce a mild load, you should see some context canceled errors produced.
package main
import (
"context"
"fmt"
"log"
"math/rand"
"net/http"
"time"
"github.com/jackc/pgx/v4/pgxpool"
)
type application struct {
DB *pgxpool.Pool
}
type Task struct {
ID string
Name string
Status string
}
//HTTP GET /create
func (app *application) create(w http.ResponseWriter, r *http.Request) {
fmt.Println(r.URL.Path, time.Now())
task := &Task{Name: fmt.Sprintf("Task #%d", rand.Int()%1000), Status: "pending"}
// -------- problem code here ----
// This line works and does not generate any 'context canceled' errors
//ctx, cancel := context.WithTimeout(context.Background(), 100*time.Second)
// However, this linegenerates 'context canceled' errors under mild load
ctx, cancel := context.WithTimeout(r.Context(), 100*time.Second)
// -------- end -------
defer cancel()
err := app.insertTask(ctx, task)
if err != nil {
fmt.Println("insert error:", err)
return
}
fmt.Fprintf(w, "%+v", task)
}
func (app *application) insertTask(ctx context.Context, t *Task) error {
stmt := `INSERT INTO task (name, status) VALUES ($1, $2) RETURNING ID`
row := app.DB.QueryRow(ctx, stmt, t.Name, t.Status)
err := row.Scan(&t.ID)
if err != nil {
return err
}
return nil
}
func main() {
rand.Seed(time.Now().UnixNano())
db, err := pgxpool.Connect(context.Background(), "postgres://test:test123#localhost:5432/test")
if err != nil {
log.Fatal(err)
}
log.Println("db conn pool created")
stmt := `CREATE TABLE IF NOT EXISTS public.task (
id uuid NOT NULL DEFAULT gen_random_uuid(),
name text NULL,
status text NULL,
PRIMARY KEY (id)
); `
_, err = db.Exec(context.Background(), stmt)
if err != nil {
log.Fatal(err)
}
log.Println("task table created")
defer db.Close()
app := &application{
DB: db,
}
mux := http.NewServeMux()
mux.HandleFunc("/create", app.create)
log.Println("http server up at localhost:4444")
err = http.ListenAndServe(":4444", mux)
if err != nil {
log.Fatal(err)
}
}
TLDR: Using r.Context() works fine in production, testing using Browser is a problem.
An HTTP request gets its own context that is cancelled when the request is finished. That is a feature, not a bug. Developers are expected to use it and gracefully shutdown execution when the request is interrupted by client or timeout. For example, a cancelled request can mean that client never see the response (transaction result) and developer can decide to roll back that transaction.
In production, request cancelation does not happen very often for normally design/build APIs. Typically, flow is controlled by the server and the server returns the result before the request is cancelled.
Multiple Client requests does not affect each other because they get independent go-routine and context. Again, we are talking about happy path for normally designed/build applications. Your sample app looks good and should work fine.
The problem is how we test the app. Instead of creating multiple independent requests, we use Browser and refresh a single browser session. I did not check what exactly is going on, but assume that the Browser terminates the existing request in order to run a new one when you click ctrl-R. The server sees that request termination and communicates it to your code as context cancelation.
Try to test your code using curl or some other script/utility that creates independent requests. I am sure you will not see cancelations in that case.

Parse/Validate JWT token from AzureAD in golang

I have Azure AD setup with OAuth2 and have it issuing a JWT for my web app. On subsequent requests, I want to validate the JWT that was issued. I'm using github.com/dgrijalva/jwt-go to do so however it always fails.
token, err := jwt.Parse(tokenString, func(token *jwt.Token) (interface{}, error) {
if _, ok := token.Method.(*jwt.SigningMethodRSA); !ok {
return nil, fmt.Errorf("unexpected signing method: %v", token.Header["alg"])
}
return []byte("bW8ZcMjBCnJZS-ibX5UQDNStvx4"), nil
})
if err != nil {
return nil, err
}
I'm picking at random the kid claim from the public keys listed by MS here https://login.microsoftonline.com/common/discovery/v2.0/keys so I'm lost as this isn't working.
Has anyone done this before or have any pointers?
The repository you are using is no longer maintained as pointed out by the README.
I've been using it's official replacement, https://github.com/golang-jwt/jwt, and I have never experienced any problem. You should try it.
Annoyingly it was a Azure AD config issue and out of the box it will generate a JWT token for MS Graph and the whole auth process succeeds but when you try to validate the token it fails for some reason. Once you have setup Azure AD correctly for your app with a correct scope it validates properly. I blogged about the specifics here - https://blog.jonathanchannon.com/2022-01-29-azuread-golang/
The asset located at https://login.microsoftonline.com/common/discovery/v2.0/keys is what's known as a JWKS, JSON Web Key Set. If all you want to do is authenticate tokens signed by this service, you can use something similar to the below code snippet. I've authored a package just for this use case: github.com/MicahParks/keyfunc
Under the hood, this package will read and parse the cryptographic keys found in the JWKS, then associate JWTs with those keys based on their key ID, kid. It also has some logic around automatically refreshing a remote JWKS resource.
package main
import (
"context"
"log"
"time"
"github.com/golang-jwt/jwt/v4"
"github.com/MicahParks/keyfunc"
)
func main() {
// Get the JWKS URL.
jwksURL := "https://login.microsoftonline.com/common/discovery/v2.0/keys"
// Create a context that, when cancelled, ends the JWKS background refresh goroutine.
ctx, cancel := context.WithCancel(context.Background())
// Create the keyfunc options. Use an error handler that logs. Refresh the JWKS when a JWT signed by an unknown KID
// is found or at the specified interval. Rate limit these refreshes. Timeout the initial JWKS refresh request after
// 10 seconds. This timeout is also used to create the initial context.Context for keyfunc.Get.
options := keyfunc.Options{
Ctx: ctx,
RefreshErrorHandler: func(err error) {
log.Printf("There was an error with the jwt.Keyfunc\nError: %s", err.Error())
},
RefreshInterval: time.Hour,
RefreshRateLimit: time.Minute * 5,
RefreshTimeout: time.Second * 10,
RefreshUnknownKID: true,
}
// Create the JWKS from the resource at the given URL.
jwks, err := keyfunc.Get(jwksURL, options)
if err != nil {
log.Fatalf("Failed to create JWKS from resource at the given URL.\nError: %s", err.Error())
}
// Get a JWT to parse.
//
// This wasn't signed by Azure AD.
jwtB64 := "eyJraWQiOiJlZThkNjI2ZCIsInR5cCI6IkpXVCIsImFsZyI6IlJTMjU2In0.eyJzdWIiOiJXZWlkb25nIiwiYXVkIjoiVGFzaHVhbiIsImlzcyI6Imp3a3Mtc2VydmljZS5hcHBzcG90LmNvbSIsImlhdCI6MTYzMTM2OTk1NSwianRpIjoiNDY2M2E5MTAtZWU2MC00NzcwLTgxNjktY2I3NDdiMDljZjU0In0.LwD65d5h6U_2Xco81EClMa_1WIW4xXZl8o4b7WzY_7OgPD2tNlByxvGDzP7bKYA9Gj--1mi4Q4li4CAnKJkaHRYB17baC0H5P9lKMPuA6AnChTzLafY6yf-YadA7DmakCtIl7FNcFQQL2DXmh6gS9J6TluFoCIXj83MqETbDWpL28o3XAD_05UP8VLQzH2XzyqWKi97mOuvz-GsDp9mhBYQUgN3csNXt2v2l-bUPWe19SftNej0cxddyGu06tXUtaS6K0oe0TTbaqc3hmfEiu5G0J8U6ztTUMwXkBvaknE640NPgMQJqBaey0E4u0txYgyvMvvxfwtcOrDRYqYPBnA"
// Parse the JWT.
var token *jwt.Token
if token, err = jwt.Parse(jwtB64, jwks.Keyfunc); err != nil {
log.Fatalf("Failed to parse the JWT.\nError: %s", err.Error())
}
// Check if the token is valid.
if !token.Valid {
log.Fatalf("The token is not valid.")
}
log.Println("The token is valid.")
// End the background refresh goroutine when it's no longer needed.
cancel()
// This will be ineffectual because the line above this canceled the parent context.Context.
// This method call is idempotent similar to context.CancelFunc.
jwks.EndBackground()
}

Gokit: Validate request/payload in transport layer

I am using go-kit to create an RPC endpoint. I am creating an endpoint like this
httptransport.NewServer(
endPoint.MakeGetBlogEndPoint(blogService),
transport.DecodeGetBlogRequest,
transport.EncodeGetBlogResponse
Below is my DecodeGetBlogRequest function
func DecodeGetBlogRequest(c context.Context, r *http.Request) (interface{}, error) {
vars := mux.Vars(r)
id, err := strconv.Atoi(vars["id"])
if err != nil {
return nil, err
}
req := endPoint.GetBlogRequest{
ID: id,
}
return req, nil
}
What I want to do is validate the HTTP request in this function and if found invalid, send a response with a valid error code from here only, without passing it to the service layer. i.e. If ID is not a valid number, return 400 Bad Request response from here.
But as I don't have a ResponseWriter reference in this function, I am not sure how to do it.
I am following this example from go-kit docs
https://gokit.io/examples/stringsvc.html
Is it a valid assumption that request/payload should be validated in the transport layer only and the service layer should only be called if the request/payload is valid? If yes, how to do so in this example?
You could use ServerErrorEncoder which returns Server options (can be found in github.com/go-kit/kit/transport/server.go).
Basically in your transport layer, apart from the Decode and Encode functions, you can define an YourErrorEncoderFunc() function which could look like the following. This will catch any error thrown in the transport layer.
YourErrorEncoderFunc(_ context.Context, err error, w http.ResponseWriter).
You will need to attach this function as an option in your endpoint registration like:
ABCOpts := []httptransport.ServerOption{
httptransport.ServerErrorEncoder(YourErrorEncoderFunc),
}
r.Methods("GET").Path("/api/v1/abc/def").Handler(httptransport.NewServer(
endpoints.GetDataEndpoint,
DecodeGetRequest,
EncodeGetResponse,
ABCOpts...,
))
This will stop at transport layer if your request validation is invalid and throw and error in the http response based of whatever format you've written in YourErrorEncoderFunc().
Not 100% sure if this applies to go-kit grpc as well:
You have an error return variable. Use that to indicate there was a problem. In the go grpc module there is a status package to return errors with status codes. If you return an error with a status code, the grpc layer will take the code from the error and send it back.
For example:
func DecodeGetBlogRequest(c context.Context, r *http.Request) (interface{}, error) {
vars := mux.Vars(r)
id, err := strconv.Atoi(vars["id"])
if err != nil {
return nil, status.Error(codes.InvalidArgument, err.Error())
}
req := endPoint.GetBlogRequest{
ID: id,
}
return req, nil
}
Note also that grpc uses different status codes. In Go they are located in the codes package.

Resources