Prometheus HTTP API - POST Request - http-post

We are working with Prometheus HTTP API, and we're sending GET requests to the following endpoint:
/api/v1/query
At the beginning things worked as expected, but recently when our queries got bigger, request-URI became too large.
Docs says that it is possible to send POST request to the same endpoint, and pass the query parameter directly in the request body, instead of passing a query param as part of the URL..
This should solve our problem, but I couldn't find any example or guidelines explaining how to do it.
URL query length is limited, so we are looking for a way to send the query as part of the body :
End-point : http://server:8082/api/v1/query
Body :
{
"query": "count(count(node_cpu_seconds_total{instance=~\"iServer.*\",job=\"events_prometheus\"}) by (cpu))"
}
Response Error :
{
"status": "error",
"errorType": "bad_data",
"error": "invalid parameter 'query': parse error at char 1: no expression found in input"
}
Just to mention that sending the same query, as a query param, will work and give us the expected results.

You can URL-encode these parameters directly in the request body by using the POST method and Content-Type: application/x-www-form-urlencoded header. This is useful when specifying a large query that may breach server-side URL character limits.
If you use some programming language, you should build these requests like in the example below.
Example code in golang:
func main() {
formData := url.Values{
"query": {"sum(temperature) by(status)"},
"time": {"1670859244"},
}
client := &http.Client{}
req, err := http.NewRequest("POST", "http://localhost:8428/api/v1/query", strings.NewReader(formData.Encode()))
if err != nil {
log.Fatalln(err)
}
req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
resp, err := client.Do(req)
if err != nil {
log.Fatalln(err)
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
log.Fatalln(err)
}
log.Println(string(body))
}
In the response, you should see something like this.
{"status":"success","data":{"resultType":"vector","result":[]}}
But in the result, there should be some vector with data.
An example of the response can check here
https://docs.victoriametrics.com/keyConcepts.html#range-query

Related

HTTP call error to shopify when sending array in json request

This is an http post response to create a new shopify blog.
blog := Blog{
Title: fileNameWithoutExtSliceNotation(doc.Name()),
Template_suffix: "hit_parader_issue_template",
Metafields: []BlogMeta{
{
Key: "filter_tag_list",
Value: "Popular",
Type: "single_line_text_field",
Namespace: "global",
},
},
}
blogJSON, err := json.Marshal(blog)
blogReq, err := http.NewRequest("POST", newBlogQuery, bytes.NewBuffer(blogJSON))
blogReq.Header.Add("X-Shopify-Access-Token", TOKEN)
blogReq.Header.Add("Content-Type", "application/json")
resp1, err := client.Do(blogReq)
my structs are like below
type Blog struct {
Title string `json:"title"`
Template_suffix string `json:"template_suffix"`
Metafields []BlogMeta
}
type BlogMeta struct {
Key string
Value string
Type string
Namespace string
}
im getting an error saying
{"errors":{"blog":"Required parameter missing or invalid"}}
These are my blow metafield definitions.
I have all the necessary fields listed in the following api call. Please help.
Im using below API
https://shopify.dev/api/admin-rest/2022-10/resources/blog#post-blogs
NOTE: the API documentation says that I need to present a array for the metafields. But I think my code is correct. IS there something im missing? Any help would be appreciated.
Looking at the API docs it seems that the blog JSON object should be nested in a root object, i.e. it should be {"blog": {"title": ...,}} rather than {"title": ...,}. If that's the case then that would also explain the error message.
To fix that you can do something like this:
blog := &Blog{
// ...
}
data, err := json.Marshal(map[string]any{"blog": blog})
if err != nil {
return err
}
r, err := http.NewRequest("POST", url, bytes.NewReader(data))
if err != nil {
return err
}
r.Header.Add("X-Shopify-Access-Token", TOKEN)
r.Header.Add("Content-Type", "application/json")
resp, err := client.Do(r)
if err != nil {
return err
}
// ...

What whould be the best way to forward a request by adding headers?

I just started to use Golang and I want to remake my already working NodeJS/TypeScript app in Go.
One endpoint of my API simply adds server-side generated authorization headers and sends a request to a remote API. Basically filling those headers for me by calling my API instead of the remote API.
This is what I am currently writing
func Endpoint(ctx *fiber.Ctx) error {
url := "https://api.twitch.tv" + ctx.OriginalURL()
req, _ := http.NewRequest(http.MethodGet, url, nil)
req.Header.Set("Authorization", "Bearer ---------")
req.Header.Set("Client-Id", "---------")
client := &http.Client{}
res, err := client.Do(req)
// temporary error handling
if err != nil {
log.Fatalln(err)
}
body, err := ioutil.ReadAll(res.Body)
// temporary error handling
if err != nil {
log.Fatalln(err)
}
var forwardedBody interface{}
json.Unmarshal(body, &forwardedBody)
return ctx.Status(fiber.StatusOK).JSON(forwardedBody)
}
I'd like to know if I am on the right steps, because making a request, parsing the JSON response with ioutil then unmarshall it to send it back seems kind of overboard for the simplicity of what I am trying to achieve ?
Edit: Thank you for the help, this is what I will be going for
func Endpoint(ctx *fiber.Ctx) error {
url := "https://api.twitch.tv" + ctx.OriginalURL()
req, _ := http.NewRequest(http.MethodGet, url, nil)
req.Header.Set("Authorization", "Bearer ---------")
req.Header.Set("Client-ID", "---------")
client := &http.Client{}
res, err := client.Do(req)
if err != nil {
return ctx.SendStatus(fiber.StatusBadRequest)
}
ctx.Set("Content-Type", "application/json; charset=utf-8")
return ctx.Status(res.StatusCode).SendStream(res.Body)
}
You can use httputil.ReverseProxy. Which takes a base URL and forwards requests to the base URL, concatenating the path.
ReverseProxy is an HTTP Handler that takes an incoming request and sends it to another server, proxying the response back to the client.
http.Handle("/", &httputil.ReverseProxy{
Director: func(r *http.Request) {
r.URL.Scheme = "https"
r.URL.Host = "go.dev"
r.Host = r.URL.Host
r.Header.Set("X-Foo", "Bar")
},
})
If you are not serving this from the root path / you can use StripPrefix.
http.HandleFunc("/foo/", http.StripPrefix("/foo/", proxy)
There is also a helper function NewSingleHostReverseProxy, which possibly removes the need to configure the proxy struct yourself. But I think it will be better to set the Host header along with your custom header.
You don't need to attempt to parse the data as JSON. This will be problematic if any of your endpoints don't return JSON, anyway, so just inject the body directly into the response:
body, err := ioutil.ReadAll(res.Body)
// temporary error handling
if err != nil {
log.Fatalln(err)
}
// Inject the body from the inner response into the actual response so it can be returned
ctx.Response().SetBody(body)
return cx.Status(fiber.StatusOK)

Structuring HTTP POST request to include req query but q values in body of request

I'm currently attempting to make a POST request using the HTTP package in Go. In the body of the request, it needs a 'code' pulled from the query of the API call to complete the request.
However, I first need to declare the req to add the query URL values in to do this. So I'm stuck with declaring q as url.Values{}, passing that in to the body of my post, and then having to add the values after the initial HTTP declaration.
But because I'm passing q in to the request before adding these values, the request URL doesn't include them when I'm sending the request. So I'm essentially just sending a blank query (I think).
So how can I get around this and pass in the query details to my http request but access the query value?
Hopefully that makes sense - it's confusing!
Here's my code:
func Fetch(w http.ResponseWriter, r *http.Request) {
client := &http.Client{}
q := url.Values{}
req, err := http.NewRequest("POST", "https://auth.truelayer-sandbox.com/connect/token", strings.NewReader(q.Encode()))
if err != nil {
log.Print(err)
fmt.Println("Error was not equal to nil at first stage.")
os.Exit(1)
}
q.Add("grant_type", "authorization_code")
q.Add("id", os.Getenv("ID"))
q.Add("secret", os.Getenv("SECRET"))
q.Add("redirect_uri", "https://callback.com")
q.Add("query-param", req.URL.Query().Get("query-param"))
req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
resp, err := client.Do(req)
if err != nil {
fmt.Println("Error sending request to server")
os.Exit(1)
}
respBody, _ := ioutil.ReadAll(resp.Body)
w.WriteHeader(resp.StatusCode)
w.Write(respBody)
}

Youtube Thumbnail upload Failing with API

I am trying to upload a thumbnail for youtube video in this way:
import (
"bufio"
"fmt"
"io/ioutil"
"log"
"net/http"
"os"
)
func main() {
url := "https://www.googleapis.com/youtube/v3/thumbnails/set?videoId=kU7okI-_vvU&key=[API_KEY]Type=media"
imageRef, err := os.Open("test.png")
if err != nil {
log.Fatal("os.Open", err)
}
rd := bufio.NewReader(imageRef)
req, err := http.NewRequest("POST", url, rd)
if err != nil {
log.Fatal("http.NewRequest", err)
}
log.Println(req.Body)
req.Header.Add("authorization", "Bearer [ACCESS_TOKEN]")
req.Header.Add("content-type", "image/png")
res, err := http.DefaultClient.Do(req)
if err != nil {
log.Fatal("http.DefaultClient", err)
}
defer res.Body.Close()
body, err := ioutil.ReadAll(res.Body)
if err != nil {
log.Fatal("ioutil.ReadAll", err)
}
fmt.Println(res)
fmt.Println(string(body))
}
I am getting this response:
"error": {
"code": 400,
"message": "The request does not include the image content.",
"errors": [
{
"message": "The request does not include the image content.",
"domain": "youtube.thumbnail",
"reason": "mediaBodyRequired",
"location": "body",
"locationType": "other"
}
]
}
}
I am including the image body in the POST request. Yet the respose says "The request does not include the image content.". Can anyone please help with this.
API requires that max file size be 2MB and I have ensured this.
Thank you.
PS: Although not shown in the code, result is tested with error handling.
Using bare HTTP methods for invoking YouTube Data API can be quite tricky at times.
You should have been employing the Google API Client Library for Go instead. See also the official Go Quickstart.
In case you stick with your current code, you need to have the following:
the invoking URL should contain the parameter uploadType=media, and
the Content-Type header should be passed on to the HTTP call with a value of kind image/png (as already suggested above).
the Content-Length header should also be set by your code (since that's not done within the NewRequestWithContext function; see also this function's doc, as #Peter indicated below).
Also you have to change your URL, since, according to the official doc, the API endpoint's URL is:
https://www.googleapis.com/upload/youtube/v3/thumbnails/set.
Do notice that your URL is missing the /upload/ path component.

Unexpected error response from GitHub API 422 when attempting to create issue

When posting an issue to GitHub API V3 I am getting an unexpected response. Namely 422 Unprocessable Entity. However the detail of the error is for the Search endpoint rather that the POST create endpoint.
{"message":"Validation Failed","errors":[{"resource":"Search","field":"q","code":"missing"}],"documentation_url":"https://developer.github.com/v3/search"}
My instinct is that I have messed up the json but it is pretty simple and I can't see the issue.
I have tried various solutions posted here and elsewhere but not found what I am doing wrong. This is a coding exercise rather than anything intended for production but driving me mildly insane.
Tested in Debug what the Request body is just before being posted.
{"title":"Hello World","body":"dfsdfsdf\n"}
Tried removing the body as it is optional, same issue.
Tested in Debug that request is of type POST
Tested in Debug that authorization header is correct.
Removed authorization key and received 401 as expected.
The posting function:
func CreateIssue (issue *NewIssue) (*IssueDetailsResult, error){
issueJson, err := json.Marshal(issue)
if err != nil {
log.Fatal(err)
os.Exit(1)
}
req, err := http.NewRequest("POST", github.IssuesURL, bytes.NewBuffer(issueJson))
req.Header.Set("Authorization", "token "+os.Getenv("UPGITUSER"))
req.Header.Set( "Content-Type", "application/json")
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
os.Exit(1)
}
if resp.StatusCode != http.StatusCreated {
bodyBytes, _ := ioutil.ReadAll(resp.Body)
body := string(bodyBytes)
resp.Body.Close()
return nil, fmt.Errorf("create issue failed:%s", resp.Status + "\ntext: " + body)
}
var result IssueDetailsResult
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
resp.Body.Close()
return nil, err
}
resp.Body.Close()
return &result, nil
}
Would expect 201 from GitHubAPI.
The response is a strong indicator that the request is being sent to the wrong endpoint.
You can use net/http/httputil's DumpRequestOut to inspect the requests you are about to send and to ensure that they are what you expect them to be.

Resources