I am writing a program by Go.
In this program, I access to a website and in this website, it will print a string. I want to get this string for next process.
For example:
I access by curl and the returned string will like that:
curl localhost:4000
abc_example
I need to get "abc_example" for next process in my program.
Now, this problem was solved.
Actually, my result will be a JSON like that:
{"name":"xyz_example"}
How can I parse this string and just get "xyz_example"
I am a newbie in Go. May you help me.
Thank you!
Here's an example of reading the response from an HTTP request.
I would recommend reading up on the documentation for the http package, and maybe a simple tutorial like this one.
package main
import (
"fmt"
"net/http"
"io/ioutil"
)
func main() {
//make a request
response, err := http.Get("https://mdtf.org")
if err != nil {
fmt.Println("error making request: ", err)
return
}
//make sure the response body gets closed
defer response.Body.Close()
//read the bytes
responseBytes, err := ioutil.ReadAll(response.Body)
if err != nil {
fmt.Println("error reading response bytes: ", err)
return
}
//turn the response bytes into a string
responseString := string(responseBytes)
//print it or something
fmt.Println(responseString)
}
Related
I'm building an API that scrapes some data off a webpage.
To do so, i need to send a GET request to a home page, scrape a 'RequestVerificationToken' from the HTML, then send another POST request to the same URL with a username, password, and the RequestVerificationToken.
I've been able to do this previously with Python:
session_requests = requests.session()
result = session_requests.get(LOGIN_URL)
parser = createBS4Parser(result.text)
return parser.find('input', attrs={'name': '__RequestVerificationToken'})["value"]
pageDOM = session_requests.post(
LOGIN_URL,
data=requestPayload, //RequestVerificationToken is in here
headers=requestHeaders
)
It seems like when i reuse the session_requests variable in Python, it's reusing the previous instance of the HTTP request.
However, when i try to do this in Go, I get an error due to an invalid token. I assume that this is because for the POST request, Go is using a new instance.
Is there any way I can get the same behavior from Go as I was with Python?
package main
import (
"fmt"
"log"
"github.com/gocolly/colly"
"github.com/gocolly/colly/proxy"
)
func main() {
//initiates the configuration
c := colly.NewCollector(colly.AllowURLRevisit())
//defining the proxy chain
revpro, err := proxy.RoundRobinProxySwitcher("socks5://127.0.0.1:9050", "socks5://127.0.0.1:9050")
if err != nil {
log.Fatal(err)
}
c.SetProxyFunc(revpro)
//parsing the required field from html we are extracting the csrf_token required for the login
c.OnHTML("form[role=form] input[type=hidden][name=CSRF_TOKEN]", func(e *colly.HTMLElement) {
csrftok := e.Attr("value")
fmt.Println(csrftok)
//posting the csrf value along with password
err := c.Post("https://www.something.com/login.jsp", map[string]string{"CSRF_TOKEN": csrftok, "username": "username", "password": "password"})
if err != nil {
log.Fatal(err)
}
return
})
//The website to visit
c.Visit("https://www.something.com/login.jsp")
//maintaining the connection using clone not initiating a callback request
d := c.Clone()
d.OnHTML("a[href]", func(e *colly.HTMLElement) {
link := e.Attr("href")
fmt.Printf("Link found: %q -> %s\n", e.Text, link)
})
d.Visit("https://skkskskskk.htm")
}
I'm writing some middleware and I need to be able to log the response body content even when the destination is using TLS encryption.
I have a handler chain within which I store the response body in an intermediate buffer, so that I can read it more than once. This is based on the excellent example provided by icza (Golang read request body).
In my handler func, I'm doing this....
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
// Print the response body to stdout
fmt.Printf("Dest HTTP response body: %s\n", body)
bRdr := bytes.NewReader(body)
n, err := io.Copy(w, bRdr) // Copy the entire response body into our outgoing response
What I'm finding is that I get readable output when connection to a destination not using TLS, but when connected to a destination using TLS, it seems the response body is still encrypted, though the Copy into the final response to the originator results in the originator receiving valid response body content.
Is this the expected behaviour for reads of the response body with an encrypted path?
Can I decrypt this data to be able make it readable? I've read the http, tls and crypto package documentation, but have not found any clues.
I'm not sure if I understand the problem but here is me calling an https google link and printing the output.
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"golang.org/x/net/http2"
)
func main() {
client := &http.Client{Transport: transport2()}
res, err := client.Get("https://www.google.com")
if err != nil {
log.Fatal(err)
}
body, err := ioutil.ReadAll(res.Body)
if err != nil {
log.Fatal(err)
}
res.Body.Close()
fmt.Printf("Code: %d\n", res.StatusCode)
fmt.Printf("Body: %s\n", body)
}
func transport2() *http2.Transport {
return &http2.Transport{
DisableCompression: true,
AllowHTTP: false,
}
}
Thanks all for your comments. Travis seems to have identified the issue I'm having. It appears the response body I'm reading is gzip encoded (the response contains "Content-Encoding: gzip"). In order to verify that this was the case, I had to explicitly remove the "Accept-Encoding: gzip" header that was in the originating request before forwarding and also configure the Transport to set "DisableCompression: true". Once I made both of those changes, I then see responses with no "Content-Encoding" header and the body I read is human readable.
Update
Since I'm not able to achieve this using the approach in this question, I created my own library to do the same thing (link). It doesn't rely on go-ethereum package but use the normal net/http package to do JSON RPC request.
I still love to know what I did wrong in my approach below.
Definitions:
owner = public variable in contract with address type
contract = smart-contract that has owner
This is the curl request to get the owner of a contract. I managed to get the owner. (JSON RPC docs)
curl localhost:8545 -X POST \
--header 'Content-type: application/json' \
--data '{"jsonrpc":"2.0", "method":"eth_call", "params":[{"to": "0x_MY_CONTRACT_ADDRESS", "data": "0x8da5cb5b"}, "latest"], "id":1}'
{"jsonrpc":"2.0","id":1,"result":"0x000000000000000000000000_OWNER"}
But when I try to replicate it in Golang (code below), I got json: cannot unmarshal string into Go value of type main.response error. (go-ethereum code that I use)
package main
import (
"fmt"
"log"
"os"
"github.com/ethereum/go-ethereum/rpc"
)
func main() {
client, err := rpc.DialHTTP(os.Getenv("RPC_SERVER"))
if err != nil {
log.Fatal(err)
}
defer client.Close()
type request struct {
To string `json:"to"`
Data string `json:"data"`
}
type response struct {
Result string
}
req := request{"0x_MY_CONTRACT_ADDRESS", "0x8da5cb5b"}
var resp response
if err := client.Call(&resp, "eth_call", req, "latest"); err != nil {
log.Fatal(err)
}
fmt.Printf("%v\n", resp)
}
What did I miss here?
Expected result:
Address in string format. E.g. 0x3ab17372b25154400738C04B04f755321bB5a94b
P/S — I'm aware of abigen and I know it's better and easier to do this using abigen. But I'm trying to solve this specific issue without using abigen method.
You can solve the problem best using the go-ethereum/ethclient:
package main
import (
"context"
"log"
"github.com/ethereum/go-ethereum"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/ethclient"
)
func main() {
client, _ := ethclient.Dial("https://mainnet.infura.io")
defer client.Close()
contractAddr := common.HexToAddress("0xCc13Fc627EFfd6E35D2D2706Ea3C4D7396c610ea")
callMsg := ethereum.CallMsg{
To: &contractAddr,
Data: common.FromHex("0x8da5cb5b"),
}
res, err := client.CallContract(context.Background(), callMsg, nil)
if err != nil {
log.Fatalf("Error calling contract: %v", err)
}
log.Printf("Owner: %s", common.BytesToAddress(res).Hex())
}
If you look at the client library code, you'll see that the JSON RPC response object is already disassembled and either an error is returned on failure, or the actual result parsed: https://github.com/ethereum/go-ethereum/blob/master/rpc/client.go#L277
The parser however already unwrapped the containing "result" field. Your type still wants to do an additional unwrap:
type response struct {
Result string
}
Drop the outer struct, simply pass a string pointer to the client.Call's first parameter.
Your response struct doesn't show the data that the json of the response has
try this
type response struct {
Jsonrpc string `json:"jsonrpc"`
ID int `json:"id"`
Result string `json:"result"`
}
json: cannot unmarshal string into Go value of type main.response error. I got similar type error when i was unmarshaling a response. It was because the response was actually json string, i mean it had Quotation " as first character. So to be sure you also encountered the same problem, please printf("%v",resp.Result) before unmarshaling in here https://github.com/ethereum/go-ethereum/blob/1ff152f3a43e4adf030ac61eb5d8da345554fc5a/rpc/client.go#L278.
I have some code (see below) written in Go which is supposed to "fan-out" HTTP requests, and collate/aggregate the details back.
I'm new to golang and so expect me to be a nOOb and my knowledge to be limited
The output of the program is currently something like:
{
"Status":"success",
"Components":[
{"Id":"foo","Status":200,"Body":"..."},
{"Id":"bar","Status":200,"Body":"..."},
{"Id":"baz","Status":404,"Body":"..."},
...
]
}
There is a local server running that is purposely slow (sleeps for 5 seconds and then returns a response). But I have other sites listed (see code below) that sometime trigger an error as well (if they error, then that's fine).
The problem I have at the moment is how best to handle these errors, and specifically the "timeout" related errors; in that I'm not sure how to recognise if a failure is a timeout or some other error?
At the moment I get a blanket error back all the time:
Get http://localhost:8080/pugs: read tcp 127.0.0.1:8080: use of closed network connection
Where http://localhost:8080/pugs will generally be the url that failed (hopefully by timeout!). But as you can see from the code (below), I'm not sure how to determine the error code is related to a timeout nor how to access the status code of the response (I'm currently just blanket setting it to 404 but obviously that's not right - if the server was to error I'd expect something like a 500 status code and obviously I'd like to reflect that in the aggregated response I send back).
The full code can be seen below. Any help appreciated.
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"sync"
"time"
)
type Component struct {
Id string `json:"id"`
Url string `json:"url"`
}
type ComponentsList struct {
Components []Component `json:"components"`
}
type ComponentResponse struct {
Id string
Status int
Body string
}
type Result struct {
Status string
Components []ComponentResponse
}
var overallStatus string = "success"
func main() {
var cr []ComponentResponse
var c ComponentsList
b := []byte(`{"components":[{"id":"local","url":"http://localhost:8080/pugs"},{"id":"google","url":"http://google.com/"},{"id":"integralist","url":"http://integralist.co.uk/"},{"id":"sloooow","url":"http://stevesouders.com/cuzillion/?c0=hj1hfff30_5_f&t=1439194716962"}]}`)
json.Unmarshal(b, &c)
var wg sync.WaitGroup
timeout := time.Duration(1 * time.Second)
client := http.Client{
Timeout: timeout,
}
for i, v := range c.Components {
wg.Add(1)
go func(i int, v Component) {
defer wg.Done()
resp, err := client.Get(v.Url)
if err != nil {
fmt.Printf("Problem getting the response: %s\n", err)
cr = append(cr, ComponentResponse{
v.Id,
404,
err.Error(),
})
} else {
defer resp.Body.Close()
contents, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Printf("Problem reading the body: %s\n", err)
}
cr = append(cr, ComponentResponse{
v.Id,
resp.StatusCode,
string(contents),
})
}
}(i, v)
}
wg.Wait()
j, err := json.Marshal(Result{overallStatus, cr})
if err != nil {
fmt.Printf("Problem converting to JSON: %s\n", err)
return
}
fmt.Println(string(j))
}
If you want to fan out then aggregate results and you want specific timeout behavior the net/http package isn't giving you, then you may want to use goroutines and channels.
I just watched this video today and it will walk you through exactly those scenarios using the concurrency features of Go. Plus, the speaker Rob Pike is quite the authority -- he explains it much better than I could.
https://www.youtube.com/watch?v=f6kdp27TYZs
I am adding this for completes, as the correct answer was provided by Dave C in the comments of the accepted answer.
We can try to cast the error to a net.Error and check if it is a timeout.
resp, err := client.Get(url)
if err != nil {
// if there is an error check if its a timeout error
if e, ok := err.(net.Error); ok && e.Timeout() {
// handle timeout
return
}
// otherwise handle other types of error
}
The Go 1.5 release solved this issue by being more specific about the type of error it has handled.
So if you see this example https://github.com/Integralist/Go-Requester/blob/master/requester.go#L38 you'll see that I'm able to apply a regex pattern to the error message to decipher if the error was indeed a timeout or not
status := checkError(err.Error())
func checkError(msg string) int {
timeout, _ := regexp.MatchString("Timeout", msg)
if timeout {
return 408
}
return 500
}
Im using web.go (http://webgo.io/) for writing a simple web app that accepts json in a POST request and after parsing it returns the result. Im having trouble reading the json from ctx.Params object.
Below is the code i have so far
package main
import (
"github.com/hoisie/web";
"encoding/json"
)
func parse(ctx *web.Context, val string) string {
for k,v := range ctx.Params {
println(k, v)
}
//Testing json parsing
mapB := map[string]int{"apple": 5, "lettuce": 7}
mapD, _ := json.Marshal(mapB)
return string(mapD)
}
func main() {
web.Post("/(.*)", parse)
web.Run("0.0.0.0:9999")
}
Though the post request gets registered i dont see anything printed on the command line for the json i posted. How can i fix this ?
Thank You
The reason you're not getting any JSON data from the body of the POST request is because hoisie/web reads form data into .Params, as seen here:
req.ParseForm()
if len(req.Form) > 0 {
for k, v := range req.Form {
ctx.Params[k] = v[0]
}
}
In order to fix this, you'll need to add something that can parse the raw body of the response. You should just be able to use ctx.Body to access the raw body, since it implements *http.Request and doesn't redefine Body in the Context struct.
For example, this should work:
json := make(map[string]interface{})
body, err := ioutil.ReadAll(ctx.Body)
if err != nil {
// Handle Body read error
return
}
err = json.Unmarshal(body, &json)
if err != nil {
// Handle JSON parsing error
return
}
// Use `json`