Testing echo handlers with path variables - go

I’m testing one of my http handlers.
A curl like this works: curl localhost:8080/v1/parts/2bb834c9-8e17-4e2c-80b9-a20b80732899.
But the corresponding test fails as the handler is not able to extract the id from the URL.
Please have a look at this test file. I don’t understand why my curl command works but the newRequest on line 17 fails.
1 package part
2
3 import (
4 "net/http"
5 "net/http/httptest"
6 "testing"
7
8 "github.com/labstack/echo/v4"
9 "github.com/stretchr/testify/assert"
10 )
11
12 var handler *Handler
13
14 func TestGetPart(t *testing.T) {
15 e := echo.New()
16
17 req := httptest.NewRequest("GET", "/v1/parts/2bb834c9-8e17-4e2c-80b9-a20b80732899", nil)
18 rec := httptest.NewRecorder()
19 c := e.NewContext(req, rec)
20
21 if assert.NoError(t, handler.handleGetPart(c)) {
22 assert.Equal(t, http.StatusOK, rec.Code)
23 }
24 }
go test -v ./internal/handler/part
=== RUN TestGetPart
2020/09/08 08:51:10 UUID PASSED:
time="2020-09-08T08:51:10+02:00" level=error msg="Could not decode string to uuid"
TestGetPart: handler_test.go:21:
Error Trace: handler_test.go:21
Error: Received unexpected error:
invalid UUID length: 0
Test: TestGetPart
--- FAIL: TestGetPart (0.00s)
The handler
ID := ctx.Param("id")
log.Println("UUID PASSED: ", ID)
uuid, err := uuid.Parse(ID)
if err != nil {
logrus.Error("Could not decode string to uuid")
return err
}
// Fetch data.json from S3 bucket
filename := helper.PartFileName(uuid)
content, err := a.GetObject(filename)
if err != nil {
logrus.Error(err)
return ctx.JSON(http.StatusNotFound, "file not found")
}
return ctx.String(http.StatusOK, content)
}
Any my route:
func Register(router *echo.Echo, handler *Handler) {
router.GET("/v1/parts/:id", handler.handleGetPart)
}

You should set a path in this way:
e := echo.New()
req := httptest.NewRequest("GET", "/", nil)
rec := httptest.NewRecorder()
c := e.NewContext(req, rec)
c.SetPath("/:id")
c.SetParamNames("id")
c.SetParamValues("2bb834c9-8e17-4e2c-80b9-a20b80732899")
// rest of the test goes below

Related

Golang database/sql hangs with SetMaxOpenConns

I have pasted a minimally reproducible test code. In short, with SetMaxOpenConns set to 10, the program hangs forever after 10. I found this relevant thread from way back when, but it seems resolved and tested: https://github.com/golang/go/issues/6593
Note that by commenting out the SetMaxOpenConns the code runs normally.
What am I doing wrong? or should I open a new issue?
1 package main
2
3 import (
4 "database/sql"
5 "log"
6 "time"
7 _ "github.com/lib/pq"
8 )
9
10 func main(){
11 // Establish db connection
12 db, err := sql.Open("postgres", "host=0.0.0.0 port=5432 user=postgres password=password dbname=test sslmode=disable")
13 if err != nil {
14 log.Fatal(err)
15 }
16
17 db.SetMaxOpenConns(10) // commenting this line will resolve the problem
18 db.SetMaxIdleConns(10)
19 db.SetConnMaxLifetime(10 * time.Second)
20
21 // Query more than max open; note that hangs forever
22 for i:=0; i<12; i++ {
23 rows, err := Query(db)
24 if err != nil {
25 log.Fatal(err)
26 }
27 log.Println(i)
28 log.Println(rows)
29 }
30 }
31
32 func Query(db *sql.DB) (*sql.Rows, error){
33 stmt, err := db.Prepare("SELECT * FROM test;")
34 if err != nil {
35 log.Fatal(err)
36 }
37
38 defer stmt.Close()
39
40 rows, err := stmt.Query()
41 if err != nil {
42 log.Fatal(err)
43 }
44
45 return rows, nil
46 }
You need to either fully iterate through the result set with rows.Next and/or call rows.Close(); as per the docs:
Close closes the Rows, preventing further enumeration. If Next is called and returns false and there are no further result sets, the Rows are closed automatically and it will suffice to check the result of Err. Close is idempotent and does not affect the result of Err.
Something like:
for i:=0; i<12; i++ {
rows, err := Query(db)
if err != nil {
log.Fatal(err)
}
log.Println(i)
log.Println(rows)
if err = rows.Close(); err != nil {
panic(err)
}
}
For this to be useful you need to iterate through the rows (see the example in the docs).
The connection to the database will remain in use until the result set is closed (at which point it is returned to the pool). Because you are doing this in a loop you will end up with 10 active result sets and when you call Query() again the sql package will wait for a connection to become available (which will never happen).
Note that because your query has no parameters (and you are only using the stmt once) calling Prepare has no benefit; the following is simpler and will have the same result:
func Query(db *sql.DB) (*sql.Rows, error) {
return db.Query("SELECT * FROM test;")
}

Read Exif Metadata with Go

Does anyone have an example on using "github.com/gohugoio/hugo/resources/images/exif" to extract metadata from a local image using Go?
I looked through the docs and since I'm new to Go I'm not 100% sure if I'm doing things write. I do read the image, but I'm not sure what the next step would be.
fname := "image.jpg"
f, err := os.Open(fname)
if err != nil {
log.Fatal("Error: ", err)
}
(Edit 1)
Actually I think I found a solution:
d, err := exif.NewDecoder(exif.IncludeFields("File Type"))
x, err := d.Decode(f)
if err != nil {
log.Fatal("Error: ", err)
}
fmt.Println(x)
however, the question is how do I know what fields are available? File Type for example returns <nil>
Looks like Hugo uses github.com/rwcarlsen/goexif.
The documentation of the package on go.dev shows Exif.Walk can walk the name and tag for every non-nil EXIF field.
Eg, a small program:
package main
import (
"fmt"
"log"
"os"
"github.com/rwcarlsen/goexif/exif"
"github.com/rwcarlsen/goexif/tiff"
)
type Printer struct{}
func (p Printer) Walk(name exif.FieldName, tag *tiff.Tag) error {
fmt.Printf("%40s: %s\n", name, tag)
return nil
}
func main() {
if len(os.Args) < 2 {
log.Fatal("please give filename as argument")
}
fname := os.Args[1]
f, err := os.Open(fname)
if err != nil {
log.Fatal(err)
}
x, err := exif.Decode(f)
if err != nil {
log.Fatal(err)
}
var p Printer
x.Walk(p)
}
Example:
$ go run main.go IMG_123.JPG
ResolutionUnit: 2
YCbCrPositioning: 2
Make: "Canon"
Model: "Canon IXUS 255 HS"
ThumbJPEGInterchangeFormat: 5620
PixelYDimension: 3000
FocalPlaneResolutionUnit: 2
GPSVersionID: [2,3,0,0]
ExifVersion: "0230"
WhiteBalance: 1
DateTime: "2016:10:04 17:27:56"
CompressedBitsPerPixel: "5/1"
... etc ...
Orientation: 1
MeteringMode: 5
FocalLength: "4300/1000"
PixelXDimension: 4000
InteroperabilityIFDPointer: 4982
FocalPlaneXResolution: "4000000/244"
XResolution: "180/1"
ComponentsConfiguration: ""
ShutterSpeedValue: "96/32"
ApertureValue: "101/32"
ExposureBiasValue: "-1/3"
FocalPlaneYResolution: "3000000/183"
SceneCaptureType: 0

Get JSON paths to value from JSON

I'm trying to write a func to get all paths to values from a yaml file and I don't know know how that possible, here is my code:
func getpath(fileyaml) string {
if _, err := os.Stat(fileyaml); err == nil {
bfile, err := ioutil.ReadFile(fileyaml)
bjson, err := yaml.YAMLToJSON(bfile)
if err != nil {
fmt.Printf("YAMLToJSON err: %v\n", err)
}
json := string(bjson)
println json
paths := ? // don't know
return path
here my yaml file :
sentinel:
number: 3
server:
number: 7
config:
fere_size: 5
lcmea:
eza_ze: all
my function will convert it to a json: {"config":{"fere_size":5},"lcmea":{"eza_hooks":"all"},"sentinel":{"number":3},"server":{"number":7}}
the output that i want :
sentinel.number=3, server.number=3,config.fere_size=5,lcmea.eza_ze=all
how to parse this json in order to get this desired output?
I'm using "github.com/tidwall/gjson" to read the yaml and convert it to json
This is an example of how you could do it: https://play.golang.org/p/7yLq_PDLdXF
It is pretty naive and definitely could be improved, but it may give you an idea about how to parse the yaml file and then print the output in your desired format:
package main
import (
"fmt"
"log"
"strings"
"github.com/go-yaml/yaml"
)
var data = `
sentinel:
number: 3
server:
number: 7
config:
fere_size: 5
lcmea:
eza_ze: all
`
func main() {
m := make(map[string]map[string]interface{})
err := yaml.Unmarshal([]byte(data), &m)
if err != nil {
log.Fatalf("error: %v", err)
}
out := []string{}
for k, v := range m {
for j, i := range v {
out = append(out, fmt.Sprintf("%s.%v=%v", k, j, i))
}
}
fmt.Println(strings.Join(out, ", "))
}
It will return:
$ go run main.go
sentinel.number=3, server.number=7, config.fere_size=5, lcmea.eza_ze=all
It is using https://github.com/go-yaml/yaml, check more examples on the README.md

gocql can not unmarshal blob into *[20]uint8

I have a question related to using blob type as partition key.
I use it, as I need to save hash value.
(hash value returns binary data. usually as hexadecimal.)
I tried a select query with gocql, however it failed with following error.
Is there any way to get a successful result for this kind of query?
Your advice highly appreciated!!
-- result
hash_value: [208 61 222 22 16 214 223 135 169 6 25 65 44 237 166 229 50 5 40 221]
/ hash_value: ?=??߇?A,???2(?
/ hash_value: 0xd03dde1610d6df87a90619412ceda6e5320528dd
string
2018/03/22 10:03:17 can not unmarshal blob into *[20]uint8
-- select.go
package main
import (
"fmt"
"log"
"crypto/sha1"
"reflect"
"github.com/gocql/gocql"
)
func main() {
cluster := gocql.NewCluster("10.0.0.1")
cluster.Keyspace = "ks"
cluster.Consistency = gocql.Quorum
cluster.ProtoVersion = 4
cluster.Authenticator = gocql.PasswordAuthenticator{
Username: "cassandra",
Password: "cassandra",
}
session, _ := cluster.CreateSession()
defer session.Close()
text := "text before hashed"
data := []byte(text)
hash_value := sha1.Sum(data)
hexa_string := fmt.Sprintf("0x%x", hash_value)
fmt.Println("hash_value: ", hash_value)
fmt.Println(" / string: ", string(hash_value[:]))
fmt.Println(" / column1: ", hexa_string)
fmt.Println(reflect.TypeOf(hexa_string))
// *** select ***
var column1 int
returned_hash := sha1.Sum(data)
//if err := session.Query(`SELECT hash_value, column1 FROM sample WHERE hash_value= ? LIMIT 1`,
// hexa_string).Consistency(gocql.One).Scan(&returned_hash, &column1); err != nil {
if err := session.Query(`SELECT hash_value, column1 FROM sample WHERE hash_value=0xd03dde1610d6df87a90619412ceda6e5320528dd`).Consistency(gocql.One).Scan(&returned_hash, &column1); err != nil {
//fmt.Println(err)
log.Fatal(err)
}
fmt.Println("comment: ", returned_hash, user_id)
}
-- table definition --
CREATE TABLE IF NOT EXISTS ks.samle (
hash_value blob,
column1 int,
...
PRIMARY KEY((hash_value), column1)
) WITH CLUSTERING ORDER BY (column1 DESC);
I fixed the problem by changing the type of variable: returned_hash.
returned_hash (var to store returned result) should be []byte.
I understands as follows.
marshal: convert the data given in the code to a type cassandra can handle.
unmarshal: convert the data cassandra returned back to a type golang-code can handle.
the original error means latter pattern doesn't go well. so the type of returned_hash must be wrong.
Correct me if I'm wrong. thanks.
package main
import (
"fmt"
"log"
"crypto/sha1"
"reflect"
"github.com/gocql/gocql"
)
func main() {
cluster := gocql.NewCluster("127.0.0.1")
cluster.Keyspace = "browser"
cluster.Consistency = gocql.Quorum
//cluster.ProtoVersion = 4
//cluster.Authenticator = gocql.PasswordAuthenticator{
// Username: "cassandra",
// Password: "cassandra",
//}
session, _ := cluster.CreateSession()
defer session.Close()
text := "text before hashed"
data := []byte(text)
hash_value := sha1.Sum(data)
hexa_string := fmt.Sprintf("0x%x", hash_value)
fmt.Println("hash_value: ", hash_value)
fmt.Println(" / string(hash_value): ", string(hash_value[:]))
fmt.Println(" / hexa(hash_value): ", hexa_string)
fmt.Println(reflect.TypeOf(hexa_string))
// *** select ***
var column1 int
//returned_hash := sha1.Sum(data)
//var returned_hash *[20]uint8
var returned_hash []byte
if err := session.Query(`SELECT hash_value, column1 FROM sample WHERE hash_value=? LIMIT 1`,
hash_value[:]).Consistency(gocql.One).Scan(&returned_hash, &column1); err != nil {
//if err := session.Query(`SELECT hash_value, column1 FROM sample WHERE hash_value=0xd03dde1610d6df87a90619412ceda6e5320528dd`).Consistency(gocql.One).Scan(&returned_hash, &column1); err != nil {
log.Fatal(err)
}
fmt.Printf("Returned: %#x %d \n", returned_hash, column1)
}

gopacket parsing Dot11 layer

I'm trying to use gopacket to parse the packets of a .pcap file and pretty much to get all the information in it, until now I get either truncated information or an error IF I try to use a filter.
package main
import (
"fmt"
"github.com/google/gopacket"
"github.com/google/gopacket/pcap"
//"github.com/google/gopacket/layers"
"log"
)
var (
pcapFile string = "myFile.pcap"
handle *pcap.Handle
err error
)
func main() {
// Open file instead of device
handle, err = pcap.OpenOffline(pcapFile)
if err != nil { log.Fatal(err) }
defer handle.Close()
// Loop through packets in file
packetSource := gopacket.NewPacketSource(handle, handle.LinkType())
for packet := range packetSource.Packets() {
fmt.Println
}
}
This returns:
PACKET: 122 bytes, wire length 122 cap length 122 # 2017-06-11 02:57:03.133873 +0100 WEST
- Layer 1 (36 bytes) = RadioTap {Contents=[..36..] Payload=[..86..] Version=0 Length=36 Present=2684370991 TSFT=661956589449 Flags=FCS Rate=1 Mb/s ChannelFrequency=2412 MHz ChannelFlags=CCK,Ghz2 FHSS=0 DBMAntennaSignal=-91 DBMAntennaNoise=0 LockQuality=0 TxAttenuation=0 DBTxAttenuation=0 DBMTxPower=0 Antenna=0 DBAntennaSignal=0 DBAntennaNoise=0 RxFlags= TxFlags= RtsRetries=0 DataRetries=0 MCS= AMPDUStatus=ref#0 VHT=}
- Layer 2 (24 bytes) = Dot11 {Contents=[..24..] Payload=[..58..] Type=DataQOSData Proto=0 Flags=TO-DS,WEP DurationID=0 Address1=11:22:33:44:55:66 Address2=00:11:22:33:44:55 Address3=11:22:33:44:55:66 Address4= SequenceNumber=0 FragmentNumber=0 Checksum=4262477891}
- Layer 3 (58 bytes) = Dot11WEP {Contents=[..58..] Payload=[]}
PACKET: 116 bytes, wire length 116 cap length 116 # 2017-06-11 02:57:03.243457 +0100 WEST
- Layer 1 (18 bytes) = RadioTap {Contents=[..18..] Payload=[..102..] Version=0 Length=18 Present=18478 TSFT=0 Flags= Rate=1 Mb/s ChannelFrequency=2417 MHz ChannelFlags=CCK,Ghz2 FHSS=0 DBMAntennaSignal=-25 DBMAntennaNoise=0 LockQuality=0 TxAttenuation=0 DBTxAttenuation=0 DBMTxPower=0 Antenna=1 DBAntennaSignal=0 DBAntennaNoise=0 RxFlags= TxFlags= RtsRetries=0 DataRetries=0 MCS= AMPDUStatus=ref#0 VHT=}
- Layer 2 (24 bytes) = Dot11 {Contents=[..24..] Payload=[..74..] Type=DataQOSData Proto=0 Flags=TO-DS,WEP DurationID=314 Address1=00:11:22:33:44:55 Address2=11:22:33:44:55:66 Address3=00:11:22:33:44:55 Address4= SequenceNumber=0 FragmentNumber=0 Checksum=412506031}
- Layer 3 (74 bytes) = Dot11WEP {Contents=[..74..] Payload=[]}
I would like to see for example the SSID of the packets or more info inside each layer but everytime I try to drill down the items I get:
RadioTap
Dot11
Dot11WEP
RadioTap
Dot11
Dot11WEP
CODE FOR THE ABOVE OUTPUT
package main
import (
"fmt"
"github.com/google/gopacket"
"github.com/google/gopacket/pcap"
//"github.com/google/gopacket/layers"
"log"
)
var (
pcapFile string = "myFile.pcap"
handle *pcap.Handle
err error
)
func main() {
// Open file instead of device
handle, err = pcap.OpenOffline(pcapFile)
if err != nil { log.Fatal(err) }
defer handle.Close()
// Loop through packets in file
packetSource := gopacket.NewPacketSource(handle, handle.LinkType())
for packet := range packetSource.Packets() {
for _, p := range packet.Layers() {
for _, b := range p.LayerType() {
fmt.Println(b)
}
}
}
}
But in reality I would like to Know the SSID/BSSID and the flags inside the packtet from Dot11 layer.
package main
import (
"fmt"
"github.com/google/gopacket"
"github.com/google/gopacket/pcap"
"github.com/google/gopacket/layers"
"log"
)
var (
pcapFile string = "Network_Join_Nokia_Mobile.pcap"
handle *pcap.Handle
err error
)
func main() {
// Open file instead of device
handle, err = pcap.OpenOffline(pcapFile)
if err != nil { log.Fatal(err) }
defer handle.Close()
// Loop through packets in file
packetSource := gopacket.NewPacketSource(handle, handle.LinkType())
for packet := range packetSource.Packets() {
fmt.Println(packet.Metadata().CaptureInfo.Timestamp)
dot11 := packet.Layer(layers.LayerTypeDot11)
if nil != dot11 {
dot11, _ := dot11.(*layers.Dot11)
// the flags are empty in many of the packets of this example capture file
fmt.Printf("BSSID: %v Flags: %+v\n", dot11.Address3, dot11.Flags)
}
dot11info := packet.Layer(layers.LayerTypeDot11InformationElement)
// some wlan frames contain these with the SSID, usually beacons, probes and association requests
if nil != dot11info {
dot11info, _ := dot11info.(*layers.Dot11InformationElement)
if dot11info.ID == layers.Dot11InformationElementIDSSID {
fmt.Printf("SSID: %q\n", dot11info.Info)
}
}
fmt.Printf("\n")
}
}
Part of output with example file Network_Join_Nokia_Mobile.pcap from Wireshark:
2000-01-01 00:05:04.913478 +0000 UTC
BSSID: 00:01:e3:41:bd:6e Flags: Retry
SSID: "martinet3"
Note that the capture file you used in your question does not contain a frame with the layer that contains the SSID.

Resources