I'm attempting to implement some Go code to solve a problem in which I need to sufficiently obfuscate a known integer value by converting it into a seemingly random hexadecimal string, when provided a known key value as an additional input parameter. The resulting hexadecimal string needs to always be the same number of characters in length (ideally, <= 32 characters).
Furthermore, using the same key string value, I need to un-obfuscate the hexadecimal string back into the original integer. For additional context, I'd like to satisfy the following function signatures (but am open to alternative methods, if necessary):
func Scramble(key string, value int32) string {
// TODO: Given a known key and value, generate a sufficiently unpredictable hexadecimal string.
}
func Unscramble(key string, value string) int32 {
// TODO: Given a known key and value, generate the integer that created the hexadecimal string.
}
func main() {
key := "Something super secret!"
scrambled := Scramble(key, 135)
fmt.Printf("Scrambled: %s\n", scrambled) // Scrambled: a1dec128b590b9ec3281110d6d188c26
unscrambled := Unscramble(key, scrambled)
fmt.Printf("Unscrambled: %d\n", unscrambled) // Unscrambled: 135
}
I think XOR'ing may be the right direction, but I'm unsure and not particularly familiar with the topic yet.
Any insight/direction would be greatly appreciated! Please let me know if I can provide any additional context/clarifications.
There are many native or external packages to achieve what you want, but if you want to implement this yourself for a learning experience, you can try the following tack:
Rather than shuffle your data back and forth between string and int32 format - keep the data in its raw type and use Stringer methods to convert to hex - and helper methods/functions to convert to the desired type. This simplifies the scrambling/unscrambling logic - as the input types are the same for both.
// Code custom type so we can add stringer methods
type Code uint32
// String converts code to hex string format
func (c Code) String() string {
return fmt.Sprintf("%x", uint32(c))
}
// CodeFromString gets a code from a hex string
func CodeFromString(hexs string) (Code, error) {
ui, err := strconv.ParseUint(hexs, 16, 32)
if err != nil {
return 0, err
}
return Code(ui), nil
}
// XOR scrambles/unscrambles
func XOR(key, value Code) Code {
return key ^ value
}
And to use:
keyHex := "74490a85"
valueHex := "d195c729"
value, _ := CodeFromString(valueHex)
key, _ := CodeFromString(keyHex)
scrambled := XOR(key, value)
unscrambled := XOR(key, scrambled)
Playground Example: https://play.golang.org/p/y5pbac_f8Z1
Related
Im learning to create REST APIs using Go. Here's where I am stuck.
When user sends a CREATE request:
From the Slice of articles, I need to take the last article
Convert the ID(originally string) to Integer
Increment the Integer and convert it back to string and save it
Article Struct
type Article struct {
Id string `json:"id"`
Title string `json:"title"`
Desc string `json:"desc"`
Content string `json:"content"`
}
Here's the logic
// get the last id and convert it to integer and increment
lastId, err := strconv.ParseInt(Articles[len(Articles) - 1].Id, 10, 64)
lastId = lastId + 1
if err != nil {
http.Error(w, "Internal Server Error", http.StatusInternalServerError)
}
response := []Article{
{
Id: strconv.Itoa(lastId),// 👈 ERROR
Title: articleBody.Title,
Desc: articleBody.Desc,
Content: articleBody.Content,
},
}
ERROR
cannot use lastId (variable of type int64) as int value
in argument to strconv.Itoa compiler (IncompatibleAssign)
Go has a strong typing system so Int32 and Int64 are not compatible type. Try to convert lastId into an int when calling Itoa:
response := []Article{
{
Id: strconv.Itoa(int(lastId)),
Title: articleBody.Title,
Desc: articleBody.Desc,
Content: articleBody.Content,
},
}
Edit:
As mention by #kostix in his answer, beware of overflows when converting int type (see his answer for details).
A safer solution would be something like this:
newId := int(lastId)
if int64(newId) != lastId {
panic("overflows!")
}
response := []Article{
{
Id: strconv.Itoa(newId),
Title: articleBody.Title,
Desc: articleBody.Desc,
Content: articleBody.Content,
},
}
The language specification says:
uint either 32 or 64 bits
int same size as uint
This means, on a particular platform/version of Go int may be the same size as int32, and this is the reason why Go would not silently allow you to pass a value of type int64 as an argument of type int.
Moreover, a plain type conversion int(lastId) suggested in another answer should be taken with a grain of salt: what happens when your program is compiled and int ends up having 32 bits in size in the compiled code, and a particular lastId value is outside the number range supported by a signed 32-bit integer, say, 2,147,483,648?
Again, the spec says:
When converting between integer types, if the value is a signed integer, it is sign extended to implicit infinite precision; otherwise it is zero extended. It is then truncated to fit in the result type's size. For example, if v := uint16(0x10F0), then uint32(int8(v)) == 0xFFFFFFF0. The conversion always yields a valid value; there is no indication of overflow.
Hence the code
var i64 int64 = 2_147_483_648
var i32 = int32(i64)
fmt.Println(i32)
prints
-2147483648
And when this value is passed to strconv.Itoa, it returns "-2147483648" — quite possibly not what you would expect.
So, in a robust code, you ought to watch out when doing such type conversions, and either check the converted value for sanity, like in
v := int(lastId)
if int64(v) != lastId {
panic("ouch!")
}
or merely use the largest convenient type via strconv.FormatInt.
I am a long time python user moving to Go, and I still have some issues to reacquire basic skill to manage typing and pointer.
I have a program receiving event from RabbitMq (But the problem would be the same no matter what transport we are talking about). One of the even contain an optional field F1 typed as int.
My understanding is, if the field is not present in the event, then go will default it to 0. But 0 is a valid value for that field, and I need to differentiate cases where the value is 0, and cases where the value is non defined.
I thought to make my field a *int to actually have "nil" as a value. But then when when a receive an event, will F1 be set to the actual pointed value, or the value address from the sender?
Do I have any other alternative?
In most cases, using a pointer to the value makes sense. E.g
type RabbitMessage struct {
F1 *int `json:"f1"`
}
The exact details of how this will work depends on how you serialise your data before sending it over RabbitMQ. If you are using JSON, then there should be no issue with this as both a null value, and an omitted value, will be represented in Go as nil. When the value is provided, it will be set to the value you expect (it will not use the address from the sender).
If you control only the receiver program, then AFAICT you can not differentiate between an int that has been automatically initialized to 0 by go from an int that has been set to 0 by the sender.
If you can modify the sender program though, an alternative could be to add a boolean field along with your int, telling whether the int is set or not. Then on the receiving end you can check whether the boolean is true or not.
You can also send a pointer to an int:
type Message struct {
Value *int `json:"value"`
}
message := Message{Value: 4}
Be aware though that when unmarshalling this you'll get an int pointer you'll need to dereference.
"Do I have any other alternative?" -- Yes, you can define a custom type, similar to sql.NullInt64.
type OptionalInt struct {
Int int
IsValid bool
}
func NewOptionalInt(i int) OptionalInt {
return OptionalInt{Int: i, IsValid: true}
}
func (o *OptionalInt) UnmarshalJSON(data []byte) error {
if string(data) != "null" {
if err := json.Unmarshal(data, &o.Int); err != nil {
return err
}
o.IsValid = true
}
return nil
}
func (o OptionalInt) MarshalJSON() ([]byte, error) {
if o.IsValid {
return json.Marshal(o.Int)
}
return json.Marshal(nil)
}
I have a list downloaded from a website in XML. I am trying to filter the list by discarding lines that contain a certain string and building the same type of list without the lines containing the string.
I have a struct type that's having another struct.
I'm trying to use regexp and replaceall, and failing at replaceall.
func (*Regexp) ReplaceAll
func (re *Regexp) ReplaceAll(src, repl []byte) []byte
There may be an entirely simpler way to filter a list to a new list that I'm missing somewhere, but I've found this as the closest possible solution so far. Please share other ways to grep and delete lines to a new list. The list is a byte at body and downloaded as a xml.
type PeopleList struct {
Peoples []Person `xml:"peoples>person"`
}
type Person struct {
ADD string `xml:"add,attr"`
Loc string `xml:"loc,attr"`
Har string `xml:"har,attr"`
Name string `xml:"name,attr"`
Country string `xml:"country,attr"`
Num string `xml:"num,attr"`
ADD2 string `xml:"add2,attr"`
Distance float64
func fetchPeopleList(userinfo Userinfo) PeopleList {
var p byte
jam, err := http.Get(string(peoplelisturl))
iferror (err)
body, err := ioutil.ReadAll(jam.Body)
peeps := body
reg := regexp.MustCompile("(?m)[\r\n]+^.*BAD:.*$")
rep := reg.ReplaceAll(peeps, p) // Here fails probably because of my syntax. Error: cannot use p (variable of type byte) as []byte value in argument to re.ReplaceAll
fmt.Println(rep)
iferror (err)
defer jam.Body.Close()
Finally, I would like a new list in the same format as the first, only without the lines containing the string.
Your question says you want to "discard lines", but Replace/ReplaceAll, as their names suggest, are for replacing matched patterns. Your regex is also a simple substring match, so the obvious solution would seem to be reading the file line by line and - as your title says - discarding lines containing the substring.
func fetchPeopleList(userinfo Userinfo) PeopleList {
jam, err := http.Get(string(peoplelisturl))
iferror (err)
br := bufio.NewReader(jam.Body)
defer jam.Body.Close()
for {
line,err := br.ReadString('\n')
if !strings.Contains(line, "BAD:") {
fmt.Println(line) // or whatever you want to do with non-discarded lines
}
if err != nil {
break
}
}
I'm using gorm, and it allows many data types like int, uint, int8, uint8 ....
Then I have a plugin in template like this:
f["UNIX2STR"] = func(t interface{}, f string) string {
switch t.(type) {
case int:
return time.Unix(int64(t.(int)), 0).Format(f)
case uint:
return time.Unix(int64(t.(uint)), 0).Format(f)
case uint8:
return time.Unix(int64(t.(uint8)), 0).Format(f)
case *int:
return time.Unix(int64(*t.(*int)), 0).Format(f)
case *uint:
return time.Unix(int64(*t.(*uint)), 0).Format(f)
case *uint8:
return time.Unix(int64(*t.(*uint8)), 0).Format(f)
.....
default:
return ""
}
// return time.Unix(int64(t), 0).Format(f)
}
It converts all integer types to formatted string.
So what am I suppose to do? List all gorm supported int types and cast it to int64?
I have searched many days for solution convert interface{} to its true type without using type assertion but didn't work.
I've not used gorm, but I think that something like this could solve your problem:
func formatUnix(t interface{}, f string) (string, error) {
timestampStr := fmt.Sprint(t)
timestamp, err := strconv.ParseInt(timestampStr, 10, 64)
if err != nil {
return "", err
}
return time.Unix(timestamp, 0).Format(f), nil
}
Rather than listing all potential types, it simply converts the interface{} to a string using fmt.Sprint() and then convert the string to int64 using strconv.ParseInt().
Based on your comments, it sounds like you're concerned with converting any numeric type to a string. This is easily done with fmt.Sprint:
stringValue := fmt.Sprint(i) // i is any type
But this has nothing to do with GORM.
On the other hand, if your problem is that GORM is returning an unpredictable type, just change your select statement to always return a string. For example, for MySQL, something like:
SELECT CAST(someNumberColumn AS VARCHAR) AS stringColumn
or
SELECT CAST(someNumberColumn AS INT) AS intColumn
I do not think that this is a problem with go or gorm. I am a bit baffled that you save your unix timestamps in many different formats. BTW, A unix timestamp is 32 bit, so there is no point in converting (and saving in the first place) any 8 bit ints.
A solution would be to use a unified data type (int64) for all timestamps in your structs. After that your formatting func can accept int64 instead ofinterface{}, without the need of any type assertions.
I have a listener which receives protobuf messages. However it doesn't know which type of message comes in when. So I tried to unmarshal into an interface{} so I can later type cast:
var data interface{}
err := proto.Unmarshal(message, data)
if err != nil {
log.Fatal("unmarshaling error: ", err)
}
log.Printf("%v\n", data)
However this code doesn't compile:
cannot use data (type interface {}) as type proto.Message in argument to proto.Unmarshal:
interface {} does not implement proto.Message (missing ProtoMessage method)
How can I unmarshal and later type cast an "unknown" protobuf message in go?
First, two words about the OP's question, as presented by them:
proto.Unmarshal can't unmarshal into an interface{}. The method signature is obvious, you must pass a proto.Message argument, which is an interface implemented by concrete protobuffer types.
When handling a raw protobuffer []byte payload that didn't come in an Any, ideally you have at least something (a string, a number, etc...) coming together with the byte slice, that you can use to map to the concrete protobuf message.
You can then switch on that and instantiate the appropriate protobuf concrete type, and only then pass that argument to Unmarshal:
var message proto.Message
switch atLeastSomething {
case "foo":
message = &mypb.Foo{}
case "bar":
message = &mypb.Bar{}
}
_ = proto.Unmarshal(data, message)
Now, what if the byte payload is truly unknown?
As a foreword, consider that this should seldom happen in practice. The schema used to generate the protobuffer types in your language of choice represents a contract, and by accepting protobuffer payloads you are, for some definitions of it, fulfilling that contract.
Anyway, if for some reason you must deal with a completely unknown, mysterious, protobuffer payload in wire format, you can extract some information from it with the protowire package.
Be aware that the wire representation of a protobuf message is ambiguous. A big source of uncertainty is the "length-delimited" type (2) being used for strings, bytes, repeated fields and... sub-messages (reference).
You can retrieve the payload content, but you are bound to have weak semantics.
The code
With that said, this is what a parser for unknown proto messages may look like. The idea is to leverage protowire.ConsumeField to read through the original byte slice.
The data model could be like this:
type Field struct {
Tag Tag
Val Val
}
type Tag struct {
Num int32
Type protowire.Type
}
type Val struct {
Payload interface{}
Length int
}
And the parser:
func parseUnknown(b []byte) []Field {
fields := make([]Field, 0)
for len(b) > 0 {
n, t, fieldlen := protowire.ConsumeField(b)
if fieldlen < 1 {
return nil
}
field := Field{
Tag: Tag{Num: int32(n), Type: t },
}
_, _, taglen := protowire.ConsumeTag(b[:fieldlen])
if taglen < 1 {
return nil
}
var (
v interface{}
vlen int
)
switch t {
case protowire.VarintType:
v, vlen = protowire.ConsumeVarint(b[taglen:fieldlen])
case protowire.Fixed64Type:
v, vlen = protowire.ConsumeFixed64(b[taglen:fieldlen])
case protowire.BytesType:
v, vlen = protowire.ConsumeBytes(b[taglen:fieldlen])
sub := parseUnknown(v.([]byte))
if sub != nil {
v = sub
}
case protowire.StartGroupType:
v, vlen = protowire.ConsumeGroup(n, b[taglen:fieldlen])
sub := parseUnknown(v.([]byte))
if sub != nil {
v = sub
}
case protowire.Fixed32Type:
v, vlen = protowire.ConsumeFixed32(b[taglen:fieldlen])
}
if vlen < 1 {
return nil
}
field.Val = Val{Payload: v, Length: vlen - taglen}
// fmt.Printf("%#v\n", field)
fields = append(fields, field)
b = b[fieldlen:]
}
return fields
}
Sample input and output
Given a proto schema like:
message Foo {
string a = 1;
string b = 2;
Bar bar = 3;
}
message Bar {
string c = 1;
}
initialized in Go as:
&test.Foo{A: "A", B: "B", Bar: &test.Bar{C: "C"}}
And by adding a fmt.Printf("%#v\n", field) statement at the end of the loop in the above code, it will output the following:
main.Field{Tag:main.Tag{Num:1, Type:2}, Val:main.Val{Payload:[]uint8{0x41}, Length:1}}
main.Field{Tag:main.Tag{Num:2, Type:2}, Val:main.Val{Payload:[]uint8{0x42}, Length:1}}
main.Field{Tag:main.Tag{Num:1, Type:2}, Val:main.Val{Payload:[]uint8{0x43}, Length:1}}
main.Field{Tag:main.Tag{Num:3, Type:2}, Val:main.Val{Payload:[]main.Field{main.Field{Tag:main.Tag{Num:1, Type:2}, Val:main.Val{Payload:[]uint8{0x43}, Length:1}}}, Length:3}}
About sub-messages
As you can see from the above the idea to deal with a protowire.BytesType that may or may not be a message field is to attempt to parse it, recursively. If it succeeds, we keep the resulting msg and store it in the field value, if it fails, we store the bytes as-is, which then may be a proto string or bytes. BTW, if I'm reading correctly, this seems what Marc Gravell does in the Protogen code.
About repeated fields
The code above doesn't deal with repeated fields explicitly, but after the parsing is done, repeated fields will have the same value for Field.Tag.Num. From that, packing the fields into a slice/array should be trivial.
About maps
The code above also doesn't deal with proto maps. I suspect that maps are semantically equivalent to a repeated k/v pair, e.g.:
message Pair {
string key = 1; // or whatever key type
string val = 2; // or whatever val type
}
If my assumption is correct, then maps can be parsed with the given code as sub-messages.
About oneofs
I haven't yet tested this, but I expect that information about the union type are completely lost. The byte payload will contain only the value that was actually set.
But what about Any?
The Any proto type doesn't fit in the picture. Contrary to what it may look like, Any is not analogous to, say, map[string]interface{} for JSON objects. And the reason is simple: Any is a proto message with a very well defined structure, namely (in Go):
type Any struct {
// unexported fields
TypeUrl string // struct tags omitted
Value []byte // struct tags omitted
}
So it is more similar to the implementation of a Go interface{} in that it holds some actual data and that data's type information.
It can hold itself arbitrary proto payloads (with their type information!) but it can not be used to decode unknown messages, because Any has exactly those two fields, type url and a byte payload.
To wrap up, this answer doesn't provide a full-blown production-grade solution, but it shows how to decode arbitrary payloads while preserving as much original semantics as possible. Hopefully it will point you in the right direction.
As you've seen, and the commenters have pointed out, you can't use proto.Unmarshal to interface{} because, the method expects a type Message which implements an interface MessageV1.
Protobuf messages are typed and correspond to method invocations ("comes in") and the implementation cannot take generic types of protobuf but specific protobufs:
func (s *server) M(ctx context.Context, _ *pb.Foo) (*pb.Bar, error)
The solution is to envelope your generic types as Any within a specific type perhaps Envelope:
message Envelope {
google.protobuf.Any content = 1;
...
}
The content is then transmitted as a []byte (see Golang anypb.Any) and the implementation (anypb) includes methods to pack|unpack these.
The 'trick' with Any is that messages include a [TypeURL] that uniquely identifies the message so that the receiver knows how to e.g. Unmarshal it.