I'd like to write a Codec plugin to enable LogStash to decode a binary data format.
The official documentation for writing a Codec shows that I need to define a decode method that accepts a single parameter: a variable called data.
I'm new to both LogStash and Ruby. Having worked mostly with statically typed languages, I'm unsure how to learn more about the data variable. I assume that it's analogous to an InputStream-type object, allowing me to read data as it becomes available, but I'm not sure.
Questions:
What type is the data object? What methods does it have?
How do Ruby developers typically go about investigating variables like this? I'm not sure I see a way to figure it out without writing a skeleton plugin and dumping a string representation of data to STDOUT.
Thanks!
The documentation for writing an input plugin hints at this. From the run() method section:
data = $stdin.sysread(16384)
#codec.decode(data) do |event|
decorate(event)
event.set("host", #host) if !event.include?("host")
queue << event
end
The data variable is a Ruby String, which is being used as a buffer of arbitrary bytes. I have verified this by creating a skeleton plugin and inspecting the value at runtime.
This seems to be cause for caution: the bytes provided to your codec's decode method are not guaranteed to be a complete event.
Related
I was trying to encode and decode HTTP responses. to deal with the body I created a custom ReadCloser with its own UnmarshalBinary and MarshalBinary methods. The gob output was inconsistent with the output of the UnmarshalBinary
I also created a sample repo to demonstrate the same - https://github.com/slayerjain/gob-decode-issue.
I've also created an issue on the golang repo - https://github.com/golang/go/issues/51645
Thanks to a user on Reddit I found the solution. The problem is that in the UnmarshalBinary method I need to create a copy of the byte array. Else it'll be populated with other data since it's a pointer.
ref: https://www.reddit.com/r/golang/comments/tddjdd/gob_is_appending_gibberish_to_my_object/
I am using net library in go and I want to make RPC call:
Client.Call("action", []string{"arg1", "arg2"}, &response)
But in JSON I see:
{"method":"action","params":[["arg1","arg2"]],"id":0}
Notice that arguments are enclosed with double square brackets.
In my case I need params to be a simple list:
{"method":"action","params":["arg1","arg2"],"id":0}
Any ideas how to accomplish this?
The codec that Go's JSON RPC uses on top of the rpc.Client will take whatever param you send and encode that as the first element of the array it uses for the params.
So the encoded request will always have a top level array with just one element, which will contain the params you sent, as you already noted.
See the WriteRequest function here:
https://golang.org/src/net/rpc/jsonrpc/client.go#L57
To achieve what you want, you can implement a custom rpc.ClientCodec.
The interface is documented here:
https://golang.org/pkg/net/rpc/#ClientCodec
You can borrow almost all of the implementation for the default JSON codec here:
https://golang.org/src/net/rpc/jsonrpc/client.go
And modify the params attribute of the request to read:
Params interface{} `json:"params"`
Then when writing your WriteRequest based on the standard one, you can just assign your params to the request params:
c.req.Params[0] = param
You can then use the rpc.NewClientWithCodec to create a client using your custom codec:
https://golang.org/pkg/net/rpc/#NewClientWithCodec
I'm doing some stuff with Net-SNMP library. Basically what I do is based on the sample application Simple_Application. What is not clear for me though is that part of code:
for (vars = response->variables; vars; vars = vars->next_variable) {
// process variable
}
I did a lot of testing, read this post as well and it seems to me that you mostly get a scalar value with SNMP request. So the question is: when you get more than one variable as a response?
Each request may contain a number of (scalar) variable names, and the response message will have corresponding variable bindings for each requested variable. So looping through them does make sense in that use case.
SNMP also allows the "get-next" request, which has similar semantics,
and even the "get-bulk" requests, which may return a large number of variables.
You can find examples of each request type in RFC 1905 (see sections 4.2.1 and 4.2.2 in particular).
I have serialized an object in YAML and send it to a remote worker.
The worker doesent have the object definition so i get a YAML::Object.
How can i access the field inside it?
A text field seems like that base64 encoded, how can i decode that? (no, decode64 not works).
you can pass the object as something "known between both sides" (like an openstruct or hash) or give the description to the client.
It would be interesting to have a serialization format that also serialized the class and its methods...I'll have to think about that one...
try c["bar"]
you can also see all the provided keys using c.keys
In a little app I'm building, I'm using the the twitter_oauth gem (source of the methods I'm using), which incidentally means I'm dealing with the JSON ruby gem.
I'm using the messages method, whose source is as follows:
def messages(page=1)
oauth_response = access_token.get("/direct_messages.json?page=#{page}")
JSON.parse(oauth_response.body)
end
It parses the JSON produced by Twitter, using the JSON.parse method. Now, what I want to do is filter the response, so as to show only the messages sent by a certain user. In other words, I want to be able to get an accounts message's on a per user basis.
I went through the JSON gem docs, but couldn't find an easy way to do this. I normally don't work with JSON (I prefer XML), but since the Twitter_oauth gem relies on it, I'm forced to learn it (unless I change the gem's source code or overwrite it - not of my preference).
Does any one know a pragmatic way of sorting JSON in Ruby?
Why do you have to work with JSON at all? JSON.parse gives you back deserialized structures (Ruby arrays and hashes), so all you have to do (I presume, you haven't pasted sample JSON output here) is to do select on whatever output JSON.parse (or, in your case, messages) method returns.