In an effort to speed up frequently repeated runs of a particular script in my chain, I started serializing to disk custom objects that otherwise take too much time to create aggregately.
Using built-in Yaml and/or Marshal.
Yaml serializes fine to a seemingly healthy text file but produces the following error when trying to deserialize:
b2 = YAML::load(File.open("browserObj.yaml", 'r'))
Syck::TypeError: Invalid Regular expression: "/\\A\\s*\n ([a-zA-Z][-+.a-zA-Z\\d]*): ...and many more strange lines
However even trying to save to a binary file via Marshal errors:
puts File.open("browserObj.bin", 'w').write Marshal::dump($browser)
TypeError: can't dump TCPServer
# Marshal::dump($browser, File.open("browserObj.bin", 'wb')) # same error
By deliberately not doing $browser.close at the end I have the option to keep this TCPServer alive and running after the lifetime of my Ruby script.
Any thoughts on how I can get away with this? I promise upon successful reloading to double-check the validity of any sockets/inner objects and simply re-initialize a whole new object if I have to.
Related
For context, I'm someone with zero experience in Ruby - I just asked my Senior Dev to copy-paste me some of his Ruby code so I could try to work with some APIs that he ended up putting off because he was too busy.
So I'm using an API wrapper called zoho_hub, used as a wrapper for Zoho APIs (https://github.com/rikas/zoho_hub/blob/master/README.md).
My IDE is VSCode.
I execute the entire length of the code, and I'm faced with this:
[Done] exited with code=0 in 1.26 seconds
The API is supposed to return a paginated list of records, but I don't see anything outputted in VSCode, despite the fact that no error is being reflected. The last 2 lines of my code are:
ZohoHub.connection.get 'Leads'
p "testing"
I use the dummy string "testing" to make sure that it's being executed up till the very end, and it does get printed.
This has been baffling me for hours now - is my response actually being outputted somewhere, and I just can't see it??
Ruby does not print anything unless you tell it to. For debugging there is a pretty printing method available called pp, which is decent for trying to print structured data.
In this case, if you want to output the records that your get method returns, you would do:
pp ZohoHub.connection.get 'Leads'
To get the next page you can look at the source code, and you will see the get request has an additional Hash parameter.
def get(path, params = {})
Then you have to read the Zoho API documentation for get, and you will see that the page is requested using the page param.
Therefore we can finally piece it together:
pp ZohoHub.connection.get('Leads', page: NNN)
Where NNN is the number of the page you want to request.
While working with the open source ELK stack, we have run into an issue where one of the Logstash inputs snmptrap is formatting data in a way that is unusable for us. Within the SNMPv1_Trap class there is an instance variable called agent_address which is stored as a SNMP::IpAddress. For anyone familiar with the way SNMP works, the agent address is extremely important in determining where a SNMP trap originated from when using trap relays on your network.
The problem can be seen when you take a look at an event generated by Logstash upon receiving a trap. Mainly, the inspect method of the agent_address variable is dumping data that does not match anything valid.
A sample event looks kind of like this:
#<SNMP::SNMPv1_Trap:0x2db53346 #enterprise=[1.3.6.1.4.1.6827.10.17.3.1.1.1], #timestamp=#<SNMP::TimeTicks:0x2a643dd1 #value=0>, #varbind_list=[#<SNMP::VarBind:0x2d5043a5 #name=[1.0], #value=#<SNMP::Integer:0x29fb6a4a #value=1>>], #specific_trap=1000, #source_ip=\"192.168.87.228\", #agent_addr=#<SNMP::IpAddress:0x227a4011 #value=\"\\xC0\\xA8V\\xFE\">, #generic_trap=6>
We know however, that the IpAddress object used in SNMP::SNMPv1_Trap is able to return us a nicely formatted string representing the IPv4 address it is storing.
For example:
require 'snmp'
include SNMP
address = IpAddress.new(192.168.86.254)
puts address
will yield 192.168.86.254 whereas:
require 'snmp'
include SNMP
address = IpAddress.new(192.168.86.254)
puts address.inspect
will yield:
#<SNMP::IpAddress:0x0000000168ae88 #value="\xC0\xA8V\xFE">
This is the expected behaviour of an object whose .inspect method has not been overridden.
Obviously the IPv4 address in #value is not useful to us, it has only three valid hex sequences (xC0=192, xA8=168, xFE=254) and also contains an invalid hex sequence ('V'). The same thing occurs whenever an octet string representing an IPv4 address is sent as a variable binding as well, which suggests some strange encoding.
Unfortunately, aside from writing our own SNMP input, there is no interface level access to this object. The object we receive via 'event' contains the inspect string, not the object itself. Therefore, the easiest apparent way to get the information we need would be to reconstruct the SNMPv1_Trap object and then make our own calls to it via Object.#send.
If I have the raw, unformatted and default string dump returned by Object.#inspect, is there any way to physically recreate the object used to make this inspect dump on the fly?
For example, given the string dump:
#<Integer:0x2737476 #value=1>
is it possible to recreate an Integer object with a field whose value is 1?. If this is possible, is there also a way to recreate nested objects the same way? For example, given the string:
#<SNMP::SNMPv1_Trap:0x2ef73621 #value=1, #agent_address=#<SNMP::IpAddress:0x0000000168ae88 #value="\xC0\xA8V\xFE">>
Would it possible to have an object that looks like the following?
SNMP::SNMPv1_Trap{
#value : 1
#agent_address : SNMP::IpAddress{
#value : 1
}
}
If I have the raw, unformatted and default string dump returned by Object#inspect, is there any way to physically recreate the object used to make this inspect dump on the fly?
No. inspect is intended for debugging purposes to be read by humans.
It is not guaranteed to be machine-readable. It is not guaranteed to be the same across different Ruby versions. It is not guaranteed to be the same across different Ruby implementations. It isn't even guaranteed to be the same across different versions of the same Ruby implementation implementing the same Ruby version. Heck, I don't even think it is guaranteed to be the same across two runs!
It is not a serialization format.
There are plenty of serialization formats specifically for Ruby (Marshal) or generically (XML, YAML, JSON, and of course ASN.1), but inspect isn't it.
I'm using splunk-client to extract results from splunk. Here's the code:
query = "sourcetype=collection #{order_id}"
search = #splunk_client.search(query)
search.wait
The search is happening fine, and it seems like I'm doing everything according to the example (https://github.com/cbrito/splunk-client), but I get this error on the 'search.wait' line:
Undefined namespace prefix: //s:key[#name='isDone']
Any ideas what could be going wrong? Running these commands in irb works fine. Is there some sort of blocking issue?
There is currently very little error checking which occurs within the gem itself. The reason for the error is that wait looks for the status of the isDone key to change to true.
Since your credentials were not properly setup in the first place, the gem creates a search object with an invalid session. The search does not initially fail, because enough response came back from Splunk that Nokogiri processes it into an object without a Splunk search sid.
In the future I should likely raise an exception if a proper sid is not returned to avoid confusion.
Source: I wrote the gem.
I found out the issue -- the splunk client wasn't authenticating properly, and so search was actually a broken SplunkJob object (with a nil username and authentication key). It's strange that there was no error raised until the wait command, but upon inspecting the search object, one of the fields stated that the object was malformed.
I keep getting this error from Chef but can't find any documentation or other people who have had it.
What are the likely root causes?
Some more info would be helpful here. What workflow are you going down when you see this?
I'm going to make an assumption that it's not a knife call. I attempted to put in some debug around the source of your error in the chef-gem and called data bag list and data bag show. Neither seemed to hit the mixin code.
The following is the source of your error in the chef gem under mixin/language
def data_bag(bag)
DataBag.validate_name!(bag.to_s)
rbag = DataBag.load(bag)
rbag.keys
rescue Exception
Log.error("Failed to list data bag items in data bag: #{bag.inspect}")
raise
end
Now I'm at a loss as to what is accessing that mixin code because all other references to data_bag() in the gem refer to the code around the data_bag_item object.
Is this custom code you've created? Is there a chance you are referencing the wrong module?
You normally get this error when Chef cannot find the data bag "id"
Say I would like to load the following data_bag
data_bags
apps
mywebserver.json
apps
recipes
default.rb
[mywebserver.json]
{
"id": "mywebserver"
}
[default.rb]
data_bag_item("apps", "mywebserver") # The id specified in the json
I believe chef does not care for the data_bag_item file name but only cares for the "id" specified in one of the data bag item json files.
I'm dealing with a SOAP webservice call from a server that is expecting to receive method calls with the paramaters in the format of:
<urn:offeringId> 354 </urn:offeringId>
But SOAP::RPC::Driver is generating messages in the form of:
<offeringId xsi:type = "xsd:int">354</offeringId>
The server keeps erroring when it gets these messages (especially since it's expecting offeringId to be a custom type internal to itself, not an int).
Is there anyway to configure the driver to format things the way the server is expecting it. Is the server even doing SOAP? I'm having trouble finding reference to that style of formating for SOAP (I know it DOES work though, because SOAPUI works just fine with that type of message).
-Jenny
Edit: I've got at least part of it solved. the RPC::Driver (obviously) uses the RPC standard, whereas apparently the server I'm trying to talk to is doing "document". Now, when I look at RPC::Driver's API, I'm seeing a method named "add_document_method". That SOUNDS to me like it might be what I want, but I can't figure out what paramaters to give it. The examples I've seen around the net don't make much sense to me, things like:
def GetNamePair(response)
response.account.each do |x|
class << x
attr :configuration, true
end
x.configuration = Hash[*x.a.map do |y|
[y.__xmlattr[XSD::QName.new(nil, 'n')], String.new(y)]
end.flatten]
end
end
mNS = 'urn:zimbraAdmin'
drv.add_document_method('GetAllAdminAccountsRequest', mNS, [XSD::QName.new(mNS, 'GetAllAdminAccountsRequest')],
[XSD::QName.new(mNS, 'GetAllAdminAccountsResponse')] )
puts YAML.dump(GetNamePair(drv.GetAllAdminAccountsRequest([]))
All I really know is that I have a method that takes in certain parameters.... I really don't get why, if this method does what I think it does, it has to be more complicated. Isn't this just a matter of taking the exact same data and formating it differently? I'm so confused....
Okay, what I ended up doing was using SOAP:RPC:Drivers add_document_method, which requires me to give it the wsdl, namespace, etc, and then give it the attributes later as a single input hash thingy (and gives me the output in a similar format). It worked, it just wasn't as clean as add_rpc_method (which is waht add_method defaults to)
-Jenny