MongoDB Ruby driver and mongoS - ruby

I am trying to use the MongoDB Ruby driver to pull information from a cluster via mongoS. Here is what I've done.
#mongo_client = Mongo::Connection.new('mongoshost', 27320)
#db = #mongo_client.db("thedatabase")
#auth = #db.authenticate("username", "password")
if(#mongo_client)
print "Successfully connected to mongos\n"
else
print "Connection failed"
end
if(#auth == true)
print "Auth successful\n"
else
print "Auth failed"
end
collection = #db.collection("thecollection")
puts collection.find.to_a
When I run this, everything works up until the point that it tries to print the documents in the collection. Here's what my output looks like:
C:\Code\whatever>ruby getdata.rb
Successfully connected to mongos
Auth successful
C:/Ruby200/lib/ruby/gems/2.0.0/gems/bson-1.9.2/lib/bson/bson_c.rb:24:in `deseria
lize': time must be positive (ArgumentError)
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/bson-1.9.2/lib/bson/bson_c.rb:2
4:in `deserialize'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/networkin
g.rb:223:in `read_documents'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/networkin
g.rb:169:in `receive'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/networkin
g.rb:133:in `receive_message'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/cursor.rb
:497:in `block in send_initial_query'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/util/logg
ing.rb:55:in `block in instrument'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/util/logg
ing.rb:20:in `instrument'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/util/logg
ing.rb:54:in `instrument'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/cursor.rb
:493:in `send_initial_query'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/cursor.rb
:478:in `refresh'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/cursor.rb
:124:in `next'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/cursor.rb
:290:in `each'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/cursor.rb
:314:in `to_a'
from C:/Ruby200/lib/ruby/gems/2.0.0/gems/mongo-1.9.2/lib/mongo/cursor.rb
:314:in `to_a'
from getdata.rb:27:in `<main>'
The "time must be positive" error had a few search results, but nothing that helped me figure out what the issue is. One of the fields in these documents does hold a negative number for a date, but I'm not sure why that's an issue since it's reading it and not writing (For example, one of the fields looks like so: {"md" : Date(-62135596800000)})
Is this an issue with the driver, is my code bad, or do I need to figure this out in another manner? I'm pretty new to ruby so any help would be appreciated!

Your data is bad as you have already noted. The problem is that the driver is trying to expand the date value ( which is really just a number internally to mongo within a BSON timestamp field ) into a real DateTime object.
When it tries to do this on read, the number is invalid an the inflation fails. If you fix the date value the problem will be corrected.
Note: If you do this in the shell use the ISOate helper to put a correct value in. If you use code then use a DateTime object. Don't use strings as you will mess your data up further.

Related

Ruby enumeration has unexpected second pass

This is making me crazy. I'm trying to loop through email messages using ".each do |one_of|…end", but inserting one innocuous statement breaks the enumeration.
Here is an almost-stand-alone snippet. The authentication info is set up immediately preceding, as is the "require net/imap"
imap = Net::IMAP.new(IMAP_server)
imap.authenticate('LOGIN', IMAP_login, IMAP_pass)
imap.select(IMAP_folder)
[1,2,3,5].each do |mnum|
#message_ids = imap.search(['SUBJECT', 'NETGEAR R7000 Log']).each do |mnum|
puts(mnum)
msg = imap.fetch(mnum, 'BODY[TEXT]')
msg[0].values[1]['BODY[TEXT]'].each_line do |full_entry|
line = String.new(full_entry)
recType = line[/^\[.*\] /]
# recType = recType.tr('[]','')
puts recType
end
puts
puts
end
It was working (and not) when I was looping through message ID numbers, but I replaced that statement with a literal array to eliminate a potential problem source.
Running it with the ".tr" statement commented out produces expected results, lines like:
1
[DHCP IP: (192.168.1.9)]
[DoS attack: FIN Scan]
[DHCP IP: (192.168.1.8)]
[Admin login]
[Admin login failure]
[Admin login]
[DHCP IP: (192.168.1.7)]
[DHCP IP: (192.168.1.5)]
[Time synchronized with NTP server]
[Internet connected]
2
[LAN access from remote]
[LAN access from remote]
[LAN access from remote]
[UPnP set event: Public_UPNP_C3]
[UPnP set event: Public_UPNP_C3]
[UPnP set event: Public_UPNP_C3]
…
Now un-comment the ".tr" line, to remove the brackets, and it goes all the way through the first iteration, but then:
1
DHCP IP: (192.168.1.9)
DoS attack: FIN Scan
DHCP IP: (192.168.1.8)
Admin login
Admin login failure
Admin login
DHCP IP: (192.168.1.7)
DHCP IP: (192.168.1.5)
Time synchronized with NTP server
Internet connected
Traceback (most recent call last):
4: from /Users/jan/bin/MassageNetgear.rb:39:in `<main>'
3: from /Users/jan/bin/MassageNetgear.rb:39:in `each'
2: from /Users/jan/bin/MassageNetgear.rb:43:in `block in <main>'
1: from /Users/jan/bin/MassageNetgear.rb:43:in `each_line'
/Users/jan/bin/MassageNetgear.rb:45:in `block (2 levels) in <main>': undefined method `tr' for nil:NilClass (NoMethodError)
So, somehow, putting this one statement in is causing the second iteration of mnum to fail. It doesn't even print "2" for the second iteration. Line 45 in the traceback is the newly inserted statement, but it doesn't even get that far! Adding another puts as the first statement in the inner loop does nothing.
I've done this before to concatenate a bunch of emails containing router log messages, but I then modified that code, and I somehow broke it! I've cut and re-typed the problem statement, thinking there may have been invisibles in there, but I'm using BBEdit, which shows such things.
Please, point out where I'm doing something stupid!
Thanks!
nil prints as an empty string, so it's possible there is actually a last line which isn't matching the regexp and which is printing as an empty string. In that case, tr would fail since nil actually doesn't have a tr method.
You could use a conditional assignment, which will not execute the right-hand side if recType is falsy.
recType = line[/^\[.*\] /]
recType &&= recType.tr('[]','')
puts recType
Or (if you want to actually treat it as an empty string), you can simply unconditionally call to_s on it.
recType = line[/^\[.*\] /].to_s
recType = recType.tr('[]','')
puts recType

How to specify time for youtube api in ruby

I'm writing Ruby script to communicate with Youtube Streaming API to create live stream and live broadcast. So far, I have been able to write code to insert new stream and it works but I'm facing problem while inserting broadcast. I'm getting the following error:
/usr/local/rvm/gems/ruby-2.3.0/gems/google-api-client-0.8.0/lib/google/api_client.rb:652:in `block (2 levels) in execute!': Scheduled start time is required (Google::APIClient::ClientError)
from /usr/local/rvm/gems/ruby-2.3.0/gems/retriable-1.4.1/lib/retriable/retry.rb:27:in `perform'
from /usr/local/rvm/gems/ruby-2.3.0/gems/retriable-1.4.1/lib/retriable.rb:15:in `retriable'
from /usr/local/rvm/gems/ruby-2.3.0/gems/google-api-client-0.8.0/lib/google/api_client.rb:635:in `block in execute!'
from /usr/local/rvm/gems/ruby-2.3.0/gems/retriable-1.4.1/lib/retriable/retry.rb:27:in `perform'
from /usr/local/rvm/gems/ruby-2.3.0/gems/retriable-1.4.1/lib/retriable.rb:15:in `retriable'
from /usr/local/rvm/gems/ruby-2.3.0/gems/google-api-client-0.8.0/lib/google/api_client.rb:626:in `execute!'
from ruby_script.rb:44:in `insert_broadcast'
from ruby_script.rb:98:in `<main>'
So, its complaining that I haven't mentioned the scheduled start time. Here's the relevant code:
# Create a liveBroadcast resource and set its title, scheduled start time,
# scheduled end time, and privacy status.
def insert_broadcast(client, youtube, options)
insert_broadcast_response = client.execute!(
api_method: youtube.live_broadcasts.insert,
parameters: {
part: 'snippet,status'
},
body_object: {
snippet: {
title: options[:broadcast_title],
scheduledStartTime: options[:state_time],
scheduledEndTime: options[:end_time]
},
status: {
privacyStatus: options[:privacy_status]
}
}
)
p "Broadcast: #{insert_broadcast_response.data.id}"
return insert_broadcast_response.id
end
options = {
stream_title: 'stream',
broadcast_title: 'dumdum',
start_time: '2018-01-30T00:00:00.000Z',
end_time: '2018-01-30T00:01:00.000Z',
privacy_status: 'private'
}
The format of date time I have borrowed from this code sample which is in Python. I tried with start_time: Time.now hoping Ruby date time object might work but it didn't. How do I solve this?
Edit: Here's the complete script.
Fix the typo as per #Simple-Lime's answer, and if its still doesn't work try the following:
You need to format your time correctly:
Time.now output is 2017-07-27 07:35:38 +0000 which is wrong.
You need to change it to datetime format of ISO-8601 standard with decimal notation of the timezone with Time#iso8601:
require 'time'
# example time format: '2014-01-30T00:00:00.000Z'
Time.now.getutc.iso8601(3)
# => "2017-07-27T07:30:33.742Z"
In your code you have scheduledStartTime: options[:state_time], and are passing it into the method as options[:start_time]

Mongo Ruby Driver Cursor not found error

I find documents from mongodb using ruby driver, collect them to array and iterate on them to update the same document as in the code below:
crawlarray = ##mongoclient[:crawlarray].find({searchresults:[]},:timeout => false).limit(500)
crawlarray.each do |elm|
finalsearchstring = elm['searchstring']
if elm["searchresults"].blank?
ap "SEARCHING: #{finalsearchstring}"
results = searchG(finalsearchstring)
elm["searchresults"] = results
##mongoclient[:crawlarray].update_one({"_id" => elm['_id']}, elm)
else
ap "ALREADY SEARCHED: #{finalsearchstring}"
end
end
There are 90K records but as you see I just get 500 to not to get the error.Everytime, after about 150 iterations I get this error;
D, [2016-08-02T22:32:08.853065 #10098] DEBUG -- : MONGODB | 127.0.0.1:27017 | posluga-dev.getMore | FAILED | Cursor not found, cursor id: 463388278686 (43) | 0.008009s
/Users/apple/.rvm/gems/ruby-2.2.1/gems/mongo-2.2.7/lib/mongo/operation/result.rb:256:in `validate!': Cursor not found, cursor id: 463388278686 (43) (Mongo::Error::OperationFailure)
from /Users/apple/.rvm/gems/ruby-2.2.1/gems/mongo-2.2.7/lib/mongo/operation/executable.rb:36:in `block in execute'
from /Users/apple/.rvm/gems/ruby-2.2.1/gems/mongo-2.2.7/lib/mongo/server/connection_pool.rb:107:in `with_connection'
from /Users/apple/.rvm/gems/ruby-2.2.1/gems/mongo-2.2.7/lib/mongo/server/context.rb:63:in `with_connection'
from /Users/apple/.rvm/gems/ruby-2.2.1/gems/mongo-2.2.7/lib/mongo/operation/executable.rb:34:in `execute'
from /Users/apple/.rvm/gems/ruby-2.2.1/gems/mongo-2.2.7/lib/mongo/cursor.rb:163:in `block in get_more'
from /Users/apple/.rvm/gems/ruby-2.2.1/gems/mongo-2.2.7/lib/mongo/retryable.rb:51:in `call'
from /Users/apple/.rvm/gems/ruby-2.2.1/gems/mongo-2.2.7/lib/mongo/retryable.rb:51:in `read_with_retry'
from /Users/apple/.rvm/gems/ruby-2.2.1/gems/mongo-2.2.7/lib/mongo/cursor.rb:162:in `get_more'
from /Users/apple/.rvm/gems/ruby-2.2.1/gems/mongo-2.2.7/lib/mongo/cursor.rb:88:in `each'
from /Users/apple/.rvm/gems/ruby-2.2.1/gems/mongo-2.2.7/lib/mongo/collection/view/iterable.rb:44:in `each'
Can anybody help me to sort it out?
Use 'no_cursor_timeout' option along with the find query while using Mongo Ruby Driver.
This will disable all cursor timeouts. By default MongoDB tries to kill all cursors which have been inactive for more than 10 mins.
For more information, check this post.

Ruby Watir: when_present not appropriately waiting until things are present?

I'm having an issue getting when_present to actually wait until the element is present in certain cases. The problem occurs only intermittently, but it always happens when changing pages.
Example: I want to click save on the current page, then enter text into a field on the subsequent page.
If I do:
$browser.button(:value => "Save").click
sleep 3
$browser.text_field(:label => "Name").when_present.set "Leelluu"
tt works fine.
However, if I do:
$browser.button(:value => "Save").click
$browser.text_field(:label => "Name").when_present.set "Leelluu"
it times out/crashes immediately after clicking "Save".
How do I fix this?
(I know, it seems like I have fixed it by using sleep, but my boss has told me that the sleeps are wasting time. He says I need to remove all of them from the code and find another way.)
Updating with a real-world example..... I enter a search text, then click the search button, then want to select the checkbox next to the leading checkbox on the returned list.
$browser.text_field(:id => /.*search*./).set "Snow Leopard"
$browser.button(:id => "save_filter_PricebookEntry").when_present.click
$browser.checkbox(:id => "allBox").when_present.set
I get an "element not clickable" (even though it has a when_present) on it.
unknown error: Element is not clickable at point (251, 482). Other element would receive the click:
...
(Session info: chrome=31.0.1650.57) (Driver info: chromedriver=2.2,platform=Windows NT 6.1 SP1 x86_64)
["D:/Ruby193/lib/ruby/gems/1.9.1/gems/selenium-webdriver-2.35.1/lib/selenium/webdriver/remote/response.rb:51:in `assert_ok'", "D:/Ruby193/lib/ruby/gems/1.9.1/gems/selenium-webdriver-2.35.1/lib/selenium/webdriver/remote/response.rb:15:in `initialize'", "D:/Ruby193/lib/ruby/gems/1.9.1/gems/selenium-webdriver-2.35.1/lib/selenium/webdriver/remote/http/common.rb:59:in `new'", "D:/Ruby193/lib/ruby/gems/1.9.1/gems/selenium-webdriver-2.35.1/lib/selenium/webdriver/remote/http/common.rb:59:in `create_response'", "D:/Ruby193/lib/ruby/gems/1.9.1/gems/selenium-webdriver-2.35.1/lib/selenium/webdriver/remote/http/default.rb:66:in `request'", "D:/Ruby193/lib/ruby/gems/1.9.1/gems/selenium-webdriver-2.35.1/lib/selenium/webdriver/remote/http/common.rb:40:in `call'", "D:/Ruby193/lib/ruby/gems/1.9.1/gems/selenium-webdriver-2.35.1/lib/selenium/webdriver/remote/bridge.rb:634:in `raw_execute'", "D:/Ruby193/lib/ruby/gems/1.9.1/gems/selenium-webdriver-2.35.1/lib/selenium/webdriver/remote/bridge.rb:612:in `execute'", "D:/Ruby193/lib/ruby/gems/1.9.1/gems/selenium-webdriver-2.35.1/lib/selenium/webdriver/remote/bridge.rb:369:in `clickElement'", "D:/Ruby193/lib/ruby/gems/1.9.1/gems/selenium-webdriver-2.35.1/lib/selenium/webdriver/common/element.rb:54:in `click'", "D:/Ruby193/lib/ruby/gems/1.9.1/gems/watir-webdriver-0.6.4/lib/watir-webdriver/elements/checkbox.rb:26:in `set'", "D:/Ruby193/lib/ruby/gems/1.9.1/gems/watir-webdriver-0.6.4/lib/watir-webdriver/wait.rb:100:in `method_missing'", "D:/Desktop/jardine/utils.rb:143:in `addQLIs'", "D:/Desktop/jardine/noSleepTest.rb:31:in `block in '", "D:/Desktop/jardine/framework.rb:39:in `execute'", "D:/Desktop/jardine/framework.rb:61:in `block in runTestCases'", "D:/Desktop/jardine/framework.rb:60:in `each'", "D:/Desktop/jardine/framework.rb:60:in `runTestCases'", "jardine.rb:66:in `
'"]

Why I cannot set the value of this hash key?

task = {:project=>1000,
:order=>0,
:partial_image=>nil,
:options=>{
:height=>50,
:width=>50,
:start_row=>1,
:start_column=>1,
:end_row=>50,
:end_column=>50,
:scene=>0}}
project = redis.hget('active_projects', task[:project])
=>
{:name=>"Pov",
:tasks=>
{0=>
{:project=>1000,
:order=>0,
:partial_image=>nil,
:options=>
{:height=>50,
:width=>50,
:start_row=>1,
:start_column=>1,
:end_row=>50,
:end_column=>50,
:scene=>"blabla"
}
}
},
:id=>1000,
:image=>"",
:options=>
{:height=>100,
:width=>50,
:scene=>"blabla"
}
}
task[:partial_image] = 'blablabla'
project[:tasks][task[:order]] = task # this is line 37
Failure/Error: completed_task = DPovray::Task.perform(task)
TypeError:
can't convert Symbol into Integer
# ./lib/jobs/job.rb:37:in `[]'
# ./lib/jobs/job.rb:37:in `block in perform'
# ./lib/jobs/job.rb:35:in `perform'
# ./spec/task_spec.rb:22:in `block (4 levels) in <top (required)>'
The code is in https://github.com/Nerian/DPovray
The test that fails can be run with rspec spec/task_spec.rb
Actually project = redis.hget('active_projects', task[:project]) is returning a string, not a ruby hash. So that is why it fails.
I am playing with https://github.com/nateware/redis-objects to see if I can do what I want to do.
Also, instead of doing:
Redis.new.hset('active_projects', active_project[:id], active_project)
You can do:
Redis.new.hset('active_projects', active_project[:id], Marshal.dump(active_project))
And it just works, thanks to hash marshaling.
Nonetheless, I do not consider this a good solution. I don't like to use Marshaling as it is much difficult do debug by looking at the database.
Also I just got a:
incompatible marshal file format (can't be read)
format version 4.8 required; 123.58 given
So let us discover a different approach...
edit:
Now I am playing with JSON.dump and JSON.parse. They seem a better approach.
Edit:
I ended up encapsulating this hash into a real object. So I have Project class and a Task class. In each one I define the methods to_json and self.json_create(o) to that they can be convert to and from JSON.
It works quite well.

Resources