Storing tweets using tweetstream in MongoDB fails with following error - ruby

Question

I have just started to learn ruby and concept of mongodb. This is the script that I am trying to run
require 'rubygems'
require 'tweetstream'
require 'mongo'
TweetStream.configure do |config|
config.consumer_key = '<key>'
config.consumer_secret = '<secret>'
config.oauth_token = '<token>'
config.oauth_token_secret = '<token_secret'
config.auth_method = :oauth
end
connection = Mongo::Connection.new
db = connection.db("tweetsDB")
tweets = db.collection("tweets")
client = TweetStream::Client.new
client.on_error do |message|
puts message
end
client.follow(<user_id>,<user_id>) do |status|
id = tweets.insert(status, :safe => true)
end
NOTE: I have removed all the static private values with in the script above for this post.
Version of Mongo, bson, bson_ext - 1.7.0
error message
NoMethodError: undefined method `has_key?' for #<Twitter::Tweet:0x7f21cd14cf08>
/var/lib/gems/1.8/gems/bson-1.7.0/lib/bson/types/object_id.rb:93:in `create_pk'
/var/lib/gems/1.8/gems/mongo-1.7.0/lib/mongo/collection.rb:360:in `insert'
/var/lib/gems/1.8/gems/mongo-1.7.0/lib/mongo/collection.rb:360:in `collect!'
/var/lib/gems/1.8/gems/mongo-1.7.0/lib/mongo/collection.rb:360:in `insert'
tracker.rb:28
/var/lib/gems/1.8/gems/tweetstream-2.3.0/lib/tweetstream/client.rb:525:in `call'
/var/lib/gems/1.8/gems/tweetstream-2.3.0/lib/tweetstream/client.rb:525:in `invoke_callback'
/var/lib/gems/1.8/gems/tweetstream-2.3.0/lib/tweetstream/client.rb:533:in `yield_message_to'
/var/lib/gems/1.8/gems/tweetstream-2.3.0/lib/tweetstream/client.rb:471:in `respond_to'
/var/lib/gems/1.8/gems/tweetstream-2.3.0/lib/tweetstream/client.rb:411:in `connect'
/var/lib/gems/1.8/gems/em-twitter-0.2.1/lib/em-twitter/connection.rb:296:in `call'
/var/lib/gems/1.8/gems/em-twitter-0.2.1/lib/em-twitter/connection.rb:296:in `invoke_callback'
/var/lib/gems/1.8/gems/em-twitter-0.2.1/lib/em-twitter/connection.rb:143:in `handle_stream'
/var/lib/gems/1.8/gems/em-twitter-0.2.1/lib/em-twitter/connection.rb:193:in `on_body'
/var/lib/gems/1.8/gems/em-twitter-0.2.1/lib/em-twitter/connection.rb:192:in `each'
/var/lib/gems/1.8/gems/em-twitter-0.2.1/lib/em-twitter/connection.rb:192:in `on_body'
/var/lib/gems/1.8/gems/em-twitter-0.2.1/lib/em-twitter/connection.rb:74:in `<<'
/var/lib/gems/1.8/gems/em-twitter-0.2.1/lib/em-twitter/connection.rb:74:in `receive_data'
/var/lib/gems/1.8/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in `run_machine'
/var/lib/gems/1.8/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in `run'
/var/lib/gems/1.8/gems/tweetstream-2.3.0/lib/tweetstream/client.rb:385:in `start'
/var/lib/gems/1.8/gems/tweetstream-2.3.0/lib/tweetstream/client.rb:128:in `filter'
/var/lib/gems/1.8/gems/tweetstream-2.3.0/lib/tweetstream/client.rb:106:in `follow'
tracker.rb:27

Answer 1

If you want to blindly dump everything into MongoDB then you can just do a :
client.follow(<user_id>,<user_id>) do |status|
data = status.to_hash
id = tweets.insert(data)
end
If you want to be a bit more selective then you can try something like :
# only add the following fields to the database
ONLY = %w{created_at text geo coordinate id_str}
client.follow(<user_id>,<user_id>) do |status|
data = status.to_hash.select{|k,v| ONLY.include?(k.to_s)}
id = tweets.insert(data)
end
Or :
# add everything except the following fields to the database
EXCEPT = %w{entities}
client.follow(<user_id>,<user_id>) do |status|
data = status.to_hash.reject{|k,v| EXCEPT.include?(k.to_s)}
id = tweets.insert(data)
end

Answer 2

It seems looking at the docs that Client.follow just returns a generic ruby object. You have to take the relevant fields from the object and pull them out into json (or, more advisable, a mongoid object) to send to the database.

Answer 3

has_key? is a method available in the Hash superclass. What it's trying to do is make a json string to pass to Mongo. Except that you are passing a string, has_key? is not part of the String class. Just convert it to a hash and you should be good to go.
client.follow(<user_id>,<user_id>) do |status|
id = tweets.insert({:status => status.text}, :safe => true)
end