I have some code written in Ruby 1.9.2 patch level 136 and I'm have an issue where when I perform a find via the _id in the raw ruby mongo driver I get a nil when trying to use a value from a csv file. Here's the code:
require 'mongo'
require 'csv'
require 'bson'
# Games database
gamedb = Mongo::Connection.new("localhost", 27017).db("gamedb")
#games = gamedb.collection("games")
# Loop over CSV data.
CSV.foreach("/tmp/somedata.csv") do |row|
puts row[0] # Puts the ObjectId
#game = #games.find( { "_id" => row[0] } ).first
puts #game.inspect
end
The CSV file looks like this:
_id,game_title,platform,upc_db_match,upc
4ecdacc339c7d7a2a6000002,TMNT,PSP,TMNT,085391157663
4ecdacc339c7d7a2a6000004,Super Mario Galaxy,Wii,Super Mario Galaxy,045496900434
4ecdacc339c7d7a2a6000005,Beowulf,PSP,Beowulf,097363473046
The first column is the objectId in Mongo that I already have. If I perform a local find from the mongo command line the values in the first column, I get the data I want. However, the code above returns nil on the #game.inspect call.
I've tried the following variations, which all produce nil:
#game = #games.find( { "_id" => row[0].to_s } ).first
#game = #games.find( { "_id" => row[0].to_s.strip } ).first
I've even tried building the ObjectId with the BSON classes as such:
#game = #games.find( { "_id" => BSON::ObjectId(row[0]) } ).first
or
#game = #games.find( { "_id" => BSON::ObjectId("#{row[0]}") } ).first
Both of which output the following error:
/Users/donnfelker/.rvm/gems/ruby-1.9.2-p136#upc-etl/gems/bson-1.4.0/lib/bson/types/object_id.rb:126:in `from_string': illegal ObjectId format: _id (BSON::InvalidObjectId)
from /Users/donnfelker/.rvm/gems/ruby-1.9.2-p136#upc-etl/gems/bson-1.4.0/lib/bson/types/object_id.rb:26:in `ObjectId'
from migrate_upc_from_csv.rb:14:in `block in <main>'
from /Users/donnfelker/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/csv.rb:1768:in `each'
from /Users/donnfelker/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/csv.rb:1202:in `block in foreach'
from /Users/donnfelker/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/csv.rb:1340:in `open'
from /Users/donnfelker/.rvm/rubies/ruby-1.9.2-p136/lib/ruby/1.9.1/csv.rb:1201:in `foreach'
from migrate_upc_from_csv.rb:10:in `<main>'
The crazy thing is, if I manually create the BSON ObjectId by hand it works (as shown below):
#game = #games.find( { "_id" => BSON::ObjectId("4ecdacc339c7d7a2a6000004") } ).first
When I run #game.inspect I get my data back, as I would expect. However, If I change this to use row[0], I get nil.
Why? What am I doing wrong?
System Details
$ gem list
*** LOCAL GEMS ***
bson (1.4.0)
bson_ext (1.4.0)
mongo (1.4.0)
RVM Version: rvm 1.6.9
Ruby Version: ruby 1.9.2p136 (2010-12-25 revision 30365) [x86_64-darwin10.6.0]
Mongo Version:
[initandlisten] db version v1.8.2, pdfile version 4.5
[initandlisten] git version: 433bbaa14aaba6860da15bd4de8edf600f56501b
Again, why? What am I doing wrong here? Thanks!
The first row is not being read as a header, to do that pass in :headers => true like this:
require 'csv'
# Loop over CSV data.
CSV.foreach("/tmp/somedata.csv", :headers => true) do |row|
puts row[0] # Puts the ObjectId
end
If you do not pass the :headers parameter in you can see the first row[0] object is the string "_id":
_id
4ecdacc339c7d7a2a6000002
4ecdacc339c7d7a2a6000004
4ecdacc339c7d7a2a6000005
When you include it, you are golden:
4ecdacc339c7d7a2a6000002
4ecdacc339c7d7a2a6000004
4ecdacc339c7d7a2a6000005
Are you sure your CSV parsing code isn't treating the headers as a first line of data and actually tries to do BSON::ObjectId("_id")? The error message kinda looks like it. Try with FasterCSV.foreach('/tmp/somedata.csv', :headers => true) and using row['_id'] (IIRC you'll still have to use BSON::ObjectID).
Related
What is the correct way to view the output of the puts statements below? My apologies for such a simple question.... Im a little rusty on ruby. github repo
require 'active_support'
require 'active_support/core_ext'
require 'indicators'
my_data = Indicators::Data.new(Securities::Stock.new(:symbol => 'AAPL', :start_date => '2012-08-25', :end_date => '2012-08-30').output)
puts my_data.to_s #expected to see Open,High,Low,Close for AAPL
temp=my_data.calc(:type => :sma, :params => 3)
puts temp.to_s #expected to see an RSI value for each data point from the data above
Maybe check out the awesome_print gem.
It provides the .ai method which can be called on anything.
An example:
my_obj = { a: "b" }
my_obj_as_string = my_obj.ai
puts my_obj_as_string
# ... this will print
# {
# :a => "b"
# }
# except the result is colored.
You can shorten all this into a single step with ap(my_obj).
There's also a way to return objects as HTML. It's the my_obj.ai(html: true) option.
Just use .inspect method instead of .to_s if you want to see internal properties of objects.
I have a Sinatra application connecting to MongoDB via Mongoid. It was working fine until a point where I started getting a weird error that I can't find anywhere on stackoverflow or anywhere.
Note that the same code is deployed on Heroku and it works fine with the same database version Mongo 3.0.7, ruby version 2.0.0 which I tried locally as well as 2.2.3. I get the error when I run shotgun through the web and when running Rspec:
Bible::Book#get_book_id is case-insensitive
Failure/Error: book[0]._id
Module::DelegationError:
Mongoid::Contextual#each delegated to context.each, but context is nil: #<Mongoid::Criteria
selector: {"title"=>"genesis"}
options: {}
class: Bible::Book
embedded: false>
# /Users/issa/.rvm/gems/ruby-2.2.3/gems/mongoid-5.0.0/lib/mongoid/contextual.rb:20:in `rescue in each'
# /Users/issa/.rvm/gems/ruby-2.2.3/gems/mongoid-5.0.0/lib/mongoid/contextual.rb:20:in `each'
# /Users/issa/.rvm/gems/ruby-2.2.3/gems/mongoid-5.0.0/lib/mongoid/criteria.rb:554:in `entries'
# /Users/issa/.rvm/gems/ruby-2.2.3/gems/mongoid-5.0.0/lib/mongoid/criteria.rb:554:in `method_missing'
# ./app/bible/book.rb:52:in `get_book_id'
# ./spec/bible/book_spec.rb:11:in `block (3 levels) in <top (required)>'
# ------------------
# --- Caused by: ---
# NoMethodError:
# undefined method `each' for nil:NilClass
# /Users/issa/.rvm/gems/ruby-2.2.3/gems/mongo-2.1.2/lib/mongo/cluster.rb:114:in `initialize'
My Rspec code:
RSpec.describe Bible::Book do
describe "#get_book_id" do
it "finds the id of genesis" do
expect(Bible::Book.get_book_id 'genesis').to eq 1
end
end
end
The code implementation is this:
module Bible
class Book
include Mongoid::Document
store_in collection: "bible_books"
field :title, type: String
field :testament, type: String
...
def self.get_book_id book_title
book = where(:title => book_title.downcase.strip)
return unless book
book[0]._id
end
end
end
My Gemfile has this part for MongoDB:
# MongoDB
...
gem 'mongoid', '~> 5.0'
gem 'bson_ext'
gem 'bson'
It's really strange that I get this error only on my machine, and just started out of the blue. I am using OS X El Capitan. I can access the mongodb collection from command line without issues and can run the same equivalent query to get the result I want:
> use bible
switched to db bible
> db.bible_books.find({"title": "genesis"})
{ "_id" : 1, "title" : "genesis", "testament" : "old", "chapters" : 50, "doc_type" : "book", "canon_type" : "canonical", "canon_order" : 1, "full_name" : { "en" : "The book of Genesis", "ar" : "سفر التكوين" }, "short_name" : { "en" : "Genesis", "ar" : "التكوين" }, "abbr_name" : { "ar" : "تك", "en" : "Gen" } }
I tried other collections, other queries, other variations, and still get the same error. I reverted to an older code from github that worked for sure, still the same issue, so it's clear to me that there's nothing wrong in my code or configuration. I tried different ruby versions from 2.0.0 - 2.2.3, different mongoid versions from 4 - 5 ... again the same versions of code and installation work fine on Heroku, but not my machine :(
I got similar error when wrong production configuration url was specified for mongoid: uri: <%= ENV['MONGOLAB_URI'] %>
I'm not sure if this is the source of your problem but I see a couple odd things in your get_book_id class method.
First of all, this will never leave book.nil?:
book = where(:title => book_title.downcase.strip)
That will leave you with a Mongoid::Contextual in book, the query may or may not find anything but book won't be nil in either case. That means that your:
return unless book
on the next line doesn't do anything useful. That also means that book[0] can be nil in here:
book[0]._id
You're also calling the _id method when you'd usually call id.
I think your method would be better off like this:
def self.get_book_id book_title
book = find_by(:title => book_title.downcase.strip)
return unless book
book.id
end
find_by(query) is just a short way of saying where(query).first so book = find_by(...) will leave you with a nil in book if it can't find anything and a single Bible::Book in book if it can find something.
If you have ActiveSupport available, you could also say:
find_by(:title => book_title.downcase.strip).try(:id)
to hide the nil check. You could also throw in an only call to only pull the _id out of MongoDB:
book = only(:id).find_by(:title => book_title.downcase.strip)
return unless book
book.id
# or, if ActiveSupport is around, just this:
only(:id).find_by(:title => book_title.downcase.strip).try(:id)
Say I have the following class:
class Buyer < ActiveRecord::Base
attr_accesible :first_name, :last_name
and the following in a CSV file:
First Name,Last Name
John,Doe
Jane,Doe
I want to save the contents of the CSV into the database. I have the following in a Rake file:
namespace :migration do
desc "Migrate CSV data"
task :import, [:model, :file_path] => :environment do |t, args|
require 'csv'
model = args.model.constantize
path = args.file_path
CSV.foreach(path, :headers => true,
:converters => :all,
:header_converters => lambda { |h| h.downcase.gsub(' ', '_') }
) do |row|
model.create!(row.to_hash)
end
end
end
I am getting an undefined method 'downcase' for nil:NilClass. If I exclude the header converters then I get unknown attribute 'First Name'. What's the correct syntax for converting a header from, say, First Name to first_name?
After doing some research here in my desktop, it seems to me the error is for something else.
First I put the data in my "a.txt" file as below :
First Name,Last Name
John,Doe
Jane,Doe
Now I ran the code, which is saved in my so.rb file.
so.rb
require 'csv'
CSV.foreach("C:\\Users\\arup\\a.txt",
:headers => true,
:converters => :all,
:header_converters => lambda { |h| h.downcase.gsub(' ', '_') }
) do |row|
p row
end
Now running the :
C:\Users\arup>ruby -v so.rb
ruby 1.9.3p448 (2013-06-27) [i386-mingw32]
#<CSV::Row "first_name":"John" "last_name":"Doe">
#<CSV::Row "first_name":"Jane" "last_name":"Doe">
So everything is working now. Now let me reproduce the error :
I put the data in my "a.txt" file as below ( just added a , after the last column) :
First Name,Last Name,
John,Doe
Jane,Doe
Now I ran the code, which is saved in my so.rb file, again.
C:\Users\arup>ruby -v so.rb
ruby 1.9.3p448 (2013-06-27) [i386-mingw32]
so.rb:5:in `block in <main>': undefined method `downcase' for nil:NilClass (NoMethodError)
It seems, in your header row, there is blank column value which is causing the error. Thus if you have a control to the source CSV file, check there the same. Or do some change in your code, to handle the error as below :
require 'csv'
CSV.foreach("C:\\Users\\arup\\a.txt",
:headers => true,
:converters => :all,
:header_converters => lambda { |h| h.downcase.gsub(' ', '_') unless h.nil? }
) do |row|
p row
end
A more general answer, but if you have code that you need to process as text, and sometimes you might get a nil in there, then call to_s on the object. This will turn nil into an empty string. eg
h.to_s.downcase.gsub(' ', '_')
This will never blow up, whatever h is, because every class in ruby has the to_s method, and it always returns a string (unless you've overridden it to do something else, which would be unadvisable).
Passing :symbol to :header_converters will automatically convert to strings to snake case as well.
options = {:headers => true,
:header_converters => :symbol}
CSV.foreach(filepath, options) ...
#<CSV::Row first_name:"John" last_name:"Doe">
#<CSV::Row first_name:"Jane" last_name:"Doe">
Hello Im using HTTParty to call for a remote json file that I need to extract the URL's to use in one of my tests..
the json format goes something like:
"manifest" : {
"header" : {
"generated" : "xxxxxxxxxxxxxxx",
"name" : "xxxxxxxxxxx",
"version" : "1.0.0"
},
"files" : [ {
"file" : "blimp.zip",
"url" : "http://www.xxx.xx/restaurants_blimp.zip",
"checksum" : "ee98c9455b8d7ba6556f53256f95"
}, {
"file" : "yard.zip",
"url" : "www.xxx.xx/yard.zip",
"checksum" : "e66aa3d123f804f34afc622b5"
}
on irb I can get all the sub hashes inside example: ['manifest']['files'] and I can only get the url if I expecify which one.. like for example puts file['manifest']['files']['1']['url'] <-- this does work on irb but since I need to get ALL url's this is why I use .each but it gives me a cant convert to string error or similar
#!/usr/bin/env ruby
require 'httparty'
HOST=ARGV[0]
ID=ARGV[1]
VERSION=ARGV[2]
class MyApi
include HTTParty
end
file = MyApi.get("http://#{HOST}/v1/dc/manifest/#{ID}/#{VERSION}")
file.each do |item|
puts item['manifest']['files']['url']
end
not working but I can on IRB do a:
puts item['manifest']['files'][2]['url'] <-- and this will give me the url but with the .each will just complaint about cant convert to string or similar
Try the following:
#!/usr/bin/env ruby
require 'httparty'
(HOST, ID, VERSION) = ARGV
class MyApi
include HTTParty
format :json
end
response = MyApi.get("http://#{HOST}/v1/dc/manifest/#{ID}/#{VERSION}")
puts response.inspect
The addition of the format :json tells HTTParty to parse the response as JSON. Then you'll get a hash you can iterate over properly.
Try:
file['manifest']['files'].each do |item|
puts item['url']
end
What am I doing wrong here? I know the _id is in the database but I get empty result.
#b = coll.find("_id" => "4db2ebee90036f010b000001")
Thanks
Use this:
coll.find(:_id => BSON::ObjectId('4db2ebee90036f010b000001')).each do |data|
puts data.inspect
end
#b will contain a cursor, not the result. You also need to use an object id proper.
You probably want this:
#b = coll.find_one(:_id => BSON::ObjectId('4db2ebee90036f010b000001'))
With Ruby 1.9.3 and mongoid 3.0.19
#coll = Coll.find( hash["_id"] )
or
#coll = Coll.find( "511296d2dfa18f07fa000009" )
find the record. Will only ever be one, _id is the primary key, it can never be double.
I would use something like first which returns a object since you have bigger problems if your primary id is duplicated in your database. The syntax is depending on your mongo gem version this one is for 2.1.0.
your_id = '4db2ebee90036f010b000001'
db = Client.new([ "localhost:27017" ], :database => "db")
coll = db[:testCollection]
res = coll.find(:_id => BSON::ObjectId(your_id)).first
Using, Ruby version 2.3.1p112, mongo (gem) 2.4.2 and BSON (gem) 4.2.2
The following worked for me
client = Mongo::Client.new(['127.0.0.1:3001'], :database=>'dbname')
collection = client[:users]
user = collection.find({_id:'XY3h5R7aJkh5FxFhJ'}).first