I've got a lot of lat / lon points in a csv file, I've created a table which has a point in the 4326 projection (table postcode, field location)
I'm building data like this:-
factory = ::RGeo::Cartesian.preferred_factory(:has_z_coordinate => false)
p = factory.point(data_hash[:latitude], data_hash[:longitude])
and storing p in the location field.
The issue then is that I want to find "near" records to a given point.
I've seen some promising code at:-
https://github.com/rgeo/activerecord-postgis-adapter/blob/master/test/spatial_queries_test.rb
so I wrote the following:-
factory = ::RGeo::Cartesian.preferred_factory(:has_z_coordinate => false)
p = factory.point(53.7492, 1.6023)
res = Postcode.where(Postcode.arel_table[:location].st_distance(p).lt(1000));
res.each do |single|
puts single.postcode
end
But I'm getting exceptions (unsupported: RGeo::Cartesian::PointImpl)
I assume I'm needing to do some converting or something, any pointers appreciated!
I think your problem lies in the factory you use. Try to generate point from a spherical factory:
p = RGeo::Geographic.spherical_factory(srid: 4326).point(53.7492, 1.6023)
Also check rails logs to see the output query and run it manually in PG. Make sure that the query runs without problems.
Related
I am new to Ruby and Mongo and am working with twitter data. I'm using Ruby 1.9.3 and Mongo gems.
I am querying bulk data out of Mongo, filtering out some documents, processing the remaining documents (inserting new fields) and then writing new documents into Mongo.
The code below is working but runs relatively slow as I loop through using .each and then insert new documents into Mongo one at a time.
My Question: How can this be structured to process and insert in bulk?
cursor = raw.find({'user.screen_name' => users[cur], 'entities.urls' => []},{:fields => params})
cursor.each do |r|
if r['lang'] == "en"
score = r['retweet_count'] + r['favorite_count']
timestamp = Time.now.strftime("%d/%m/%Y %H:%M")
#Commit to Mongo
#document = {:id => r['id'],
:id_str => r['id_str'],
:retweet_count => r['retweet_count'],
:favorite_count => r['favorite_count'],
:score => score,
:created_at => r['created_at'],
:timestamp => timestamp,
:user => [{:id => r['user']['id'],
:id_str => r['user']['id_str'],
:screen_name => r['user']['screen_name'],
}
]
}
#collection.save(#document)
end #end.if
end #end.each
Any help is greatly appreciated.
In your case there is no way to make this much faster. One thing you could do is retrieve the documents in bulks, processing them and the reinserting them in bulks, but it would still be slow.
To speed this up you need to do all the processing server side, where the data already exist.
You should either use the aggregate framework of mongodb if the result document does not exceed 16mb or for more flexibility but slower execution (much faster than the potential your solution has) you can use the MapReduce framework of mongodb
What exactly are you doing? Why not going pure ruby or pure mongo (well that's ruby too) ? and Why do you really need to load every single attribute?
What I've understood from your code is you actually create a completely new document, and I think that's wrong.
You can do that with this in ruby side:
cursor = YourModel.find(params)
cursor.each do |r|
if r.lang == "en"
r.score = r.retweet_count + r.favorite_count
r.timestamp = Time.now.strftime("%d/%m/%Y %H:%M")
r.save
end #end.if
end #end.each
And ofcourse you can import include Mongoid::Timestamps in your model and it handles your created_at, and updated_at attribute (it creates them itself)
in mongoid it's a little harder
first you get your collection with use my_db then the next code will generate what you want
db.models.find({something: your_param}).forEach(function(doc){
doc.score = doc.retweet_count + doc.favorite_count
doc.timestamp = new Timestamp()
db.models.save(doc)
}
);
I don't know what was your parameters, but it's easy to create them, and also mongoid really do lazy loading, so if you don't try to use an attribute, it won't load that. You can actually save a lot of time not using every attribute.
And these methods, change the existing document, and won't create another one.
So I have 2 tables actor and actor2role. The latter is a lookup (junction) table to relate actor, role and dvd. I need to create a query with aliases, so I have this method:
def self.remove_duplicate_by_id(id)
offendingActor = self.find(id).actor # get the actor's name
ids = self.find_by_sql("SELECT MIN(id) AS minId, MAX(id) AS maxId, actor FROM `dvd_actor` WHERE actor = '#{offendingActor}'")
rolesForOffender = ids.actor2role # throws error
end
The problem is that ids is not an ActiveRecord object so I can't use the actor2role method (which is a relationship I've established between the 2 tables in Rails and works when you do something like Actor.first.actor2role.
so the questions is: Am I doomed to do this manually and then issue another sql query to recreate what the actor2role method would accomplish or is there some way to do this with Rails objects?
I'd really like to do it all natively if possible because I also have to issue these queries:
UPDATE dvd_actor2role SET actorid = $d->minId WHERE actorId = $d->maxId");
DELETE FROM dvd_actor2role WHERE actorId = $d->maxId LIMIT 1");
Is this even possible?
In the end I went with this which seems to do the trick. If anyone can spot any code that could be optimized, or something inherently wrong (and feels like chiming in) please feel free to comment.
actorObject = self.find_by_id(id) # get the object because we need it below for other queries
offendingActor = actorObject.actor
ids = self.select("MIN(id) AS minId, MAX(id) AS maxId, id, actor").find_by_actor(offendingActor)
rolesForOffender = actorObject.actor2role
rolesForOffender.each do |r|
# first find out if the relationship already exists or we get a SQL error for the foreign key relationship.
exists = Actor2role.where("actorId = ? AND roleId = ?", ids.minId, r.roleId)
if exists.nil?
Actor2role.update_all("actorId = #{ids.minId}, actorId = #{ids.maxId}")
end
end
self.destroy(ids.maxId) # delete this guy in actor table
end
How would I modify the following code:
var result = from p in Cache.Model.Products
from f in p.Flavours
where f.FlavourID == "012541-5-5-5-651"
select p;
So that f.FlavourID is supplied a range of ID's as a supposed to just one value as shown in the above example?
Given the following ERD Model:
Products* => ProdCombinations <= *Flavours
ProdCombinations is a junction/link table and simply has one composite key in there.
Of the top of my head
string [] ids = new[]{"012541-5-5-5-651", "012541-5-5-5-652", "012541-5-5-5-653"};
var result = from p in Cache.Model.Products
from f in p.Flavours
where ids.Contains(f.FlavourID)
select p;
There are some limitations, but an array of ids has worked for me before. I've only actually tried with SQL Server backend, and my IDs were integers.
As I understand it, Linq needs to translate your query into SQL, and it's only possible sometimes. For example it's not possible with IEnumerable<SomeClass>, which produces a runtime error, but possible with a collection of simple types.
I have records with a 'resource' field which can contain multiple resources. When I return this data, I need to iterate over this field and return an individual record for each value in the field. I am currently using sinatra and am able to interate over the fields okay, but I am having difficulty replacing the field in the json array.
For example
event: Name
resources: resourceA, resourceB, resourceC
This record needs to be returned as 3 uniqe records/events with only one resource per record.
With the code listed below, I am getting three records, but all three records are coming back with the same resource value (resourceC)
Here is my code
docs = #db.view('lab/events', :startkey => params[:startDate], :endkey => endSearch)['rows']
rows = Array.new
docs.each do |doc|
resources = doc['value']['resources'].split(",")
resources.each do |r|
doc['value']['resources'] = r
rows.push(doc['value'])
end
end
Any help is greatly appreciated.
Thanks
Chris
if you use the ruby gem "json" you can convert the json string to a hash
require 'json'
converted_hash = JSON(json_string).to_hash
This should be much easier to manage.
You can then turn the hash to a JSON string:
new_json_string = converted_hash.to_json
Basically what is happening is ruby is seeing all three records as the same record so as the hash value is updated on one record, it impacts all other records that were created from the same doc. To get around this, I acutally needed to create a duplicate record each time through and modify it's value.
docs = #db.view('lab/events', :startkey => params[:startDate], :endkey => endSearch)['rows']
rows = Array.new
docs.each do |doc|
resources = doc['value']['resources'].split(",")
resources.each do |r|
newDoc = doc['value'].dup # <= create a duplicate record and update the value
newDoc["resources"] = r
rows.push(newDoc)
end
end
I created a strongly-typed dataset in the dataset designer. The DataSet has a Table called FocusOffsetsTable and that table has four colums; SerialNumber, Filter, Wheel and Offset. I use the ReadXml() method of the DataSet class to load the strongly typed data from the xml file into the dataset. That seems to be working just fine.
I am trying to use a LINQ expression to try to get a Single row from this table but I can't seem to get the syntax correct. I want to use the Single() or SingleOrDefault() method to get just one row of data at a time but I am not sure how.
I have tried this FocusOffsets.FocusOffsetsTableRow x = FocusOffsetData.FocusOffsetsTable. but the Single() method is not available here. I also tried this...
FocusOffsets.FocusOffsetsTableRow x = (from offset in FocusOffsetData.FocusOffsetsTable
where offset.SerialNumber == mydevice.SerialNumber
where offset.Wheel == WheelID
where offset.Filter == FilterNum
select offset).Single();
but the Single method is not available here either.
I have done this before with tables in a SQL database before but this is my first time using a dataset from the dataset designer.
Have you added a using statement for System.Linq and included a reference to System.Data.DataSetExtensions. I think (but can't confirm since I'm on my Mac), that you ought to be able to do:
var x = FocusOffsetData.FocusOffsetsTable
.AsEnumerable()
.SingleOrDefault( o => o.SerialNumber == mydevice.SerialNumber
&& o.Wheel = WheelID
&& o.Filter = FilterNum );