Ruby and PostgreSQL text matching - ruby

Question

I am trying to do some text matching in Ruby using PostgreSQL.
This is my code:
require 'active_record'
require 'yaml'
require 'pg'
require 'pry'
$config = '
adapter: postgresql
database: edgar
username: YYYYY
password:
host: 127.0.0.1'
ActiveRecord::Base.establish_connection(YAML::load($config))
class Doc < ActiveRecord::Base; end
class Filing < ActiveRecord::Base; end
#Searching database edgar and table for key words
#install gem install pg_search
class Filing < ActiveRecord:: Base
include PgSearch
end
class Filing < ActiveRecord::Base
pg_search_score:search_eightks,
:against => [:cancer, heart attack]
end
I have a few questions:
How do I search the eightks table of my database?
How do I search for multiple words? I want to see if a document contains the word "cancer" OR "heart attack". It doesn't need both, just one or the other.
This is what the list of relations in the database looks like:
edgar=# \d
List of relations
Schema | Name | Type | Owner
--------+-----------------------+----------+------------------
public | crsp_ccm_lookup | table | YYYY
public | docs | table | YYYY
public | docs_downloaded | table | YYYY
public | docs_id_seq | sequence | YYYY
public | document_types | table | YYYY
public | document_types_id_seq | sequence | YYYY
public | eightks | table | YYYY
public | filings | table | YYYY
public | filings_for_run | table | YYYY
public | filings_id_seq1 | sequence | YYYY
public | indices | table | YYYY
public | indices_id_seq | sequence | YYYY
public | scraper_groups | table | YYYY
public | scraper_groups_id_seq | sequence | YYYY
public | ws | view | YYYY
public | ws_table | table | YYYY
public | z_docs_10_ks | table | YYYY
(17 rows)
Ideally when a text document containing these words is found I would like to COPY it to a new folder.
Any help is greatly appreciated.

Answer 1

A couple of things regarding the code above specifically:
Not quite sure why you're opening the class twice in a row (and also a third time above that)
I believe pg_search_scope (note the p instead of r) is the correct term to use (I don't see pg_search_score in the code for pg_search)
So something like this:
class Filing < ActiveRecord::Base
include PgSearch
pg_search_scope:search_eightks,
:against => [:cancer, heart attack]
end
However, I am wondering if you really want to use full text search here, as that is a reasonably niche use case for Postgres.
For example, if the docs are just of type text, you could run a regex against them to check (or one of the string functions). Now, if they're large documents, or if you wanted to do some more advanced things like ranking the search results (in a somewhat Solr like fashion), then full-text may be worth a look.
What is the overall picture of what you're trying to accomplish here? (Not how, but what/why?) Will also need more info as to what the definitions of the tables involved are, and also what version of Postgres.
Also, not quite sure what you mean by copy it to a new folder. folder doesn't mean anything in Postgres. Are you mixing Postgres terms with terms from your application logic?
Edit in response to comment from OP:
Based on your comments about there being 200,000 documents (which sounds like a fairly small dataset, unless the documents themselves are absolutely enormous), and you wanting to check to see if there are a few keywords contained in them, assuming the documents are text (which it sounds like they are) I would recommend using a Postgres regex.
Assuming, for the sake of an example, that the document text is stored in a column called contents in the docs table, you could do something like this:
SELECT *
FROM docs
WHERE contents ~* 'heart attack|cancer|illness';
ActiveRecord lets you use raw SQL via the connection.execute method, where you would pass the above query as a string.
You can then process them further as desired.
I'm not saying you won't ever need to use full text search for this sort of thing, it's just that I wouldn't recommend starting with that, especially based on the information you've provided regarding your use case and data.
Edit in response to follow-up comment from OP:
The code you said you were using:
require
Select * from eightks where contents ~* 'heart attack | cancer |illness'
end
is not valid Ruby. You would need a begin to go with the end, instead of the require, and the SQL needs to be a string which is passed to your Active Record object's connection.execute method. There doesn't appear to be a connection object listed in your code above. Possibly, it would be an instance of the Filing class you're defining.
Also, the column contents will only work if you actually have that in your table definition. That was just for the example -- you'll need to adapt it to fit your specific table.
If you have additional issues getting the raw SQL to work in your scenario, it should be split off into a separate question, as this one is getting pretty long and veering into separate (although related) issues. You can link this question for additional context.