I'm attempting to write a simple Ruby/Nokogiri scraper to get event information from multiple pages and then output it to a CSV that is attached to an email sent out weekly.
I have completed the scraping components and the CSV component and it's working perfectly. However, I now realize that I need to know when new events are added, which means I need some sort of database. Ideally I would just store this locally.
I've dabbled a bit with using the ruby gem 'sequel', but the data does not seem to persist beyond the running of the program. Do I need to download some database software to work with 'sequel'? Also I'm not using the Rails framework, just Ruby.
Any and all guidance is deeply appreciated!
I'm guessing you did Sequel.sqlite, as in the first example in the Sequel README, which creates an in-memory SQLite database. To create a database in your filesystem instead of memory, just pass it a path, e.g.:
Sequel.sqlite("./my-database.db")
This is, of course, assuming that you have the sqlite3 gem installed. If the given file doesn't exist, it will be created.
This is covered in the Sequel docs.
Related
I'm running a very basic Sinatra server, which simply shows a Chartkick graph of some data I have through the Sequel gem. I'm noticing that the data on the chart doesn't seem to update unless I quit the Sinatra server script and rerun it. I don't really understand how that would be possible... the only non-normal thing option I'm using when reading my database using Sequel is the read-only option.. would that cause this?
It turns out, from reading another post on here:
First, by default, multiple processes can have the same SQLite
database open at the same time, and several read accesses can be
satisfied in parallel.
In case of writing, a single write to the database locks the database
for a short time, nothing, even reading, can access the database file
at all.
Beginning with version 3.7.0, a new “Write Ahead Logging” (WAL) option
is available, in which reading and writing can proceed concurrently.
By default, WAL is not enabled. To turn WAL on, refer to the SQLite
documentation.
I currently have script A, which maintains a connection to the DB file and writes to it regularly, and script B, which is my Sinatra server that reads information from that DB file. I worked around this issue by using a block connection in my Sinatra script. I don't know how to turn on WAL with Sequel though...
Im looking for a way to store data in selenium for use in future tests.
Im using jenkins, maven + selenium and testng.
How can i store some data, lets say i want to run test, get some data from website (weather forecast). Store it somewhere and next day run test to check if forecast match todays weather.
I can store it in txt file, and parse by regex but im sure there is better way to do it?
You have to consider what "selenium webdriver" is in this context. It is a Java app. It "exists" only when it is running. Once the run stops, it is purged from memory, including all data it held. If you are using JUnit or TestNG (as you specified), then this data is purged even more frequently: after every the class.
To accomplish what you are asking, you will need something external to your tests. You can certainly utilize a txt file as you suggested. A spreadsheet might do for your purposes. Most applications utilize an entire database.
You also mentioned Jenkins. This will make the external storage a more interesting problem, as Jenkins often purges the current working directory before each run.
I did something silly and made hundreds of records in Development environment in CloudKit. A previous thread mentioned that the records could be downloaded into a file and re-uploaded to the Production environment. Is there any other way I could do this, and if not, how would I go about downloading in records and storing it into a file?
Thanks in advance!
There is no option do do it in one run. You need one app that is connected to the development environment for reading your records. Then if you want to write to the production environment, you can only do that by re-signing your app. So indeed you first need to download all data, store them somewhere, and then writing them back to your production database.
Since the CKRecord complies to the NSCoding protocol you can write the results of your query directly to a file using:
NSKeyedArchiver.archiveRootObject(records, toFile: filePath)
Then if you want to read that file you can use:
result = NSKeyedUnarchiver.unarchiveObjectWithFile(filePath)
Time ago I discovered a ruby gem that allows you to create database dumps with particular rules.
Inside a file you defined wich tables to dump, which records to skip and which fields to scramble in a nifty ruby DSL.
I can't remember the name of the tool, do you know what I'm talking about?
After searching for an hour I finally found it.
It's called ocelot, here is the homepage: http://exussum.heroku.com/projects/ocelot
I'm pretty sure the answer is "no" but I thought I'd check.
Background:
I have some legacy data in Access, need to get it into MySQL, which will be the DB server for a Ruby application that uses this legacy data.
Data has to be processed and transformed. Access and MySQL schemas are totally different. I want to write a rake task in Ruby to do the migration.
I'm planning to use the techniques outlined in this blog post: Using Ruby and ADO to Work with Access Databases. But I could use a different technique if it solves the problem.
I'm comfortable working on Unix-like computers, such as Macs. I avoid working in Windows because it fills me with deep existential horror.
Is there a practical way that I can write and run my rake task on my Mac and have it reach across the network to the grunting Mordor that is my Windows box and delicately pluck the data out like a team of commandos rescuing a group of hostages? Or do I have to just write this and run it on Windows?
Why don't you export it from MS-Access into Excel or CSV files and then import it into a separate MySQL database? Then you can rake the new one to your heart's content.
Mac ODBC drivers that open Access databases are available for about $30.00
http://www.actualtechnologies.com/product_access.php is one. I just run access inside vmware on my mac and expore to csv/excel as CodeSlave mentioned.
ODBC might be handy in case you want to use the access database to do a more direct transfer.
Hope that helps.
I had a similar issue where I wanted to use ruby with sql server. The best solution I found was using jruby with the java jdbc drivers. I'm guessing this will work with access as well, but I don't know anything about access