I am a newbie in Ruby coming from web development with mainly PHP/SQL. I was thinking about how I store preferences in my application. For instance, if I want to store a path as default_path and have that set also when the user restarts the application.
In the web world one would probably store this in a database or XML. Database seems overkill for a standalone application. But I am unsure wheter XML/YAML/Other-Write-Format is the way to go. And if so, where should I store these preferences? Should they be, for instance on a Mac, in ~/Library/MyAppName?
I like using YAML because it's very easily read/written by a lot of languages, making it possible for several apps to share the same configuration info. It's a well documented standard so there should be very little chance of data falling into a hole with it.
Also, because it's easy for a human to understand, and doesn't take any special tools to change, it works nicely for any data that might occasionally change in an app, either for fine-tuning or to enable special behaviors.
A little creative coding on your part that periodically checks the last modified time of the YAML file could make it so your app would modify its behavior on the fly as the prefs file is tweaked. I had a big app I didn't want to shut down for changes and set up that behavior. It ran three weeks straight, and I tweaked its operating parameters via its config file. It would read the file every minute and inherit any changes to its parameters on the fly.
Databases are a good way to store parameters/preferences if it's a centralized server or web-based app. For something distributed that runs on individual machines it makes no sense.
Ruby gives you another method for storing data called Marshaling. This will let you store a class/object to a file and reconstitute it later. If all of your user preferences are stored in a single object (or you can create an object which can hold all of the data that you need), it may be easiest to marshal the data instead of writing import/export routines to a text-based format or trying to pull in an additional library or gem.
As to where on the disk to store the data, that's up to you. Most platforms have a standard location for storing application data based on whether it's available to a single user or all users. It's usually safest to follow the common practice on your target platform of choice.
Update: The simplest example of marshaling would probably be this: Say that you have a class called UserPrefs that you use to store all of your user preferences. You can use the following code to store the preferences data into a file:
my_prefs = UserPrefs.new
# ... Fill in the 'my_prefs' object with the user's preferences, etc ...
# Store the object into a file
File.open("user_prefs.data", "wb") do |file|
Marshal.dump(my_prefs, file)
end
The next time that you load the application, you can restore those preferences using the following:
# Load prefs from file
my_prefs = nil
File.open("user_prefs.data", "rb") {|f| my_prefs = Marshal.load(f)}
At this point, the my_prefs object should be exactly the same as it was when the marshaling code was originally run. This essentially lets you take a 'snaphot' of an object at one point in time (say, when your program shuts down) and restore it later (say, when your program loads). Internally, all of the data in the structure is encoded into a single string and that string is what is stored to disk; the Marshal module simply takes care of the encoding and decoding for you.
Here is another example of using marshaling to store and retrieve data.
The default encode/decode routines built into the Marshal module are usually sufficient for most data-storing classes. Particularly complex classes may have problems, and if that is the case then you can define your own encode and decode methods (the first link includes an example of defining custom methods).
Some types of data, however, cannot be marshaled (things like handles to open files, Proc objects, etc) since they don't normally persist across Ruby sessions. If you are needing to marshal a class that includes members like this that Marshal doesn't like, you can use custom encode/decode functions to marshal the rest of the class and omit the problematic members.
I saw some applications using ruby gconf2
Related
In a Mac OS X app (Cocoa), I'm copying some images from my app to others using a NSDraggingSession. The NSDraggingItem makes use of an object that implements the protocol NSPasteboardItemDataProvider, to provide the data when the user drops it.
As I'm dealing with images, the types involved are: NSPasteboardTypePNG, kPasteboardTypeFileURLPromise, kUTTypeFileURL, com.adobe.photoshop-image and public.svg-image. These images are in a remote location, so before I can provide them to the pasteboard, I have to download them from the Internet.
I implement the method - pasteboard(pasteboard:item:provideDataForType:) doing something like this:
If the type requested is kPasteboardTypeFileURLPromise, I get the paste location and build and set in the pasteboard the URL string with the location where the file is supposed to be written in the future.
If the type requested is kUTTypeFileURL, I download the file, specify a temporal location and write the downloaded file to that location. Then, I set in the pasteboard the URL string of the location.
If the type requested is one of the others, I download the file and set the plain NSData in the pasteboard.
All these operations are performed on the main thread, producing some lags that I want to get rid of.
I've tried to perform these operations on a background thread, and come back to the main thread to set the final data in the pasteboard, but this doesn't work because the method finishes before.
Does anyone know a way to achieve it?
Promises of pasteboard types are usually meant to be an alternative format of data that you already have, where you want to avoid the expense in time and memory of converting before it's necessary. I don't think it's really appropriate to use it to defer downloading any of the data, at all. For one thing, the download could fail when it's ultimately requested. For another, it could take an arbitrarily long time, as you're struggling with now.
So, I think you should download the data in advance. Either keep it in memory or save it to a temporary file. Use promised types, if appropriate, to deliver it in different forms, but have it on hand in advance.
Which methods and system calls should I hook into, so I can replace 'how' an OS X app (the target) reads and writes to/from the HD?.
How may I determine that list of functions or system calls?.
Adding more context:
This is a final project and I'm looking for advise. The goal is to alter the behavior of an OS X app, adding it data encryption and decryption capabilities.
Which tools could I use to achieve my goal, and why?
For instance, assume the target app is Text Edit. Instead of saving "hello world" as plain text in a .txt file in the HD, it'll save: "ifmmnXxnpme". Opening the file will show the original text.
I think its better to get more realistic or at least conscious of what you want to do.
The lowest level in software is a kernel module on top of the storage modules, that "encrypt" the data.
In Windows you can stack drivers, so conceptually you simply intercept the call for a read/write, edit it and pass it down the driver stack.
Under BSD there is an equivalent mechanism surely, but I don't know precisely what it is.
I don't think you want to dig into kernel programming.
At the lowest level from an user space application point of view, there are the system calls.
The system calls used to write and read are respectively the number 3 and 4 (see here), in BSD derived OS, like OS X, they becomes 2000003h and 2000004h (see here).
This IA32e specific since you are using Apple computers.
Files can be read/written by memory mapping them, so you would need to hijack the system call sys_mmap too.
This is more complex as you need to detect page faults or any mechanism used to implement file mapping.
To hijack system calls you need a kernel module again.
The next upper level of abstraction is the runtime, that probably is the Obj C runtime (up to data, Swift still use Obj C runtime AFAIK).
An Obj C application use the Cocoa Framework and can read/write to file with calls like [NSData dataWithContentOfFile: myFileName] or [myData writeToFile: myFileName atomically:myAtomicalBehavior].
There are plenty of Cocoa methods that write to or read from file, but internally the framework will use few methods from the Obj C runtime.
I'm not an expert of the internals of Cocoa, so you need to take a debugger and look what the invocation chain is.
Once you have found the "low level" methods that read or write to files you can use method swizzling.
If the target app load your code as part of a library, this is really simple, otherwise you need more clever techniques (like infecting or manipulating the memory of the other process directly). You can google around for more info.
Again to be honest this is still a lot of work, although manageable.
You may consider to simply hijack a limited set of Cocoa methods, for example the writeToFile of NSData or similar for NSString and consider the project a work in progress demo.
A similar question has been asked and answered here.
So. Before I get singleton pattern hate on this message hear me out. I'd love to hear ideas. I'm making a program that I think I need to use core data for, because later I want the status of some variables to be easily accessible from OS X, and multiple iOS devices.
What I'm making is an OS X program that will control phidgets (phidgets.com) to control and listen for status changes in real world objects. Example: whether a motor is turned on or not. Turn a motor on and off. Turn on status lights, etc.
I originally thought I'd just make global variables that I change, poll and manipulate in order to have a central status board for the logic of the program to work off of. But, because of the engineering that is put into core data every year by apple, I am assuming making this work with core data will allow me to more easily have options to sync this later with iOS devices that could control or monitor the said status' remotely.
Is there a nifty way you can imagine to:
-startup the program, confirm there is only one entity of type "SystemStatus", if there isn't one, make one. is there is one, we continue and are able to let the program update it's attributes with status of the real world objects it's controlling.
using core data was something I thought of also, because it will allow me a place to persist stored history of data gathered too. Example: motor bearing temperature over time.
If you ensure that access to this object is done through your API, Core Data becomes an implementation detail behind the getter method of the singleton object. There are no facilities in Core Data to tell it to create only one object, but if you ensure access to the object is done through a wrapper of your own, you can fetch it on demand and if it doesn't exist, you can insert it, save, and pass it to the caller.
An important thing to consider when using Core Data objects is multithreading. Passing the same object to multiple threads is very error-prone and requires locking mechanisms (or use of Apple's block-based API). This is not very straightforward for what you describe. Consider either a wrapper object which uses Core Data objects internally (wraps access to properties in block-based API) or using a different approach than Core Data.
I want to pass an instance of an object between two Ruby processes. Specifically, I want to pass an instance of a Selenium WebDriver from one process to another process. The reason I want to do this is because it takes a lot of time for Ruby to create this object, but I want it to be used by the other process.
I've found some related questions here and here that seem to point towards using DRb, but I've been unable to find any useful examples or sample code.
Is there a tool other than DRb that I should be using? Does anyone have an example similar to this that I could copy from?
It looks like you're going to have to use DRb, although the documentation for it seems to be lacking. There is however an interesting article here. You might also want to consider purchasing The dRuby Book by Masatoshi Seki to get a better idea of how to do this effectively.
Another option to investigate if you are not looking at simultaneous access, but you just want to send the object from one process to another, is to serialize (that is, encode in a way that Ruby can read) the object with YAML (for a human readable file) or Marshall (for a binary encoded file) and send it using a pipe. This was mentioned in another answer that has since been deleted.
Note that either of these solutions require modifying the Selenium code heavily since the objects you want to manipulate neither support copying, nor simultaneous access natively.
TL;DR
Most queue or distributed processes are going to require some sort of serialization to work properly. If you want to pass objects rather than messages, then this will a limiting factor in how you approach the problem.
DRb
I don't know if you can marshal a WebDriver object. If you can't, then DRb may be a good choice for your distributed Ruby programs because it supports DRbObject references for things that can't be marshaled. There are some examples provided in the DRb documentation.
Selenium Wire Protocol
Depending on what you're really trying to do, it may be worth taking a closer look at using the remote bindings for the Remote WebDriver client/server, or Selenium's JSON Wire Protocol as an alternative to passing objects between processes.
Other Alternatives: Fixtures, Factories, Stubs, and Mocks
Whether or not these work in your specific case will depend a lot on why you want to pass objects instead of simply driving the remote server. If it's largely an issue of how long it takes to build your object, then the serialization/de-serialization cycle may not necessarily be faster in all cases.
You might want to revisit why your object is so slow to create. If gathering and processing the data for it is what's taking too long, you can use some sort of test fixture or factory to trim that time, either by using a smaller set of fixed data, or using a pre-serialized object that's optimized for speed.
You might also consider whether you actually need real data or objects for your test at all. In many cases, you can speed up your tests a lot by stubbing methods or creating mock objects that will return the values you need for your integration tests without needing to perform expensive calculations or long-running operations.
There are certainly cases where you need to drive the full stack and perform acceptance tests on real data. Even then, you may be able to devise a set of fixture data that will take less time or memory to process. It's certainly worth at least thinking about.
ETL is pretty common-place. Data is out there somewhere so you go get it. After you get it, it's probably in a weird format so you transform it into something and then load it somewhere. The only problem I see with this method is you have to write the transform rules. Of course, I can't think of anything better. I supposed you could load whatever you get into a blob (sql) or into a object/document (non-sql) but then I think you're just delaying the parsing. Eventually you'll have to parse it into something structured (assuming you want to). So is there anything better? Does it have a name? Does this problem have a name?
Example
Ok, let me give you an example. I've got a printer, an ATM and a voicemail system. They're all network enabled or I can give you connectivity. How would you collect the state from all these devices? For example, the printer dumps a text file when you type status over port 9000:
> status
===============
has_paper:true
jobs:0
ink:low
The ATM has a CLI after you connect on port whatever and you can type individual commands to get different values:
maint-mode> GET BILLS_1
[$1 bills]: 7
maint-mode> GET BILLS_5
[$5 bills]: 2
etc ...
The voicemail system requires certain key sequences to get any kind of information over a network port:
telnet> 7,9*
0 new messages
telnet> 7,0*
2 total messages
My thoughts
Printer - So this is pretty straight-forward. You can just capture everything after sending "status", split on lines and then split on colons or something. Pretty easy. It's almost like getting a crap-formatted result from a web service or something. I could avoid parsing and just dump the whole conversation from port 9000. But eventually I'll want to get rid of that equal signs line. It doesn't really mean anything.
ATM - So this is a bit more of a pain because it's interactive. Now I'm approaching expect or a protocol territory. It'd be better if they had a service that I could query these values but that's out of scope for this post. So I write a client that gets all the values. But now if I want to collect all the data, I have to define what all the questions are. For example, I know that the ATM has more bills than $1 and $5 so I'd have a complete list like "BILLS_1 BILLS_5 BILLS_10 BILLS_20". If I ask all the questions then I have an inventory of the ATM machine. Of course, I still have to parse out the results and clean up the text if I wanted to figure out how much money is left in the ATM machine. So I could parse the results and figure out the total at data collection time or just store it raw and make sense of it later.
Voicemail - This is similar to the ATM machine where it's interactive. It's just a bit weirder because the key sequences/commands aren't "get key". But essentially it's the same problem and solution.
Future Proof
Now what if I was going to give you an unknown device? Like a refrigerator. Or a toaster. Or anything? You'd have to write "connectors" ahead of time or write a parser afterwards against some raw field you stored earlier. Maybe in the case of these very limited examples there's no alternative. There's no way to future-proof. You just have to understand the new device and parse it at collection or parse it after the fact (your stored blob/object/document).
I was thinking that all these systems are text driven so maybe you could create a line iterator type abstraction layer that simply requires the device to split out lines. Then you could have a text processing piece that parses based on rules. For the ATM device, you'd have to write something that "speaks ATM" and turns it into lines which the iterator would then take care of. At this point, hopefully you'd be able to say "I can handle anything that has lines of text".
But then what will you call these rules for parsing the text? "Printer rules" might as well be called "printer parser" which is the same to me as "printer transform". Is there a better term for all of this?
I apologize for this question being so open ended. :)
When your sources of information are as disparate as what you illustrate then you have no choice but to implement the Transform in order to bring the items into a common data repository. Usually your data sources won't be this extreme, the data will all be related in some way but you may be retrieving it from different sources (some might come from a nicely structured database, some more might come from an Excel or XML or text file, some more might come from a web service call, etc).
When coding up a custom ETL application, a common pattern that is used is the Provider model, this enables you to write a whole bunch of custom providers to load/query and then transform the data. All the providers will implement a common interface with some relatively common function definitions (for example QueryData(), TransformData()), but the implementation of those methods will be wildly different depending on the data source being dealt with - the interface just gives a common way to deal with all the different providers. You can then use an XML configuration file to dictate which providers to run and any other initial settings they may require. Tools like SSIS abstract this stuff away for you by giving you a nice visual designer, but you can still get down and dirty and write your own code which it calls.
Now what if I was going to give you an unknown device? Like a refrigerator. Or a toaster.
No problem, i would just write a new provider, which can sit in its very own assembly (dll), so it can be shipped (or modified, upgraded, etc) in isolation to any other providers i already have. Or if i was using SSIS then i would write a new DTS package.
I was thinking that all these systems are text driven so maybe you could create a line iterator type abstraction layer ... Then you could have a text processing piece that parses based on rules.
Absolutely - you can have a base class containing common functionality which several different providers can implement, and each provider can use its own set of rules which could be coded into it or they can be contained in an external configuration file.
So I could parse the results and figure out the total at data collection time or just store it raw and make sense of it later.
Use whichever approach makes sense for the data you are grabbing. It is also quite common for an ETL process to dump its data into a staging area (like some staging tables in a database) while the data is all being aggregated and accumulated, and then further process it to link related data and perform calculations. In the case of your ATM it may not be necessary to calculate a cash balance at ETL time because you can easily calculate it at any time in the future.