Its my understanding that when I make a gem I'm compiling my ruby code into some form of executable, right? Does this mean that unless someone used introspection techniques (which is an acceptable risk to me), my source code is secure?
A gem is not a compiled executable. It's not compiled at all. Ruby is interpreted. Creating a gem just bundles the necessary files together, much like a zip file or tar archive.
If you want your gem secure you should keep it out of rubygems.org. You can set up your own private gem server or you can just include your gem in projects that need it.
While it is possible to compile Ruby code into an executable or shared library using Ruby's C API, that has nothing to do with gems.
A gem is just a collection of Ruby code (which could be regular scripts or compiled libraries) in a nice package for use with the rubygems package manager. It makes no effort to hide/protect the code.
I think
gem unpack
can extract your code. Never tried to do it and see if it's "human-readable" but you can try it before publishing your gem ;)
Its my understanding that when I make a gem I'm compiling my ruby code into some form of executable, right?
No. It's just a zipfile with some metadata. The contents of the zipfile are exactly what you put into them.
Does this mean that unless someone used introspection techniques (which is an acceptable risk to me), my source code is secure?
This depends on what you mean by "secure", but is completely orthogonal to RubyGems.
If you mean that "it can't be stolen", then that is already guaranteed for you by copyright law. Unless you live in a really weird country, software is protected by copyright automatically from the moment you write it.
If you mean "cannot be reverse engineered", then that is impossible. If you want people to be able to run your program, then you must give it to them in a format that can be understood by the CPU. Humans are much smarter than computers, so, if the program can be understood by the CPU, then it can also be understood by a human.
There are two common ways around this, which I will call the "Nintendo way" and the "Google way".
The Nintendo way is to give the user the CPU as well as the program, therefore, the user's CPU doesn't have to understand it. However, that model is still flawed. As long as you give the user something, he can figure it out. In the end, it's all just maths and physics, which can be understood. And users are pretty clever. Note that, for example, most game consoles were not cracked by evil crime syndicates trying to steal the code or pirating games, no, they were cracked by students wanting to run Linux or BSD on their hardware.
The Google way is to give the user nothing. You type something in the search box, Google sends you back the results, but at no time does the software leave Google's datacenter.
I have a custom build and deployment script which work over SSH and deploy to servers (on running MacOS). The bash script does a lot of simple things like copying files, backing up the old ones and applying the correct SQL scripts for a forward moving database. But there are some advanced things like starting a remote SQL upgrade procedure which can be disconnected from and once the deployment script is started again it only goes forward if the SQL script has been applied completely (in short there is some flow control happening and bash is not really ideal for such stuff)
The script is already huge and is a mess since bash is not meant for such kind of detailed logic. Can you recommend some tools, libraries which would make things easier.
For what you tell us, I think you need a deployment tool, rather than a configuration management tool.
To simplify, I'll distinguish the two like this:
A deployment tool is a 'push' tool: When you press the button, the required actions are run to make the deployment. It's a one-step process (it can have multiple actions, but it's launched once).
A configuration management tool is usually a 'pull' tool, where your servers periodically check if their configuration is exactly as the CM server tells them to be - and apply changes, if needed. You configure your servers once, and after that the system assures that all is as it should be. It is also a great tool to easily clone systems.
For deployment tools, I personally know Fabric, a great Python tool. But there is also Capistrano in the Ruby world. I don't know of any others.
For CM tools, Puppet and Chef seem to be the preferred choice of people nowadays. Cfengine is an older tool, which had some problems (I don't know if that has changed).
Here are my recommendations:
Puppet
Chef
cfengine
These are all free (as in beer) and allow you to do what you're wanting. They will require you to adapt your current bash script into modules to fit their design/framework. It's a bit of work, but in the long run it tends to be better since the frameworks take care of error checking, converging configurations and a lot of other things you'd have to manually insert into your own code were you doing this yourself.
I've also used Opsware previously for this sort of thing, but that costs a fair bit of cash and, for what you're trying to do, does not offer significantly more benefit.
In some cases moving from a bash-script to an complete solution is not as straightforward as many cloudservices claim.
With 'dont try new things when your on a deadline' in mind:
it could also be a good timing to refactor your bashscripts.
I have done automated, repeatable deployments in the past using PaaS or just using GIT/SVN hooks using deployogi (which is bash) : https://github.com/coderofsalvation/deployogi
I understand your situation, but Im not sure whether its fair to say that the bash-language implies 'a mess' and 'complex'.
Every language allows to hide complexity no?
I guess code (in whatever language) gets overly complex when time does not allow us to refactor :)
PaaS is great. But always needed? I think not.
Locked. This question and its answers are locked because the question is off-topic but has historical significance. It is not currently accepting new answers or interactions.
Sometimes it feels that my company is the only company in the world using Ruby but not Ruby on Rails, to the point that Rails has almost become synonymous with Ruby.
I'm sure this isn't really true, but it'd be fun to hear some stories about non-Rails Ruby usage out there.
One of the huge benefits of Ruby is the ability to create DSLs very easily. Ruby allows you to create "business rules" in a natural language way that is usually easy enough for a business analyst to use. Many Ruby apps outside of web development exist for this purpose.
I highly recommend Googling "ruby dsl" for some excellent reading, but I would like to leave you with one post in particular. Russ Olsen wrote a two part blog post on DSLs. I saw him give a presentation on DSLs and it was very good. I highly recommend reading these posts.
I also found this excellent presentation on Ruby DSLs by Obie Fernandez. Highly recommended reading!
I use Ruby extensively in my work, and none of it is Rails (or even web) based.
My domain is usually client-side Windows applications (wxRuby GUI) and scripts, automating Excel, Internet Explorer, SQL Server queries and report generation (win32ole COM automation). I also use the sqlite, pdf-writer, and gruff libraries for various data munging and graph generation tasks.
Rails' success has been great for Ruby, but I agree that Rails has received so much attention that Ruby's value beyond the web is often overlooked.
We are mainly a C++ shop, but we've found several areas where Ruby has proven quite useful. Here are a few:
Code Generation - Built several DSLs to generate C++/Java/C# code from single input files
Build Support
scripts to generate Makefiles for unix from Visual Studio Project Files
scripts for building projects and formatting the output for Cruise Control
scripts for running our unit tests and formatting the output for Cruise Control
scripts for manipulating Visual Studio projects and solutions from the command line
Integration Tests - We can crank out tests much quicker and cleaner using Ruby than C++
QA's entire testing suite is written in Ruby
Ruby is basically my go to tool for where it makes sense. And it makes sense in a lot of places.
Google Sketchup uses Ruby as an embedded scripting language. You can use it to perform all sorts of 3d modeling and import/export tasks. The scripting works with the free version and there's even decent documentation.
Ruby with a homebrew extension written in C++ does all the heavy pixel pushing for my photography processing. I was using Python+numpy but when doing artsy stuff, Ruby is just more fun. Also the relative lack of, or lesser maturity of, good image processing libraries makes me feel less like i'm reinventing wheels. I am clueless about Rails, other than i've heard of it, have a fuzzy idea what it is, and actually have a book on it (unopened)
We use Watir (Ruby library) to test our .net web application.
Check out Shoes, a simple API for building GUIs in Ruby aimed at novice programmers.
Or you could use Ruby to make music ala Giles Bowkett's Archaeopteryx. This presentation by Giles about Archaeopteryx is one of the best presentations ever. I highly recommend it.
RubyCocoa and MacRuby. Possible to make full Cocoa-based GUI apps without Rails. And then you get to use Interface Builder, too.
I worked on a museum project last year that used a lot of Ruby. (http://http://ourspace.tepapa.com/home)
The part that I spent most of my time on was an interactive floor map. The Map on the floor has sensors so when people walk on it lights are triggered and displays in the wall show images or videos and audio tracks are played.
All the control code for this part of the exhibit is ruby. I wrote C interfaces with ruby wrappers to communicate with the floor sensors and the lighting controllers. The system queries a MYSQL database for the media files to be displayed and then tells computers in the walls to play the media via UDP.
It's the most reliable part of the entire exhibit.
Ruby was used for the other major part of the exhibit, the Wall though I didn't have much to do with that. Most of the graphics were prototyped in ruby using interfaces to OpenGL, a bit of Cocoa and a physics library before being ported to pure Obj-C.
Puppet and Chef: DevOps
I didn't see a mention of Puppet or Chef in the 30 answers that preceded my arrival. Ruby appears to dominate current work in cloud automation and is the base, extension, and templating language of these two big players. They are used primarily to distribute system and application configuration information for server arrays and for general IT workstation management.
The DevOps field is quite Ruby-aware. Today, Perl has a competitor. While a really simple script may often still be written directly for sh(1), a complex task now might be done in Ruby rather than Perl.
The only site I've done with Ruby at work is using Rails, but I'd like to try Merb.
Other than that I do a lot of little utility programs in Ruby - for instance an app that reads RSS feeds and imports new posts into a dabase.
It's fun, so I also write some dumb stuff just because it's so quick. Yesterday I wrote an app to play the Monty Hall problem 100,000 times to help a friend convince her professor that switching is the correct strategy.
I almost take insult that ruby is a rails thing. It is like back when CGI was the latest trend and everyone figured that if you knew perl you must be doing it only because you programmed CGI apps. Ruby is just a scripting language for me, although not as mature as python so I somewhat regret having to jump through some of its hoops and recent changes, I still like it and use it. Although I work in a java shop and therefore groovy is the ideal choice for a scripting language, I still use ruby at home and for throw away scripts that aren't needed to be shared at work.
I was considering getting into RoR from all the buzz and how quick/simple it is, but after looking over rails I didn't see anything at all that was amazing or even the least bit innovative or rapidly fast about its development compared to any other framework. The only benefit I saw was that I could code in ruby, which would be nice, but initial setup, server maintenance and scaling is more difficult, thus re-offsetting the pleasure of coding in ruby.
I created a presentation -- coincidentally named Off The Rails -- to discuss Rack-based web applications:
https://github.com/alexch/Off-The-Rails
The git repo includes slides in Markdown format and sample code (in the form of running applications and middleware). Here's the abstract:
Ruby on Rails is the most popular web application framework for Ruby. But it's not the only one! If you think Rails is too big, or too opinionated, or too anything, you might be happy to learn about the new generation of so-called microframeworks built on Rack. And since Rails 3 is itself a Rack app, you don't have to give up Rails to get the benefit of Sinatra routes or Grape APIs.
And here are some references:
This talk lives at https://github.com/alexch/off-the-rails
Yehuda's #10 Favorite Thing About Ruby
Rack
rack-test
rack-client
Sinatra
Grape
Vegas
Siesta
Rerun
Hope you find it useful!
I'm mostly a Web developer, and I learned Ruby to use Rails, but I like the language so much that I started developing a desktop Swing application in Ruby, using JRuby and Monkeybars. I'm competent in Java, but don't much like using it, and the Swing API is horrible, so putting Ruby on top has been a big win.
We mainly use rails, but we have plenty of other non-rails ruby things - for example a standalone authentication daemon thing for centralized authentication of users, and an 'image processing server' which runs arbitrary numbers of ruby processes to process images in parallel.
Oh, and don't forget good old Rake :-)
Ruby is also used for Desktop application. Especially the use of JRuby to develop Swing desktop application.
I've used Ruby at work for
A data extractor, generating csv files from binary output.
A .ini file generator, turning a simple syntax into a repetitive .ini format.
A simple TCP/IP server, acting as stand-in for the customer's system during testing.
We use Ruby to implement our test automation software. This includes test framework and driver code for Selenium RC, WATIR and AutoIT.
Ruby is powerful enough to create comprehensive applications that can interface with Test tools like Selenium or WATIR, while at the same time reading from data files, interacting with a remote Windows UI and performing near transparent network communication. All while running on Windows or Linux.
The uncluttered syntax makes it ideal for new and inexperienced programmers to read. While its totally OO nature makes it easy for these same programmers to apply good (recently learned) OO techniques, from the start.
The flexible nature of Ruby's syntax also makes the use and creation of DSLs much easier. This allows less-technical people to get invovled, read and possibly create there own tests.
I have used Ruby for code generation of C# and T-SQL stored procedures in a project with unstable requirements. The data model was encoded in a YAML file and .erb templates were used for the classes and stored procedures. It also allowed for a much more DRY solution than would have been possible with straight C# as repetitve code could be factored out into a single method in the code generator.
Where I work, we use Ruby to do a number of different one-off type batch jobs. One example of that is a job that interacts with Amazon's S3 service. At the time, the Ruby S3 library was probably the easiest one out there for us to get up and running in a short amount of time.
I wrote an order processing expert system (see DSL answer as well), converted 100k lines of customer specific perl into about 10k lines of ruby handling dozens of customers. No web components at all, no Rails.
I am a webdriver user. ruby is used by webdriver for automating the build process thanks to rake. see http://code.google.com/p/webdriver/ for details
Heh, great question.
I used Ruby to convert Excel spreadsheet airport facility data to sqlite3 for the android phone platform while making an app for pilots.
I use Ruby with Sinatra which is much simpler than Rails. I did use Rails but just found that it has turned into a bit of a monster, although Rails is still amazing compared to web frameworks available for Java.
The main feature of Ruby that I love however is "eval" and "method_missing", which Rails actually uses for example in ActiveRecord so that you can use the amazing "find_by-field-name-" queries.
I used Ruby for a lot of back-end code simply because I was the only person who was tasked to do it and needed a nice clean language that allowed me to be very productive and write easy to maintain code. I find Ruby allows me to do that easier than Perl and Python. Other people's mileage might vary on that but it works well for me.
Besides that, I like how Sequel and Nokogiri work. I also used ActiveRecord for a while separately from Rails.
We use some Ruby for file manipulation but have not been able to incorporate rails yet.
I've used Ruby a lot professionally for quick scripts for things like shuffling files around. I'm the same way in that I was using Ruby first before touching Rails at all.
In Boulder there was an excellent group of Ruby users who met monthly. This point was made - that Ruby does have an existence beside its use in Rails. Plain Ruby users do exist, are begging for attention, have neat things to show, and can find each other at user group meetings.
They also had better pizza than the Python group, who met also the same day of the month. Can only pick one...
While we do have several Rails apps at work, we also use Ruby for some fairly intensive non-web stuff.
We've got an SMS delivery daemon, which pulls messages from a queue and then delivers them, and credit card processing daemon which other apps can call out to, which makes sure there's a central audit trail.
What would be the best way of inserting functionality into a binary application (3d party, closed source).
The target application is on OSX and seems to have been compiled using gcc 3+. I can see the listing of functions implemented in the binary and have debugged and isolated one particular function which I would like to remotely call.
Specifically, I would like to call this function - let's call it void zoomByFactor(x,y) - when I receive certain data from a complex HIDevice.
I can easily modify or inject instructions into the binary file itself (ie. the patching does not need to occur only in RAM).
What would you recommend as a way of "nicely" doing this?
Edit:
I do indeed need to entire application. So I can't ditch it and use a library. (For those who need an ethical explanation: this is a proprietary piece of CAD software whose company website hasn't been updated since 2006. I have paid for this product (quite a lot of money for what it is, really) and have project data which I can not easily migrate away from it. The product suits me just fine as it is, but I want to use a new HID which I recently got. I've examined the internals of the application, and I'm fairly confident that I can call the correct function with the relevant data and get it to work properly).
Here's what I've done so far, and it is quite gheto.
I've already modified parts of the application through this process:
xxd -g 0 binary > binary.hex
cat binary.hex | awk 'substitute work' > modified.hex
xxd -r modified.hex > newbinary
chmod 777 newbinary
I'm doing this kind of jumping through hoops because the binary is almost 100 megs large.
The jist of what I'm thinking is that I'd jmp somewhere in the main application loop, launch a thread, and return to the main function.
Now, the questions are: where can I insert the new code? do I need to modify symbol tables? alternatively, how could I make a dylib load automatically so that the only "hacking" I need to do is inserting a call to a normally loaded dylib into the main function?
For those interested in what I've ended up doing, here's a summary:
I've looked at several possibilities. They fall into runtime patching, and static binary file patching.
As far as file patching is concerned, I essentially tried two approaches:
modifying the assembly in the code
segments (__TEXT) of the binary.
modifying the load commands in the
mach header.
The first method requires there to be free space, or methods you can overwrite. It also suffers from extremely poor maintainability. Any new binaries will require hand patching them once again, especially if their source code has even slightly changed.
The second method was to try and add a LC_ LOAD_ DYLIB entry into the mach header. There aren't many mach-o editors out there, so it's hairy, but I actually modified the structures so that my entry was visible by otool -l. However, this didn't actually work as there was a dyld: bad external relocation length at runtime. I'm assuming I need to muck around with import tables etc. And this is way too much effort to get right without an editor.
Second path was to inject code at runtime. There isn't much out there to do this. Even for apps you have control over (ie. a child application you launch). Maybe there's a way to fork() and get the initialization process launched, but I never go that.
There is SIMBL, but this requires your app to be Cocoa because SIMBL will pose as a system wide InputManager and selectively load bundles. I dismissed this because my app was not Cocoa, and besides, I dislike system wide stuff.
Next up was mach_ inject and the mach_star project. There is also a newer project called
PlugSuit hosted at google which seems to be nothing more than a thin wrapper around mach_inject.
Mach_inject provides an API to do what the name implies. I did find a problem in the code though. On 10.5.4, the mmap method in the mach_inject.c file requires there to be a MAP_ SHARED or'd with the MAP_READ or else the mmap will fail.
Aside from that, the whole thing actually works as advertised. I ended up using mach_ inject_ bundle to do what I had intended to do with the static addition of a DYLIB to the mach header: namely launching a new thread on module init that does its dirty business.
Anyways, I've made this a wiki. Feel free to add, correct or update information. There's practically no information available on this kind of work on OSX. The more info, the better.
In MacOS X releases prior to 10.5 you'd do this using an Input Manager extension. Input Manager was intended to handle things like input for non-roman languages, where the extension could popup a window to input the appropriate glyphs and then pass the completed text to the app. The application only needed to make sure it was Unicode-clean, and didn't have to worry about the exact details of every language and region.
Input Manager was wildly abused to patch all sorts of unrelated functionality into applications, and often destabilized the app. It was also becoming an attack vector for trojans, such as "Oompa-Loompa". MacOS 10.5 tightens restrictions on Input Managers: it won't run them in a process owned by root or wheel, nor in a process which has modified its uid. Most significantly, 10.5 won't load an Input Manager into a 64 bit process and has indicated that even 32 bit use is unsupported and will be removed in a future release.
So if you can live with the restrictions, an Input Manager can do what you want. Future MacOS releases will almost certainly introduce another (safer, more limited) way to do this, as the functionality really is needed for language input support.
I believe you could also use the DYLD_INSERT_LIBRARIES method.
This post is also related to what you were trying to do;
I recently took a stab at injection/overriding using the mach_star sources. I ended up writing a tutorial for it since documentation for this stuff is always so sketchy and often out of date.
http://soundly.me/osx-injection-override-tutorial-hello-world/
Interesting problem. If I understand you correctly, you'd like to add the ability to remotely call functions in a running executable.
If you don't really need the whole application, you might be able to strip out the main function and turn it into a library file that you can link against. It'll be up to you to figure out how to make sure all the required initialization occurs.
Another approach could be to act like a virus. Inject a function that handles the remote calls, probably in another thread. You'll need to launch this thread by injecting some code into the main function, or wherever else is appropriate. Most likely you'll run into major issues with initialization, thread safety, and/or maintaining proper program state.
The best option, if its available, is to get the vendor of your application to expose a plugin API that lets you do this cleanly and reliably in a supported manner.
If you go with either hack-the-binary route, it'll be time consuming and brittle, but you'll learn a lot in the process.
On Windows, this is simple to do, is actually very widely done and is known as DLL/code injection.
There is a commercial SDK for OSX which allows doing this: Application Enhancer (free for non-commercial use).