Working on an app that has almost 200 gems. Has anyone figured out how to isolate gems that are not used so they can be taken out of the mix.
Due to the dynamic nature of Ruby, it's not possible to know for sure if a gem is or isn't used without testing. Although it is bad practice to load them in ad-hoc without a good reason, it is possible to require a gem at any point in the execution of the program. They do not need to be loaded up-front.
Although there might be advantages to loading gems on demand, for instance, keeping a lower memory footprint and reducing launch times, it does make it difficult to determine if or where they are actually used.
There isn't always a correlation between a gem name and the methods it uses. While many have a namespace that's easily grepped for, some just add methods to existing classes which can complicate tracking them down, especially if they go so far as to patch out old methods with new ones that have the same name.
If you are able to exercise a large portion of the application through your unit, functional, and integration tests it might be possible to use ruby-prof to at least get a sense of which gems are used. That could make identifying candidates for removal easier.
Related
On a site like www.codewars.com, one can run ruby in a sort of sandbox, almost identical to IRB.
How does this actually work?
If the submitted code is eval()d, what's preventing me from submitting a system("rm -rf *") or redefining basic functions so that 50% of the time Array.sort actually runs Array.shuffle?
The simplest and safest solution is to run the Ruby code on separate computer, which you wipe and re-install after every run. This is, however, also a pretty heavyweight solution.
More lightweight, but (almost) as safe, would be using a virtual machine or a container instead of a whole separate computer, and e.g. using a read-only filesystem with a ramfs overlay, which you umount after every run. (Or just throw away and recreate the container.)
You could also use JRuby together with the JVM's security features (or IronRuby with the CLI's). The JVM has sandboxing features for JVM programs, and after all, JRuby is just a Java program like any other.
Lastly, you could write your own Ruby implementation with sandboxing in mind, or modify an existing one. The three options above are fairly simple, this one is hard, because most Ruby implementations aren't designed for sandboxing. TryRuby.Com worked this way, for example, and it took a significant amount of time to update it for Ruby 1.9, because it was originally based on a modified version of MRI, but MRI doesn't support Ruby 1.9. So, the implementation had to be switched to YARV, and a lot of the modifications to make it sandboxing-safe had to re-implemented from scratch. (The JRuby/IronRuby option above is similar to this, but you push off the work of making the implementation sandbox-safe to someone else, e.g. Oracle or Microsoft.)
A not-so-safe but also simple solution would be to run the interpreter under a restricted user account.
Of course, you can combine multiple approaches for defense-in-depth, for example, running a sandboxed interpreter under a restricted user account on a separate VM.
What does not work is to statically analyze the code before running it. The pesky Halting Problem bites us here.
Its my understanding that when I make a gem I'm compiling my ruby code into some form of executable, right? Does this mean that unless someone used introspection techniques (which is an acceptable risk to me), my source code is secure?
A gem is not a compiled executable. It's not compiled at all. Ruby is interpreted. Creating a gem just bundles the necessary files together, much like a zip file or tar archive.
If you want your gem secure you should keep it out of rubygems.org. You can set up your own private gem server or you can just include your gem in projects that need it.
While it is possible to compile Ruby code into an executable or shared library using Ruby's C API, that has nothing to do with gems.
A gem is just a collection of Ruby code (which could be regular scripts or compiled libraries) in a nice package for use with the rubygems package manager. It makes no effort to hide/protect the code.
I think
gem unpack
can extract your code. Never tried to do it and see if it's "human-readable" but you can try it before publishing your gem ;)
Its my understanding that when I make a gem I'm compiling my ruby code into some form of executable, right?
No. It's just a zipfile with some metadata. The contents of the zipfile are exactly what you put into them.
Does this mean that unless someone used introspection techniques (which is an acceptable risk to me), my source code is secure?
This depends on what you mean by "secure", but is completely orthogonal to RubyGems.
If you mean that "it can't be stolen", then that is already guaranteed for you by copyright law. Unless you live in a really weird country, software is protected by copyright automatically from the moment you write it.
If you mean "cannot be reverse engineered", then that is impossible. If you want people to be able to run your program, then you must give it to them in a format that can be understood by the CPU. Humans are much smarter than computers, so, if the program can be understood by the CPU, then it can also be understood by a human.
There are two common ways around this, which I will call the "Nintendo way" and the "Google way".
The Nintendo way is to give the user the CPU as well as the program, therefore, the user's CPU doesn't have to understand it. However, that model is still flawed. As long as you give the user something, he can figure it out. In the end, it's all just maths and physics, which can be understood. And users are pretty clever. Note that, for example, most game consoles were not cracked by evil crime syndicates trying to steal the code or pirating games, no, they were cracked by students wanting to run Linux or BSD on their hardware.
The Google way is to give the user nothing. You type something in the search box, Google sends you back the results, but at no time does the software leave Google's datacenter.
I'm using the excellent Fog gem to access just the Rackspace Cloud Files service. My challenge is that I'm trying to keep the service that is accessing Cloud Files lightweight, and it seems that Fog through its flexibility has a lot of dependencies and code I'll never need.
Has anybody tried to build a slimmed down copy of Fog, just to include a subset of providers, and therefore limit the dependencies? For example, for the Rackspace Cloud Files API exclusively, I'd expect to be able to handle everything without net-ssh, net-scp, nokogiri gems, and all the unused code for Amazon, Rackspace and 20 other providers that are not being used. I'm hoping to avoid upgrading the gem every time one of those unused providers notices a bug, while keeping my memory footprint down.
I'd appreciate any experience anybody may have in doing this, or advice from anybody familiar with building Fog in what I can and can't rip out.
If I'm just using the wrong gem, then that's equally fine. I'll move to something more focused.
I'm the maintainer of fog so I'll chime in to help fill in some of the explanation/gaps. The short answer is, yeah it's a lot of stuff, but mostly it shouldn't impact you negatively.
First off, fog grew pretty organically over time, so it did get bigger than intended. One of the ways that we contend with this is that we rather aggressively avoid requiring/loading files until really needed. So although you have to download lots of provider files you won't use to install fog, they shouldn't actually end up in memory. That was the simplest thing we could do in order to keep things "just working" while also reducing the memory usage (and load time).
The release schedule doesn't tend to be too crazy (on average about once a month) and tends to include a mix of stuff across most of the providers. So I'd expect you won't have too much churn here (outside of emergency/security type fixes which might warrant shortening the normal cycle).
So, that hopefully provides some insight in to the state of the art. We have also discussed starting to split things out more in the longer term. I think if/when that happens we would end up with something like fog-rackspace for all the rackspace related things. And then they could share things through fog-core or similar. We have a rough outline, but it is a pretty big undertaking without a huge upside, so it isn't something we have really actively begun on.
Hope that helps, certainly happy to discuss further if you have further questions or concerns.
I work for Rackspace on, among other things, our Ruby SDKs. You're using the right gem. Fog is our official Ruby API.
This is possibly something that could be done by introducing another gemspec into the project that builds from only fog core and the Rackspace-specific files. Though this would be unconventional and make #geemus' (the gem maintainer) gem release process more complicated––especially should other providers start to do the same. Longer term, this would serve to divert the fog community away from acting as a unified API.
I'm looking for a simple Ruby framework for a non-web project. I'm very familiar with Rails, so when I write pure Ruby I miss:
The different environments (development, test, production)
The console rails c
And many other utilities provided by rake or rails
I know I can require ActiveSupport and friends, but it's not what I need. It's mostly the development framework that I miss.
One thing I investigated is to do my project as a gem (even if I don't need a gem at the end). Using jeweler for instance provides versioning. I'm sure there are better ways but I can't find any. What would you use ?
Alright, that is somewhat of a three-part question. A threstion, or triesti... actually that doesn't work. Threstion it is then.
One thing to understand about Rails is that it isn't magical. It's code, written by mortal humans who have faults (such as oversized-egos) but who are also incredibly damn smart. Therefore, we can understand it after perhaps a little bit of studying their work.
1. Rails environments are lovely
The code behind the Rails environment management is actually quite simple. You have a default environment which is set to development by this line in railties/lib/rails/commands/server.rb. You could do much the same thing in your application. Run a bit of initialization code that defines YourAwesomeThing.environment = ENV["AWESOME_ENV"] || "development". Then it's a matter of requiring config/environments/#{YourAwesomeThing.environment}.rb where you need it. Then it's a matter of having configuration in those files that modify how your library works depending on the environment.
I couldn't find the line that does that in 3-1-stable (it's gone walkies), so here's one that I think does it in the 3-0-stable branch..
As for the lovely methods such as Rails.env.production?, they are provided by the ActiveSupport::StringInquirer class.
2. Rails console is the Second Coming
I think rails console is amazing. I don't want to create a new file just to manually test some code like I would have done back in the Black Days of PHP. I can jump into the console and try it out there and then type exit when I'm done. Amazing. I think Rails developers don't appreciate the console nearly enough.
Anyway, this is just an irb session that has the Rails environment loaded before it! I'll restate it again: Rails is not magical. If you typed irb and then inside that irb session typed require 'config/environment' you would get (at least as far as I am aware...) an identical thing to the Rails console!
This is due to the magic that goes on in the initialization part of Rails which is almost magic, but not and I've explained it in the Initialization Guide Which Will Be Finished And Updated For Rails 3.1 Real Soon I Promise Cross My Heart And Hope to Die If Only I Had The Time!
You could probably learn a lot by reading that text seemingly of a similar length to a book written by George R. R. Martin or Robert Jordan. I definitely learned a lot writing it!
3. The Others
This is where I falter. What other utilities do you need from Rake / Rails? It's all there, you just need to pick out the parts you need which is actually quite simple... but I can't tell you how to do that unless you go ahead and paint those targets.
Anyway, this is the kind of question that is exciting for me to answer. Thanks for giving me that bit of entertainment :)
Edit
I just noticed the BONUS fourth question at the end of your question. What would be the point of versioning your project? Why not use a version control system such as Git to do this and just git tag versions as you see fit? I think that would make the most sense.
First of all, Rails goal as a web framework is to let you mostly forget about the low-level details of a web based application (interacting with a web server, a database, etc.) and provide you with high-level abstractions which allow you to concentrate yourself on the actual job: Getting your application done.
There is such a huge diversity of non-web applications, that it is simply impossible to have a framework that "makes them all".
It is useful to understand what a framework is, and why and when you should use it.
Let's say you would have to build a dog house. You would buy some wood, nails and a hammer to build it the way you want from scratch, or you could go to Walmart and buy a DIY doghouse kit.
You will get a manual with detailed instructions on how to build your dog's "dream doghouse", and you will get it faster done than when making it from scratch, but you will be limited to the wood that comes with it and may be forced to use their nails or screws, and in some cases you would also need to buy a special screw driver.
The DIY kit is your framework. Some smart guys already provided you with the basic tools of making your doghouse, you only had to figure out (or not, if you had a manual) how to do it given their "framework".
Now, let's say you would build a skyscraper. The materials, tools and conventions (the framework) used would probably not be the same as they would when building a doghouse, and it would be over-kill to use a "skyscraper framework" to build a doghouse.
You wouldn't write Twitter client with the same set of tools and conventions as you would when writing a banking application with a billion concurrent users.
Rails is just one framework which provides you with a set of many, many good (and useful) conventions to write a web application, but that doesn't necessarily mean that it's the ultimate way to go. You can choose another framework, and maybe you will like the conventions for writing your application even more than the ones provided by Rails; or you could write all from scratch which would not limit you to what is provided by a specific framework and maybe make it even better.
To answer your question: Think of what you want to write and don't let yourself be narrowed down by the set of tools and conventions provided by a framework, as they will surely be if you choose this path.
Buy yourself some books about Design Patterns and read some great software, and then you may understand why the authors did not choose an application framework. Indeed, they've developed their own conventions and defined the behavior of what it should do; and once again, the "Rails way" isn't the only way, which doesn't make it the bad way. :)
I would appreciate constructive criticism as I really tried to provide a good answer.
Have you looked at Monkeybars? It's a Rails-inspired MVC framework that runs on top of JRuby and Swing to build desktop applications. I like it a lot.
any usable ruby code to interact with the safe browsing API from google?
i did search at google but i didn't find any mature and solid code.
I have 3 points:
(0) I'd say that This looks alright, as does this
(1) Having used quite a few ruby gems for various obscure things, I find bugs all the time. It helps the open source community and the world if you find a gem, fix a bug, and let the rest of the world benefit by submitting a pull request. Tests make the life of a contributor sooooooooooo much easier, and guarantee that your fix works, so use gems with extensive tests where possible, even if they are not mature and you half-expect them to fail.
(2) From experience, gems which have lots of objects encapsulating something can sometimes be counterproductive. This has tripped me up in the case of the ruby mail gem and the tire gem (though that's not to say that they are not good and incredibly useful gems.). This applies to you if you only need to make one type of API call, say, and take a simple action. Using the simplest gem is sometimes advantageous, and for this purpose you might not need to use any gem at all! Just write a class that uses Net::HTTP to call the HTTP API: https://developers.google.com/safe-browsing/lookup_guide