Installing gems from behind a corporate firewall - ruby

I suspect that the corporate firewall is preventing gems from getting installed. I have HTTP_PROXY defined and I'm able to view remote gems via the following command:
jruby -S gem list -r
But when I go to install a gem, I get a 404:
jruby -S gem install rails
Is there a good workaround for resolving this issue other than maintaining an internal gem repository?

for ruby gems, placing this in my gem.bat this works for me
#"%~dp0ruby.exe" "%~dpn0" %* --http-proxy http://domainname.ccc.com:8080
for jruby gems this works
#"%~dp0jruby.exe" "%~dpn0" %* -p http://domainname.ccc.com:8080
Also setting the environment variable works like
set http-proxy=http://domainname.ccc.com:8080
or if your proxyserver needs authentication
set http-proxy=http://user:password#host:port)
EDIT: for folks who have a very restrictive firewall or no internet accesss (eg on a server) you can do the following: install the gem(s) on a pc who has free access to internet, afterward you check the folder C:\Ruby193\lib\ruby\gems\1.9.1\cache and copy all the gems with a date last modified after the moment you did your install. On the target pc you copy them in the same folder or if you clean up afterward in your bin folder and start the install there with
c:\ruby193\bin\gem.bat install --local gemname-x.x.x.gem
gemname-x.x.x.gem being the gem filename of the base gem you want to install. The gem with all its dependencies should install. This is for windows, but other OS's can use the same technique, just adapt a few things.
Success

Look in Internet Explorer proxy settings to find the name of your proxy server.
Use the name of your proxy server in the -p option to the gem command.
gem update rails -p http://mylocalproxy.mycompany.com
If this doesn't work, you can always set your own proxy server up on the internet somewhere.

I'm not an expert with Ruby. None the less I might give the following "generic" advices (useful for lots of situations with installers) :
Check which protocol is the installation running over. Some installer download package from mirror instead of main site, and some mirrors might be using FTP or something else instead of HTTP. Thus you might need to update your settings accordingly, and check with your admin that the corporate proxy is able to retrieve data from FTP sites.
Check which user the installer is running as (some of the installer have to escalate privileges to "root". And therefore inherit a different environment with another HTTP_PROXY defined).
Check what the installer program is using to download the packages (lots of distribution package managers rely on curl/libcurl which stores its proxy parameters in a rc file). Same warning as 2 regarding the users whose homedir should contain the rc file.
Transparent proxying : it is possible to transfer on-the-fly HTTP/FTP requests initially targeting the web to a proxy. Either the corporate proxy (if it can work with such an installation) or some mini local proxy which will then cascade to the corporate one.
Setup a quick VPN (either Tunnel other PPP) over SSH (see SSH's man page) between your machine and a special machine within the corporation which is allowed to see the web, then update your routing tables to route traffic through the VPN.
You can also use corkscrew to connect through SSH (and thus be able to install a VPN over SSH like 5) with a machine outside, on the other side of the corporate firewall, like your own at home. (this uses the HTTPS "CONNECT" mode of the proxy to access a SSH outside). Just make sure that the admins are- or at least one person in charge is- kept in loop and don't panic.
Desperate measure : create a tunnel between two machines running each http tunnel, over a proxy which only allows HTTP requests. Then use the tunnel to establish SSH VPN.
These solutions aren't specific to Ruby but can help you through any desperate solution behind an over-restrictive firewall.
1 to 4 should work in most situation. You'll have to resort to 5 to 7 if confronted with some bizarre and/or paranoid firewalls. 7 is rather extreme but always works, although sometimes with catastrophic performance.

Simple thing first: are you in the same shell session when you tried both the list and the install? Maybe you forget to set the env second time around?
Try setting http_proxy instead of HTTP_PROXY.
Make sure the proxy setting is a valid URL (see below)
Do you have multiple proxy options? Open IE and type wpad in the address bar. You might see some other possibilities.
Is your proxy authenticated? If so, do you have your credentials in the URL. Mine is like this (on windows):
set http_proxy=http://myuserid:mypassword#internetproxy:3128
Does your your password have any punctuation characters? Try it with just numbers and letters, as : and # (and possibly others) are significant in the URL string itself.

You can use this tool:
http://ntlmaps.sourceforge.net/
gems don't support proxy authentication, so going through a secondary proxy (like, say, YOUR machine) that does the authentication for you and lets gems think there is no authentication happening should make your life easier.

Another stupid question, but what's the corporate proxy software ? Some Windows-based software tend to used weird authentication standards (NTLM hashes, etc.) which aren't supported by all clients (wget doesn't support it, for example), but are supported by lots of browsers.
Thus even if you tried writing the username and password into the proxy URL, it won't work when you try to download and install a packages, although it works pretty well when you try to display a page in FireFox. I've had similar problems with some corporate network and my distro's package manager.
In these situation, you might use something like ntlmaps. You use it as a local proxy which will then cascade the requests to the corporate proxy. The good thing is that ntlmaps will be able to authenticate the weird NTLM password with the corporate proxy, and all your applications will be able to connect to ntlmaps, even those which don't support NTLM.

I've retested this since the gemcutter.org folks made the mode to Amazon's Cloudfront and am able to download gems again through my employer's firewall.

I described a procedure to get around any restrictions of a corporate firewall in:
How do I update Ruby Gems from behind a Proxy (ISA-NTLM)
It basically uses an ssh tunnel and the programm tsocks.

1.GO to the drive in which you installed ruby.
2.you will find a folder named "ruby200-x64" or something similar to it based on which version you installed.
3.inside the folder open ...bin/
4. find gem.bat, open it with any text editor you will find a bunch of settings.
5. after #"%~dp0ruby.exe" "%~dpn0" %* place you proxy server address and port.
**
#"%~dp0ruby.exe" "%~dpn0" %* --http-proxy
http://domainname.ccc.com:8080
**

Related

Configure perforce to use proxy server

As the title says, I need to configure perforce to use a proxy server. To clarify, I am not talking about using Perforce Proxy, but to actually connect to the perforce server (in this case perforce.assembla.com:1666) through a proxy.
Looks like you need a SOCKS proxy.
There are plenty of open source solutions depending on the platform you are using. https://wiki.debian.org/SOCKS
The only way i've found so far to do this is by installing an extra software that forces perforce to use a socks proxy.
1) you set up whatever proxy you want to use
2) you install AllProxy
3) Add your proxy to the proxy list
4) In Manage Rules add a rule.
- On Proxy choose your proxy
- On programs add perforce. Search for p4v.exe (or whatever) (its easier if perforce is already running).
- On remote destination specify the port, or leave to any.
Then use perforce as normal.
Note: there's a default route that goes direct (not using proxy).
Also for HTTP proxy you can set it globally under Windows Proxy settings (on Win10), or use AllProxy.
Note2: This info is based on this answer.
Note3: AllProxy is the only listed soft that worked for me. You could still try the longer route.

How to configure Atom to use proxy file (pac) settings?

On Windows, is it possible to configure Atom to use the system default proxy instead of manually specifying the proxy settings in the .apmrc file?
I recently started using the atom-editor for Windows (Windows 7 Pro) and so far I like it. I've been searching for some time to try and get the editor to be able to connect to the outside world and haven't met with any success.
Our network (I'm not a network expert) is configured to use a proxy server (internal) to access the internet. On my system, this is accomplished in the proxy settings with an 'Automatic Configuration Script', for example: http://internal-server-name/sub-dir/file-name.pac.
Many resources have suggested manually setting the proxy information in the .apmrc file:
Not able to connect to atom.io for themes and packages
here:https://discuss.atom.io/t/is-there-any-proxy-settings/710/19
https://github.com/atom/atom/issues/1807
These do not answer my question as I need the system defaults to work
I tried to configure Atom to do this same thing but when I change the .apmrc file to have:
http-proxy = http://internal-server-name/sub-dir/file-name.pac
https-proxy = https://internal-server-name/sub-dir/file-name.pac
strict-ssl = false
I get the following error (presumably because the above address is not the proxy itself):
tunneling socket could not be established, cause=Parse Error
When I leave the .apmrc file unmolested, I receive the following error when trying to view packages:
getaddrinfo ENOTFOUND
Even if I had a proxy server IP address and port number to specify, not only would i not want to hard-code my credentials in a config file but I'm pretty sure the authentication is somehow set up to use Kerberos tokens anyway, so I wouldn't be able to specify them at all.
Interestingly the Atom updates says that my installed packages (out of the box) are up to date, which would seem to indicate that this portion is working while the packages and themes do not.
Here are some screenshots for reference:
No .apmrc changes
Proxy server specified in .apmrc
Edit 08/31/2015
In response to some of the feedback, I tried to view the .pac file and extract the proxy server address information. I was able to get the information about the proxy out of the file and put it in the .apmrc file but it had no affect on the outcome (I still receive the same errors). I have also since tried using CNTLM to no avail.
Also note that my proxy does require authentication as stated above. I've been doing further research on this issue and it may be a combination of proxy settings for the APM and proxy settings for GitHub. See the following relevant article:
https://discuss.atom.io/t/error-running-apm-install-behind-proxy/14812
Ideally, I'd like a solution that doesn't require a complete machine configuration or a fragile multi-config file setup.
A .pac file is just a javascript file that browsers can run to programmatically determine which proxy to use. This is useful if your network needs to use different proxies to access different resources. If you open up that file, you'll probably be able to make sense of it (it's just javascript, after all) and figure out which proxy your network is using for general access to the web. Set that as your proxy in your .apmrc file and it should work.
Things get more complicated if your proxy requires some kind of authentication. If that's your problem, let me know. I have some experience dealing with it.
Authenticating
OK, so you need to authenticate. Then your situation is probably pretty similar to mine. You are correct in using CNTLM, we just need to configure it properly. Your cntlm.ini file will look something like this:
Username <your windows username>
Domain <your domain name>
Password <leave this blank>
PassLM <get this by running `cntlm -H` on the command line>
PassNT <get this by running `cntlm -H` on the command line>
PassNTLMv2 <get this by running `cntlm -H` on the command line>
Proxy <Your proxy address like ip_address:port>
Proxy <If you have multiple proxies, you may list them each on a new line>
NoProxy localhost, 127.0.0.*, <any others that should bypass the proxy>
Listen <a local port to listen on (I use 53128)>
There's something a bit funny that you should know about this file (cntlm.ini). When you install cntlm, it's included in the program's folder but it actually will only work if it's located at C:\Program Files (x86)\cntlm\cntlm.ini. I guess it's probably more accurate to say that it should be in the cntlm folder inside the PROGRAMFILES directory so you should double check that this is the case.
Anyway, having done all that, you can now start cntlm (instructions in the readme). Next, you need to configure apm to actually use your local proxy. Your .apmrc should look like this:
http-proxy = http://localhost:<port # from cntlm.ini>
https-proxy = https://localhost:<port # from cntlm.ini>
strict-ssl = false
You'd think this would be enough but there's more that might help. I had a lot of problems with npm until I found that https doesn't work well through cntlm so I needed to change my registry from https://registry.npmjs.org/ to http://registry.npmjs.org/. Since apm also uses the same registry, I decided to change that too:
apm config set registry http://registry.npmjs.org/
And then... it seemed to also help to go into Atom's config.cson and set core.proxy to http://localhost:<port # from cntlm.ini>
And after all that, I'd expect things to work for you. If it's cool and fine and Atom is working well, there's one more thing that you might want to do that will make most (all?) other apps (npm, git, etc) work well. On the command line, run:
netsh winhttp set proxy localhost:<port # from cntlm.ini> "localhost, 127.0.0.*"
This will save you the trouble of configuring most (all?) other apps one-by-one. For some reason, apm does need to be configured specifically. I guess it doesn't use the winhttp settings for some reason? I don't know the answer to that.
New Info 2016/03/18
I recently noticed that there's one additional thing you might need to do for some apps. You should set the following environment variables to your local proxy as well:
HTTP_PROXY
HTTPS_PROXY
FTP_PROXY
http_proxy
https_proxy
ftp_proxy
It may seem like overkill to set both upper and lower case but I recently installed MSys2 and found that it looks for the lower case versions so just set them all and save yourself some trouble.
You should download the file http://internal-server-name/sub-dir/file-name.pac and open it with a text editor, then find the actual proxy address and port inside it (near a PROXY keyword).
I used this for my trouble with npm and .npmrc
c:\users\%user%\.atom\.apmrc (add if not exist) and put this
proxy=http://YOUR-DOMAIN**%5C**USER-NAME:PASSWORD#YOUR-PROXY-SERVER:PORT/

Not able to access page data, using anemone with socksify gem and Tor

I ve written a ruby script using anemone gem to crawl a website. The script runs fine when used directly.
But I would like to use socksify gem so that all TCP calls from the script is routed with socks5. I did the following for the same:
Installed and started Tor project and it is running in my machine
Installed socksify gem
ran the following command socksify_ruby localhost 9050 myscript.rb as given here
However anemone does not detect any page in this case. Please let me know what mistake I am doing.
There are a number of problems that could be causing this to happen. First, if ntp is not running on your machine, and the time is off by even a little bit, you will not be able do use the socks server to do anything complicated. This happened to me. You need to install ntp and make sure it has synced before doing anything.
Second, you may find that a lot of this commands like socksify are obsolete. The best way I have found to make sure that everything happens through the socks port without dns leakage is by using curl, which has bindings for many languages. You can carefully watch the traffic with tcpdump to make sure it isn't leaking, and it is watertight in my experience.
I'd also suggest that you look at torsocks, which has recently been updated by dgoulet on github. This replaces tsocks, which the outdated socksify_ruby is based on.
Finally, hidden services have been under great strain lately, because a bot has decided to start up a few million Tor clients. Make sure you can connect with the Tor Browser Bundle, assuming the project you are working on is trying to crawl hidden service.
You didn't actually say that this project involves Tor or hidden services, but you did tag it with Tor.

Can gitlab be installed with Cherokee web server?

I've looked all over and can't figure out if you could use Cherokee instead of Apache or Nginx for gitlab. I'd rather not run multiple webservers (and imagine that they could conflict anyway). I'm giving this a shot on Ubuntu Server 12.10.
For the record, I've already installed gitlab with this guide up to the Nginx section (with all default settings other than passwords, email addresses, and hostname). I'd like to install gitlab at git.mydomain.com and I would prefer for the local server files to be located at /var/www/git.mydomain.com, as I keep all of my domains under /var/www/.
Since you already have all of the Ruby config done, you just need to hook cherokee
up for hosting RoR by following this guide http://cherokee-project.com/doc/cookbook_ror.html
My only problem turned out to be an issue with Ruby. Once that was resolved, I set up gitlab to use a port (though sockets should work too). Everything seems to work pretty well, except for an issue with pushing over HTTPS, but that might have something to do with my local Eclipse/eGit install.
So yes, gitlab will work with Cherokee.

A script that download a file and rename

I'm facing an little issue here. At the place I live, they shape the download speed by extension (using delaypool). Is there any script that I can run on my web that will let me enter the URL of files that I wanted to download, then it will download the file and rename it to "originalfilename.abc" (because .abc is not shaped)then save it on my web where I can download it. By the way, I have a paid webhosting service.
Thanks
I tried the SSH (my web hosting does indeed provide me with shell access) but all I get is a blank page in my browser. No error. Please advice.
Also, reason I choose script on the server instead of SSH because I though that SSH would be slower than direct HTTP download from my webserver. Can anyone point that out if I'm right or wrong with my thinking.
Thanks
You can write a vbs script or even batch file script that will go to a designated URL, and then download the file, then rename it. Then you can script FTP commands to upload to your webhosting service (I'm sure it has an FTP site for access). You could load this as a scheduled task, or run it manually.
I'd do either one of these routes.
A simpler (once its set up anyway ;) option I can think of is going through a secure tunnel. Whilst this is not quite answering your question I believe this to be simpler while achieving the same thing.
Get an SSH Client (Putty) and get a free Proxy. If your web-server has an SSH-server you can use it as Proxy as well of course, I am using my modified router at home as proxy via DynDns, but the Tor-Network will work, if very slow, so do other official free and paid proxy servers. If you are using an application to download that does not have the option to specify a proxy, get Proxifier Portable.
Use Putty to create the tunnel. Here some how-tos:
http://oldsite.precedence.co.uk/nc/putty.html
http://www.techrepublic.com/blog/security/use-putty-as-a-secure-proxy-on-windows/421
http://kimmo.suominen.com/docs/proxy-through-ssh/
And set your application to use your proxy (or actually to use putty which connects via SSH to your proxy) by entering 127.0.0.1:1080 into the proxy settings. Alternatively, if the app does not have an option to enter proxy settings, add 127.0.0.1:1080 to your Proxifier proxy list and add the applications that are supposed to use that connection to Proxifier.
Now you can do pretty much everything without anyone eavesdropping your connection stream being able to tell what it is, as the connection stream is SSH encrypted. This includes surfing websites that your provider/company/mother has blocked, download anything - even if blocked by IP/name/whatever-filters and even play MMORPGs from work (something which I do not recommend because it will get you fired and there is always some way for someone to figure it out, just saying it is possible to do even in secured company/school networks as Port 22 (SSH) is usually one of the 2 Ports which are open on pretty much any network (the other one being port 80)).
Its a wee bit of a pain to set up. Once it is working though, you can even put it on a usb-stick and use it pretty much anywhere as long as you remember what proxy to connect to. And you wont have to rewrite scripts to try to circumvent the delaypool thingy.

Resources