gem install Errno::ENOLCK when installing gem - ruby

Question

Whenever I try to install a gem with gem install [gem-name], I keep getting this error:
ERROR: While executing gem ... (Errno::ENOLCK)
No locks available # rb_file_flock - /home/abod/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/bundler-1.7.12.gemspec
I did gem install bundler just for illustration, but the Errno::ENOLCK will occur regardless of the gem I'm trying to install. I've searched around and can't seem to find anything about this error.
Here is the out put from gem env, in case that helps.
RubyGems Environment:
- RUBYGEMS VERSION: 2.4.5
- RUBY VERSION: 2.2.0 (2014-12-25 patchlevel 0) [x86_64-linux]
- INSTALLATION DIRECTORY: /home/abod/.rvm/gems/ruby-2.2.0#global
- RUBY EXECUTABLE: /home/abod/.rvm/rubies/ruby-2.2.0/bin/ruby
- EXECUTABLE DIRECTORY: /home/abod/.rvm/gems/ruby-2.2.0#global/bin
- SPEC CACHE DIRECTORY: /home/abod/.gem/specs
- SYSTEM CONFIGURATION DIRECTORY: /home/abod/.rvm/rubies/ruby-2.2.0/etc
- RUBYGEMS PLATFORMS:
- ruby
- x86_64-linux
- GEM PATHS:
- /home/abod/.rvm/gems/ruby-2.2.0#global
- GEM CONFIGURATION:
- :update_sources => true
- :verbose => true
- :backtrace => false
- :bulk_threshold => 1000
- REMOTE SOURCES:
- https://rubygems.org/
- SHELL PATH:
- /home/abod/.rvm/gems/ruby-2.2.0#global/bin
- /home/abod/.rvm/rubies/ruby-2.2.0/bin
- /usr/local/bin
- /usr/bin
- /usr/local/sbin
- /usr/sbin
- /home/abod/.rvm/bin
- /home/abod/.rvm/bin
At the suggestion of #joelparkerhenderson, here are the output of some commands that may be helpful.
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 2063389
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 32768
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 2063389
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
$ uname -a
Linux biohen36 3.14.23-100.fc19.x86_64 #1 SMP Thu Oct 30 18:36:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
$ stat /home/abod/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/bundler-1.7.12.gemspec
File: ‘/home/abod/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/bundler-1.7.12.gemspec’
Size: 1025 Blocks: 8 IO Block: 1048576 regular file
Device: 2fh/47d Inode: 2285894 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 730/ abod) Gid: ( 1013/ wommack)
Access: 2015-02-03 01:12:35.573192019 -0500
Modify: 2015-02-02 13:53:41.227813208 -0500
Change: 2015-02-02 13:53:41.227813208 -0500
Birth: -
$ fuser -m /home/abod/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/bundler-1.7.12.gemspec
/home/abod/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/bundler-1.7.12.gemspec: 57788c
There were more processes accessing the file, but I killed them and tried to install again, but I still got the lock error. I'm not out of disk space either.
Here is some more info:
This is on a compute cluster. The home directory /home/abod is symlinked somewhere on /mnt which mounts to some other machine on the network.
The output from mount has been requested by a few commenters. Here are the lines regarding the home directory:
fileserver:/homes on /home type nfs (rw,noatime,nodiratime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,nocto,noacl,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=128.175.xxx.xxx,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=128.175.xxx.xxx)
-hosts on /net/fileserver/homes type autofs (rw,relatime,fd=13,pgrp=1368,timeout=300,minproto=5,maxproto=5,offset)
fileserver:/homes on /net/fileserver/homes type nfs (rw,nosuid,nodev,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=128.175.xxx.xxx,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=128.175.xxx.xxx)
Edit
I found this ticket on the rubygems github page that is related to my issue: https://github.com/rubygems/rubygems/issues/1176

Answer 1

Note: I'm offering a bounty to help the OP, because we need more expert help.
Here are some general ideas to get you started...
Is it possible that you're out of system resources, such as disk space?
Or your system is limiting the resources you can use?
Or trying to write to a volume where you don't have permissions or a lock daemon?
If no one has a better answer for you, by the time you read this, can you try running these commands then post the output in your question?
$ df
$ ulimit -a
$ uname -a
$ stat /home/abod/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/bundler-1.7.12.gemspec
$ fuser -m /home/abod/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/bundler-1.7.12.gemspec
If you want to kill any process accessing that file, that may be holding locks:
$ fuser --kill /home/abod/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/bundler-1.7.12.gemspec
You can discover if the issue is with Ruby vs. gem by running this:
path = "/home/abod/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/bundler-1.7.12.gemspec"
f = File.open(path, File::RDWR|File::CREAT, 0644){|f|
f.flock(File::LOCK_EX)
f.flock(File::LOCK_UN)
}
If the above code fails, then the most likely explanation is that you're using a filesystem that doesn't support file locking. This would be consistent with your home directory being mounted to some other filesystem, such as using NFS without a lock daemon.
Edit: OP writes:
Running the code you suggest: test.rb:4:in flock': No locks available # rb_file_flock - /home/abod/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/bundler-1.7.12.gems‌​pec (Errno::ENOLCK) from test.rb:4:inblock in ' from test.rb:3:in open' from test.rb:3:in'
That looks to me like your filesystem doesn't provide locking.
I have two suggestions and I'm also adding a bounty to your question to help attract expert attention.
If you have any filesystem space on that computer that isn't a mounted compute cluster, try installing a gem there. You can set the GEM_HOME environment variable like this:
GEM_HOME=/foo/bar gem install <name>
If you can use a similar system, such as the same OS on VirtualBox or VMWare, then you can use the bundler gem, a Gemfile, and the command bundle pack. This creates a directory full of the gems listed in the Gemfile. You can then copy this directory anywhere you want, for example by using rsync, or scp, or sftp, etc. (This doesn't help much though with native gems, such as pg for PostgreSQL - you would need to be sure to install all the PostgresSQL dependencies and headers on your own).

Answer 2

Ask your Sys Admin to unmount and mount the partition; the NFS Lock Daemon messes up at times and this will reset it.
(Not exactly an elegant or long-term solution, I know).