Working following the direction suggested after a recent question of mine (Parallel processing: #everywhere, distributions and types) I tried to set up a module using #everywhere, with a type depending on a package.
However, this arises a different question - as suggested in meta I created a new post. Let's say I have the following module (in test.jl):
#everywhere module test
using Distributions
type TypeDistrib{T <: Float64}
d::Distributions.Normal{T}
end
include('./folder1/fun1.jl');
export TypeDistrib, fun1
end
where fun1 is a function (in fun1.jl) and a simple script (in script.jl):
include("./testdir/test.jl");
using test
In addition, let's assume that these file are stored as follows:
-> example/
- script.jl
-> testdir/
- test.jl
-> folder1/
- fun1.jl
When I run it using example as my working dir without #everywhere, then it works with no errors. If I run it with #everywhere it says that the file fun1.jl doesn't exist.
I noticed that when I use #everywhere the relative path is linked with my working dir and not with the directory in which the module is stored. Am I missing something? Could you please clarify how I should set correctly the relative path using #everywhere - or more generally in case my settings are not accurate?
Related
I'm trying to load a C shared library within Ruby using Fiddle.
Here is a minimal example:
require 'fiddle'
require 'fiddle/import'
module Era
extend Fiddle::Importer
dlload './ServerApi.so'
extern 'int era_init_lib()'
extern 'void era_deinit_lib()'
extern 'int era_process_request(const char* request, char** response)'
extern 'void era_free(char* response)'
end
Era.era_init_lib
begin
# ...
ensure
Era.era_deinit_lib
end
The shared library loads without issues. However when I call Era.era_init_lib it tries to load additional libraries (Network.so and Protobuf.so). I have these file located in the current working directory (in the same directory as ServerApi.so).
However when I try to execute the code above I receive the following error:
! Failed to load library: /home/username/.rvm/rubies/ruby-2.6.5/bin/Network.so, error: /home/username/.rvm/rubies/ruby-2.6.5/bin/Network.so: cannot open shared object file: No such file or directory
If I place the file at the location the error describes everything works fine.
My guess is that the C working directory of fiddle is different from the Ruby working directory. I would like to keep the project files within the project and not in the Ruby installation directory.
How can I use Network.so from my project folder?
All the *.so files are provided by a third-party. I do not have the source and as a result cannot change these files. The function signatures are provided by the documentation.
Searching for Network.so in the strace gives me these results:
readlink("/proc/self/exe", "/home/username/.rvm/rubies/ruby-2."..., 4096) = 44
openat(AT_FDCWD, "/home/username/.rvm/rubies/ruby-2.6.5/bin/Network.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
futex(0x7fcc16666d90, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x7fcc16b44520, FUTEX_WAKE_PRIVATE, 2147483647) = 0
write(2, "! Failed to load library: ", 26! Failed to load library: ) = 26
write(2, "/home/username/.rvm/rubies/ruby-2."..., 50/home/username/.rvm/rubies/ruby-2.6.5/bin/Network.so) = 50
write(2, ", error: ", 9, error: ) = 9
write(2, "/home/username/.rvm/rubies/ruby-2."..., 109/home/username/.rvm/rubies/ruby-2.6.5/bin/Network.so: cannot open shared object file: No such file or directory) = 109
write(2, "\n", 1) = 1
I've also written a C script which does the same thing which works perfectly fine when the files are dropped into the same directory. So it might be the fault of the library, which I assume checks the location of the current running program, then tries to load the library from that folder. This would explain the behavior when ran as a Ruby script (since it runs as part of the Ruby program), whereas a C binary runs standalone.
For those that want to re-create the (Linux) issue. You can download the necessary files from here. Which gives you the server-linux-x86_64.sh file.
Supported distros are: Suse, Ubuntu, Debian, Red Hat and CentOS but others may also work fine.
You can either run the installer, which should place the files in /opt/eset/RemoteAdministrator/Server. Or, assuming most of you don't want to install the full application you can run the following command:
sed '1,/^# Start of TAR\.GZ file #$/d' server-linux-x86_64.sh | sed '1d' > server-linux-x86_64.tar.gz
Which removes all the installer instructions from the .sh file and only leaves the binary .tar.gz data, writing it to server-linux-x86_64.tar.gz.
Copy the files ServerApi.so, Protobuf.so and Network.so into a directory of your liking. Create a Ruby script (with the question code) in the same directory and run the script.
Because ServerApi.so checks /proc/self/exe for the location of all subsequent files to load, and it is very difficult to modify this target by normal means, it is easier to just modify ServerApi.so itself so that it uses something else besides proc for the source.
If we run strings ServerApi.so, we can verify that the location to check is stored inside a string in ServerApi.so:
strings ServerApi.so | grep 'proc/self/exe'
B/proc/self/exe
So now all we need to do is modify this string to something else that works for us.
The easiest way to modify the string is to replace it with something that is exactly the same length as the original. This way we do not have to worry about changing the end-of-string zero padding or accidentally changing the total size of ServerApi.so.
Here we can see a suitable candidate could be /tmp/scriptexe:
/proc/self/exe
/tmp/scriptexe <- same length
So let's do that:
sed -e 's/proc\/self\/exe/tmp\/scriptexe/' ServerApi.so > ServerApi_Mod.so
Now we can verify the change:
strings ServerApi_Mod.so | grep scriptexe
B/tmp/scriptexe
Next we need to create /tmp/scriptexe to actually point to our Ruby script:
ln -s /the/full/path/to/our/ruby/script.rb /tmp/scriptexe
Then we modify our script:
dlload './ServerApi_Mod.so
Now we can run it as normal:
ruby script.rb
And everything should work.
If we read the strace output we see that the library obtains the current executable location from /proc/self/exe, and then searches subsequent libraries from there.
/proc/self/exe is not easily modifiable, but by using a hard link to a Ruby executable in the current directory we can trick it to point to a new folder.
Problem is making a hard link requires root.
In any case, here is a self-contained solution (note that it will ask for root password the first time you run it, in order to create the hard link).
Put this at the top of your script:
# Obtain path to current executable
exe = File.readlink("/proc/self/exe")
# Check if we are running the hard-liked version
if !exe.match /localruby/
if !File.exist?('localruby')
# Create a hard link to the current Ruby exe using sudo
system("sudo ln #{exe} localruby")
end
puts "Restarting..."
# In order to prevent infinite busy loop in case of some mishap
sleep 1
# Rerun self using the hard-linked Ruby executable.
# This will make /proc/self/exe point to the hard-link, which then
# allows the ESET library to search for .so files in current folder.
exec('./localruby', File.expand_path(__FILE__))
end
require 'fiddle'
require 'fiddle/import'
# ...rest of your script goes here...
A simple solution without any extra Ruby code is to just create the hard link manually, and then always run the script with ./localruby myscript.rb, instead of using the normal ruby myscript.rb.
I have recently started studying parallel processing in Julia and I am having an issue which I don't really understand how to fix.
After having executed Julia with julia -p 4 I want to load the Distributions module in all the processes and I would like to define a Type which depends on Distributions.
The following apparently works correctly when I include it:
#everywhere using Distributions
type TypeDistrib{T <: Float64}
d::Distributions.Normal{T}
end
If I write exactly the same code as a module, then it doesn't:
module test
#everywhere using Distributions
type TypeDistrib{T <: Float64}
d::Distributions.Normal{T}
end
export TypeDistrib
end
The above gives the following error message:
ERROR: LoadError: UndefVarError: Distributions not defined
in include_from_node1(::String) at ./loading.jl:488
in include_from_node1(::String) at /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib:?
while loading /***/test.jl, in expression starting on line 4
Would you please clarify what I am not doing correctly here?
Note: I replaced the full path in the error message with *** since it was confusing.
#everywhere something evaluates something in the Main module across all processes. And in this case, not in test module, and therefore the error in the second case and not in the first.
Perhaps
#everywhere module test
using Distributions
type TypeDistrib{T <: Float64}
d::Distributions.Normal{T}
end
export TypeDistrib
end
does what is meant.
In the Julia NMF package a verbose option provides information on convergence using the #printf macro.
How can I access this output without rewriting the NMF package io?
To rephrase, having a function f() containing the macro #printf, how can I access the output outside f()?
This does seem like useful functionality to have: I would suggest that you file an issue with the package.
However, as a quick hack, something like the following should work:
oldout = STDOUT
(rd,wr) = redirect_stdout()
start_reading(rd)
# call your function here
flush_cstdio()
redirect_stdout(oldout)
close(wr)
s = readall(rd)
close(rd)
s
Does python have a simple function for checking if a directory is an actual directory or if it's just an alias to another directory? I'm trying to list all files/folders in a directory but because of these alias folders, I'm getting a lost of stuff that looks like this:
/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bundle/Home/bundle/Home/bundle/Home/bundle/Home/bundle/Home/bundle/Home/bundle/Home/bundle/Home/bundle/Home/bundle/Home/bundle
I know I can write a function that will compare paths and quit if it seems like I'm going in circles, but is there a simple function that does exactly that that I'm not aware of?
E.g.
os.isAlias( …pathname… )
Thanks!
Here's a version of os.path.realpath that works on Mac aliases as well as on symbolic links under Python 2:
from Carbon import File
def osx_realpath (path):
return File.FSResolveAliasFile(path, True)[0].as_pathname()
If you call osx_realpath on each directory before you recurse into it you should avoid duplication. Alternatively you could define something like
def is_osx_realpath (path):
return path == osx_realpath(path)
Here you have to worry a little about false negatives, however. If you filter for is_osx_realpath and the path you start with is an alias, your program will stop without looking at anything.
So far I don't know of a way to do this under Python 3. I have a question here where I'm hoping for an answer. Right now I can't do better than using subprocess.call to invoke something that does the check on the command line.
EDIT: I should add that not only is Carbon.File not available in Python 3, but it is deprecated and so is best avoided in Python 2 as well--however it's the most pragmatic solution I know of for Python 2 at present.
EDIT 2: here is a way to check if a file is an alias that I believe to be Python 3-friendly. However, I don't have code to resolve the alias. I believe you need PyObjC installed.
from AppKit import NSWorkspace
def is_alias (path):
uti, err = NSWorkspace.sharedWorkspace().typeOfFile_error_(
os.path.realpath(path), None)
if err:
raise Exception(unicode(err))
else:
return "com.apple.alias-file" == uti
(source)
The answer above is incorrect.
While it is true that Finder reports symlinks as alias, they are distinct things.
Symlinks are a basic feature of UNIX, but alias are a Apple only feature.
If you doubt this create a symlink to a directory and an alias. The symlink will be small typically 50-100 bytes, whereas the alias can be several MB.
os.path.islink( … ) will report symlinks, but not alias.
I am not sure how you would find them in Python, but the following link shows other methods.
https://stackoverflow.com/a/21151368/838253
You can check whether a file or directory is an alias with the GetFileInfo command in Mac OS X. GetFileInfo -aa foo prints a line with "1" if foo is an alias and "0" if not.
import subprocess
def is_alias(path):
return subprocess.check_output(["GetFileInfo", "-aa", path]) == "1\n"
Seems a little sad to spawn a process for every check, but I think this works with versions of Mac OS X since probably 10.4.4 (2006), 32-bit, 64-bit, Python 2 and Python 3. The version of GetFileInfo I have (from 2009) is a "universal" i386 + PPC binary.
GetFileInfo is part of Xcode, which is large, but you can download the command-line tools separately (see the "Separate Download" section here).
https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/GetFileInfo.1.html
Old question, but I just ran into this myself.
I have no perfect method for checking if the file is an alias, however, if using mimetypes, python will return None for an alias or a symlink. Might be useful in some situations. I've only tested this in python 3.8 on macOS Big Sur.
import mimetypes
for idx, f in enumerate(filepaths):
type = mimetypes.guess_type(f)[0]
print(f"type is: {type}")
returns (without my added comments):
type is: None # <-- Folder Alias
type is: None # <-- File Alias
type is: text/x-python
type is: None # <-- Folder Alias
type is: video/mp4
type is: image/png
type is: None # <-- Folder Alias
type is: None # <-- Symlink
type is: image/png
type is: application/zip
type is: image/png
type is: image/jpeg
type is: None # <-- Symlink
I ran some files through exiftool just to see what types they returned, and aliases and symlinks both showed the following:
File Type : ALIAS
File Type Extension : alias
MIME Type : application/x-macos
You might be able to init the mimetypes for these, but haven't tested and not sure if it will give false positives if anything else shows up as application/x-macos
Assuming I compile my own fresh Ruby (MRI 1.9.3), what is the default LOAD_PATH, and how is that computed?
On my machine, the initial load path looks like this:
$ ruby -e 'puts $LOAD_PATH'
/Users/matt/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1
/Users/matt/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1/x86_64-darwin10.8.0
/Users/matt/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby
/Users/matt/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/vendor_ruby/1.9.1
/Users/matt/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/vendor_ruby/1.9.1/x86_64-darwin10.8.0
/Users/matt/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/vendor_ruby
/Users/matt/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1
/Users/matt/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/x86_64-darwin10.8.0
Armed with grep, an investigation into the Ruby source leads to the definition of ruby_initial_load_paths[] in version.c (this is on Ruby 1.9.3). The first of these that apply (neither NO_INITIAL_LOAD_PATH or RUBY_SEARCH_PATH have been set) is RUBY_SITE_LIB2. Looking at the defines above that definition we see:
#define RUBY_SITE_LIB2 RUBY_SITE_LIB "/"RUBY_LIB_VERSION
and in turn:
#define RUBY_SITE_LIB RUBY_LIB_PREFIX"/site_ruby"
Following this chain of defines, it becomes clear that this corresponds to the first entry in my load path above. Similarly the other constants that go into this variable correspond to the other load path entries.
The ruby_initial_load_paths[] variable is used in ruby_init_loadpath_safe() in ruby.c, where the actual load path is set up for the process.
So the answer to your question is that the initial load path is set at compile time with some #defines, according to how the build has been configured.
Your compilation create a lib directory
on *nix based systems it is generally
/usr/lib/ruby/X.Y.Z
X.Y.Z represents version number of your ruby C api version.