For some obscure reason I have written a bash script which generates some source code, then compiles it, using
... whatever ... | gcc -x c -o /dev/stdout
Now, I want to execute the result on the compilation. How can I make that happen? No use of files please.
As Charles Duffy said, to execute a binary, you'd have to tell your operating system (which seems to be a Unix variant) to load and execute something – and Unix systems only take files to execute them directly.
What you could do is have a process that prepares a memory region containing the ELF binary, fork and jump into that region - but even that is questionable, considering that there's CPU support to suppress exactly that operation (R^X). Basically, what you need is a runtime linker, and shells do not (and also: should not) include something like that.
Let's drop the Bash requirement (which really just sounds like you're trying to find an obvious hole in an application that is older than I am grumpy):
Generally, requiring ELF (which is a file format) and avoiding files at the same time is a tad complicated. GCC generates machine code. If you just want to execute known machine code, put it into some buffer, build a function pointer to that and call it. Simple as that. However, you'd obviously don't have all the nice relocation and dynamic linking that the process of executing an ELF binary or loading a shared object (dlopen) would have.
If you want that, I'd look in the direction of things like LLVM – I know, for a fact, that there's people building "I compile C++ at runtime and execute it" with LLVM as executing instance, and clang as compiler. In the end, what your gcc|something is is really just JIT – an old technology :)
If your goal is to not write to the filesystem at all, then neither bash nor any other UNIX program will be able to help you execute an ELF from a pipe - execve only takes a path to a regular file as its filename and will fail (setting errno to EACCES) if you pass it a special file (device or named pipe) or a directory.
However, if your goal is to keep the executable entirely in RAM and not touch the hard disk (perhaps because the disk is read-only) you can do something with the same effect on your machine by using tmpfs, which comes with many UNIX-like systems (and is used in Linux to implement semaphores) and allows you to create a full-permissions filesystem that resides entirely in RAM:
$ sudo mount -t tmpfs -o size=10M tmpfs /mnt/mytmpfs
You can then write your binary to that:
... whatever ... | gcc -x c -o /mnt/mytmpfs/program.out
/mnt/mytmpfs/program.out
and bash will load it for you as if it was on disk.
Note, however, that you do still need enough RAM onboard the device to store and execute the program - though due to the nature of most executable binaries, you would need that anyway.
If you don't want to leave the program behind on your ramdisk (or normal disk, if that is acceptable) for others to find, you can also delete the file immediately after starting to execute it:
/mnt/mytmpfs/program.out &
rm /mnt/mytmpfs/program.out
The name will disappear immediately, but the process will internally hold a hard link to that file, then release that hard link when it terminates, allowing the file to be immediately deleted from disk. (However, the storage won't actually be freed until the program exits, and the program will not be able to exec itself either).
Related
Say I source a bash script from within $PROMPT_COMMAND, which is to say at every time enter is pressed, which makes it quite often, does bash optimize this somehow when the file wasn't changed?
EDIT:
Just to clarify, I only ask about loading the script's content from disk, not optimizing the code itself.
An example an optimization one could manually do is check if the sourced file has the same modified date and size[1], if so, then not read the file from disk again and use an already parsed script from memory and execute that instead. If that file contains only bash function definitions then one could also imagine an optimization where these definitions need not be changed (reevaluated) at all - given that the contents are the same.
Is checking file size and modified date sufficient to determine if a file has changed? It can certainly be subverted but given that this is what rsync does by default then it surely is a method to consider.
[1] If a filesystem also stores checksums for files then this would be an even better way to determine if a file on disk has or hasn't changed.
Just to avoid misunderstandings regarding the term optimization:
It seems you are concerned with the time it takes to load the sourced file from the disk (this special form of optimization is usually called caching)
... not about the time it takes to executed an already loaded file (optimization as done by compilers, e.g. gcc -O2)
As far as I know, bash neither caches file contents nor does it optimize scripts. Although the underlying file system or operating system may cache files, bash would have to parse the cached file again; which probably takes longer than loading it from a modern disk (e.g. an SSD).
I wouldn't worry too much about such things unless they actually become a problem for you. If they do, you can easily ...
Cache the script yourself
Wrap the entire content of the sourced file in a function definition. Then source the file once on shell startup. After that, you can run the function from memory.
define-my-prompt-command.sh
my_prompt_command() {
# a big script
}
.bashrc
source define-my-prompt-command.sh
PROMPT_COMMAND=my_prompt_command
You can try to add following snippet to your script :
if ! type reload-sourced-file > /dev/null 2>&1 ; then
echo "This is run when you first source the file..."
PROMPT_COMMAND=$'reload-sourced-file\n'"$PROMPT_COMMAND"
absolute_script_path=$(realpath -e $BASH_SOURCE)
script_previous_stat=$(stat -c "%B%y%z" $absolute_script_path)
fi
reload-sourced-file(){
local stat=$(stat -c "%B%y%z" "$absolute_script_path")
if ! test "$stat" = "$script_previous_stat"; then # Re-source when stat changes.
script_previous_stat="$stat"
echo "You'll see this message for the following Re-sourcings."
source "$absolute_script_path"
fi
}
the script will be re-sourced when stat changes. Hopefully the stat is cached by the file system.
Is there a way to see the original code of a executable sh script. (I am very new to Linux and trying to understand what things do and such.)
If you know how I need very clear step by step process so I can just type i the commands and run them.
Thanks for your help. Trying to learn (Windows man for 25 years here)
A shell script specifically can be seen in the original text form by simply printing the contents of the file:
cat disk-space.sh.x
Several caveats:
If you mean an executable rather than a script the situation is different. Scripts are read by an interpreter at runtime, which then executes it line by line. Executables may be either scripts or ELF binaries. The latter have been transformed from the original source code to a machine readable form which is very much harder to read for humans.
The extension of the file (.sh.x or .x) does not control whether the file contents are executed as a binary or script.
If the file really is a script it may have been obfuscated, meaning that the source code on your system has deliberately been changed to make the resulting file hard to read.
I'm working a small JS project and trying to get a script to run, which compiles some source files that are written in our own "language x".
To run the compiler normally you would use the command ./a.out < source.x And it would print out success or compilation errors etc.
In the case now, I'm trying to working between two directories and using this command:
sudo ~/Documents/server/xCompiler/./a.out < ~/Documents/server/xPrograms/source.x
But this produces no output into the terminal at all and doesn't affect the output files. Is there somthing I'm doing wrong with the use of <? I'm planning to use it in child_process.exec within a node server later.
Any help would be appreciated, I'm a bit stumped.
Thanks.
Redirection operators (<, >, and others like them) describe operations to be performed by the shell before your command is run at all. Because these operations are performed by the shell itself, it's extremely unlikely that they would be broken in a way specific to an individual command: When they're performed, the command hasn't started yet.
There are, however, some more pertinent ways your first and second commands differ:
The second (non-working) one uses a fully-qualified path to the compiler itself. That means that the directory that the compiler is found in and the current working directory where the compiler is running can differ. If the compiler looks for files in or in locations relative to its current working directory, this can cause a failure.
The second uses sudo to escalate privileges to run the compiler. This means you're running as a different user, with most environment variables cleared or modified (unless explicitly whitelisted in /etc/sudoers) during the switch -- and has widespread potential to break things depending on details of your compiler's expectations about its runtime environment beyond what we can reasonably be expected to diagnose here.
That first one, at least, is amenable to a solution. In shell:
xCompile() {
(cd ~/Documents/server/xCompiler && exec ./a.out "$#")
}
xCompile < ~/Documents/server/xPrograms/source.x
Using exec is a performance optimization: It balances the cost of creating a new subshell (with the parenthesis) by consuming that subshell to launch the compiler rather than launching it as a subprocess.
Calling the node child_process.exec(), you can simply pass the desired runtime directory in the cwd argument, so no shell function is necessary.
I am trying to execute a binary file on a xeon phi coprocessor, and it is coming back with "bash: cannot execute binary file". So I am trying to find how to either view an error log or have it display what's happening when I tell it to execute that is causing it not work. I have already tried bash --verbose but it didn't display any additional information. Any ideas?
You don't specify where you compiled your executable nor where you tried to execute from.
To compile a program on the host system to be executed directly on the coprocessor, you must either:
if using one of the Intel compilers, add -mmic to the compiler
command line
if using gcc, use the cross-compilers provided with the MPSS
(/usr/linux-k1om-4.7) - note, however, that the gcc compiler does not
take advantage of vectorization on the coprocessor
If you want to compile directly on the coprocessor, you can install the necessary files from the additional rpm files provided for the coprocessor (found in mpss-/k1om) using the directions from the MPSS user's guide for installing additional rpm files.
To run a program on the coprocessor, if you have compiled it on the host, you must either:
copy your executable file and required libraries to the coprocessor
using scp before you ssh to the coprocessor yourself to execute the
code.
use the micnativeloadex command on the host - you can find a man page
for that on the host.
If you are writing a program using the offload model (part of the work is done using the host then some of the work is passed off to the coprocessor), you can compile on the host using the Intel compilers with no special options.
Note, however, that, regardless of what method you use, any libraries to be used with an executable for the coprocessor will need themselves to be built for the coprocessor. The default libraries exist but any libraries you add, you need to build a version for the coprocessor in addition to any version you make for the host system.
I highly recommend the articles you will find under https://software.intel.com/en-us/articles/programming-and-compiling-for-intel-many-integrated-core-architecture. These articles are written by people who develop and/or support the various programming tools for the coprocessor and should answer most of your questions.
Update: What's below does NOT answer the OP's question - it is one possible explanation for the cannot execute binary file error, but the fact that the error message is prefixed with bash: indicates that the binary is being invoked correctly (by bash), but is not compatible with the executing platform (compiled for a different architecture) - as #Barmar has already stated in a comment.
Thus, while the following contains some (hopefully still somewhat useful) general information, it does not address the OP's problem.
One possible reason for cannot execute binary file is to mistakenly pass a binary (executable) file -- rather than a shell script (text file containing shell code) -- as an operand (filename argument) to bash.
The following demonstrates the problem:
bash printf # fails with '/usr/bin/printf: /usr/bin/printf: cannot execute binary file'
Note how the mistakenly passed binary's path prefixes the error message twice; If the first prefix says bash: instead, the cause is most likely not a problem of incorrect invocation, but one of trying to a invoke an incompatible binary (compiled for a different architecture).
If you want bash to invoke a binary, you must use the -c option to pass it, which allows you to specify an entire command line; i.e., the binary plus arguments; e.g.:
bash -c '/usr/bin/printf "%s\n" "hello"' # -> 'hello'
If you pass a mere binary filename instead of a full path - e.g., -c 'program ...' - then a binary by that name must exist in one of the directories listed in the $PATH variable that bash sees, otherwise you'll get a command not found error.
If, by contrast, the binary is located in the current directory, you must prefix the filename with ./ for bash to find it; e.g. -c './program ...'
I ask because I recently made a change to a KornShell (ksh) script that was executing. A short while after I saved my changes, the executing process failed. Judging from the error message, it looked as though the running process had seen some -- but not all -- of my changes. This strongly suggests that when a shell script is invoked, the entire script is not read into memory.
If this conclusion is correct, it suggests that one should avoid making changes to scripts that are running.
$ uname -a
SunOS blahblah 5.9 Generic_122300-61 sun4u sparc SUNW,Sun-Fire-15000
No. Shell scripts are read either line-by-line, or command-by-command followed by ;s, with the exception of blocks such as if ... fi blocks which are interpreted as a chunk:
A shell script is a text file containing shell commands. When such a
file is used as the first non-option argument when invoking Bash, and
neither the -c nor -s option is supplied (see Invoking Bash), Bash
reads and executes commands from the file, then exits. This mode of
operation creates a non-interactive shell.
You can demonstrate that the shell waits for the fi of an if block to execute commands by typing them manually on the command line.
http://www.gnu.org/software/bash/manual/bashref.html#Executing-Commands
http://www.gnu.org/software/bash/manual/bashref.html#Shell-Scripts
It's funny that most OS'es I know, do NOT read the entire content of any script in memory, and run it from disk. Doing otherwise would allow making changes to the script, while running. I don't understand why that is done, given the fact :
scripts are usually very small (and don't take many memory anyway)
at some point, and shown in this thread, people would start making changes to a script that is already running anyway
But, acknowledging this, here's something to think about: If you decided that a script is not running OK (because you are writing/changing/debugging), do you care on the rest of the running of that script ? you can go ahead making the changes, save them, and ignore all output and actions, done by the current run.
But .. Sometimes, and that depends on the script in question, a subsequent run of the same script (modified or not), can become a problem since the current/previous run is doing an abnormal run. It would typically skip some stuff, or sudenly jump to parts in the script, it shouldn't. And THAT may be a problem. It may leave "things" in a bad state; particularly if file manipulation/creation is involved.
So, as a general rule : even if the OS supports the feature or not, it's best to let the current run finish, and THEN save the updated script. You can change it already, but don't save it.
It's not like in the old days of DOS, where you actually have only one screen in front of you (one DOS screen), so you can't say you need to wait on run completion, before you can open a file again.
No they are not and there are many good reasons for that.
One of the things you should keep in mind is that a shell is not an interpreter even if there are some similarities. Shells are designed to work with a stream of commands. Either from the TTY ,a PIPE, FIFO or even a socket.
The shell reads from its resource line by line until a EOF is returned by the kernel.
The most shells have no extra support for interpreting files. they work with a file as they would work with a terminal.
In fact this is considered to be a nice feature because you can do interesting stuff like this How do Linux binary installers (.bin, .sh) work?
You can use a binary file and prepend shell scripts. You can't do this with an interpreter. because it parses the whole file or at least it would try it and fail. A shell would just interpret it line by line and doesnt care about the garbage at the end of the file. You just have to make sure the execution of the script gets terminated before it reaches the binary part.