I'm a bioinformatician, new in the community and quite new about working with bash-commands.
I recently encountered a very trivial error message but for me the issue is a bit complex to fix.
Briefly, when I launch a script with the qsub command (from the master node ) the job does not work and I find the following error message in the 'log' file:
Fatal error: cannot open file
'/data/users/genethongandolfi/scripts/multi454.mse/multi454fasta.manip.r':
No such file or directory
This sounds quite strange for me since the path to the script file called 'multi454fasta.manip.r' is correct (I already checked with the 'find' command).
I also tried to move the script into the home directory /home/genethongandolfi/scripts and the error message changes: the job runs because the system finds the script, but not the input file in the usual path /data/users/genethongandolfi/analysis/etc... . It seems to be something for which the /data/users/... path is not recognized when I launch a job.
There are a couple of reasons why this could be the case:
The file location on the slave node is different from the master
The file permissions on the slave do not permit access to the file
If you can, try logging into the slave node, change to the user running the job, and check the file location and permissions.
Had the same error for a simple c program in form of an .exe
Removing the .exe from the shell script did eventually fix it.
So instead of ./program.exe write ./program
Related
Through the terminal I ran
yarn create xxxxxxx-app in a directory that had whitespace in the URL, but it failed because it couldn't understand the complete URL. So I changed to a directory that didn't have any spaces in it to run the same command and it failed using the same internal path as the previous.
The package installs the scripts and binaries where it should but when it tries to run them, it tries to use the original URL that has the whitespace and fails
Example in a directory with or without whitespace:
runs: C:\Users\First Last\AppData\Local\Yarn\bin\xxxxxxxxx-app
"C:\Users\First" isn't recognized as an internal or external command,
error Command failed.
So internally it uses the same command as the one that failed orginially even though it was run in another directory
both respond as follows:
Exit code: 1
Command: C:\Users\First Last\AppData\Local\Yarn\bin\xxxxxxxxxx-app
Arguments: ./xxxxxxxblog
Directory: C:\Users\First_Last\Github\David-Ventures1
I found that the problem is that the package uses my user account no matter where I try to run the script, and unfortunately my user directory has whitespace in it.
So I need to remove the whitespace.... I tried a couple of times a little while ago but nearly bricked my computer...
I am running Shell script on Oozie using Hue, when the workflow job is submitted the following error is shown:
Cannot run program "FileManipulation.sh"
(in directory "/hadoop/mapred/local/taskTracker/root/jobcache/job_201503271756_0277/attempt_201503271756_0277_m_000000_0/work"):
java.io.IOException: error=2, No such file or directory
What could be the reason ?
Please check directory permission containing FileManipulation.sh file. Give all read, write and execute permissions to directory and its file. Also add path to the shell script in both "Shell command" as well as "Files", sometimes this may cause same problem.
I am trying to run a bash file from install4j6. install4j does indeed try to run the bash file but it just returns an error at the end of the installation. The error is very generic and has no code reference or anything that will help me determine a solution - just a message that says "Error while executing file."
The only thing I can provide is how I have it setup in install4j6 since I am pretty sure that's my issue.
The bash file is defined in the root of my installation directory distribution tree and is named set_permissions.sh. For the sake of eliminating permissions being a cause, the file permission mode is set to 777 (both in install4j and on the file system).
I believe the issue is related to what I have set as my "working directory". I currently have it set to just ".". Is there a way to debug this further? Maybe get an actual error as to why it's not executing?
Ok, first a few things to check:
make sure that you're running the batch file after the install files step (you mention it being at the root of your install)
best to have the wait for termination checked and a variable for the return code.
redirect stderr to the log file (just in case)
As for working directory, . should work, but you can change it to ${installer:sys.installationDir} to make sure that it references the installation directory chosen by the user. You can also set the executable in the form of ${installer:sys.installationDir}\set_permissions.sh
Also, try and run just your shell script to make sure that it works :)
I am invoking a bash shell script using oozie editor in Hue.
I used the shell action in the workflow and tried below different options in shell command:
Uploaded the shell script using 'choose a file'
Gave local directory path where shell script is present
Gave HDFS path where shell script is present
But all these options gave following error:
Cannot run program "sec_test_oozie.sh" (in directory "/data/hadoop/yarn/local/usercache/user/appcache/application_1399542362142_0086/container_1399542362142_0086_01_000002"): java.io.IOException: error=2, No such file or directory
How should I give the shell script execution command?
Where the shell script file should be residing?
You need add file "sec_test_oozie.sh" in oozie shell step. In add files
I think you are creating the file from windows machine which is adding extra line break characters.You need to convert the shell script file to Unix format.I also faced the same issue.Then I created the file from a Linux system and it started working.The error is misguiding.
I want to extend the #SergioRG answer. Oozie, at least with Cloudera's Hue interface is very counterintuitive.
To run a script file, three conditions should be met:
the file is on the HDFS file system, in a folder accessible by Oozie
the file should be indicated in the shell command field
the file should be added with any other dependent file in the "Files+" part of the task card.
I wonder why they didn't add by default the script file you are calling.
Edit: please also check in advanced options (the gear in the left upper corner) if you need to set the path variable (eg. PATH=/usr/local/bin:/usr/bin).
Did you edit sec_test_oozie.sh with the Hue File Browser? Depending on your Hue version it might have corrupted it: hue-list
I encountered the same problem, and the problem was that the script echoed some irrelevant line while the workflow tried to parse it as a property line. Oozie gave a very irrelevant error message of java.io.IOException: error=2, No such file or directory which only added confusion.
You will need to use <file> to add your script.
If you used <capture-output/> then you must make sure that your script prints only "key=value" lines, like java properties, otherwise you will get the error you see java.io.IOException: error=2, No such file or directory with some path pointing to .../yarn/local/usercache/...
We had this issue on a test script, basically if you use an editor that adds wierd characters or line ending to the file, it'll throw this error because the script cannot be used in the container.
Try using nano file.sh to see if any strange characters appear. Then push it back to hdfs with hdfs dfs -put file.sh /path/you/need
Removing the #!/bin/bash from my shell script helped me
"No such a file or directory" oozie cannot locate the file. Please check the AddPath setting in the command.
In the edit node seciton, get the oozie application hdfs path.
Upload the shell script in hdfs oozie application path.
In the oozie edit node step, Shell command - specify the shell script name which is uploaded.
Below that there would be option to AddPath, then add files, add the shell script which was uploaded in the hdfs path.
I'm trying to instal Extension:Scribuntu, but when I try to run simple command, like #invoke I get the error:
Lua error: Internal error: The interpreter exited with status 127.
This is what I found in error logfile:
sh: /var/lib/php-exec//sh /alt/home/webmaster.plast/html/Wiki/extensions/Scribunto/engines/LuaStandalone/lua_ulimit.sh 7 8 51200 \\/alt/home/webmaster.plast/html/Wiki/extensions/Scribunto/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua\: No such file or directory
So my question is: can You tell me if the paths are correct? I've checked the files and they are all in place. Paths also are absolute so I have no idea where the problem is. Are those "\" before path correct?
I have never used shell before so i don't even know the correct syntax. Thank You for simple explanation.
There should be no "\" in the path, but it's possible that this is an artifact of the error reporting/logging machanism. Do check though.
Another thing to check is that the user executing the code actually has the permissions needed to read and run the lua executable (and the directory where it resides, and all parent directories). For apache under Debian or Ubuntu, that user would typically be www-data. It should be fine to make the lua executable readable and runnable for all users.