I am running Shell script on Oozie using Hue, when the workflow job is submitted the following error is shown:
Cannot run program "FileManipulation.sh"
(in directory "/hadoop/mapred/local/taskTracker/root/jobcache/job_201503271756_0277/attempt_201503271756_0277_m_000000_0/work"):
java.io.IOException: error=2, No such file or directory
What could be the reason ?
Please check directory permission containing FileManipulation.sh file. Give all read, write and execute permissions to directory and its file. Also add path to the shell script in both "Shell command" as well as "Files", sometimes this may cause same problem.
Related
I am a newbie to Airflow and trying to create a simple task of executing a bash file(whose job is just to create a directory). I have given the full path of the bash file to be executed (with a space at the end) in bash_command. However, upon triggering the DAG from the UI, I see no errors in the log as well as no folder created with the name specified in the bash file.
Can someone please help me fix the issue?
When BashOperator executes, Airflow will create a temporary directory as the working directory and executes the bash command. When the execution finishes, the temporary directory will be deleted.
To keep the directory created from the bash command, you can either
specify an absolute path outside of the working directory, or
change your working directory to a place outside of the temporary directory.
I am creating a test directory in the Airflow home directory.
p = BashOperator(
task_id='create_dir',
bash_command='pwd; mkdir $AIRFLOW_HOME/test; ls -al',
)
I want to execute a shell script as a step on EMR that loads a tarball, unzips it and runs the script inside. I chose this setup to stay as vendor-agnostic as possible.
My script is
#!/bin/sh
aws s3 cp s3://path_to_my_bucket/name_of.tar.gz .
tar -xzf name_of.tar.gz
. main_script.sh
Where main_script.sh is part of the tarball along with a number of other packages, scripts and config files.
If I run this script as a Hadoop user on the master node, everything works as intended. Added as a step via the command-runner.jar, I get errors, no matter what I try.
What I tried so far (and the errors):
running the script as above (file not found "main_script.sh")
hardcoding the path to be the Hadoop users home directory (permission denied on main_script.sh)
dynamically getting the path where script lives (using this) and giving this path as an argument for the tar -C option and invoking main_script.sh explicitly from this path (another permission denied on main_script.sh)
What is the proper way of loading a bash script into the master node and executing it?
As a bonus, I am wondering why the command-runner.jar is set up so different from the spark step, which runs as the Hadoop user in the Hadoop user directory.
you can use script-runner.jar with region
JAR location : s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar
Arguments : s3://your_bucket/your_shell_script.sh
Refer below link for more info
https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hadoop-script.html
I am running a Shell script with Oozie. First I uploaded the script to HDFS, then script should forward its logs to a log file in the same directory where this script is stored in HDFS, meaning the generated log file should be in the HDFS.
Anyone knows how to achieve this goal?
...a log file in the same directory where this script is stored in HDFS...
The Oozie Shell action contains a <file> element with the HDFS path of the script. But the way it works is not what you seem to think:
Oozie aks YARN to allocate a container somewhere
Oozie asks YARN to download some files to the container's private local filesystem (in CWD), and especially the <file> stuff
finally, Oozie asks YARN to run the
local version of the script
Bottom line: the script that is executed has no way to know its original HDFS directory. The Action must pass that directory explicitly as a script argument, or as an env variable.
Assuming that you use an env variable, the obvious solution to run the archive part is something like
hdfs dfs -appendToFile ./MySession.log $LOG_ARCHIVE_DIR/archive.log
I'm a bioinformatician, new in the community and quite new about working with bash-commands.
I recently encountered a very trivial error message but for me the issue is a bit complex to fix.
Briefly, when I launch a script with the qsub command (from the master node ) the job does not work and I find the following error message in the 'log' file:
Fatal error: cannot open file
'/data/users/genethongandolfi/scripts/multi454.mse/multi454fasta.manip.r':
No such file or directory
This sounds quite strange for me since the path to the script file called 'multi454fasta.manip.r' is correct (I already checked with the 'find' command).
I also tried to move the script into the home directory /home/genethongandolfi/scripts and the error message changes: the job runs because the system finds the script, but not the input file in the usual path /data/users/genethongandolfi/analysis/etc... . It seems to be something for which the /data/users/... path is not recognized when I launch a job.
There are a couple of reasons why this could be the case:
The file location on the slave node is different from the master
The file permissions on the slave do not permit access to the file
If you can, try logging into the slave node, change to the user running the job, and check the file location and permissions.
Had the same error for a simple c program in form of an .exe
Removing the .exe from the shell script did eventually fix it.
So instead of ./program.exe write ./program
I am invoking a bash shell script using oozie editor in Hue.
I used the shell action in the workflow and tried below different options in shell command:
Uploaded the shell script using 'choose a file'
Gave local directory path where shell script is present
Gave HDFS path where shell script is present
But all these options gave following error:
Cannot run program "sec_test_oozie.sh" (in directory "/data/hadoop/yarn/local/usercache/user/appcache/application_1399542362142_0086/container_1399542362142_0086_01_000002"): java.io.IOException: error=2, No such file or directory
How should I give the shell script execution command?
Where the shell script file should be residing?
You need add file "sec_test_oozie.sh" in oozie shell step. In add files
I think you are creating the file from windows machine which is adding extra line break characters.You need to convert the shell script file to Unix format.I also faced the same issue.Then I created the file from a Linux system and it started working.The error is misguiding.
I want to extend the #SergioRG answer. Oozie, at least with Cloudera's Hue interface is very counterintuitive.
To run a script file, three conditions should be met:
the file is on the HDFS file system, in a folder accessible by Oozie
the file should be indicated in the shell command field
the file should be added with any other dependent file in the "Files+" part of the task card.
I wonder why they didn't add by default the script file you are calling.
Edit: please also check in advanced options (the gear in the left upper corner) if you need to set the path variable (eg. PATH=/usr/local/bin:/usr/bin).
Did you edit sec_test_oozie.sh with the Hue File Browser? Depending on your Hue version it might have corrupted it: hue-list
I encountered the same problem, and the problem was that the script echoed some irrelevant line while the workflow tried to parse it as a property line. Oozie gave a very irrelevant error message of java.io.IOException: error=2, No such file or directory which only added confusion.
You will need to use <file> to add your script.
If you used <capture-output/> then you must make sure that your script prints only "key=value" lines, like java properties, otherwise you will get the error you see java.io.IOException: error=2, No such file or directory with some path pointing to .../yarn/local/usercache/...
We had this issue on a test script, basically if you use an editor that adds wierd characters or line ending to the file, it'll throw this error because the script cannot be used in the container.
Try using nano file.sh to see if any strange characters appear. Then push it back to hdfs with hdfs dfs -put file.sh /path/you/need
Removing the #!/bin/bash from my shell script helped me
"No such a file or directory" oozie cannot locate the file. Please check the AddPath setting in the command.
In the edit node seciton, get the oozie application hdfs path.
Upload the shell script in hdfs oozie application path.
In the oozie edit node step, Shell command - specify the shell script name which is uploaded.
Below that there would be option to AddPath, then add files, add the shell script which was uploaded in the hdfs path.