I have a simple script that creates a loop around another script and directly gives the parameters and arguments to that script - here comes the loop into play since the script is supposed to run over several files. The way I wrote it it's currently not working so how should I attach these parameters? I'm fairly new to bash so any help will be appreciated a lot!
#!/bin/bash
SCRIPT_PATH="xx.sh"
for x in {001..031}; do
"$SCRIPT_PATH" /data/raw/"$x"_AE data/processed/"$x"_AE 5 --info
done
There may have a syntax issue in your script, the first path is starting with '/' (/data/raw/...) so it is absolute, but it is NOT the case of the second one data/processed/...; is is intentional?
Ensure there is NO directory/path issue (where is xx.sh located ?)
Ensure the user who launches the script has access permissions on /data directories and sub-directories
Let me know if it fixes your issue?
Related
at the end of a bash script I have some simple code to list csv files in a certain directory:
hdfs dfs -ls $target_dir/*csv
But it does not return the list of csv files that I know are there. The only way I can get it to work is by putting that line in another .sh file and running in from within the parent script.
Any ideas why, and a way round it? thanks
I tried running the same script and it worked for me. Try executing your parent script by adding #!/bin/bash as your first line in the script. It works if you specifically denote it as a bash script.
Using set -x told me that my $target_dir variable was empty when I tried to run the line in my question. The problem is having /*csv on the end of it. Everything works fine without that. $target_dir has the correct value in it and lists ALL the files in the directory. I just can't narrow it down to csv ones. In this case, that happens to be fine.
I'm trying to run several sets of commands in parallel on a few remote hosts.
I've created a script that constructs these commands, and then writes the output in a local file, something along the lines of:
ssh <me>#<ip1> "command" 2> ./path/to/file/newFile1.txt & ssh <me>#<ip2>
"command" 2> ./path/to/file/newFile2.txt & ssh <me>#<ip2> "command" 2>
./path/to/file/newFile3.txt; ...(same repeats itself, with new commands and new
file names)...
My issue is that, when my script runs these commands, I am getting the following errors:
bash: ./path/to/file/newFile1.txt: No such file or directory
bash: ./path/to/file/newFile2.txt: No such file or directory
bash: ./path/to/file/newFile3.txt: No such file or directory
...
These files do NOT exist but will be written. That being said, the directory paths are valid.
The strange thing is that, if I copy and paste the whole big command, then it works without any issue. I'd rather have it automated tho ;).
Any ideas?
Edit - more information:
My filesystem is the following:
- home
- User
- Desktop
- Servers
- Outputs
- ...
I am running the bash script from home/User/Desktop/Servers.
The script creates the commands that need to be run on the remote servers. First thing first, the script creates the directories where the files will be stored.
outputFolder="./Outputs"
...
mkdir -p ${outputFolder}/f{fileNumb}
...
The script then continues to create the commands that will be called on remotes hosts, and their respective outputs will be placed in the created directories.
The directories are there. Running the commands gives me the errors, however printing and then copying the commands into the same location works for some reason. I have also tried to give the full path to directory, still same issue.
Hope I've been a bit clearer.
If this is the exact error message you get:
bash: ./path/to/file/newFile1.txt: No such file or directory
Then you'll note that there's an extra space between the colon and the dot, so it's actually trying to open a file called " ./path/to/file/newFile1.txt" (without the quotes).
However, to accomplish that, you'd need to use quotes around the filename in the redirection, as in
something ... 2> " ./path/to/file/newFile1.txt"
Or the first character would have to something else than a regular space. A non-breaking space perhaps, possible something that some editor might create if you hit alt-space or such.
I don't believe you've shown enough to correctly answer the question.
This doesn't look like a problem with ssh, but the way you are calling the (ssh) commands.
You say that you are writing the commands into a file... presumably you are then running that file as a script. Could you show the code you use to do that. I believe that's your problem.
I suspect you have made a false assumption about the way the working directory changes when you run a script. It doesn't. You are listing relative paths, so its important to know what they are relative to. That is the most likely reason for it working when you copy and paste it... You are executing from a different working directory.
I am new to bash scripting and was building my script based on another one I had seen. I was "running" the command by simply calling the variable where the command was stored:
$cmd
Solved by using:
eval $cmd
instead. My bad, should have given the full script from the start.
I'm writing a shell script. what it does is it will create a file by the input that is received from the user. Now, i want to add the feature called "view a file" for my current script. Now, it's unreasonal to retype it again since i've already had a script that helps
I know it's crazy when it is possible to it with normal shell command. I'm actually writing a script that help me to create pages that are generated from the touch command. (this pages had attached date, author name, subjects, and title).
The question is how to call a another script or inhere another script?
Couple of ways to do this. My prefered way is by using source
You can -
Call your other script with the source command (alias is .) like this: source /path/to/script.
Make the other script executable, add the #!/bin/bash line at the top, and the path where the file is to the $PATH environment variable. Then you can call it as a normal command.
Use the bash command to execute it: /bin/bash /path/to/script
I have an ultra-simple script that houses the Linux program shred, and it contains the parameters that have always worked from the command line (bash). Specifically 'shred -uzn 35'
The script, named D, has execute permissions set.
When I run the script, bash prints an error:
$ D some_file_to_delete
shred: missing file operand
I realize that the solution to the problem is probably as simple as the program itself. Please help?
Thanks in advance.
EDIT: The error "missing file operand" was due to the fact that the script was not set to take arguments, such as via "$#". Also, as stated in the accepted answer, I agree that an alias makes total sense for such a scenario (much more sense than, say, a script somewhere in $PATH).
Since you are using a script, not an alias, you need to pass the arguments through
shred -uzn 35 "$#"
In this case, however, I suggest you do make it an alias. In your .bashrc file, add this:
alias D='shred -uzn 35'
I have a ruby script containing system command like http://gist.github.com/235833, while I ran this script from shell, it works correctly, but when I added it to my cron job list, it doesn't work any more, the cron job is like:
10/* * * * * cd /home/hekin; /usr/bin/ruby my_script.rb
any idea what's going wrong with what i've done? Thank you.
Thank you all for your answers.
It's my mistake.
Since I'm using ssh key forwarding on the local machine, while I executed the script from the shell, the ssh key forwarding related environment variables are all sitting there, but from cron job context, those environment variables are missing.
Try to separate the things that might go wrong. The ones I can think of are:
The cron syntax - is the time value given legal and fitting your shell?
Permissions - execute permissions and read permissions for the relevant directory and file
Quoting - what scope does cron cover? Does it run only the first command?
In order to dissect this, I suggest you first run a really simple cron job, like 'ls'. Next run a single-liner script. Next embed your commands in a shell-script file. Somewhere along these lines you should find the problem.
The problem is your environment. While testing in your shell its fully equipped and boosted by your shell environment. While running under cron its very, very stripped down.
Where is the destination "." for your script? I guess it will be "/" and may not "$HOME" thus your script won't be able to write at that location and fails. Try using an absolut path for the destination.