at the end of a bash script I have some simple code to list csv files in a certain directory:
hdfs dfs -ls $target_dir/*csv
But it does not return the list of csv files that I know are there. The only way I can get it to work is by putting that line in another .sh file and running in from within the parent script.
Any ideas why, and a way round it? thanks
I tried running the same script and it worked for me. Try executing your parent script by adding #!/bin/bash as your first line in the script. It works if you specifically denote it as a bash script.
Using set -x told me that my $target_dir variable was empty when I tried to run the line in my question. The problem is having /*csv on the end of it. Everything works fine without that. $target_dir has the correct value in it and lists ALL the files in the directory. I just can't narrow it down to csv ones. In this case, that happens to be fine.
Related
I have text a file with a list of s3 objects, in the form of:
prefix_x/prefix_y/file_name_1
prefix_w/prefix_z/file_name_88
etc...
I wrote a bash script to delete all of these objects, as follows:
#! /bin/bash
LIST_OF_PATHS=$1
while read FILE_PATH; do
aws s3 rm s3://bucket-name/$FILE_PATH
done < $LIST_OF_PATHS
The script doesn't seem to delete the objects (they appear in the UI and in the terminal when lsing them with the CLI).
Further details and things I've already tried:
deleting the objects manually with similar command in the CLI - it works.
adding ls command to the loop provides no output, whereas typing this command manually on the very same files does give output.
adding sleep 0.1 to each iteration of the loop didn't help either.
of course the script runs - I see the output: delete: s3://bucket-name/prefix_x/prefix_y/file_name_1, but the file doesn't actually get deleted.
running a simpler bash script with the same command and a specific file name (not inside a loop) does delete successfully.
What might be the problem?
Solved!
The issue was that in a script, bash expects the newline char to be '\n', but my input file contained '\r' at the end of each line. More on this can be found here:
https://superuser.com/questions/489180/remove-r-from-echoing-out-in-bash-script/489191
Many thanks to #Barmar, whose comment helped me see this and debug the issue.
I simply changed input file itself, and the script ran perfectly as it was.
Check your AWS permissions on the S3 service.
Check your AWS command line tool configuration.
Check your s3 bucket configuration.
Run your AWS s3 command without using a loop, see if the command is run successfully, before using a loop
It is possible to supply batch commands directly with the -b flag, but if the commands become very long, this is no longer an option. Is there a way to give the path to an .scm script that was written to a file, without having to move the file into the scripts directory?
No as far as I know. What you give in the -b flag is a Scheme statement, which implies your function has already been loaded by the script executor process. You can of course add more directories that are searched for scripts using Edit>Preferences>Folders>Scripts.
If you write your script in Python the problem is a bit different since you can alter the Python path before loading the script code but the command line remains a bit long.
I have a simple script that creates a loop around another script and directly gives the parameters and arguments to that script - here comes the loop into play since the script is supposed to run over several files. The way I wrote it it's currently not working so how should I attach these parameters? I'm fairly new to bash so any help will be appreciated a lot!
#!/bin/bash
SCRIPT_PATH="xx.sh"
for x in {001..031}; do
"$SCRIPT_PATH" /data/raw/"$x"_AE data/processed/"$x"_AE 5 --info
done
There may have a syntax issue in your script, the first path is starting with '/' (/data/raw/...) so it is absolute, but it is NOT the case of the second one data/processed/...; is is intentional?
Ensure there is NO directory/path issue (where is xx.sh located ?)
Ensure the user who launches the script has access permissions on /data directories and sub-directories
Let me know if it fixes your issue?
I'm trying to run several sets of commands in parallel on a few remote hosts.
I've created a script that constructs these commands, and then writes the output in a local file, something along the lines of:
ssh <me>#<ip1> "command" 2> ./path/to/file/newFile1.txt & ssh <me>#<ip2>
"command" 2> ./path/to/file/newFile2.txt & ssh <me>#<ip2> "command" 2>
./path/to/file/newFile3.txt; ...(same repeats itself, with new commands and new
file names)...
My issue is that, when my script runs these commands, I am getting the following errors:
bash: ./path/to/file/newFile1.txt: No such file or directory
bash: ./path/to/file/newFile2.txt: No such file or directory
bash: ./path/to/file/newFile3.txt: No such file or directory
...
These files do NOT exist but will be written. That being said, the directory paths are valid.
The strange thing is that, if I copy and paste the whole big command, then it works without any issue. I'd rather have it automated tho ;).
Any ideas?
Edit - more information:
My filesystem is the following:
- home
- User
- Desktop
- Servers
- Outputs
- ...
I am running the bash script from home/User/Desktop/Servers.
The script creates the commands that need to be run on the remote servers. First thing first, the script creates the directories where the files will be stored.
outputFolder="./Outputs"
...
mkdir -p ${outputFolder}/f{fileNumb}
...
The script then continues to create the commands that will be called on remotes hosts, and their respective outputs will be placed in the created directories.
The directories are there. Running the commands gives me the errors, however printing and then copying the commands into the same location works for some reason. I have also tried to give the full path to directory, still same issue.
Hope I've been a bit clearer.
If this is the exact error message you get:
bash: ./path/to/file/newFile1.txt: No such file or directory
Then you'll note that there's an extra space between the colon and the dot, so it's actually trying to open a file called " ./path/to/file/newFile1.txt" (without the quotes).
However, to accomplish that, you'd need to use quotes around the filename in the redirection, as in
something ... 2> " ./path/to/file/newFile1.txt"
Or the first character would have to something else than a regular space. A non-breaking space perhaps, possible something that some editor might create if you hit alt-space or such.
I don't believe you've shown enough to correctly answer the question.
This doesn't look like a problem with ssh, but the way you are calling the (ssh) commands.
You say that you are writing the commands into a file... presumably you are then running that file as a script. Could you show the code you use to do that. I believe that's your problem.
I suspect you have made a false assumption about the way the working directory changes when you run a script. It doesn't. You are listing relative paths, so its important to know what they are relative to. That is the most likely reason for it working when you copy and paste it... You are executing from a different working directory.
I am new to bash scripting and was building my script based on another one I had seen. I was "running" the command by simply calling the variable where the command was stored:
$cmd
Solved by using:
eval $cmd
instead. My bad, should have given the full script from the start.
I have a bash script which is pretty simple (or so I thought - but I don't write them very often):
cp -f /mnt/storage/vhosts/domain1.COM/private/auditbaseline.php /mnt/storage/vhosts/domain1.COM/httpdocs/modules/mod_monitor/tmpl/audit.php
cp -f /mnt/storage/vhosts/domain1.COM/private/auditbaseline.php /mnt/storage/vhosts/domain2.org/httpdocs/modules/mod_monitor/tmpl/audit.php
The script copies the contents of auditbaseline to both domain 1 and domain 2.
For some reason it won't work. When I have the first line in on its own it's okay but when I add the second line I can't get it to work it locks up the scripts and they can't be accessed.
Any help would be really appreciated.
Did you perhaps create this script on a Windows machine? You should make sure that there are no CRLF line breaks in the file. Try using dos2unix (http://www.linuxcommand.org/man_pages/dos2unix1.html) to convert the file in that case.