I have been following the answers given in these questions
Shellscript Looping Through All Files in a Folder
How to iterate over files in a directory with Bash?
to write a bash script which goes over files inside a folder and processes them. So, here is the code I have:
#!/bin/bash
YEAR="2002/"
INFOLDER="/local/data/datasets/Convergence/"
for f in "$INFOLDER$YEAR*.mdb";
do
echo $f
absname=$INFOLDER$YEAR$(basename $f)
# ... the rest of the script ...
done
I am receiving this error: basename: extra operand.
I added echo $f and I realized that f contains all the filenames separated by space. But I expected to get one at a time. What could be the problem here?
You're running into problems with quoting. In the shell, double-quotes prevent word splitting and wildcard expansion; generally, you don't want these things to happen to variable's values, so you should double-quote variable references. But when you have something that should be word-split or wildcard-expanded, it cannot be double-quoted. In your for statement, you have the entire file pattern in double-quotes:
for f in "$INFOLDER$YEAR*.mdb";
...which prevents word-splitting and wildcard expansion on the variables' values (good) but also prevents it on the * which you need expanded (that's the point of the loop). So you need to quote selectively, with the variables inside quotes and the wildcard outside them:
for f in "$INFOLDER$YEAR"*.mdb;
And then inside the loop, you should double-quote the references to $f in case any filenames contain whitespace or wildcards (which are completely legal in filenames):
echo "$f"
absname="$INFOLDER$YEAR$(basename "$f")"
(Note: the double-quotes around the assignment to absname aren't actually needed -- the right side of an assignment is one of the few places in the shell where it's safe to skip them -- but IMO it's easier and safer to just double-quote all variable references and $( ) expressions than to try to keep track of where it's safe and where it's not.)
Just quote your shell variables if they are supposed to contain strings with spaces in between.
basename "$f"
Not doing so will lead to splitting of the string into separate characters (see WordSplitting in bash), thereby messing up the basename command which expects one string argument rather than multiple.
Also it would be a wise to include the * outside the double-quotes as shell globbing wouldn't work inside them (single or double-quote).
#!/bin/bash
# good practice to lower-case variable names to distinguish them from
# shell environment variables
year="2002/"
in_folder="/local/data/datasets/Convergence/"
for file in "${in_folder}${year}"*.mdb; do
# break the loop gracefully if no files are found
[ -e "$file" ] || continue
echo "$file"
# Worth noting here, the $file returns the name of the file
# with absolute path just as below. You don't need to
# construct in manually
absname=${in_folder}${year}$(basename "$file")
done
just remove "" from this line
for f in "$INFOLDER$YEAR*.mdb";
so it looks like this
#!/bin/bash
YEAR="2002/"
INFOLDER="/local/data/datasets/Convergence/"
for f in $INFOLDER$YEAR*.mdb;
do
echo $f
absname=$INFOLDER$YEAR$(basename $f)
# ... the rest of the script ...
done
Related
I have the following array:
SPECIFIC_FILES=('resources/logo.png' 'resources/splash.png' 'www/img/logo.png' 'www/manifest.json')
And the following variable:
CUSTOMER=default
How can I loop through my array and generate strings that would look like
resources/logo_default.png
depending on the variable.
The below uses parameter expansion to extract the relevant substrings, as also described in BashFAQ #100:
specific_files=('resources/logo.png' 'resources/splash.png' 'www/img/logo.png' 'www/manifest.json')
customer=default
for file in "${specific_files[#]}"; do
[[ $file = *.* ]] || continue # skip files without extensions
prefix=${file%.*} # trim everything including and after last "."
suffix=${file##*.} # trim everything up to and including last "."
printf '%s\n' "${prefix}_$customer.$suffix" # concatenate results of those operations
done
Lower-case variable names are used here in keeping with POSIX-specified conventions (all-caps names are used for variables meaningful to the operating system or shell, whereas variables with at least one lower-case character are reserved for application use; setting a regular shell variable overwrites any like-named environment variable, so the conventions apply to both classes).
Here's a solution with sed:
for f in "${SPECIFIC_FILES[#]}"; do
echo "$f" | sed "s/\(.*\)\.\([^.]*\)/\1_${CUSTOMER}.\2/p"
done
If you know that there is only one period per filename, you can use expansion on each element directly:
$ printf '%s\n' "${SPECIFIC_FILES[#]/./_"$CUSTOMER".}"
resources/logo_default.png
resources/splash_default.png
www/img/logo_default.png
www/manifest_default.json
If you don't, Charles' answer is the robust one covering all cases.
Trying to loop through all files in a directory, check them for the existence of a string, and add it if it doesn’ exist. This is what I have:
#!/bin/bash
FILES=*
for f in $FILES
do
echo "Processing $f file..."
if grep -Fxq '<?xml version="1.0" encoding="UTF-8"?>' $f
then
continue
else
echo '<?xml version="1.0" encoding="UTF-8"?>' | cat - $f > temp && mv temp $f
fi
done
… but the script stops after the first loop. Any ideas why?
A simpler solution would be to use the inplace edit option -i of the sed tool like below
sed -i '1{/^<?xml version="1.0" encoding="UTF-8"?>/!{
s/^/<?xml version="1.0" encoding="UTF-8"?>\n/}}' /path/to/files/*
What are we doing above
The inplace option -i with sed makes any change to the file written to the file.
By 1{} we are processing just the first line of the file
The /^<?xml version="1.0" encoding="UTF-8"?>/! part checks if the string is NOT(note the ! at the end) present in the beginning of the line.
If the above condition is not true we substitute the beginning of the line (^) with <?xml version="1.0" encoding="UTF-8"?>\n using
s/^/<?xml version="1.0" encoding="UTF-8"?>\n/
The rest is closing the curly brackets in the correct order :)
That said, in your original script I see variables like FILES. It is discouraged to use uppercase variables as user variables as they are reserved as environment variables and might lead to conflict. So use files instead.
Again doing
file=*
has the implication of [ word splitting ] and produce undesired results if you have non standard files that contain spaces or even new lines. What you could do is
files=( * ) # This put the files in an array
for file in "${files[#]}" # Double quoting the array prevents word splitting
do
# Do something with "$file" but why bother when you've a one-liner with sed? ;-)
done
Note: For sed manual visit [ here ]
I want to clear some things up about word splitting and filename expansion I saw here in the comments.
When using variable assignment, quoting Bash Reference Manual, only the following expansions are done: tilde expansion, parameter expansion, command substitution, arithmetic expansion. This means that there really is just an asterisk in your variable $files as there is no filename expansion taking place. So at this point you don't need to worry about newlines, spaces etc. because there are no actual files in your variable. You can see this with declare -p files.
This is the reason you don't have to quote when assigning to a variable.
var=$othervariable
is the same as:
var="$othervariable"
Now, when you use your variable $files in the for loop for f in $files (note that you cannot quote $files here because the filename expansion wouldn't take place) that variable gets expanded and undergoes word splitting. But the actual value is JUST the asterisk and word splitting won't do anything with the result! Quoting the manual again:
After word splitting, unless the -f option has been set (see The Set
Builtin), Bash scans each word for the characters ‘*’, ‘?’, and ‘[’.
If one of these characters appears, then the word is regarded as a
pattern, and replaced with an alphabetically sorted list of filenames
matching the pattern (see Pattern Matching).
Meaning of this is that filename expansion is done after variable expansion and word splitting. So the files expanded by the filename expansion won't be split by IFS! Therefore, the following code works just fine:
#!/usr/bin/env bash
files=*
for f in $files; do
echo "<<${f}>>"
done
and correctly outputs:
<<file with many spaces>>
<<filewith* weird characters[abc]>>
<<normalfile>>
A shorter version of this is obviously to use for f in * instead of the variable $files. You also definitely want to quote any usage of $f in your loop as that expansion really does undergo the word splitting.
This being said, your loop should function properly.
I am trying to copy a .nii file (Gabor3.nii) path to a variable but even though the file is found by the find command, I can't copy the path to the variable.
find . -type f -name "*.nii"
Data= '/$PWD/"*.nii"'
output:
./Gabor3.nii
./hello.sh: line 21: /$PWD/"*.nii": No such file or directory
What went wrong
You show that you're using:
Data= '/$PWD/"*.nii"'
The space means that the Data= parts sets an environment variable $Data to an empty string, and then attempts to run '/$PWD/"*.nii"'. The single quotes mean that what is between them is not expanded, and you don't have a directory /$PWD (that's a directory name of $, P, W, D in the root directory), so the script "*.nii" isn't found in it, hence the error message.
Using arrays
OK; that's what's wrong. What's right?
You have a couple of options. The most reliable is to use an array assignment and shell expansion:
Data=( "$PWD"/*.nii )
The parentheses (note the absence of spaces before the ( — that's crucial) makes it an array assignment. Using shell globbing gives a list of names, preserving spaces etc in the names correctly. Using double quotes around "$PWD" ensures that the expansion is correct even if there are spaces in the current directory name.
You can find out how many files there are in the list with:
echo "${#Data[#]}"
You can iterate over the list of file names with:
for file in "${Data[#]}"
do
echo "File is [$file]"
ls -l "$file"
done
Note that variable references must be in double quotes for names with spaces to work correctly. The "${Data[#]}" notation has parallels with "$#", which also preserves spaces in the arguments to the command. There is a "${Data[*]}" variant which behaves analogously to "$*", and is of similarly limited value.
If you're worried that there might not be any files with the extension, then use shopt -s nullglob to expand the globbing expression into an empty list rather than the unexpanded expression which is the historical default. You can unset the option with shopt -u nullglob if necessary.
Alternatives
Alternatives involve things like using command substitution Data=$(ls "$PWD"/*.nii), but this is vastly inferior to using an array unless neither the path in $PWD nor the file names contain any spaces, tabs, newlines. If there is no white space in the names, it works OK; you can iterate over:
for file in $Data
do
echo "No white space [$file]"
ls -l "$file"
done
but this is altogether less satisfactory if there are (or might be) any white space characters around.
You can use command substitution:
Data=$(find . -type f -name "*.nii" -print -quit)
To prevent multiline output, the -quit option stop searching after the first file was found(unless you're sure only one file will be found or you want to process multiple files).
The syntax to do what you seem to be trying to do with:
Data= '/$PWD/"*.nii"'
would be:
Data="$(ls "$PWD"/*.nii)"
Not saying it's the best approach for whatever you want to do next of course, it's probably not...
I want to write a script that takes a name of a folder as a command line argument and produces a file that contains the names of all subfolders with size 0 (empty subfolder). This is what I got:
#!/bin/bash
echo "Name of a folder'
read FOLDER
for entry in "$search_dir"/*
do
echo "$entry"
done
your script doesn't have the logic you intended. find command has a feature for this
$ find path/to/dir -type d -empty
will print empty directories starting from the given path/to/dir
I would suggest you accept the answer which suggests to use find instead. But just to be complete, here is some feedback on your code.
You read the input directory into FOLDER but then never use this variable.
As an aside, don't use uppercase for your private variables; this is reserved for system variables.
You have unpaired quotes in the prompt string. If the opening quote is double, you need to close with a double quote, or vice versa for single quotes.
You loop over directory entries, but do nothing to isolate just the ones which are directories, let alone empty directories.
Finally, nothing in your script uses Bash-only facilities, so it would be safe and somewhat more portable to use #!/bin/sh
Now, looping over directories can be done by using search_dir/*/ instead of just search_dir/*; and finding out which ones are empty can be done by checking whether a wildcard within the directory returns just the directory itself. (This assumes default globbing behavior -- with nullglob you would make a wildcard with no matches expand to an empty list, but this is problematic in some scenarios so it's not the default.)
#!/bin/bash
# read -p is not POSIX
read -p "Name of a folder" search_dir
for dir in "$search_dir"/*/
do
# [[ is Bash only
if [[ "$dir"/* = "$dir/*" ]]; then # Notice tricky quoting
echo "$dir"
fi
done
Using the wildcard expansion with [ is problematic because it is not prepared to deal with a wildcard expansion -- you get "too many arguments" if the wildcard expands into more than one filename -- so I'm using the somewhat more mild-tempered Bash replacement [[ which copes just fine with this. Alternatively, you could use case, which I would actually prefer here; but I've stuck to if in order to make only minimal changes to your script.
I was given a tip to use file globbing in stead of ls in Bash scripts, in my code I followed the instructions and replaced array=($(ls)) to:
function list_files() { for f in *; do [[ -e $f ]] || continue done }
array=($(list_files))
However the new function doen't return anything, am I doing something wrong here?
Simply write this:
array=(*)
Leaving aside that your "list_files" doesn't output anything, there are still other problems with your approach.
Unquoted command substitution (in your case "$(list_files)") will still be subject to "word splitting" and "pathname expansion" (see bash(1) "EXPANSION"), which means that if there are spaces in "list_files" output, they will be used to split it into array elements, and if there are pattern characters, they will be used to attempt to match and substitute the current directory file names as separate array elements.
OTOH, if you quote the command substitution with double quotes, then the whole output will be considered a single array element.