Bash Parameter Expansion - get part of directory path - bash

Lets say I have this variable
FILE=/data/tenant/dummy/TEST/logs/2020-02-03_16.20-LSTRM.log
I'm trying to get the name of the 4th sub directory, in this example TEST
job=${FILE%/*/*} # This gives me /data/tenant/dummy/TEST
Then
name=${job##*/}
This give me exactly what I want - Test
However when I try to use this in for loop like this:
for FILE in "/data/tenant/dummy/*/logs/*$year-*"; do
job=${FILE%/*/*}
echo $job # /data/tenant/dummy/TEST(and few other directories, so far so good)
name=${job##*/}
echo $name
done
The result of echo $name shows the list of files in my current directory I'm in instead of TEST

Main issue is the double quotes groups all files into a single long string:
for FILE in "/data/tenant/dummy/*/logs/*$year-*"
You can see this if you do something like:
for FILE in "/data/tenant/dummy/*/logs/*$year-*"
do
echo "+++++++++++++++++++"
echo "${FILE}"
done
This should print ++++++++++++++ once followed by a single line with a long list of file names.
To process the files individually remove the double quotes, eg:
for FILE in /data/tenant/dummy/*/logs/*$year-*
It's also good practice to wrap individual variable references in double quotes, eg:
job="${FILE%/*/*}"
echo "$job"
name="${job##*/}"
echo "$name"

Related

Bash File names will not append to file from script

Hello I am trying to get all files with Jane's name to a separate file called oldFiles.txt. In a directory called "data" I am reading from a list of file names from a file called list.txt, from which I put all the file names containing the name Jane into the files variable. Then I'm trying to test the files variable with the files in list.txt to ensure they are in the file system, then append the all the files containing jane to the oldFiles.txt file(which will be in the scripts directory), after it tests to make sure the item within the files variable passes.
#!/bin/bash
> oldFiles.txt
files= grep " jane " ../data/list.txt | cut -d' ' -f 3
if test -e ~data/$files; then
for file in $files; do
if test -e ~/scripts/$file; then
echo $file>> oldFiles.txt
else
echo "no files"
fi
done
fi
The above code gets the desired files and displays them correctly, as well as creates the oldFiles.txt file, but when I open the file after running the script I find that nothing was appended to the file. I tried changing the file assignment to a pointer instead files= grep " jane " ../data/list.txt | cut -d' ' -f 3 ---> files=$(grep " jane " ../data/list.txt) to see if that would help by just capturing raw data to write to file, but then the error comes up "too many arguments on line 5" which is the 1st if test statement. The only way I get the script to work semi-properly is when I do ./findJane.sh > oldFiles.txt on the shell command line, which is me essentially manually creating the file. How would I go about this so that I create oldFiles.txt and append to the oldFiles.txt all within the script?
The biggest problem you have is matching names like "jane" or "Jane's", etc. while not matching "Janes". grep provides the options -i (case insensitive match) and -w (whole-word match) which can tailor your search to what you appear to want without having to use the kludge (" jane ") of appending spaces before an after your search term. (to properly do that you would use [[:space:]]jane[[:space:]])
You also have the problem of what is your "script dir" if you call your script from a directory other than the one containing your script, such as calling your script from your $HOME directory with bash script/findJane.sh. In that case your script will attempt to append to $HOME/oldFiles.txt. The positional parameter $0 always contains the full pathname to the current script being run, so you can capture the script directory no matter where you call the script from with:
dirname "$0"
You are using bash, so store all the filenames resulting from your grep command in an array, not some general variable (especially since your use of " jane " suggests that your filenames contain whitespace)
You can make your script much more flexible if you take the information of your input file (e.g list.txt), the term to search for (e.g. "jane"), the location where to check for existence of the files (e.g. $HOME/data) and the output filename to append the names to (e.g. "oldFile.txt") as command line [positonal] parameters. You can give each default values so it behaves as you currently desire without providing any arguments.
Even with the additional scripting flexibility of taking the command line arguments, the script actually has fewer lines simply filling an array using mapfile (synonymous with readarray) and then looping over the contents of the array. You also avoid the additional subshell for dirname with a simple parameter expansion and test whether the path component is empty -- to replace with '.', up to you.
If I've understood your goal correctly, you can put all the pieces together with:
#!/bin/bash
# positional parameters
src="${1:-../data/list.txt}" # 1st param - input (default: ../data/list.txt)
term="${2:-jane}" # 2nd param - search term (default: jane)
data="${3:-$HOME/data}" # 3rd param - file location (defaut: ../data)
outfn="${4:-oldFiles.txt}" # 4th param - output (default: oldFiles.txt)
# save the path to the current script in script
script="$(dirname "$0")"
# if outfn not given, prepend path to script to outfn to output
# in script directory (if script called from elsewhere)
[ -z "$4" ] && outfn="$script/$outfn"
# split names w/term into array
# using the -iw option for case-insensitive whole-word match
mapfile -t files < <(grep -iw "$term" "$src" | cut -d' ' -f 3)
# loop over files array
for ((i=0; i<${#files[#]}; i++)); do
# test existence of file in data directory, redirect name to outfn
[ -e "$data/${files[i]}" ] && printf "%s\n" "${files[i]}" >> "$outfn"
done
(note: test expression and [ expression ] are synonymous, use what you like, though you may find [ expression ] a bit more readable)
(further note: "Janes" being plural is not considered the same as the singular -- adjust the grep expression as desired)
Example Use/Output
As was pointed out in the comment, without a sample of your input file, we cannot provide an exact test to confirm your desired behavior.
Let me know if you have questions.
As far as I can tell, this is what you're going for. This is totally a community effort based on the comments, catching your bugs. Obviously credit to Mark and Jetchisel for finding most of the issues. Notable changes:
Fixed $files to use command substitution
Fixed path to data/$file, assuming you have a directory at ~/data full of files
Fixed the test to not test for a string of files, but just the single file (also using -f to make sure it's a regular file)
Using double brackets — you could also use double quotes instead, but you explicitly have a Bash shebang so there's no harm in using Bash syntax
Adding a second message about not matching files, because there are two possible cases there; you may need to adapt depending on the output you're looking for
Removed the initial empty redirection — if you need to ensure that the file is clear before the rest of the script, then it should be added back, but if not, it's not doing any useful work
Changed the shebang to make sure you're using the user's preferred Bash, and added set -e because you should always add set -e
#!/usr/bin/env bash
set -e
files=$(grep " jane " ../data/list.txt | cut -d' ' -f 3)
for file in $files; do
if [[ -f $HOME/data/$file ]]; then
if [[ -f $HOME/scripts/$file ]]; then
echo "$file" >> oldFiles.txt
else
echo "no matching file"
fi
else
echo "no files"
fi
done

basic question: using a variable content as directory name in Linux

I want to make a new directory with the name of every zip file currently exists in a specific directory, in another one. I have written a 'for loop' for this purpose:
files=NL*.zip
for a in $files; do
b=echo $a;
sudo mkdir '/home/shokufeh/Desktop/normals/T1/"${b}"';
done
but it creates a directory named ${b} for the first round and make errors for next ones.
Could you please tell me what I should do?
You put your variable in simple quotes, there fore '${b}' will never be interpreted. Try this:
for a in NL*.zip; do
sudo mkdir "/home/shokufeh/Desktop/normals/T1/${a}";
done
No need for variables $files and $b.
To summarize, let's say var=3,
echo "$var" will display 3
echo '$var' will display $var
echo "'$var'" will display '3'
echo '"$var"' will display "$var"
Hopefully this makes sense. Quotes function like parentheses and brackets.

Bulk Renaming Files isnt working for me

I am running a shell script on my mac, and i am getting a "No Such file or directory.
The input is: the replacement_name, and the working dir.
The output is: changing all files in the directory from $file to $newfilename
#!/bin/sh
echo "-------------------------------------------------"
echo "Arguments:"
echo "Old File String: $1"
echo "New File Name Head: $2"
echo "Directory to Change: $3"
echo "-------------------------------------------------"
oldname="$1"
newname="$2"
abspath="$3"
echo "Updating all files in '$abspath' to $newname.{extension}"
for file in $(ls $abspath);
do
echo $file
echo $file | sed -e "s/$oldname/$newname/g"
newfilename=$("echo $file| sed -e \"s/$oldname/$newname/g\"")
echo "NEW FILE: $newfilename"
mv $abspath/$file $abspath/$newfilename
done
It seems that it doesnt like assigning the result of my 1-liner to a variable.
old_filename_string_template.dart
test_template.dart
./bulk_rename.sh: line 16: echo old_filename_string.dart| sed -e "s/old_filename_string/test/g": No such file or directory
NEW FILE:
Test Information:
mkdir /_temp_folder
touch old_filename_string_template.a old_filename_string_template.b old_filename_string_template.c old_filename_string1_template.a old_filename_string1_template.b old_filename_string1_template.c old_filename_string3_template.a old_filename_string3_template.b old_filename_string3_template.c
./convert.sh old_filename_string helloworld /_temp_folder
The double quotes here make the shell look for a command whose name (filename, alias, or function name) is the entire string between the quotes. Obviously, no such command exists.
> newfilename=$("echo $file| sed -e \"s/old_filename_string/$1/g\"")
Removing the double quotes inside the parentheses and the backslashes before the remaining ones will fix this particular error.
The construct $(command sequence) is called a command substitution; the shell effectively replaces this string with the standard output obtained by evaluating command sequence in a subshell.
Most of the rest of your script has much too few quotes; so it's really unclear why you added them here in particular. http://shellcheck.net/ is a useful service which will point out a few dozen more trivial errors. Briefly, anything which contains a file name should be between double quotes.
Try to put double quotes outside backticks/subtitutions (not INSIDE backticks/substitutions like $("..."))
newfilename="$(....)"
By the way, please consider to use the package perl rename which already does this bulk file rename very well, with Perl regex style which is easier to use. This perl rename command maybe alreay available in your (Mac) distro. See intro pages.

Bash removes underscores from my filenames?

I'm trying to move files from one S3 bucket to another and put them into a folder structure by date. Put simply all the files at the moment are going into one folder and that folder has over 500,000 files inside of and I now need to sort through all these files and put them into folders by month.
The file names are similar to:
"This_is_a_file_20150403.xml"
So I loop through all the files within an S3 bucket, tokensize and get the date. I create a yearmonth variable ignoring the day and move them into another S3 bucket. But the filename changes to:
"This is a file 20150403.xml"
So when I try to move it, AWS can't find the file. Why has bash removed the underscores from the filename? I tried temporarily storing the filename in tempFilename but it still had the underscores removed.
The code I have at the moment is:
#!/bin/bash
count=0
for filename in $(aws s3 ls s3://stagingbucket)
do
echo $filename
tempFilename=$filename
(IFS='_'; for word in $filename;
do
echo $filename
if [ "$count" -eq 2 ]; then
yearmonth=${word:0:6}
echo $tempFilename
aws s3 cp s3://stagingbucket/$filename s3://archivebucket/$yearmonth/
fi
count=$((count + 1))
done)
done
Any ideas?
Let's look at what your code actually does.
echo $foo
String-splits $foo, breaking it into pieces based on characters in IFS
Evaluates each piece as a glob expression
Passes each result of those glob expressions to echo as a separate argument
echo then prints those arguments with spaces between them.
Instead, use:
echo "$foo"
...to keep your string together -- and, likewise, quote all your other expansions as well.
For much the same reasons (unintended glob expressions), for word in $filename is evil; don't do that.
IFS=_ read -a words <<<"$filename"
for word in "${words[#]}"; do
echo "Processing $word"
done

Why can't I pass a directory as an argument to a for loop in bash?

I have a simple bash script, simple.sh, as follows:
#/usr/local/bin/bash
for i in $1
do
echo The current file is $i
done
When I run it with the following argument:
./simple.sh /home/test/*
it would only print and list out the first file located in the directory.
However, if I change my simple.sh to:
#/usr/local/bin/bash
DIR=/home/test/*
for i in $DIR
do
echo The current file is $i
done
it would correctly print out the files within the directory. Can someone help explain why the argument being passed is not showing the same result?
If you take "$1", it is the first file/directory, which is possible!
You should do it in this way:
for i in "$#"
do
echo The current file is ${i}
done
If you execute it with:
./simple.sh *
They list you all files of the actual dictionary
"$1" is alphabetical the first file/directory of your current directory, and in the for loop, the value of "i" would be e.g. a1.sh and then they would go out of the for loop!
If you do:
DIR=/home/<s.th.>/*
you save the value of all files/directories in DIR!
This is as portable as it gets, has no useless forks to ls and runs with a minimum of CPU cycles wasted:
#!/bin/sh
cd $1
for i in *; do
echo The current file is "$i"
done
Run as ./simple.sh /home/test
Your script does not receive "/home/test/*" as an argument; the shell expands the patter to the list of files that match, and your shell receives multiple arguments, one per matching file. Quoting the argument will work:
./simple.sh "/home/test/*"
Your change to using DIR=/home/test/* did what you expected because filename generation is not performed on the RHS of a variable assignment. When you left $DIR unquoted in the for loop, the pattern was expanded to the list of matching files.
How about list the file manully instead of using *:
#/usr/local/bin/bash
for i in $(ls $1)
do
echo The current file is $i
done
and type
./simple.sh /home/test/

Resources