Files created in 'testdir':
file1 file2.old file3old file4.old
Execution of 'oldfiles2 testdir':
Files in 'testdir' after 'oldfiles2' was run:
file1.old file2.old file3old.old file4.old
Error: 'for' does not seem to loop only through required filenames
Please hit to continue with the Assignment
Is the error I am hitting with a script running for school,
Here is the script below
#!/bin/bash
shopt -s extglob nullglob
dir=$1
for file in "$dir"/!(*.old)
do
[[ $file == *.old ]] || mv -- "$file" "$file.old"
done
The assignment was written by someone who doesn't know bash well. Your approach is way better.
Instead of grepping ls, you can use extglob (and also nullglob in case there are no matches):
#!/bin/bash
shopt -s extglob nullglob
for file in "$dir"/!(*.old)
do
mv -- "$file" "$file.old"
done
As demonstrated by your test validator's output, it works perfectly:
file1 does not end in .old, and so it's renamed to file1.old
file2.old ends in .old, and is not renamed.
file3old does not end in .old (old != .old), and is renamed.
file4.old ends in .old, and is not renamed.
However, the validator refuses to accept it, indicating that the validator is wrong. A common mistake for people who don't know bash well (like your professor) is to use grep -v .old or grep -v '.old$', which doesn't actually check if files end .old because . means "any character".
We can emulate this bug in the script:
#!/bin/bash
shopt -s extglob nullglob
for file in "$dir"/!(*?old*)
do
mv -- "$file" "$file.old"
done
This code is objectively wrong, but may pass the incorrect validator. Alternatively, "$dir"/!(*?old) will emulate a buggy grep anchored to the end of the line.
If I read correctly what your teacher wants, then here is a one liner using grep -v and no if statement. You can block it out in the script or leave it as a one liner.
ls | grep -v '\.old' | while read FILE; do mv "${FILE}" "${FILE}.old"; done
BTW I've tested this and it works because the "." in '\.old' is a dot (or period) and not "any character" because it's escaped with a backslash.
Here is sample output from Terminal
System1:test 123$ ls -1
file name 1
file name 2
file name.old
file.old
file1
file2
System1:test 123$ ls | grep -v '\.old' | while read FILE; do mv "${FILE}" "${FILE}.old"; done
System1:test 123$ ls -1
file name 1.old
file name 2.old
file name.old
file.old
file1.old
file2.old
System1:test 123$
Try:
#!/bin/bash
for filename in $(ls $1 | grep -v "\.old$")
do
mv $1/$filename $1/$filename.old
done
In Bash you can use character classes beginning with the inversion character ^ or ! to match all characters except the listed character. In your case:
for file in "$dir"/*.[^o][^l][^d]*; do
[ "$file" = *.old ] || mv -- "$file" "$file.old"
done
That will locate all files in $dir that do NOT have and .old extension and move the file to $file.old. For a case insensitive version:
for file in "$dir"/*.[^oO][^lL][^dD]*; do
You can use the bash [[ operator for the [[ "$file" == *.old ]] test as well, but it is less portable in practice. (character classes are also not portable). Unless a file starts potentially starts with -, there isn't any reason to include -- following mv (but it doesn't hurt either).
Related
I have a list of files stored in a text file, and if a Python file is found in that list. I want to the corresponding test file using Pytest.
My file looks like this:
/folder1/file1.txt
/folder1/file2.jpg
/folder1/file3.md
/folder1/file4.py
/folder1/folder2/file5.py
When 4th/5th files are found, I want to run the command pytest like:
pytest /folder1/test_file4.py
pytest /folder1/folder2/test_file5.py
Currently, I am using this command:
cat /workspace/filelist.txt | while read line; do if [[ $$line == *.py ]]; then exec "pytest test_$${line}"; fi; done;
which is not working correctly, as I have file path in the text as well. Any idea how to implement this?
Using Bash's variable substring removal to add the test_. One-liner:
$ while read line; do if [[ $line == *.py ]]; then echo "pytest ${line%/*}/test_${line##*/}"; fi; done < file
In more readable form:
while read line
do
if [[ $line == *.py ]]
then
echo "pytest ${line%/*}/test_${line##*/}"
fi
done < file
Output:
pytest /folder1/test_file4.py
pytest /folder1/folder2/test_file5.py
Don't know anything about the Google Cloudbuild so I'll let you experiment with the double dollar signs.
Update:
In case there are files already with test_ prefix, use this bash script that utilizes extglob in variable substring removal:
shopt -s extglob # notice
while read line
do
if [[ $line == *.py ]]
then
echo "pytest ${line%/*}/test_${line##*/?(test_)}" # notice
fi
done < file
You can easily refactor all your conditions into a simple sed script. This also gets rid of the useless cat and the similarly useless exec.
sed -n 's%[^/]*\.py$%test_&%p' /workspace/filelist.txt |
xargs -n 1 pytest
The regular expression matches anything after the last slash, which means the entire line if there is no slash; we include the .py suffix to make sure this only matches those files.
The pipe to xargs is a common way to convert standard input into command-line arguments. The -n 1 says to pass one argument at a time, rather than as many as possible. (Maybe pytest allows you to specify many tests; then, you can take out the -n 1 and let xargs pass in as many as it can fit.)
If you want to avoid adding the test_ prefix to files which already have it, one solution is to break up the sed script into two separate actions:
sed -n '/test_[^/]*\.py/p;t;s%[^/]*\.py$%test_&%p' /workspace/filelist.txt |
xargs -n 1 pytest
The first p simply prints the matches verbatim; the t says if that matched, skip the rest of the script for this input.
(MacOS / BSD sed will want a newline instead of a semicolon after the t command.)
sed is arguably a bit of a read-only language; this is already pressing towards the boundary where perhaps you would rewrite this in Awk instead.
You may want to focus on lines that ends with ".py" string
You can achieve that using grep combined with a regex so you can figure out if a line ends with .py - that eliminates the if statement.
IFS=$'\n'
for file in $(cat /workspace/filelist.txt|grep '\.py$');do pytest $file;done
I am very new to shell scripting, I have a scenario where i have many of files which is inside a folder and which has a naming convention such as (test-2020-11-19-1652.tgz - yyyy-mm-dd-hhmm), i need to compare the date(need to get from file name) and pick the latest one and need to unzip them and need to rename that particular file. i tried in many ways but end-up with an error due to beginner level.can anyone help me with this?
Expectation
In this above case i need to pick file shop_db-2020-11-19-1652.tgz because it is the latest file in the folder.and need to unzip it and rename it it shop_db
Expanding a file pattern always returns a sorted list. So it makes it possible to extract the ultimate entry you want with:
Using POSIX shell syntax:
#!/usr/bin/env sh
last() {
shift $(($# - 1))
printf %s "$1"
}
lastfile=$(last shop_db*.tgz)
if [ "$lastfile" = 'shop_db*.tgz' ]; then
lastfile=
fi
shift $(($# - 1)): Shift all arguments away except the last one.
printf %s "$1": Print the last argument since there is only one left.
Using Bash syntax:
#!/usr/bin/env bash
shopt -s nullglob
lastfile=$(printf '%s\0' shop_db*.tgz | tail -z -n1 | tr -d \\0)
shopt -s nullglob: A Bash feature to return an empty list if no file matches the pattern.
printf '%s\0' shop_db*.tgz: Print a null delimited list of files matching the shop_db*.tgz globbing pattern.
| tail -z -n1: Extract the last record from this null delimited list.
Alternate method using only Bash built-in:
#!/usr/bin/env bash
shopt -s nullglob
while read -r -d '' f && [ "$f" ]
do
lastfile=$f
done < <(
printf '%s\0' shop_db*.tgz
)
echo "$lastfile"
And finally expanding the globbing pattern into an array, and extracting the last index:
#!/usr/bin/env bash
shopt -s nullglob
array=(shop_db*.tgz)
if [ ${#array[#]} -gt 0 ]
then
lastfile=${array[-1]}
fi
echo "$lastfile"
What you require can actually be achieved by utilising awk with find:
find . -name "shop_db*.tgz" | awk 'END { "tar -xvf "$0|getline fil;system("mv "fil" shop_db") }'
Look for files starting with shop_db and ending with .tgz. Pipe the output into awk and un tar the file in verbose mode, reading the uncompressed file name into the variable fil using awk's getline. Utilise this fil variable to run the necessary move command to rename the file to shop_db using awk's system function. As the output from find will already be ordered, we utilise the END block within awk to process the last compressed file piped from find.
I have 300+ files named:
T1_0000106_FS1_MAX_5743.nii.gz T1_0000214_FS1_MAX_5475.nii.gz
T1_0000107_FS1_MAX_5477.nii.gz T1_0000215_FS1_MAX_6162.nii.gz
I would like to remove everything between T1 and _5/6*.nii.gz so:
T1_5743.nii.gz T1_5475.nii.gz
T1_5477.nii.gz T1_6162.nii.gz
I can't figure out why it isn't working; I tried (from another post):
for file in *.gz;
do new_name=$(sed 's/_[^.]*_/_g' <<< "new_name");
mv "$file" "$new_name"; done
and variations of rename/sed but nothing changes.
Problems with your script include, at least,
s/_[^.]*_/_g is not a valid sed command. You appear to want s/_[^.]*_/_/g, or in this case, s/_[^.]*_/_/ would do fine as well.
<<< "new_name" redirects the literal string new_name into sed. Possibly you mean <<< "$new_name"
Personally, though, I would not bother with sed for this job, especially if you have a large number of files. Bash is perfectly capable of doing fairly complex string manipulation itself, and your needs don't push it too hard. Consider:
for f in *.gz; do
# The value of $f with the longest trailing match to _* removed
head=${f%%_*}
# The value of $f with the longest leading match to *_ removed
tail=${f##*_}
new_name="${head}_${tail}"
# Sanity check and avoid needless errors
if [ "$f" != "$new_name" ]; then
mv "$f" "$new_name"
fi
done
You could do
for i in *_5*.nii.gz *_6*.nii.gz;do a=${i%%_*};b=${i##*_};[[ $i != $a"_"$b ]] && mv $i $a"_"$b;done
Edited Following suggestion that the file could already be renamed.
Bash's built-in string substitution and its extglob option simplify the replacement of the middle part:
#!/usr/bin/env bash
shopt -s extglob
for file in T1_*.nii.gz; do
echo mv -- "$file" "${file/_+([^.])_/_}"
done
Remove the echo or pipe the output to a shell, if it matches your expectations.
Here is the output of my own test:
mv -- T1_0000106_FS1_MAX_5743.nii.gz T1_5743.nii.gz
mv -- T1_0000107_FS1_MAX_5477.nii.gz T1_5477.nii.gz
mv -- T1_0000214_FS1_MAX_5475.nii.gz T1_5475.nii.gz
mv -- T1_0000215_FS1_MAX_6162.nii.gz T1_6162.nii.gz
I have got a directory with files in which some of then end with an underscore.
I would like to test each file to see if it ends with an underscore and then strip off the underscore.
I am currently running the following code:
for file in *;do
echo $file;
if [[ "${file:$length:1}" == "_" ]];then
mv $file $(echo $file | sed "s/.$//g");
fi
done
But it does not seem to be renaming the files with underscore. For example if i have a file called all_indoors_ I expect it to give me all_indoors.
You could use built-in string substitution:
for file in *_; do
mv "$file" "${file%_}"
done
Just use a regex to check the string:
for file in *
do
[[ $file =~ "_$" ]] && echo mv "$file" "${file%%_}"
done
Once you are sure it works as intended, remove the echo so that the mv command executes!
It may even be cleaner to use *_ so that the for will just loop over the files with a name ending with _, as hek2mgl suggests in comments.
for file in *_
do
echo mv "$file" "${file%%_}"
done
You can use which will be recursive:
while read f; do
mv "$f" "${f:0:-1}"; # Remove last character from $f
done < <(find . -type f -name '*_')
Although not a pure bash approach, you can use rename.ul (written by Larry Wall, the person behind perl). Rename is not part of the default linux environment, but is part of util-linux.
You use rename with:
rename perlexpr files
(some flags ommitted).
So you could use:
rename 's/_$//' *
if you want to remove all characters including and after the underscore.
As #hek2mgl points out, there are multiple rename commands (see here), so first test if you have picked the right one.
How can Bash rename a series of packages to remove their version numbers? I've been toying around with both expr and %%, to no avail.
Examples:
Xft2-2.1.13.pkg becomes Xft2.pkg
jasper-1.900.1.pkg becomes jasper.pkg
xorg-libXrandr-1.2.3.pkg becomes xorg-libXrandr.pkg
You could use bash's parameter expansion feature
for i in ./*.pkg ; do mv "$i" "${i/-[0-9.]*.pkg/.pkg}" ; done
Quotes are needed for filenames with spaces.
If all files are in the same directory the sequence
ls |
sed -n 's/\(.*\)\(-[0-9.]*\.pkg\)/mv "\1\2" "\1.pkg"/p' |
sh
will do your job. The sed command will create a sequence of mv commands, which you can then pipe into the shell. It's best to first run the pipeline without the trailing | sh so as to verify that the command does what you want.
To recurse through multiple directories use something like
find . -type f |
sed -n 's/\(.*\)\(-[0-9.]*\.pkg\)/mv "\1\2" "\1.pkg"/p' |
sh
Note that in sed the regular expression grouping sequence is brackets preceded by a backslash, \( and \), rather than single brackets ( and ).
I'll do something like this:
for file in *.pkg ; do
mv $file $(echo $file | rev | cut -f2- -d- | rev).pkg
done
supposed all your file are in the current directory. If not, try to use find as advised above by Javier.
EDIT: Also, this version don't use any bash-specific features, as others above, which leads you to more portability.
We can assume sed is available on any *nix, but we can't be sure
it'll support sed -n to generate mv commands. (NOTE: Only GNU sed does this.)
Even so, bash builtins and sed, we can quickly whip up a shell function to do this.
sedrename() {
if [ $# -gt 1 ]; then
sed_pattern=$1
shift
for file in $(ls $#); do
mv -v "$file" "$(sed $sed_pattern <<< $file)"
done
else
echo "usage: $0 sed_pattern files..."
fi
}
Usage
sedrename 's|\(.*\)\(-[0-9.]*\.pkg\)|\1\2|' *.pkg
before:
./Xft2-2.1.13.pkg
./jasper-1.900.1.pkg
./xorg-libXrandr-1.2.3.pkg
after:
./Xft2.pkg
./jasper.pkg
./xorg-libXrandr.pkg
Creating target folders:
Since mv doesn't automatically create target folders we can't using
our initial version of sedrename.
It's a fairly small change, so it'd be nice to include that feature:
We'll need a utility function, abspath (or absolute path) since bash
doesn't have this build in.
abspath () { case "$1" in
/*)printf "%s\n" "$1";;
*)printf "%s\n" "$PWD/$1";;
esac; }
Once we have that we can generate the target folder(s) for a
sed/rename pattern which includes new folder structure.
This will ensure we know the names of our target folders. When we
rename we'll need to use it on the target file name.
# generate the rename target
target="$(sed $sed_pattern <<< $file)"
# Use absolute path of the rename target to make target folder structure
mkdir -p "$(dirname $(abspath $target))"
# finally move the file to the target name/folders
mv -v "$file" "$target"
Here's the full folder aware script...
sedrename() {
if [ $# -gt 1 ]; then
sed_pattern=$1
shift
for file in $(ls $#); do
target="$(sed $sed_pattern <<< $file)"
mkdir -p "$(dirname $(abspath $target))"
mv -v "$file" "$target"
done
else
echo "usage: $0 sed_pattern files..."
fi
}
Of course, it still works when we don't have specific target folders
too.
If we wanted to put all the songs into a folder, ./Beethoven/ we can do this:
Usage
sedrename 's|Beethoven - |Beethoven/|g' *.mp3
before:
./Beethoven - Fur Elise.mp3
./Beethoven - Moonlight Sonata.mp3
./Beethoven - Ode to Joy.mp3
./Beethoven - Rage Over the Lost Penny.mp3
after:
./Beethoven/Fur Elise.mp3
./Beethoven/Moonlight Sonata.mp3
./Beethoven/Ode to Joy.mp3
./Beethoven/Rage Over the Lost Penny.mp3
Bonus round...
Using this script to move files from folders into a single folder:
Assuming we wanted to gather up all the files matched, and place them
in the current folder, we can do it:
sedrename 's|.*/||' **/*.mp3
before:
./Beethoven/Fur Elise.mp3
./Beethoven/Moonlight Sonata.mp3
./Beethoven/Ode to Joy.mp3
./Beethoven/Rage Over the Lost Penny.mp3
after:
./Beethoven/ # (now empty)
./Fur Elise.mp3
./Moonlight Sonata.mp3
./Ode to Joy.mp3
./Rage Over the Lost Penny.mp3
Note on sed regex patterns
Regular sed pattern rules apply in this script, these patterns aren't
PCRE (Perl Compatible Regular Expressions). You could have sed
extended regular expression syntax, using either sed -r or sed -E
depending on your platform.
See the POSIX compliant man re_format for a complete description of
sed basic and extended regexp patterns.
Here is a POSIX near-equivalent of the currently accepted answer. This trades the Bash-only ${variable/substring/replacement} parameter expansion for one which is available in any Bourne-compatible shell.
for i in ./*.pkg; do
mv "$i" "${i%-[0-9.]*.pkg}.pkg"
done
The parameter expansion ${variable%pattern} produces the value of variable with any suffix which matches pattern removed. (There is also ${variable#pattern} to remove a prefix.)
I kept the subpattern -[0-9.]* from the accepted answer although it is perhaps misleading. It's not a regular expression, but a glob pattern; so it doesn't mean "a dash followed by zero or more numbers or dots". Instead, it means "a dash, followed by a number or a dot, followed by anything". The "anything" will be the shortest possible match, not the longest. (Bash offers ## and %% for trimming the longest possible prefix or suffix, rather than the shortest.)
I find that rename is a much more straightforward tool to use for this sort of thing. I found it on Homebrew for OSX
For your example I would do:
rename 's/\d*?\.\d*?\.\d*?//' *.pkg
The 's' means substitute. The form is s/searchPattern/replacement/ files_to_apply. You need to use regex for this which takes a little study but it's well worth the effort.
better use sed for this, something like:
find . -type f -name "*.pkg" |
sed -e 's/((.*)-[0-9.]*\.pkg)/\1 \2.pkg/g' |
while read nameA nameB; do
mv $nameA $nameB;
done
figuring up the regular expression is left as an exercise (as is dealing with filenames that include spaces)
This seems to work assuming that
everything ends with $pkg
your version #'s always start with a "-"
strip off the .pkg, then strip off -..
for x in $(ls); do echo $x $(echo $x | sed 's/\.pkg//g' | sed 's/-.*//g').pkg; done
I had multiple *.txt files to be renamed as .sql in same folder.
below worked for me:
for i in \`ls *.txt | awk -F "." '{print $1}'\` ;do mv $i.txt $i.sql; done
Thank you for this answers. I also had some sort of problem. Moving .nzb.queued files to .nzb files. It had spaces and other cruft in the filenames and this solved my problem:
find . -type f -name "*.nzb.queued" |
sed -ne "s/^\(\(.*\).nzb.queued\)$/mv -v \"\1\" \"\2.nzb\"/p" |
sh
It is based on the answer of Diomidis Spinellis.
The regex creates one group for the whole filename, and one group for the part before .nzb.queued and then creates a shell move command. With the strings quoted. This also avoids creating a loop in shell script because this is already done by sed.