I have a bunch of files that need to be renamed and the new name is in a text file.
Example file name:
ASBC_Fishbone_Ia.pdf
Example entry in text file:
Ia. Propagation—Design Considerations
Expected new file name:
Ia. Propagation—Design Considerations.pdf
or
Ia._Propagation—Design_Considerations
What would be a good way of going about this using typical linux cli tools? I'm thinking some combination of ls, grep and rename?
You can try:
#!/bin/bash
# Do not allow the script to run if it's not Bash or Bash version is < 4.0 .
[ -n "$BASH_VERSION" ] && [[ BASH_VERSINFO -ge 4 ]] || exit 1
# Do not allow presenting glob pattern if no match is found.
shopt -s nullglob
# Use an associative array.
declare -A MAP=() || exit 1
while IFS=$'\t' read -r CODE NAME; do
# Maps name with code e.g. MAP['Ia']='Propagation—Design Considerations'
MAP[${CODE%.}]=$NAME
done < /path/to/text_file
# Change directory. Not needed if files are in current directory.
cd "/path/to/dir/containing/files" || exit 1
for FILE in *_*.pdf; do
# Get code from filename.
CODE=${FILE##*_} CODE=${CODE%.pdf}
# Skip if no code was extracted from file.
[[ -n $CODE ]] || continue
# Get name from map based from code.
NAME=${MAP[$CODE]}
# Skip if no new name was registered based on code.
[[ -n $NAME ]] || continue
# Generate new name.
NEW_NAME="${CODE}. $NAME.pdf"
# Replace spaces with _ at your preference. Uncomment if wanted.
# NEW_NAME=${NEWNAME// /_}
# Rename file. Remove echo if you find it correct already.
echo mv -- "$FILE" "$NEW_NAME"
done
Related
I'm running this in bash and even though there is a .txt file it prints out "no new folders to create" in the terminal.
Am I missing something?
FILES=cluster-02/*/*
for f in $FILES
do
if [[ $f == *."txt" ]]
then
cat $f | xargs mkdir -p
else
echo "No new folders to create"
fi
done;
As mentioned in the first comment, the behaviour is indeed as you might expect from your script: you run through all files, text files and other ones. In case your file is a text file, you perform the if-case and in case your file is another type of file, you perform the else-case.
In order to solve this, you might decide not to take the other files into account (only handle text files), I think you might do this as follows:
FILES=cluster-02/*/*.txt
You're looping over multiple files, so the first result may trigger the if and the second can show the else.
You could save the wildcard result in an array, check if there's something in it, and loop if so:
shopt -s nullglob
FILES=( foo/* )
if (( ${#FILES[#]} )); then
for f in "${FILES[#]}"; do
if [[ $f == *."txt" ]]; then
echo $f
fi
done
else
echo "No new folders to create"
fi
#!/usr/bin/env bash
# Create an array containing a list of files
# This is safer to avoid issues with files having special characters such
# as spaces, glob-characters, or other characters that might be cumbersome
# Note: if no files are found, the array contains a single element with the
# string "cluster-02/*/*"
file_list=( cluster-02/*/* )
# loop over the content of the file list
# ensure to quote the list to avoid the same pitfalls as above
for _file in "${file_list[#]}"
do
[ "${_file%.txt}" == "${_file}" ] && continue # skip, not a txt
[ -f "${_file}" ] || continue # check if the file exists
[ -r "${_file}" ] || continue # check if the file is readable
[ -s "${_file}" ] || continue # check if the file is empty
< "${_file}" xargs mkdir -p -- # add -- to avoid issues with entries starting with -
_c=1
done;
[ "${_c}" ] || echo "No new folders to create"
I am trying to automate the below: Any help, please.
We have 2 directories as mentioned below, whenever we get new files in Directory-1, only they should be copied and replaced into Directory-2. How to achieve this in Linux scripting. Filename remains the same but the version will be different.
Directory-1:
FileOne_2.0.0.txt
FileTwo_3.0.0.txt
Directory-2:
FileOne_1.0.0.txt
FileTwo_2.0.0.txt
FileThree_3.0.0.txt
FileFive_5.0.0.txt
Try this code (on a test setup before you trust your real directories and files with it):
#! /bin/bash -p
shopt -s extglob # Enable extended globbing ( +([0-9]) ... )
shopt -s nullglob # Globs that match nothing expand to nothing
shopt -s dotglob # Globs match files with names starting with '.'
srcdir='Directory-1'
destdir='Directory-2'
# A(n extended) glob pattern to match a version string (e.g. '543.21.0')
readonly kVERGLOB='+([0-9]).+([0-9]).+([0-9])'
# shellcheck disable=SC2231 # (Bad warning re. unquoted ${kVERGLOB})
for srcpath in "$srcdir"/*_${kVERGLOB}.txt; do
srcfile=${srcpath##*/} # E.g. 'FileOne_2.0.0.txt'
srcbase=${srcfile%_*} # E.g. 'FileOne'
# Set and check the path that the file will be moved to
destpath=$destdir/$srcfile
if [[ -e $destpath ]]; then
printf "Warning: '%s' already exists. Skipping '%s'.\\n" \
"$destpath" "$srcpath" >&2
continue
fi
# Make a list of the old versions of the file
# shellcheck disable=SC2206 # (Bad warning re. unquoted ${kVERGLOB})
old_destpaths=( "$destdir/$srcbase"_${kVERGLOB}.txt )
# TODO: Add checks that the number of old files (${#old_destpaths[*]})
# is what is expected (exactly one?)
# Move the file
if mv -i -- "$srcpath" "$destpath"; then
printf "Moved '%s' to '%s'\\n" "$srcpath" "$destpath" >&2
else
printf "Warning: Failed to move '%s' to '%s'. Skipping '%s'.\\n" \
"$srcpath" "$destpath" "$srcpath" >&2
continue
fi
# Remove the old version(s) of the file (if any)
for oldpath in "${old_destpaths[#]}"; do
if rm -- "$oldpath"; then
printf "Removed '%s'\\n" "$oldpath" >&2
else
printf "Warning: Failed to remove '%s'.\\n" "$oldpath" >&2
fi
done
done
The code is Shellcheck-clean. Two Shellcheck suppression comments are used because the unquoted expansions are necessary here.
srcdir and destdir are set to constant values. You might want to take them from command line parameters, or set them to different constant values.
The code could be made shorter by removing checks. However, moves and removes are destructive operations that can do a lot of damage if they are done incorrectly. I'd add even more checks if it was my own data.
See glob - Greg's Wiki for an explanation of the "extended globbing" used in the code.
See Parameter expansion [Bash Hackers Wiki] for an explanation of ${srcpath##*/} and ${srcfile%_*}.
mv -i is used as a double protection against overwriting an existing file.
All external commands are invoked with -- to explicitly end options, in case they are ever used with paths that begin with -.
Make sure that you understand the code and test it VERY carefully before using it for real.
source_dir=./files/0
dest_dir=./files/1/
for file in $source_dir/*
do
echo $file
echo "processing"
if [[ "1" == "1" ]]; then
mv $file $dest_dir
fi
done
Where processing and the 1 == 1 is whatever your 'prechecks' are (which you haven't told us)
If your coreutils sort is newer than or equal to v7.0 (2008-10-5) after which sort command
supports -V option (version-sort), would you please try:
declare -A base2ver base2file
# compare versions
# returns 0 if $1 equals to $2
# 1 if $1 is newer than $2
# -1 if $1 is older than $2
vercomp() {
if [[ $1 = $2 ]]; then
echo 0
else
newer=$(echo -e "$1\n$2" | sort -Vr | head -n 1)
if [[ $newer = $1 ]]; then
echo 1
else
echo -1
fi
fi
}
for f in Directory-1/*.txt; do
basename=${f##*/}
version=${basename##*_}
version=${version%.txt} # version number such as "2.0.0"
basename=${basename%_*} # basename such as "FileOne"
base2ver[$basename]=$version # associates basename with version number
base2file[$basename]=$f # associates basename with full filename
done
for f in Directory-2/*.txt; do
basename=${f##*/}
version=${basename##*_}
version=${version%.txt}
basename=${basename%_*}
if [[ -n ${base2ver[$basename]} ]] && (( $(vercomp "${base2ver[$basename]}" "$version") > 0 )); then
# echo "${base2file[$basename]} is newer than $f"
rm -- "$f"
cp -p -- "${base2file[$basename]}" Directory-2
fi
done
I'm hoping this is a simple question, since I've never done shell scripting before. I'm trying to filter certain files out of a list of results. While the script executes and prints out a list of files, it's not filtering out the ones I don't want. Thanks for any help you can provide!
#!/bin/bash
# Purpose: Identify all *md files in H2 repo where there is no audit date
#
#
#
# Example call: no_audits.sh
#
# If that call doesn't work, try ./no_audits.sh
#
# NOTE: Script assumes you are executing from within the scripts directory of
# your local H2 git repo.
#
# Process:
# 1) Go to H2 repo content directory (assumption is you are in the scripts dir)
# 2) Use for loop to go through all *md files in each content sub dir
# and list all file names and directories where audit date is null
#
#set counter
count=0
# Go to content directory and loop through all 'md' files in sub dirs
cd ../content
FILES=`find . -type f -name '*md' -print`
for f in $FILES
do
if [[ $f == "*all*" ]] || [[ $f == "*index*" ]] ;
then
# code to skip
echo " Skipping file: " $f
continue
else
# find audit_date in file metadata
adate=`grep audit_date $f`
# separate actual dates from rest of the grepped line
aadate=`echo $adate | awk -F\' '{print $2}'`
# if create date is null - proceed
if [[ -z "$aadate" ]] ;
then
# print a list of all files without audit dates
echo "Audit date: " $aadate " " $f;
count=$((count+1));
fi
fi
done
echo $count " files without audit dates "
First, to address the immediate issue:
[[ $f == "*all*" ]]
is only true if the exact contents of f is the string *all* -- with the wildcards as literal characters. If you want to check for a substring, then the asterisks shouldn't be quoted:
[[ $f = *all* ]]
...is a better-practice solution. (Note the use of = rather than == -- this isn't essential, but is a good habit to be in, as the POSIX test command is only specified to permit = as a string comparison operator; if one writes [ "$f" == foo ] by habit, one can get unexpected failures on platforms with a strictly compliant /bin/sh).
That said, a ground-up implementation of this script intended to follow best practices might look more like the following:
#!/usr/bin/env bash
count=0
while IFS= read -r -d '' filename; do
aadate=$(awk -F"'" '/audit_date/ { print $2; exit; }' <"$filename")
if [[ -z $aadate ]]; then
(( ++count ))
printf 'File %q has no audit date\n' "$filename"
else
printf 'File %q has audit date %s\n' "$filename" "$aadate"
fi
done < <(find . -not '(' -name '*all*' -o -name '*index*' ')' -type f -name '*md' -print0)
echo "Found $count files without audit dates" >&2
Note:
An arbitrary list of filenames cannot be stored in a single bash string (because all characters that might otherwise be used to determine where the first name ends and the next name begins could be present in the name itself). Instead, read one NUL-delimited filename at a time -- emitted with find -print0, read with IFS= read -r -d ''; this is discussed in [BashFAQ #1].
Filtering out unwanted names can be done internal to find.
There's no need to preprocess input to awk using grep, as awk is capable of searching through input files itself.
< <(...) is used to avoid the behavior in BashFAQ #24, wherein content piped to a while loop causes variables set or modified within that loop to become unavailable after its exit.
printf '...%q...\n' "$name" is safer than echo "...$name..." when handling unknown filenames, as printf will emit printable content that accurately represents those names even if they contain unprintable characters or characters which, when emitted directly to a terminal, act to modify that terminal's configuration.
Nevermind, I found the answer here:
bash script to check file name begins with expected string
I tried various versions of the wildcard/filename and ended up with:
if [[ "$f" == *all.md ]] || [[ "$f" == *index.md ]] ;
The link above said not to put those in quotes, and removing the quotes did the trick!
So I have this bash script which will rename all the files from the current directory. I need help modifying it so I can instead specify only certain files which will be renamed, but also still have the ability to pass it a directory instead. I'm not super familiar with bash so it's fairly confusing to me.
#!/bin/bash
#
# Filename: rename.sh
# Description: Renames files and folders to lowercase recursively
# from the current directory
# Variables: Source = x
# Destination = y
#
# Rename all directories. This will need to be done first.
#
# Process each directory’s contents before the directory itself
for x in `find * -depth -type d`;
do
# Translate Caps to Small letters
y=$(echo $x | tr '[A-Z]' '[a-z]');
# check if directory exits
if [ ! -d $y ]; then
mkdir -p $y;
fi
# check if the source and destination is the same
if [ "$x" != "$y" ]; then
# check if there are files in the directory
# before moving it
if [ $(ls "$x") ]; then
mv $x/* $y;
fi
rmdir $x;
fi
done
#
# Rename all files
#
for x in `find * -type f`;
do
# Translate Caps to Small letters
y=$(echo $x | tr '[A-Z]' '[a-z]');
if [ "$x" != "$y" ]; then
mv $x $y;
fi
done
exit 0
Your script has a large number of beginner errors, but the actual question in the title has some merit.
For a task like this, I would go for a recursive solution.
tolower () {
local f g
for f; do
# If this is a directory, process its contents first
if [ -d "$f" ]; then
# Recurse -- run the same function over directory entries
tolower "$f"/*
fi
# Convert file name to lower case (Bash 4+)
g=${f,,}
# If lowercased version differs from original, move it
if [ "${f##*/}" != "${g##*/}" ]; then
mv "$f" "$g"
fi
done
}
Notice how variables which contain file names always need to be quoted (otherwise, your script will fail on file names which contain characters which are shell metacharacters) and how Bash has built-in functionality for lowercasing a variable's value (in recent versions).
Also, tangentially, don't use ls in scripts and try http://shellcheck.net/ before asking for human debugging help.
Quite often i need to work on the newest file in a directory.
Normally i do:
ls -rt
and then open the last file in vim or less.
Now i wanted to produce an alias or function, like
lastline() {ls -rt | tail -n1}
# or
alias lastline=$(ls -rt | tail -n1)
Calling lastline outputs the newest file in the directory, which is nice.
But calling
less lastline
wants to open the file "lastline" which doesn't exist.
How do i make bash execute the function or alias, if possible without a lot of typing $() or ``?
Or is there any other way to achieve the same result?
Thanks for your help.
You're parsing ls, and this is very bad. Moreover, if the last modified “file” is a directory, you'll be lessing/viming a directory.
So you need a robust way to determine the last modified file in the current directory. You may use a helper function like the following (that you'll put in your .bashrc):
last_modified_regfile() {
# Finds the last modified regular file in current directory
# Found file is in variable last_modified_regfile_ret
# Returns a failure return code if no reg files are found
local file
last_modified_regfile_ret=
for file in *; do
[[ -f $file ]] || continue
if [[ -z $last_modified_regfile_ret ]] || [[ $file -nt $last_modified_regfile_ret ]]; then
last_modified_regfile_ret=$file
fi
done
[[ $last_modified_regfile_ret ]]
}
Then you may define another function that will vim the last found file:
vimlastline() {
last_modified_regfile && vim -- "$last_modified_regfile_ret"
}
You may even have last_modified_regfile take optional arguments: the directories where it will find the last modified regular file:
last_modified_regfile() {
# Finds the last modified regular file in current directory
# or in directories given as arguments
# Found file is in variable last_modified_regfile_ret
# Returns a failure return code if no reg files are found
local file dir
local save_shopt_nullglob=$(shopt -p nullglob)
shopt -s nullglob
(( $# )) || set .
last_modified_regfile_ret=
for dir; do
dir=${dir%/}
[[ -d $dir/ ]] || continue
for file in "$dir"/*; do
[[ -f $file ]] || continue
if [[ -z $last_modified_regfile_ret ]] || [[ $file -nt $last_modified_regfile_ret ]]; then
last_modified_regfile_ret=$file
fi
done
done
$save_shopt_nullglob
[[ $last_modified_regfile_ret ]]
}
Then you can even alter vimlastline accordingly:
vimlastline() {
last_modified_regfile "$#" && vim -- "$last_modified_regfile_ret"
}
Use command substitution like this:
lastline() { ls -rt | tail -n1; }
less "$(lastline)"
Or pipe it to xargs:
lastline | xargs -I {} less '{}'