I have many files not having consistent filenames.
For example
IMG_20200823_1.jpg
IMG_20200823_10.jpg
IMG_20200823_12.jpg
IMG_20200823_9.jpg
I would like to rename all of them and ensure they all follow same naming convention
IMG_20200823_0001.jpg
IMG_20200823_0010.jpg
IMG_20200823_0012.jpg
IMG_20200823_0009.jpg
Found out it's possible to change for file having only a number using below
printf "%04d\n"
However am not able to do with my files considering they mix string + "_" + different numbers.
Could anyone help me ?
Thanks !
With Perl's standalone rename or prename command:
rename -n 's/(\d+)(\.jpg$)/sprintf("%04d%s",$1,$2)/e' *.jpg
Output:
rename(IMG_20200823_10.jpg, IMG_20200823_0010.jpg)
rename(IMG_20200823_12.jpg, IMG_20200823_0012.jpg)
rename(IMG_20200823_1.jpg, IMG_20200823_0001.jpg)
rename(IMG_20200823_9.jpg, IMG_20200823_0009.jpg)
if everything looks fine, remove -n.
With Bash regular expressions:
re='(IMG_[[:digit:]]+)_([[:digit:]]+)'
for f in *.jpg; do
[[ $f =~ $re ]]
mv "$f" "$(printf '%s_%04d.jpg' "${BASH_REMATCH[1]}" "${BASH_REMATCH[2]}")"
done
where BASH_REMATCH is an array containing the capture groups of the regular expression. At index 0 is the whole match; index 1 contains IMG_ and the first group of digits; index 2 contains the second group of digits. The printf command is used to format the second group with zero padding, four digits wide.
Use a regex to extract the relevant sub-strings from the input and then pad it...
For each file.
Extract the prefix, number and suffix from the filename.
Pad the number with zeros.
Create the new filename.
Move files
The following code for bash:
echo 'IMG_20200823_1.jpg
IMG_20200823_10.jpg
IMG_20200823_12.jpg
IMG_20200823_9.jpg' |
while IFS= read -r file; do # foreach file
# Use GNU sed to extract parts on separate lines
tmp=$(<<<"$file" sed 's/\(.*_\)\([0-9]*\)\(\..*\)/\1\n\2\n\3\n/')
# Read the separate parts separated by newlines
{
IFS= read -r prefix
IFS= read -r number
IFS= read -r suffix
} <<<"$tmp"
# create new filename
newfilename="$prefix$(printf "%04d" "$number")$suffix"
# move the files
echo mv "$file" "$newfilename"
done
outputs:
mv IMG_20200823_1.jpg IMG_20200823_0001.jpg
mv IMG_20200823_10.jpg IMG_20200823_0010.jpg
mv IMG_20200823_12.jpg IMG_20200823_0012.jpg
mv IMG_20200823_9.jpg IMG_20200823_0009.jpg
Being puzzled by your hint at printf...
Current folder content:
$ ls -1 IMG_*
IMG_20200823_1.jpg
IMG_20200823_21.jpg
Surely is not a good solution but with printf and sed we can do that:
$ printf "mv %3s_%8s_%d.%3s %3s_%8s_%04d.%3s\n" $(ls -1 IMG_* IMG_* | sed 's/_/ /g; s/\./ /')
mv IMG_20200823_1.jpg IMG_20200823_0001.jpg
mv IMG_20200823_21.jpg IMG_20200823_0021.jpg
Consider this mockup:
function test() {
for line in $1
do
echo $line
done
for line2 in $2
do
echo $line2
done
}
# This will give me a list of IDs
list=$(find testfolder/ -type f -exec grep "ID" {} + | sed "s/^.*ID:\ //g")
list2=$(find testfolder2/ -type f -exec grep "ID" {} + | sed "s/^.*ID:\ //g")
# this will not work
test list1 list2
# this will work
for line in $line
do
echo $line
done
for line2 in $2
do
echo $line
done
The problem with this is that the variables $1 and $2 in the function, will be (of course) the first two IDs that were retrieved in list.
Is there a way to pass list and list2 to the function and use them as I would in a non function call?
The problem with Shell scripting and file names is, that the shell splits the input stream into tokens by spaces and newlines. Which characters are used, is stored in the global variable IFS, which is the abbreviation for input field separator. The problem is, that file names may contain spaces and newlines. And if you do not quote the file names correctly as you did it in your question, then the file names get split by the shell.
Problem
Create some files with space:
$ touch a\ {1..3}
If you use a globing pattern to iterate the files, everything is fine:
$ for f in a\ *; do echo ►$f◄; done
►a 1◄
►a 2◄
►a 3◄
But when you use a sub-shell, which echos the file names, they get messed:
$ for f in $(echo a\ *); do echo ►$f◄; done
►a◄
►1◄
►a◄
►2◄
►a◄
►3◄
The same happens, when you use find:
$ for f in $(find . -name 'a *'); do echo ►$f◄; done
►./a◄
►2◄
►./a◄
►3◄
►./a◄
►1◄
Solution
The best way to read a list of files is to delimit them with a character, which is normally not in a file. This is the null character $'\0'. The program find has a special action called -print0 to print the file name with a trailing null character. And the Bash function mapfile can read a list, which is delimited with null characters:
$ mapfile -d $'\0' list < <(find . -name 'a *' -print0)
Now you can write a function, which needs a list of file names.
$ inode() { for f in "$#"; do stat -c %i "$f"; done; }
And pass the list of file names correctly quoted to the function.
$ inode "${list[#]}"
2638642
2638644
2638641
And this works even with newlines in the file name.
I have a shell script with this code:
var=`hg st -R "$path"`
if [ -n "$var" ]; then
echo $var
fi
But the conditional code always executes, because hg st always prints at least one newline character.
Is there a simple way to strip whitespace from $var (like trim() in PHP)?
or
Is there a standard way of dealing with this issue?
I could use sed or AWK, but I'd like to think there is a more elegant solution to this problem.
A simple answer is:
echo " lol " | xargs
Xargs will do the trimming for you. It's one command/program, no parameters, returns the trimmed string, easy as that!
Note: this doesn't remove all internal spaces so "foo bar" stays the same; it does NOT become "foobar". However, multiple spaces will be condensed to single spaces, so "foo bar" will become "foo bar". In addition it doesn't remove end of lines characters.
Let's define a variable containing leading, trailing, and intermediate whitespace:
FOO=' test test test '
echo -e "FOO='${FOO}'"
# > FOO=' test test test '
echo -e "length(FOO)==${#FOO}"
# > length(FOO)==16
How to remove all whitespace (denoted by [:space:] in tr):
FOO=' test test test '
FOO_NO_WHITESPACE="$(echo -e "${FOO}" | tr -d '[:space:]')"
echo -e "FOO_NO_WHITESPACE='${FOO_NO_WHITESPACE}'"
# > FOO_NO_WHITESPACE='testtesttest'
echo -e "length(FOO_NO_WHITESPACE)==${#FOO_NO_WHITESPACE}"
# > length(FOO_NO_WHITESPACE)==12
How to remove leading whitespace only:
FOO=' test test test '
FOO_NO_LEAD_SPACE="$(echo -e "${FOO}" | sed -e 's/^[[:space:]]*//')"
echo -e "FOO_NO_LEAD_SPACE='${FOO_NO_LEAD_SPACE}'"
# > FOO_NO_LEAD_SPACE='test test test '
echo -e "length(FOO_NO_LEAD_SPACE)==${#FOO_NO_LEAD_SPACE}"
# > length(FOO_NO_LEAD_SPACE)==15
How to remove trailing whitespace only:
FOO=' test test test '
FOO_NO_TRAIL_SPACE="$(echo -e "${FOO}" | sed -e 's/[[:space:]]*$//')"
echo -e "FOO_NO_TRAIL_SPACE='${FOO_NO_TRAIL_SPACE}'"
# > FOO_NO_TRAIL_SPACE=' test test test'
echo -e "length(FOO_NO_TRAIL_SPACE)==${#FOO_NO_TRAIL_SPACE}"
# > length(FOO_NO_TRAIL_SPACE)==15
How to remove both leading and trailing spaces--chain the seds:
FOO=' test test test '
FOO_NO_EXTERNAL_SPACE="$(echo -e "${FOO}" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')"
echo -e "FOO_NO_EXTERNAL_SPACE='${FOO_NO_EXTERNAL_SPACE}'"
# > FOO_NO_EXTERNAL_SPACE='test test test'
echo -e "length(FOO_NO_EXTERNAL_SPACE)==${#FOO_NO_EXTERNAL_SPACE}"
# > length(FOO_NO_EXTERNAL_SPACE)==14
Alternatively, if your bash supports it, you can replace echo -e "${FOO}" | sed ... with sed ... <<<${FOO}, like so (for trailing whitespace):
FOO_NO_TRAIL_SPACE="$(sed -e 's/[[:space:]]*$//' <<<${FOO})"
There is a solution which only uses Bash built-ins called wildcards:
var=" abc "
# remove leading whitespace characters
var="${var#"${var%%[![:space:]]*}"}"
# remove trailing whitespace characters
var="${var%"${var##*[![:space:]]}"}"
printf '%s' "===$var==="
Here's the same wrapped in a function:
trim() {
local var="$*"
# remove leading whitespace characters
var="${var#"${var%%[![:space:]]*}"}"
# remove trailing whitespace characters
var="${var%"${var##*[![:space:]]}"}"
printf '%s' "$var"
}
You pass the string to be trimmed in quoted form, e.g.:
trim " abc "
A nice thing about this solution is that it will work with any POSIX-compliant shell.
Reference
Remove leading & trailing whitespace from a Bash variable (original source)
In order to remove all the spaces from the beginning and the end of a string (including end of line characters):
echo $variable | xargs echo -n
This will remove duplicate spaces also:
echo " this string has a lot of spaces " | xargs echo -n
Produces: 'this string has a lot of spaces'
Bash has a feature called parameter expansion, which, among other things, allows string replacement based on so-called patterns (patterns resemble regular expressions, but there are fundamental differences and limitations).
[flussence's original line: Bash has regular expressions, but they're well-hidden:]
The following demonstrates how to remove all white space (even from the interior) from a variable value.
$ var='abc def'
$ echo "$var"
abc def
# Note: flussence's original expression was "${var/ /}", which only replaced the *first* space char., wherever it appeared.
$ echo -n "${var//[[:space:]]/}"
abcdef
Strip one leading and one trailing space
trim()
{
local trimmed="$1"
# Strip leading space.
trimmed="${trimmed## }"
# Strip trailing space.
trimmed="${trimmed%% }"
echo "$trimmed"
}
For example:
test1="$(trim " one leading")"
test2="$(trim "one trailing ")"
test3="$(trim " one leading and one trailing ")"
echo "'$test1', '$test2', '$test3'"
Output:
'one leading', 'one trailing', 'one leading and one trailing'
Strip all leading and trailing spaces
trim()
{
local trimmed="$1"
# Strip leading spaces.
while [[ $trimmed == ' '* ]]; do
trimmed="${trimmed## }"
done
# Strip trailing spaces.
while [[ $trimmed == *' ' ]]; do
trimmed="${trimmed%% }"
done
echo "$trimmed"
}
For example:
test4="$(trim " two leading")"
test5="$(trim "two trailing ")"
test6="$(trim " two leading and two trailing ")"
echo "'$test4', '$test5', '$test6'"
Output:
'two leading', 'two trailing', 'two leading and two trailing'
From Bash Guide section on globbing
To use an extglob in a parameter expansion
#Turn on extended globbing
shopt -s extglob
#Trim leading and trailing whitespace from a variable
x=${x##+([[:space:]])}; x=${x%%+([[:space:]])}
#Turn off extended globbing
shopt -u extglob
Here's the same functionality wrapped in a function (NOTE: Need to quote input string passed to function):
trim() {
# Determine if 'extglob' is currently on.
local extglobWasOff=1
shopt extglob >/dev/null && extglobWasOff=0
(( extglobWasOff )) && shopt -s extglob # Turn 'extglob' on, if currently turned off.
# Trim leading and trailing whitespace
local var=$1
var=${var##+([[:space:]])}
var=${var%%+([[:space:]])}
(( extglobWasOff )) && shopt -u extglob # If 'extglob' was off before, turn it back off.
echo -n "$var" # Output trimmed string.
}
Usage:
string=" abc def ghi ";
#need to quote input-string to preserve internal white-space if any
trimmed=$(trim "$string");
echo "$trimmed";
If we alter the function to execute in a subshell, we don't have to worry about examining the current shell option for extglob, we can just set it without affecting the current shell. This simplifies the function tremendously. I also update the positional parameters "in place" so I don't even need a local variable
trim() {
shopt -s extglob
set -- "${1##+([[:space:]])}"
printf "%s" "${1%%+([[:space:]])}"
}
so:
$ s=$'\t\n \r\tfoo '
$ shopt -u extglob
$ shopt extglob
extglob off
$ printf ">%q<\n" "$s" "$(trim "$s")"
>$'\t\n \r\tfoo '<
>foo<
$ shopt extglob
extglob off
You can trim simply with echo:
foo=" qsdqsd qsdqs q qs "
# Not trimmed
echo \'$foo\'
# Trim
foo=`echo $foo`
# Trimmed
echo \'$foo\'
I've always done it with sed
var=`hg st -R "$path" | sed -e 's/ *$//'`
If there is a more elegant solution, I hope somebody posts it.
With Bash's extended pattern matching features enabled (shopt -s extglob), you can use this:
{trimmed##*( )}
to remove an arbitrary amount of leading spaces.
You can delete newlines with tr:
var=`hg st -R "$path" | tr -d '\n'`
if [ -n $var ]; then
echo $var
done
# Trim whitespace from both ends of specified parameter
trim () {
read -rd '' $1 <<<"${!1}"
}
# Unit test for trim()
test_trim () {
local foo="$1"
trim foo
test "$foo" = "$2"
}
test_trim hey hey &&
test_trim ' hey' hey &&
test_trim 'ho ' ho &&
test_trim 'hey ho' 'hey ho' &&
test_trim ' hey ho ' 'hey ho' &&
test_trim $'\n\n\t hey\n\t ho \t\n' $'hey\n\t ho' &&
test_trim $'\n' '' &&
test_trim '\n' '\n' &&
echo passed
There are a lot of answers, but I still believe my just-written script is worth being mentioned because:
it was successfully tested in the shells bash/dash/busybox shell
it is extremely small
it doesn't depend on external commands and doesn't need to fork (->fast and low resource usage)
it works as expected:
it strips all spaces and tabs from beginning and end, but not more
important: it doesn't remove anything from the middle of the string (many other answers do), even newlines will remain
special: the "$*" joins multiple arguments using one space. if you want to trim & output only the first argument, use "$1" instead
if doesn't have any problems with matching file name patterns etc
The script:
trim() {
local s2 s="$*"
until s2="${s#[[:space:]]}"; [ "$s2" = "$s" ]; do s="$s2"; done
until s2="${s%[[:space:]]}"; [ "$s2" = "$s" ]; do s="$s2"; done
echo "$s"
}
Usage:
mystring=" here is
something "
mystring=$(trim "$mystring")
echo ">$mystring<"
Output:
>here is
something<
This is what I did and worked out perfect and so simple:
the_string=" test"
the_string=`echo $the_string`
echo "$the_string"
Output:
test
If you have shopt -s extglob enabled, then the following is a neat solution.
This worked for me:
text=" trim my edges "
trimmed=$text
trimmed=${trimmed##+( )} #Remove longest matching series of spaces from the front
trimmed=${trimmed%%+( )} #Remove longest matching series of spaces from the back
echo "<$trimmed>" #Adding angle braces just to make it easier to confirm that all spaces are removed
#Result
<trim my edges>
To put that on fewer lines for the same result:
text=" trim my edges "
trimmed=${${text##+( )}%%+( )}
# Strip leading and trailing white space (new line inclusive).
trim(){
[[ "$1" =~ [^[:space:]](.*[^[:space:]])? ]]
printf "%s" "$BASH_REMATCH"
}
OR
# Strip leading white space (new line inclusive).
ltrim(){
[[ "$1" =~ [^[:space:]].* ]]
printf "%s" "$BASH_REMATCH"
}
# Strip trailing white space (new line inclusive).
rtrim(){
[[ "$1" =~ .*[^[:space:]] ]]
printf "%s" "$BASH_REMATCH"
}
# Strip leading and trailing white space (new line inclusive).
trim(){
printf "%s" "$(rtrim "$(ltrim "$1")")"
}
OR
# Strip leading and trailing specified characters. ex: str=$(trim "$str" $'\n a')
trim(){
if [ "$2" ]; then
trim_chrs="$2"
else
trim_chrs="[:space:]"
fi
[[ "$1" =~ ^["$trim_chrs"]*(.*[^"$trim_chrs"])["$trim_chrs"]*$ ]]
printf "%s" "${BASH_REMATCH[1]}"
}
OR
# Strip leading specified characters. ex: str=$(ltrim "$str" $'\n a')
ltrim(){
if [ "$2" ]; then
trim_chrs="$2"
else
trim_chrs="[:space:]"
fi
[[ "$1" =~ ^["$trim_chrs"]*(.*[^"$trim_chrs"]) ]]
printf "%s" "${BASH_REMATCH[1]}"
}
# Strip trailing specified characters. ex: str=$(rtrim "$str" $'\n a')
rtrim(){
if [ "$2" ]; then
trim_chrs="$2"
else
trim_chrs="[:space:]"
fi
[[ "$1" =~ ^(.*[^"$trim_chrs"])["$trim_chrs"]*$ ]]
printf "%s" "${BASH_REMATCH[1]}"
}
# Strip leading and trailing specified characters. ex: str=$(trim "$str" $'\n a')
trim(){
printf "%s" "$(rtrim "$(ltrim "$1" "$2")" "$2")"
}
OR
Building upon moskit's expr soulution...
# Strip leading and trailing white space (new line inclusive).
trim(){
printf "%s" "`expr "$1" : "^[[:space:]]*\(.*[^[:space:]]\)[[:space:]]*$"`"
}
OR
# Strip leading white space (new line inclusive).
ltrim(){
printf "%s" "`expr "$1" : "^[[:space:]]*\(.*[^[:space:]]\)"`"
}
# Strip trailing white space (new line inclusive).
rtrim(){
printf "%s" "`expr "$1" : "^\(.*[^[:space:]]\)[[:space:]]*$"`"
}
# Strip leading and trailing white space (new line inclusive).
trim(){
printf "%s" "$(rtrim "$(ltrim "$1")")"
}
Use AWK:
echo $var | awk '{gsub(/^ +| +$/,"")}1'
You can use old-school tr. For example, this returns the number of modified files in a git repository, whitespaces stripped.
MYVAR=`git ls-files -m|wc -l|tr -d ' '`
This will remove all the whitespaces from your String,
VAR2="${VAR2//[[:space:]]/}"
/ replaces the first occurrence and // all occurrences of whitespaces in the string. I.e. all white spaces get replaced by – nothing
I would simply use sed:
function trim
{
echo "$1" | sed -n '1h;1!H;${;g;s/^[ \t]*//g;s/[ \t]*$//g;p;}'
}
a) Example of usage on single-line string
string=' wordA wordB wordC wordD '
trimmed=$( trim "$string" )
echo "GIVEN STRING: |$string|"
echo "TRIMMED STRING: |$trimmed|"
Output:
GIVEN STRING: | wordA wordB wordC wordD |
TRIMMED STRING: |wordA wordB wordC wordD|
b) Example of usage on multi-line string
string=' wordA
>wordB<
wordC '
trimmed=$( trim "$string" )
echo -e "GIVEN STRING: |$string|\n"
echo "TRIMMED STRING: |$trimmed|"
Output:
GIVEN STRING: | wordAA
>wordB<
wordC |
TRIMMED STRING: |wordAA
>wordB<
wordC|
c) Final note:
If you don't like to use a function, for single-line string you can simply use a "easier to remember" command like:
echo "$string" | sed -e 's/^[ \t]*//' | sed -e 's/[ \t]*$//'
Example:
echo " wordA wordB wordC " | sed -e 's/^[ \t]*//' | sed -e 's/[ \t]*$//'
Output:
wordA wordB wordC
Using the above on multi-line strings will work as well, but please note that it will cut any trailing/leading internal multiple space as well, as GuruM noticed in the comments
string=' wordAA
>four spaces before<
>one space before< '
echo "$string" | sed -e 's/^[ \t]*//' | sed -e 's/[ \t]*$//'
Output:
wordAA
>four spaces before<
>one space before<
So if you do mind to keep those spaces, please use the function at the beginning of my answer!
d) EXPLANATION of the sed syntax "find and replace" on multi-line strings used inside the function trim:
sed -n '
# If the first line, copy the pattern to the hold buffer
1h
# If not the first line, then append the pattern to the hold buffer
1!H
# If the last line then ...
$ {
# Copy from the hold to the pattern buffer
g
# Do the search and replace
s/^[ \t]*//g
s/[ \t]*$//g
# print
p
}'
There are a few different options purely in BASH:
line=${line##+([[:space:]])} # strip leading whitespace; no quote expansion!
line=${line%%+([[:space:]])} # strip trailing whitespace; no quote expansion!
line=${line//[[:space:]]/} # strip all whitespace
line=${line//[[:space:]]/} # strip all whitespace
line=${line//[[:blank:]]/} # strip all blank space
The former two require extglob be set/enabled a priori:
shopt -s extglob # bash only
NOTE: variable expansion inside quotation marks breaks the top two examples!
The pattern matching behaviour of POSIX bracket expressions are detailed here. If you are using a more modern/hackable shell such as Fish, there are built-in functions for string trimming.
I've seen scripts just use variable assignment to do the job:
$ xyz=`echo -e 'foo \n bar'`
$ echo $xyz
foo bar
Whitespace is automatically coalesced and trimmed. One has to be careful of shell metacharacters (potential injection risk).
I would also recommend always double-quoting variable substitutions in shell conditionals:
if [ -n "$var" ]; then
since something like a -o or other content in the variable could amend your test arguments.
Here's a trim() function that trims and normalizes whitespace
#!/bin/bash
function trim {
echo $*
}
echo "'$(trim " one two three ")'"
# 'one two three'
And another variant that uses regular expressions.
#!/bin/bash
function trim {
local trimmed="$#"
if [[ "$trimmed" =~ " *([^ ].*[^ ]) *" ]]
then
trimmed=${BASH_REMATCH[1]}
fi
echo "$trimmed"
}
echo "'$(trim " one two three ")'"
# 'one two three'
This does not have the problem with unwanted globbing, also, interior white-space is unmodified (assuming that $IFS is set to the default, which is ' \t\n').
It reads up to the first newline (and doesn't include it) or the end of string, whichever comes first, and strips away any mix of leading and trailing space and \t characters. If you want to preserve multiple lines (and also strip leading and trailing newlines), use read -r -d '' var << eof instead; note, however, that if your input happens to contain \neof, it will be cut off just before. (Other forms of white space, namely \r, \f, and \v, are not stripped, even if you add them to $IFS.)
read -r var << eof
$var
eof
To remove spaces and tabs from left to first word, enter:
echo " This is a test" | sed "s/^[ \t]*//"
cyberciti.biz/tips/delete-leading-spaces-from-front-of-each-word.html
var=' a b c '
trimmed=$(echo $var)
This is the simplest method I've seen. It only uses Bash, it's only a few lines, the regexp is simple, and it matches all forms of whitespace:
if [[ "$test" =~ ^[[:space:]]*([^[:space:]].*[^[:space:]])[[:space:]]*$ ]]
then
test=${BASH_REMATCH[1]}
fi
Here is a sample script to test it with:
test=$(echo -e "\n \t Spaces and tabs and newlines be gone! \t \n ")
echo "Let's see if this works:"
echo
echo "----------"
echo -e "Testing:${test} :Tested" # Ugh!
echo "----------"
echo
echo "Ugh! Let's fix that..."
if [[ "$test" =~ ^[[:space:]]*([^[:space:]].*[^[:space:]])[[:space:]]*$ ]]
then
test=${BASH_REMATCH[1]}
fi
echo
echo "----------"
echo -e "Testing:${test}:Tested" # "Testing:Spaces and tabs and newlines be gone!"
echo "----------"
echo
echo "Ah, much better."
Removing spaces to one space:
(text) | fmt -su
Assignments ignore leading and trailing whitespace and as such can be used to trim:
$ var=`echo ' hello'`; echo $var
hello
Python has a function strip() that works identically to PHP's trim(), so we can just do a little inline Python to make an easily understandable utility for this:
alias trim='python -c "import sys; sys.stdout.write(sys.stdin.read().strip())"'
This will trim leading and trailing whitespace (including newlines).
$ x=`echo -e "\n\t \n" | trim`
$ if [ -z "$x" ]; then echo hi; fi
hi
I have a MAC address that looks like this.
01:AA:BB:0C:D0:E1
I want to convert it to lowercase and strip the leading zeros.
1:aa:bb:c:d0:e1
What's the simplest way to do that in a Bash script?
$ echo 01:AA:BB:0C:D0:E1 | sed 's/\(^\|:\)0/\1/g;s/.*/\L\0/'
1:aa:bb:c:d0:e1
\(^\|:\)0 represents either the line start (^) or a :, followed by a 0.
We want to replace this by the capture (either line start or :), which removed the 0.
Then, a second substitution (s/.*/\L\0/) put the whole line in lowercase.
$ sed --version | head -1
sed (GNU sed) 4.2.2
EDIT: Alternatively:
echo 01:AA:BB:0C:D0:E1 | sed 's/0\([0-9A-Fa-f]\)/\1/g;s/.*/\L\0/'
This replaces 0x (with x any hexa digit) by x.
EDIT: if your sed does not support \L, use tr:
echo 01:AA:BB:0C:D0:E1 | sed 's/0\([0-9A-Fa-f]\)/\1/g' | tr '[:upper:]' '[:lower:]'
Here's a pure Bash≥4 possibility:
mac=01:AA:BB:0C:D0:E1
IFS=: read -r -d '' -a macary < <(printf '%s:\0' "$mac")
macary=( "${macary[#]#0}" )
macary=( "${macary[#],,}" )
IFS=: eval 'newmac="${macary[*]}"'
The line IFS=: read -r -d '' -a macary < <(printf '%s:\0' "$mac") is the canonical way to split a string into an array,
the expansion "${macary[#]#0}" is that of the array macary with leading 0 (if any) removed,
the expansion "${macary[#],,}" is that of the array macary in lowercase,
IFS=: eval 'newmac="${macary[*]}"' is a standard way to join the fields of an array (note that the use of eval is perfectly safe).
After that:
declare -p newmac
yields
declare -- newmac="1:aa:bb:c:d0:e1"
as required.
A more robust way is to validate the MAC address first:
mac=01:AA:BB:0C:D0:E1
a='([[:xdigit:]]{2})' ; regex="^$a:$a:$a:$a:$a:$a$"
[[ $mac =~ $regex ]] || { echo "Invalid MAC address" >&2; exit 1; }
And then, using the valid result of the regex match (BASH_REMATCH):
set -- $(printf '%x ' $(printf '0x%s ' "${BASH_REMATCH[#]:1}" ))
IFS=: eval 'printf "%s\n" "$*"'
Which will print:
1:aa:bb:c:d0:e1
Hex values without leading zeros and in lowercase.
If Uppercase is needed, change the printf '%x ' to printf '%X '.
If Leading zeros are needed change the same to printf '%02x '.
In a generic shell script, I would like to use shell pattern matching to filter the lines of a text file.
I have a list of file names in files.txt:
file1.txt
file2.sh
file3.png
And I have a list of patterns in patterns.txt:
other_file.txt
file2.*
If I would have regular expressions in patterns.txt, I could do this:
$ grep -v -f patterns.txt files.txt
But I would like to use shell globbing patterns. I found the C function fnmatch but no shell/unix command to use it.
OK, this is going to be really unperformant, as POSIX sh does not even have arrays (which I would have used for caching the patterns):
while IFS= read -r filename; do
hasmatch=0
while IFS= read -r pattern; do
case $filename in ($pattern) hasmatch=1; break ;; esac
done <patterns.txt
test $hasmatch = 1 || printf '%s\n' "$filename"
done <files.txt
If you don’t need the positional arguments ($1, $2, …) you can abuse those for pattern caching though:
saveIFS=$IFS; IFS='
'; set -o noglob
set -- $(cat patterns.txt)
IFS=$saveIFS; set +o noglob
while IFS= read -r filename; do
hasmatch=0
for pattern in "$#"; do
case $filename in ($pattern) hasmatch=1; break ;; esac
done
test $hasmatch = 1 || printf '%s\n' "$filename"
done <files.txt
Be careful about whitespace there though: we set IFS to a literal newline character and nothing else, i.e. IFS='Enter'.
I’ve tested this with your dataset plus a few additions (like a a b* pattern, to test whitespace behaviour), and it seems to work for me according to the spec in the OP.