This question already has answers here:
Extracting a string between last two slashes in Bash
(6 answers)
Closed 7 years ago.
I'd like to get the name of the immediate parent directory of a given file, e.g. foo given /home/blah/foo/bar.txt, using a parameter expansion. Right now I can do it in two lines:
f="/home/blah/foo/bar.txt"
dir_name="${f%/*}"
immediate_parent="${dir_name##*/}"
But I'm very new to parameter expansions, so I assume this could be optimized. Is there a way to do it in only one line?
You can't do it with a single parameter expansion, but you can use =~, Bash's regex-matching operator:
[[ $f =~ ^.*/(.+)/.+$ ]] && immediate_parent=${BASH_REMATCH[1]}
Note: Assumes an absolute path with at least 2 components.
If calling an external utility is acceptable, awk offers a potentially simpler alternative:
immediate_parent=$(awk -F/ '{ print $(NF-1) }' <<<"$f")
As for why it can't be done with a single parameter expansion:
Parameter expansion allows for stripping either a prefix (# / ##) or a suffix (% / %%) from a variable value, but not both.
Nesting prefix- and suffix-trimming expansions, while supported in principle, does not help, because you'd need iterative modification of values, whereas an expansion only ever returns the modified string; in other words: the effective overall expansion still only performs a single expansion operation, and you're again stuck with either a prefix or a suffix operation.
Using a single parameter expansion, extracting an inner substring can only be done by character position and length; e.g., a hard-coded solution based on the sample input would be: immediate_parent=${f:11:3}
You can use arithmetic expressions and even command substitutions as parameter expansion arguments, but the pointlessness of this approach - at least in this scenario - becomes obvious if we try it; note the embedded command substitutions - the first to calculate the character position, the second to calculate the length:
immediate_parent=${f:$(d=${f%/*/*}; printf %s ${#d})+1:$(awk -F/ '{ print length($(NF-1)) }' <<<"$f")}
Related
This question already has answers here:
When to wrap quotes around a shell variable?
(5 answers)
In bash, how do I expand a wildcard while it's inside double quotes?
(1 answer)
Closed 5 months ago.
How to get correctly work?
Below line is working but need use argument in 'for'.
for f in ~/files/*/*.txt do
Code:
list ()
{
for f in $1; do
echo $f
done
}
list "~/files/*/*.txt"
list "~/files/*.txt"
Output:
~/files/*/*.txt
~/files/*.txt
The problem lies in the expansion order in bash.
There is a certain logic in what you did.
It would work with list "./dir/*/*.txt" for example.
You enclosed list argument with double quotes, so that pattern expansion is not done when calling list.
So, $1 in "list" function is literally ./dir/*/*.txt. As you wanted it to be.
Then you don't enclose $1 in the for instruction, so that it is expanded into a list of file names.
So it does what you want.
Or almost so.
Problem is that the order of expansion is :
~ then parameters then patterns (plus other irrelevant in between).
So, when $1 is substituted by its value, it is too late to substitute ~.
If you had done that (not at all a suggestion. Just an explanation) :
list ()
{
for f in ~/$1; do
echo $f
done
}
list "files/*/*.txt"
list "files/*.txt"
It would have worked. Not a solution obviously. But it helps understand what happens: "files/*.txt" is literally passed to "list".
Then
for f in ~/$1
is transformed into
for f in /home/you/$1 [~ substitution]
then transformed into
for f in /home/you/files/*.txt [parameter substitution]
then transformed into
for f in /home/you/files/a.txt /home/you/files/b.txt [pattern expansion]
Now for the solution, quoting the arguments and then using $#, as suggested in comments, would do the trick indeed.
If you don't quote the argument and call
list ~/files/*.txt
Then expansion will occur before the call.
list ~/files/*.txt
is transformed into
list /home/you/files/*.txt
then info
list /home/you/files/a.txt /home/you/files/b.txt.
Then passed to list.
But then, inside list, what you have are 2 arguments.
So indeed, for the for the for, then, you need to use "$#"
list ()
{
for f in "$#"; do
echo $f
done
}
list ~/files/list.txt
Another way, if you have a reason to want the expansion to occurs inside "list" (for example, if you may want to pass two of those patterns), would be to force the expansion after the parameter substitution.
There is no ideal way to do that.
Either you need to recode (or use external commands, such as realpath) the path expansion.
Or you use "eval" to force double evaluation of your "$1". But that is a huge security breach if the arguments come from the user (one could use $(rm -fr /) as an argument, and eval would execute it), plus it can also be tricky, if you have, for example, filenames containing "$".
If you know that the patterns will always look like your examples (maybe a tilde and some * and likes) then you could just do the tilde substitution yourself and keep the rest of the code as is
list ()
{
param=${1/#\~/$HOME}
for f in $param; do
echo $f
done
}
list "~/files/list.txt"
Not the best solution. But the one closest to yours.
tl;dr:
The problem is the order of substitution is bash. You need to understand how bash works by rewriting commands in several stages before execution.
More specifically, because ~ is expanded before parameters and variables. So if x="~/*", then echo $x means echo $x after ~ expansion (no ~ in echo $x), then echo ~/* after variable expansion ($x is replaced by its value), and then echo ~/* after * expansion (since you have to directory literally named ~, * matches nothing).
The easiest solution is to have list take many arguments, not just one, let the expansion occurs before the call to list (so not enclosing argument to list in "), and then, rewrite list by taking into account that $1 is just the first of many arguments.
If you insist on having a single argument to list, you have to deal with potential ~ yourself. Like with ${1/#\~/$HOME} if ~ are always single ~ (not ~user) at the beginning of the pattern.
This question already has answers here:
Stripping prefixes and suffixes from shell words matching a pattern
(2 answers)
Difference between ${} and $() in Bash [duplicate]
(3 answers)
Closed 1 year ago.
I have a string with the structure task_name-student_name and I want to split it into two variables:
task: containing the chunk before the -
student: containing the chunk after the -
I can get this to work with sed:
string="task_name-student_name"
student=$(echo "$string" | sed "s/.*-//")
task=$(echo "$string" | sed "s/-[^-]*$//")
However, VS Code suggests "See if you can use $(variable//search/replace) instead".
So I have two questions:
Why would $(variable//search/replace) be better
How do I get the parameter expansion to work without it being interpreted as a command?
When I try
echo $("$string"//-[^-]*$//)
or
echo $(echo $("$string"//-[^-]*$//))
I get this output:
bash: task_name-student_name//-[^-]*$//: No such file or directory
Thanks in advance!
First: for variable expansion, you want curly braces instead of parentheses. $(something) will execute something as a command; ${something} will expand something as a variable. And just for completeness, $((something)) will evaluate something as an arithmetic expression (integers only, no floating point).
As for replacing the sed with a variable expansion: I wouldn't use $(variable//search/replace} for this; there are more appropriate modifications. ${variable#pattern} will remove the shortest possible match of pattern from the beginning of the variable's value, so use that with the pattern *- to remove through the first "-":
student=${string#*-}
Similarly, ${variable%pattern} will remove from the end of the variable's value, so you can use this with the pattern -* to remove from the dash to the end:
task=${string%-*}
Note that the patterns used here are "glob" expressions (like filename wildcards), not regular expressions like sed uses; they're just different enough to be confusing. Also, the way I've written these assumes there's exactly one "-" in the string; if there's a possibility some student will have a hyphenated name or something like that, you may need to modify them.
There are lots more modifications you can do in a parameter expansion; see the bash hacker's wiki on the subject. Some of these modifications will work in other shells besides bash; the # and % modifiers (and the "greedy" versions, ## and %%) will work in any shell that conforms to the POSIX standard.
This question already has answers here:
Syntax with pound and percent sign after shell parameter name [duplicate]
(2 answers)
Closed 4 years ago.
Can anyone explain the following bash snippet?
for i in $(seq 1 1 10)
do
VAR=${2%?}$i
break;
done
It removes the trailing character from $2 (second positional parameter) and concatenates that value with $i
example:
$ v1="myvalue1x"
$ v2="myvalue2"
$ combined="${v1%?}$v2"
$ echo $combined
myvalue1myvalue2
For more info how the substitution works you can check the Parameter Expansion section of the bash manual
See the bash man page, section parameter expansion:
${parameter%word}
${parameter%%word}
Remove matching suffix pattern. The word is expanded to produce a
pattern just as in pathname expansion. If the pattern matches a
trailing portion of the expanded value of parameter, then the
result of the expansion is the expanded value of parameter with the
shortest matching pattern (the ``%'' case) or the longest matching
pattern (the ``%%'' case) deleted. If parameter is # or *, the
pattern removal operation is applied to each positional parameter
in turn, and the expansion is the resultant list. If parameter is
an array variable subscripted with # or *, the pattern removal
operation is applied to each member of the array in turn, and the
expansion is the resultant list.
Since ? matches a single character, the trailing character is removed from the second argument of the script.
This question already has answers here:
Why does shell ignore quoting characters in arguments passed to it through variables? [duplicate]
(3 answers)
Closed 6 years ago.
If i use this command in pipeline, it's working very well;
pipeline ... | grep -P '^[^\s]*\s3\s'
But if I want to set grep into variable like:
var="grep -P '^[^\s]*\s3\s'"
And if I put variable in pipeline;
pipeline ... | $var
nothing happens, like there isn't any matches.
Any help what am I doing wrong?
The robust way to store a simple command in a variable in Bash is to use an array:
# Store the command names and arguments individually
# in the elements of an *array*.
cmd=( grep -P '^[^\s]*\s3\s' )
# Use the entire array as the command to execute - be sure to
# double-quote ${cmd[#]}.
echo 'before 3 after' | "${cmd[#]}"
If, by contrast, your command is more than a simple command and, for instance, involves pipes, multiple commands, loops, ..., defining a function is the right approach:
# Define a function encapsulating the command...
myGrep() { grep -P '^[^\s]*\s3\s'; }
# ... and use it:
echo 'before 3 after' | myGrep
Why what you tried didn't work:
var="grep -P '^[^\s]*\s3\s'"
causes the single quotes around the regex to become a literal, embedded part of $var's value.
When you then use $var - unquoted - as a command, the following happens:
Bash performs word-splitting, which means that it breaks the value of $var into words (separate tokens) by whitespace (the chars. defined in special variable $IFS, which contains a space, a tab, and a newline character by default).
Bash also performs globbing (pathname expansion) on the resulting works, which is not a problem here, but can have unintended consequences in general.
Also, if any of your original arguments had embedded whitespace, word splitting would split them into multiple words, and your original argument partitioning is lost.
(As an aside: "$var" - i.e., double-quoting the variable reference - is not a solution, because then the entire string is treated as the command name.)
Specifically, the resulting words are:
grep
-P
'^[^\s]*\s3\s' - including the surrounding single quotes
The words are then interpreted as the name of the command and its arguments, and invoked as such.
Given that the pattern argument passed to grep starts with a literal single quote, matching won't work as intended.
Short of using eval "$var" - which is NOT recommended for security reasons - you cannot persuade Bash to see the embedded single quotes as syntactical elements that should be removed (a process appropriate called quote removal).
Using an array bypasses all these problems by storing arguments in individual elements and letting Bash robustly assemble them into a command with "${cmd[#]}".
What you are doing wrong is trying to store a command in a variable. For simplicity, robustness, etc. commands are stored in aliases (if no arguments) or functions (if arguments), not variables. In this case:
$ alias foo='grep X'
$ echo "aXb" | foo
aXb
I recommend you read the book Shell Scripting Recipes by Chris Johnson ASAP to get the basics of shell programming and then Effective Awk Programming, 4th Edition, by Arnold Robbins when you're ready to start writing scripts to manipulate text.
I found out that with ${string:0:3} one can access the first 3 characters of a string. Is there a equivalently easy method to access the last three characters?
Last three characters of string:
${string: -3}
or
${string:(-3)}
(mind the space between : and -3 in the first form).
Please refer to the Shell Parameter Expansion in the reference manual:
${parameter:offset}
${parameter:offset:length}
Expands to up to length characters of parameter starting at the character
specified by offset. If length is omitted, expands to the substring of parameter
starting at the character specified by offset. length and offset are arithmetic
expressions (see Shell Arithmetic). This is referred to as Substring Expansion.
If offset evaluates to a number less than zero, the value is used as an offset
from the end of the value of parameter. If length evaluates to a number less than
zero, and parameter is not ‘#’ and not an indexed or associative array, it is
interpreted as an offset from the end of the value of parameter rather than a
number of characters, and the expansion is the characters between the two
offsets. If parameter is ‘#’, the result is length positional parameters
beginning at offset. If parameter is an indexed array name subscripted by ‘#’ or
‘*’, the result is the length members of the array beginning with
${parameter[offset]}. A negative offset is taken relative to one greater than the
maximum index of the specified array. Substring expansion applied to an
associative array produces undefined results.
Note that a negative offset must be separated from the colon by at least one
space to avoid being confused with the ‘:-’ expansion. Substring indexing is
zero-based unless the positional parameters are used, in which case the indexing
starts at 1 by default. If offset is 0, and the positional parameters are used,
$# is prefixed to the list.
Since this answer gets a few regular views, let me add a possibility to address John Rix's comment; as he mentions, if your string has length less than 3, ${string: -3} expands to the empty string. If, in this case, you want the expansion of string, you may use:
${string:${#string}<3?0:-3}
This uses the ?: ternary if operator, that may be used in Shell Arithmetic; since as documented, the offset is an arithmetic expression, this is valid.
Update for a POSIX-compliant solution
The previous part gives the best option when using Bash. If you want to target POSIX shells, here's an option (that doesn't use pipes or external tools like cut):
# New variable with 3 last characters removed
prefix=${string%???}
# The new string is obtained by removing the prefix a from string
newstring=${string#"$prefix"}
One of the main things to observe here is the use of quoting for prefix inside the parameter expansion. This is mentioned in the POSIX ref (at the end of the section):
The following four varieties of parameter expansion provide for substring processing. In each case, pattern matching notation (see Pattern Matching Notation), rather than regular expression notation, shall be used to evaluate the patterns. If parameter is '#', '*', or '#', the result of the expansion is unspecified. If parameter is unset and set -u is in effect, the expansion shall fail. Enclosing the full parameter expansion string in double-quotes shall not cause the following four varieties of pattern characters to be quoted, whereas quoting characters within the braces shall have this effect. In each variety, if word is omitted, the empty pattern shall be used.
This is important if your string contains special characters. E.g. (in dash),
$ string="hello*ext"
$ prefix=${string%???}
$ # Without quotes (WRONG)
$ echo "${string#$prefix}"
*ext
$ # With quotes (CORRECT)
$ echo "${string#"$prefix"}"
ext
Of course, this is usable only when then number of characters is known in advance, as you have to hardcode the number of ? in the parameter expansion; but when it's the case, it's a good portable solution.
You can use tail:
$ foo="1234567890"
$ echo -n $foo | tail -c 3
890
A somewhat roundabout way to get the last three characters would be to say:
echo $foo | rev | cut -c1-3 | rev
Another workaround is to use grep -o with a little regex magic to get three chars followed by the end of line:
$ foo=1234567890
$ echo $foo | grep -o ...$
890
To make it optionally get the 1 to 3 last chars, in case of strings with less than 3 chars, you can use egrep with this regex:
$ echo a | egrep -o '.{1,3}$'
a
$ echo ab | egrep -o '.{1,3}$'
ab
$ echo abc | egrep -o '.{1,3}$'
abc
$ echo abcd | egrep -o '.{1,3}$'
bcd
You can also use different ranges, such as 5,10 to get the last five to ten chars.
1. Generalized Substring
To generalise the question and the answer of gniourf_gniourf (as this is what I was searching for), if you want to cut a range of characters from, say, 7th from the end to 3rd from the end, you can use this syntax:
${string: -7:4}
Where 4 is the length of course (7-3).
2. Alternative using cut
In addition, while the solution of gniourf_gniourf is obviously the best and neatest, I just wanted to add an alternative solution using cut:
echo $string | cut -c $((${#string}-2))-
Here, ${#string} is the length of the string, and the trailing "-" means cut to the end.
3. Alternative using awk
This solution instead uses the substring function of awk to select a substring which has the syntax substr(string, start, length) going to the end if the length is omitted. length($string)-2) thus picks up the last three characters.
echo $string | awk '{print substr($1,length($1)-2) }'