wrong parameters in unix shell [duplicate] - shell

I wrote a BASH file that features multiple embedded loops of the form
for P in {'0.10','0.20', [...] '0.90','1.00'}; do
for Q in {'0.10','0.20', [...] ,'0.90','1.00'}; do
[...]
I use these variables both as parameters for a command line application, and to create file names directly in BASH. I would like to create duplicates, say $P_REP=0_10 that replaces the dot by an underscore without writting a explicit switch statement for every case, or some hardcoded equivalent. The (non-elegant way) I found to go about it is to
dump the content of P,Q to a temporary file.
replace the dot by an underscore using sed 's/./_/ -i.
read the file again and load its content to the new variable.
Hence, I was wondering if it is possible to run a sed like command directly on the content of a variable?

You can do pattern substitution directly in bash:
P_REP=${P/./_}
Q_REP=${Q/./_}
From the bash(1) man page:
Paramter Expansion
${parameter/pattern/string}
Pattern substitution. The pattern is expanded to produce a pattern just as in pathname expansion. Parameter is expanded and the longest match of pattern against its value is replaced with string. If pattern begins with /, all matches of pattern are replaced with string. Normally only the first match is replaced. If pattern begins with #, it must match at the beginning of the expanded value of parameter. If pattern begins with %, it must match at the end of the expanded value of parameter. If string is null, matches of pattern are deleted and the / following pattern may be omitted. If parameter is # or *, the substitution operation is applied to each positional parameter in turn, and the expansion is the resultant list. If parameter is an array variable subscripted with # or *, the substitution operation is applied to each member of the array in turn, and the expansion is the resultant list.

John Kugelman's answer is fine for your example, but if you need to process the content of a variable with the actual sed program (or some other arbitrary command), you can do it like this:
P_REP=$(sed 's/\./_/' <<< "$P")

For loops you could use:
#!/bin/bash
P_REP=$(for P in '0.10' '0.20' '0.90' '1.00'; do echo ${P/./_} ; done)
Q_REP=$(for Q in '0.10' '0.20' '0.90' '1.00'; do echo ${Q/./_} ; done)
echo ${P_REP[#]}
echo ${Q_REP[#]}

For the exact problem you are mentionning, use John's proposition above.
I would however mention, in case you ever have to do something similar that can't be solved with bash's pattern substitution syntax, that you don't need to actually create temporary files to transform content with sed or similar commands. First, you can pipe a variable directly to a program as STDIN. Second, you may get the output of a command (oeither it's STDOUT, STDERR, or both) directly into a shell variable.
So in your example, you would have had:
for P in 0.10 0.20 [...] 0.90 1.00 ; do
for Q in 0.10 0.20 [...] 0.90 1.00 ; do
P_REP=$( sed 's/\./_/g' <<< "$P" )
Q_REP=$( sed 's/\./_/g' <<< "$Q" )
done
done
Note also that the array syntax (that is { '0.10', '0.20', ...}) is mostly specific to Bash and a very few Bash-followers. When it is easy to do so, you might prefer the more classical approach to for loops in shell, as I domonstrated above. Then your code will safetly execute in all posix-compliant shells.

Why so complicated there is simple solution
You are changing ALL substrings ALL files in Folder / Catalog
ORG="orignal_string"
DES="destination_string"
find . -type f -exec sed -i 's/'"${ORG}"'/'"${DES}"'/g' {} +

Related

bash script on specific URL string manipulation

I need to manipulate a string (URL) of which I don't know lenght.
the string is something like
https://x.xx.xxx.xxx/dontcare1/dontcare2/dontcareN/keyword/restofstring
I basically need a regular expression which returns this:
https://x.xx.xxx.xxx/keyword/restofstring
where the x is the current ip which can vary everytime and I don't know the number of dontcares.
I actually have no idea how to do it, been 2 hours on the problem but didn't find a solution.
thanks!
You can use sed as follows:
sed -E 's=(https://[^/]*).*(/keyword/.*)=\1\2='
s stands for substitute and has the form s=search pattern=replacement pattern=.
The search pattern is a regex in which we grouped (...) the parts you want to extract.
The replacement pattern accesses these groups with \1 and \2.
You can feed a file or stdin to sed and it will process the input line by line.
If you have a string variable and use bash, zsh, or something similar you also can feed that variable directly into stdin using <<<.
Example usage for bash:
input='https://x.xx.xxx.xxx/dontcare1/dontcare2/dontcareN/keyword/restofstring'
output="$(sed -E 's=(https://[^/]*).*(/keyword/.*)=\1\2=' <<< "$input")"
echo "$output" # prints https://x.xx.xxx.xxx/keyword/restofstring
echo "https://x.xx.xxx.xxx/dontcare1/dontcare2/dontcareN/keyword/restofstring" | sed "s/dontcare[0-9]\+\///g"
sed is used to manipulate text. dontcare[0-9]\+\///g is an escaped form of the regular expression dontcare[0-9]+/, which matches the word "dontcare" followed by 1 or more digits, followed by the / character.
sed's pattern works like this: s/find/replace/g, where g is a command that allowed you to match more than one instance of the pattern.
You can see that regular expression in action here.
Note that this assumes there are no dontcareNs in the rest of the string. If that's the case, Socowi's answer works better.
You could also use read with a / value for $IFS to parse out the trash.
$: IFS=/ read proto trash url trash trash trash keyword rest <<< "https://x.xx.xxx.xxx/dontcare1/dontcare2/dontcareN/keyword/restofstring"
$: echo "$proto//$url/$keyword/$rest"
https://x.xx.xxx.xxx/keyword/restofstring
This is more generalized when the dontcare... values aren't known and predictable strings.
This one is pure bash, though I like Socowi's answer better.
Here's a sed variation which picks out the host part and the last two components from the path.
url='http://example.com:1234/ick/poo/bar/quux/fnord'
newurl=$(echo "$url" | sed 's%\(https*://[^/?]*[^?/]\)[^ <>'"'"'"]*/\([^/ <>'"''"]*/^/ <>'"''"]*\)%\1\2%')
The general form is sed 's%pattern%replacement%' where the pattern matches through the end of the host name part (captured into one set of backslashed parentheses) then skips through the penultimate slash, then captures the remainder of the URL including the last slash; and the replacement simply recalls the two captured groups without the skipped part between them.

Extracting part of a string bounded by special symbols

Hello I am passing strings for example /bin/bash/Xorg.tar.gz to my script which is
for i in $*; do
echo "$(expr match "$i" '\.*\.')"
done
I expect to return Xorg only but it returns 0,any ideas why?
It seems weird that your string would be /bin/bash/Xorg.tar.gz (kinda looks like /bin/bash is a directory or something) but either way, you can use standard parameter expansion to get the part you want:
i=${i##*/}
i=${i%%.*}
First remove everything up to the last /, then remove everything from the first ..
expr match directive attempts to match complete input not partial.
However, you can use builtin BASH regex for this:
[[ "$i" =~ .*/([^./]+)\. ]] && echo "${BASH_REMATCH[1]}"
This will print Xorg for your example argument.
The immediate fix (leaving the loop aside):
$ expr '/path/to/Xorg.tar.gz' : '.*/\([^.]*\)'
Xorg
Note:
: is needed after the input string to signal a regex-matching operation.
Note: expr <string> : <regex> is the POSIX-compliant syntax; GNU expr also accepts expr match <string> <regex>, as in your attempt.
expr implicitly matches from the start of the string, so .*/ must be used to match everything up to the last /
\([^.]*\) is used to match everything up to, but not including, the first . of the filename component; note the \-escaping of the ( and ) (the capture group delimiters), which is needed, because expr only supports (the obsolescent and limited) BREs.
Using a capture group ensures that the matched string is output, whereas by default the count of matching chars. is output.
As for the regex you used:
'\.*\.': \.* matches any (possibly empty) sequence (*) of literal . chars. (\.), implicitly at the start of the string, followed by exactly 1 literal . (\.).
In other words: you tried to match 2 or more consecutive . chars. at the start of the string, which is obviously not what you intended.
Because your regex doesn't contain a capture group, expr outputs the count of matching characters, which in this case is 0, since nothing matches.
That said, calling an external utility in every iteration of a shell loop is inefficient, so consider:
Tom Fenech's helpful answer, which only uses shell parameter expansions.
anubhava's helpful answer, which only uses Bash's built-in regex-matching operator, =~
If you don't actually need a shell loop and are fine with processing all paths with a single command using external utilities, consider this:
basename -a "$#" | cut -d'.' -f1
Note: basename -a, for processing multiple filename operands, is nonstandard, but both GNU and BSD/macOS basename support it.
To demonstrate it in action:
# Set positional parameters with `set`.
$ set -- '/path/to/Xorg.tar.gz' '/path/to/another/File.with.multiple.suffixes'
$ basename -a "$#" | cut -d'.' -f1
Xorg
File

Extract parts of file path and concatenate in bash

I am new to bash scripting and my dir structure looks like below.
"/ABC/DEF/GHI/JKL/2015/01/01"
I am trying to produce the output like this - "JKL_2015-01-01".
I am trying using sed and cut and might take a while but this is needed immediately and any help is appreciated. Thanks.
i=/ABC/DEF/GHI/JKL/2015/01/01
o=`echo $i | sed -r 's|^.+/([^/]+)/([0-9]+)/([0-9]+)/([0-9]+)$|\1_\2-\3-\4|'`
i=xxx is a variable assignment, no whitespace around = allowed!
`command`
enclosed by backticks is a command substitution, which captures the standard output of the command inside as a string.
And sed is the stream editor, applying mostly regex based operations to each line from standard input, and emitting the result on standard output.
sed's s||| operation is regex based substitution. I capture 4 character groups with parens (): (non-slashes), slash, (numbers), slash, (numbers), slash, (numbers), end-of-string $. Then in the second part of the subst I print the for captured groups, separated by an underscore and 2 dashes, respectively.
There's no need to use tools that aren't built into bash for this -- using builtins is far more efficient than external tools like sed.
s="/ABC/DEF/GHI/JKL/2015/01/01"
s_re='/([^/]+)/([^/]+)/([^/]+)/([^/]+)$'
if [[ $s =~ $s_re ]]; then
name="${BASH_REMATCH[1]}_${BASH_REMATCH[2]}-${BASH_REMATCH[3]}-${BASH_REMATCH[4]}"
echo "$name"
fi
Alternately, and perhaps more readably (using string manipulation techniques documented in BashFAQ #100):
s="/ABC/DEF/GHI/JKL/2015/01/01"
s_prefix=${s%/*/*/*/*} # find the content we don't care about
s_suffix=${s#"$s_prefix"/} # strip that content
# read the rest into named variables
IFS=/ read -r category year month day <<<"$s_suffix"
# assemble those named variables into the string we care about
echo "${category}_${year}-${month}-${day}"

How truncate the ../ characters from string in bash?

How can I truncate the ../ or .. characters from string in bash
So, If I have strings
str1=../lib
str2=/home/user/../dir1/../dir2/../dir3
then how I can get string without any .. characters in a string like after truncated result should be
str1=lib
str2=/home/user/dir1/dir2/dir3
Please note that I am not interesting in absolute path of string.
You don't really need to fork a sub-shell to call sed. Use bash parameter expansion:
echo ${var//..\/}
str1=../lib
str2=/home/user/../dir1/../dir2/../dir3
echo ${str1//..\/} # Outputs lib
echo ${str2//..\/} # Outputs /home/user/dir1/dir2/dir3
You could use:
pax> str3=$(echo $str2 | sed 's?\.\./??g') ; echo $str3
/home/user/dir1/dir2/dir3
Just be aware (as you seem to be) that's a different path to the one you started with.
If you're going to be doing this infrequently, forking an external process to do it is fine. If you want to use it many times per second, such as in a tight loop, the internal bash commands will be quicker:
pax> str3=${str2//..\/} ; echo $str3
/home/user/dir1/dir2/dir3
This uses bash pattern substitution as described in the man page (modified slightly to adapt to the question at hand):
${parameter/pattern/string}
The parameter is expanded and the longest match of pattern against its value is replaced with string. If pattern begins with /, all matches of pattern are replaced with string.
If string is null, matches of pattern are deleted and the / following pattern may be omitted.
You can use sed to achieve it
sed 's/\.\.\///g'
For example
echo $str2 | sed 's/\.\.\///g'
OP => /home/user/dir1/dir2/dir3

Bash replace ls with for loop

I was given a tip to use file globbing in stead of ls in Bash scripts, in my code I followed the instructions and replaced array=($(ls)) to:
function list_files() { for f in *; do [[ -e $f ]] || continue done }
array=($(list_files))
However the new function doen't return anything, am I doing something wrong here?
Simply write this:
array=(*)
Leaving aside that your "list_files" doesn't output anything, there are still other problems with your approach.
Unquoted command substitution (in your case "$(list_files)") will still be subject to "word splitting" and "pathname expansion" (see bash(1) "EXPANSION"), which means that if there are spaces in "list_files" output, they will be used to split it into array elements, and if there are pattern characters, they will be used to attempt to match and substitute the current directory file names as separate array elements.
OTOH, if you quote the command substitution with double quotes, then the whole output will be considered a single array element.

Resources