i have a bash command that produces a list of files for which i need to alter the filename.
so i was thinking of using something like this:
mycommand | awk {mv $1 altered$1}
the problem is that the second $1 should be altered replacing with sed some regular expressions.
how can i apply sed to the second parameter?
i tried with $() and |, but it does not work.
I also tried
awk '{print $1 sed "s/[^A-Za-z0-9._-]/_/g" <<< $1}'
awk: cmd. line:1: Unexpected token
mv is not an awk command. You need shell. Try:
mycommand | while IFS= read -r f; do mv "$f" "${f//[^A-Za-z0-9._-]/_}"; done
This assumes that the file names are newline-separated. This is OK unless a file name contains a newline as part of its name. For better reliability, mycommand and the while loop should be modified to use NUL as the separator.
How it works:
while IFS= read -r f; do
This starts a loop that reads each line, in turn, into variable f.
IFS= tells the shell to keep the leading or trailing whitespace on a line. If mycommand produces superfluous leading or trailing whitespace, then remove this.
-r tells the shell to keep backslashes in the input just as they are.
mv "$f" "${f//[^A-Za-z0-9._-]/_}"
This renames the file.
done
This signals the end of the while loop.
Is subshell accepted by you? If yes, a simple way you can do as followed:
mv `mycommand | awk '{print $1}'` {altered$1}
Use rename (always installed in Debian based distros, via the required util-linux package):
rename 's/^/altered/' $(mycommand)
Related
I run the command
df -gP /data1 /data2 | grep -v File | awk '{print $1}' |
awk -F/dev/ '$0=$2' | tr '\n' '
on the AIX shell (ksh) and it prints the output below:
lv_data01 lv_data02 root#testhost:/
However, I would like the output to be printed this way. Could someone help?
lv_data01 lv_data02
Using grep … | awk … | awk … is not necessary; a single awk could do the whole job. So could sed and it might even be easier. I'd be tempted to deal with the spacing by using:
x=$(df … | sed …); echo $x
The tr command, once corrected, replaces newlines with spaces, so the prompt follows without a newline before it. The ; echo suggestion adds the missing newline; the echo $x suggestion (note no double quotes) does too.
As for the sed command:
sed -n '/File/!{ s/[[:space:]].*//; s%^.*/dev/%%p; }'
Don't print anything by default
If the line doesn't match File (doing the work of grep -v):
remove the first space (blank or tab) and everything after it (doing the work of awk '{print $1}')
replace everything up to /dev/ with nothing and print (doing the work of awk -F/dev/ '{$0=$2}')
The command substitution and capture, followed by echo, deals with spaces and newlines.
So, my suggested solution is:
x=$(df -gP /data1 /data2 | sed -n '/File/!{ s/[[:space:]].*//; s%^.*/dev/%%p; }'); echo $x
You could add unset x after the echo if you are going to be using this directly in the shell and not in a shell script. If it'll be encapsulated in a shell script, you don't have to worry about it.
I'm blithely assuming the output from df -gP won't contain a path such as this, with two occurrences of /dev:
/who/knows/dev/lv_data01/dev/bin
If that's a real problem, you can fix the sed script, but I don't think it will be. It's one thing the second awk script in the question handles differently.
I found this thread with two solutions for trimming whitespace: piping to xargs and defining a trim() function:
trim() {
local var="$*"
# remove leading whitespace characters
var="${var#"${var%%[![:space:]]*}"}"
# remove trailing whitespace characters
var="${var%"${var##*[![:space:]]}"}"
echo -n "$var"
}
I prefer the second because of one comment:
This is overwhelmingly the ideal solution. Forking one or more external processes merely to trim whitespace from a single string is fundamentally insane – particularly when most shells (including bash) already provide native string munging facilities out-of-the-box.
I am getting, for example, the wifi SSID on macOS by piping to awk (when I get comfortable with regular expressions in bash, I won't fork an awk process), which includes a leading space:
$ /System/Library/PrivateFrameworks/Apple80211.framework/Resources/airport -I | awk -F: '/ SSID/{print $2}'
<some-ssid>
$ /System/Library/PrivateFrameworks/Apple80211.framework/Resources/airport -I | awk -F: '/ SSID/{print $2}' | xargs
<some-ssid>
$ /System/Library/PrivateFrameworks/Apple80211.framework/Resources/airport -I | awk -F: '/ SSID/{print $2}' | trim
$ wifi=$(/System/Library/PrivateFrameworks/Apple80211.framework/Resources/airport -I | awk -F: '/ SSID/{print $2}')
$ trim "$wifi"
<some-ssid>
Why does piping to the trim function fail and giving it an argument work?
It is because your trim() function is expecting a positional argument list to process. The $* is the argument list passed to your function. For the case that you report as not working, you are connecting the read end of a pipe to the function inside which you need to fetch from the standard input file descriptor.
In such a case you need to read from standard input using read command and process the argument list, i.e. as
trim() {
# convert the input received over pipe to a a single string
IFS= read -r var
# remove leading whitespace characters
var="${var#"${var%%[![:space:]]*}"}"
# remove trailing whitespace characters
var="${var%"${var##*[![:space:]]}"}"
echo -n "$var"
}
for which you can now do
$ echo " abc " | trim
abc
or using a command substitution syntax to run the command that fetches the string, that you want to pass to trim() with your older definition.
trim "$(/System/Library/PrivateFrameworks/Apple80211.framework/Resources/airport -I | awk -F: '/ SSID/{print $2}')"
In this case, the shell expands the $(..) by running the command inside and replaces it with output of the commands run. So now the function sees trim <args> which it interprets as a positional argument and runs the string replacement functions directly on it.
I have some variables:
$begin=10
$end=20
how to pass them to sed command.
sed -n '$begin,$endp' filename | grep word
sed -n '10,20p' filename | grep word
The reason this doesn't work is that single quotes in shell code prevent variable expansion. The good way is to use awk:
awk -v begin="$begin" -v end="$end" 'NR == begin, NR == end' filename
It is possible with sed if you use double quotes (in which shell variables are expanded):
sed -n "$begin,$end p" filename
However, this is subject to code injection vulnerabilities because sed cannot distinguish between code and data this way (unlike the awk code above). If a user manages to set, say, end="20 e rm -Rf /;", unpleasant things can happen.
I have a bash script which checks for a string pattern in file and delete entire line i same file but somehow its not deleting the line and no throwing any error .same command from command prompt deletes from file .
#array has patterns
for k in "${patternarr[#]}
do
sed -i '/$k/d' file.txt
done
sed version is >4
when this loop completes i want all lines matching string pattern in array to be deleted from file.txt
when i run sed -i '/pataern/d file.txt from command prompt then it works fine but not inside bash
Thanks in advance
Here:
sed -i '/$k/d' file.txt
The sed script is singly-quoted, which prevents shell variable expansion. It will (probably) work with
sed -i "/$k/d" file.txt
I say "probably" because what it will do depends on the contents of $k, which is just substituted into the sed code and interpreted as such. If $k contains slashes, it will break. If it comes from an untrustworthy source, you open yourself up to code injection (particularly with GNU sed, which can be made to execute shell commands).
Consider k=^/ s/^/rm -Rf \//e; #.
It is generally a bad idea to substitute shell variables into sed code (or any other code). A better way would be with GNU awk:
awk -i inplace -v pattern="$k" '!($0 ~ pattern)' file.txt
Or to just use grep -v and a temporary file.
first of all, you got an unclosed double quote around ${patternarr[#]} in your for statement.
Then your problem is that you use single quotes in the sed argument, making your shell not evaluate the $k within the quotes:
% declare -a patternarr=(foo bar fu foobar)
% for k in ${patternarr[#]}; do echo sed -i '/$k/d' file.txt; done
sed -i /$k/d file.txt
sed -i /$k/d file.txt
sed -i /$k/d file.txt
sed -i /$k/d file.txt
if you replace them with double quotes, here it goes:
% for k in ${patternarr[#]}; do echo sed -i "/$k/d" file.txt; done
sed -i /foo/d file.txt
sed -i /bar/d file.txt
sed -i /fu/d file.txt
sed -i /foobar/d file.txt
Any time you write a loop in shell just to manipulate text you have the wrong approach. This is probably closer to what you really should be doing (no surrounding loop required):
awk -v ks="${patternarr[#]}" 'BEGIN{gsub(/ /,")|(",ks); ks="("ks")} $0 !~ ks' file.txt
but there may be even better approaches still (e.g. only checking 1 field instead of the whole line, or using word boundaries, or string comparison or....) if you show us some sample input and expected output.
You need to use double quotes to interpolate shell variables inside the sed command, like:
for k in ${patternarr[#]}; do
sed -i "/$k/d" file.txt
done
I wanted to print the name from the entire address by shell scripting. So user1#12.12.23.234 should give output "user1" and similarly 11234#12.123.12.23 should give output 11234
Reading from the terminal:
$ IFS=# read user host && echo "$user"
<user1#12.12.23.234>
user1
Reading from a variable:
$ address='user1#12.12.23.234'
$ cut -d# -f1 <<< "$address"
user1
$ sed 's/#.*//' <<< "$address"
user1
$ awk -F# '{print $1}' <<< "$address"
user1
Using bash in place editing:
EMAIL='user#server.com'
echo "${EMAIL%#*}
This is a Bash built-in, so it might not be very portable (it won't run with sh if it's not linked to /bin/bash for example), but it is probably faster since it doesn't fork a process to handle the editing.
Using sed:
echo "$EMAIL" | sed -e 's/#.*//'
This tells sed to replace the # character and as many characters that it can find after it up to the end of line with nothing, ie. removing everything after the #.
This option is probably better if you have multiple emails stored in a file, then you can do something like
sed -e 's/#.*//' emails.txt > users.txt
Hope this helps =)
I tend to use expr for this kind of thing:
address='user1#12.12.23.234'
expr "$address" : '\([^#]*\)'
This is a use of expr for its pattern matching and extraction abilities. Translated, the above says: Please print out the longest prefix of $address that doesn't contain an #.
The expr tool is covered by Posix, so this should be pretty portable.
As a note, some historical versions of expr will interpret an argument with a leading - as an option. If you care about guarding against that, you can add an extra letter to the beginning of the string, and just avoid matching it, like so:
expr "x$address" : 'x\([^#]*\)'