I have a list of alias definitions in a file I want to unalias in a batch.
The file looks like this:
please=sudo
po='git push origin'
I have come this far but I'm not sure how to pass the alias names to the unalias com
cat old.txt | cut -d = -f 1
To allow the input file to contain comments, you might do something like:
while IFS== read -r name val; do
[[ $val ]] || continue # skip any line that didn't have a "="
[[ $name =~ [#] ]] && continue # skip any line that had a # anywhere before the "="
unalias "$name"
done <old.txt
This avoids relying on any tools external to the shell itself -- all processing is done with bash-native logic. (Sometimes this is the right thing, sometimes it's not -- bash's string processing tends to be slower than general-purpose tools, but those tools typically also have significant startup-time costs, making them undesirable to run in a loop).
The while read idiom is documented in BashFAQ #1. Setting IFS== means that we split into fields when an = is seen; providing name and val means that the first field goes into name, and all subsequent fields go into val.
[[ $var =~ $regex ]] does POSIX ERE-style regex matching.
... <in.txt is both more efficient than cat in.txt | ..., and avoids triggering the issues described in BashFAQ #24 (which can happen when piping data into a loop).
This did the trick:
unalias $(cat old.txt | cut -d = -f 1)
Related
I wrote the following code in Bash:
#!/bin/sh
host=$1
regex="^(((git|ssh|http(s)?)|(git#[\w\.]+))(:(\/\/)?)([A-Za-z0-9.#:_/-]+)\.com)(.*)"
if [[ "$host" =~ $regex ]]; then
d=${BASH_REMATCH[1]}
if [[ "$d" = *github* ]]; then
return
fi
fi
die "Current repository is not stored in Github."
I want to learn how to write a better Bash code so I use the shellcheck.net.
Line 5:
if [[ "$host" =~ $regex ]]; then
^-- SC2039: In POSIX sh, [[ ]] is undefined.
Line 6:
d=${BASH_REMATCH[1]}
^-- SC2039: In POSIX sh, array references are undefined.
Line 7:
if [[ "$d" = *github* ]]; then
^-- SC2039: In POSIX sh, [[ ]] is undefined.
I'm trying to understand how to fix those warnings. I understand that in order to fix [[ ]] I need it to switch to [ ] but then I get an error due globs. Also how should I replace the =~ operator?
When you write #!/bin/sh then you shouldn't use bash-specific features like [[. But you don't need to change [[ to [ or anything like that; just change the shebang line to #!/bin/bash. Then you can use all the bash features you like.
Use grep and sed in posix.
# use grep -q to match with regex
if printf "%s\n" "$host" | grep -q '\(git\|ssh\|http\(s\)\)etc. etc. etc.'; then
# use sed to extract part of the string matching regex
d=$(printf "%s\n" "$host" | sed 's/\(g\|ssh\|http\(s\)\)etc. etc. etc./\2/')
if printf "%s\n" "$d" | grep -q github; then
return
fi
fi
Finding out proper regexes is left to others.
You could try to parse out the different parts with parameter expansions though it's going to get a bit tedious. (The link is to the Bash manual; only a few of the expansions supported by Bash are POSIX.)
Assuming the input is a valid, well-formed URL (which may or may not be warranted) maybe try
host=$1
tail=${1#*://*/}
case $tail in "$host") tail=${host#*/};; esac
case ${host%/$tail} in
*github.com) return ;;
esac
die "Current repository is not stored in Github."
(where of course we assume that this is in a context where return makes sense, and where die is defined separately, like we have to assume in the original code).
This is quite a lot simpler than the regex you presented, and definitely does not cover all the strings that the regex would be able to handle; but perhaps it doesn't have to be all that complex if we can assume that the URL has gone through some sort of validation (i.e. if it's the output from git remote it's pretty safe to assume that the user has verified it by other means already).
(This question is a follow-up on this comment, in an answer about git hooks)
I'm far too unskilled in bash (so far) to understand fully the remark and how to act accordingly. More specifically, I've been advised to avoid using bash command cat this way :
echo "$current_branch" $(cat "$1") > "$1"
because the order of operations depends on the specific shell and it could end up destroying the contents of the passed argument, so the commit message itself if I got it right?
Also, how to "save the contents in a separate step"?
Would the following make any sense?
tmp = "$1"
echo "$current_branch" $(cat $tmp) > "$1"
The proposed issue is not about overwriting variables or arguments, but about the fact that both reading from and writing to a file at the same time is generally a bad idea.
For example, this command may look like it will just write a file to itself, but instead it truncates it:
cat myfile > myfile # Truncates the file to size 0
However, this is not a problem in your specific command. It is guaranteed to work in a POSIX compliant shell because the order of operations specify that redirections will happen after expansions:
The words that are not variable assignments or redirections shall be expanded. If any fields remain following their expansion, the first field shall be considered the command name and remaining fields are the arguments for the command.
Redirections shall be performed as described in Redirection.
Double-however, it's still a bit fragile in the sense that seemingly harmless modifications may trigger the problem, such as if you wanted to run sed on the result. Since the redirection (> "$1") and command substitution $(cat "$1") are now in separate commands, the POSIX definition no longer saves you:
# Command may now randomly result in the original message being deleted
echo "$current_branch $(cat "$1")" | sed -e 's/(c)/©/g' > "$1"
Similarly, if you refactor it into a function, it will also suddenly stop working:
# Command will now always delete the original message
modify_message() {
echo "$current_branch $(cat "$1")"
}
modify_message "$1" > "$1"
You can avoid this by writing to a temporary file, and then replace your original.
tmp=$(mktemp) || exit
echo "$current_branch $(cat "$1")" > "$tmp"
mv "$tmp" "$1"
In my opinion, it's better to save to another file.
You may try something like
echo "$current_branch" > tmp
cat "$1" >> tmp # merge these into
# echo "$current_branch" $(cat "$1") > tmp
# may both OK
mv tmp "$1"
However I am not sure if my understanding is right, or there are some better solutions.
This is what I considered as the core of question. It is hard to decide the "precedence" of $() block and >. If > is executed "earlier", then echo "$current_branch" will rewrite "$1" file and drop the original content of "$1", which is a disaster. If $() is executed "earlier", then everything works as expected. However, there exists a risk, and we should avoid it.
A command group would be far better than a command substitution here. Note the similarity to Geno Chen's answer.
{
echo "$current_branch"
cat "$1"
} > tmp && mv tmp "$1"
I made a script like this:
#! /usr/bin/bash
a=`ls ../wrfprd/wrfout_d0${i}* | cut -c22-25`
b=`ls ../wrfprd/wrfout_d0${i}* | cut -c27-28`
c=`ls ../wrfprd/wrfout_d0${i}* | cut -c30-31`
d=`ls ../wrfprd/wrfout_d0${i}* | cut -c33-34`
f=$a$b$c$d
echo $f
sed "s/.* startdate=.*/export startdate=${f}/g" ./post_process > post_process2
echo command works and gives 2008042118 that is what I want but in file post_process2 is like this export startdate= and can not recall variable f. I want to produce a line like export startdate=2008042118
First -- don't use ls here -- it's both expensive in terms of performance (compared to globbing, which is performed internal to the shell without starting any external programs), and doesn't guarantee useful output for the full range of possible filenames, making its use in this context inherently bug-prone. A better way to retrieve pieces from a filename, assuming a ksh-derived shell such as bash or zsh, would look like this:
#!/bin/bash
# this is an array, but we're only going to use the first element
file=( "../wrfprd/wrfout_d0${i}"* )
[[ -e $file ]] || { echo "No file found" >&2; exit 1; }
f=${file:22:4}${file:27:2}${file:30:2}${file:33:2}
Second, don't use sed to modify code -- doing so requires that your runtime user have permission to modify its own code, and moreover invites injection vulnerabilities. Just write your content out to a data file:
printf '%s\n' "$f" >startdate.txt
...and, in your second script, to read in the value from that file:
# if the shebang is #!/bin/bash
startdate=$(<startdate.txt)
# if the shebang is #!/bin/sh
startdate=$(cat startdate.txt)
I'm new to UNIX and have this really simple problem:
I have a text-file (input.txt) containing a string in each line. It looks like this:
House
Monkey
Car
And inside my shell script I need to read this input file line by line to get to a variable like this:
things="House,Monkey,Car"
I know this sounds easy, but I just couldnt find any simple solution for this. My closest attempt so far:
#!/bin/sh
things=""
addToString() {
things="${things},$1"
}
while read line; do addToString $line ;done <input.txt
echo $things
But this won't work. Regarding to my google research I thought the while loop would create a new sub shell, but this I was wrong there (see the comment section). Nevertheless the variable "things" was still not available in the echo later on. (I cannot just write the echo inside the while loop, because I need to work with that string later on)
Could you please help me out here? Any help will be appreciated, thank you!
What you proposed works fine! I've only made two changes here: Adding missing quotes, and handling the empty-string case.
things=""
addToString() {
if [ -n "$things" ]; then
things="${things},$1"
else
things="$1"
fi
}
while read -r line; do addToString "$line"; done <input.txt
echo "$things"
If you were piping into while read, this would create a subshell, and that would eat your variables. You aren't piping -- you're doing a <input.txt redirection. No subshell, code works without changes.
That said, there are better ways to read lists of items into shell variables. On any version of bash after 3.0:
IFS=$'\n' read -r -d '' -a things <input.txt # read into an array
printf -v things_str '%s,' "${things[#]}" # write array to a comma-separated string
echo "${things_str%,}" # print that string w/o trailing comma
...on bash 4, that first line can be:
readarray -t things <input.txt # read into an array
This is not a shell solution, but the truth is that solutions in pure shell are often excessively long and verbose. So e.g. to do string processing it is better to use special tools that are part of the “default” Unix environment.
sed ':b;N;$!bb;s/\n/,/g' < input.txt
If you want to omit empty lines, then:
sed ':b;N;$!bb;s/\n\n*/,/g' < input.txt
Speaking about your solution, it should work, but you should really always use quotes where applicable. E.g. this works for me:
things=""
while read line; do things="$things,$line"; done < input.txt
echo "$things"
(Of course, there is an issue with this code, as it outputs a leading comma. If you want to skip empty lines, just add an if check.)
This might/might not work, depending on the shell you are using. On my Ubuntu 14.04/x64, it works with both bash and dash.
To make it more reliable and independent from the shell's behavior, you can try to put the whole block into a subshell explicitly, using the (). For example:
(
things=""
addToString() {
things="${things},$1"
}
while read line; do addToString $line ;done
echo $things
) < input.txt
P.S. You can use something like this to avoid the initial comma. Without bash extensions (using short-circuit logical operators instead of the if for shortness):
test -z "$things" && things="$1" || things="${things},${1}"
Or with bash extensions:
things="${things}${things:+,}${1}"
P.P.S. How I would have done it:
tr '\n' ',' < input.txt | sed 's!,$!\n!'
You can do this too:
#!/bin/bash
while read -r i
do
[[ $things == "" ]] && things="$i" || things="$things","$i"
done < <(grep . input.txt)
echo "$things"
Output:
House,Monkey,Car
N.B:
Used grep to tackle with empty lines and the probability of not having a new line at the end of file. (Normal while read will fail to read the last line if there is no newline at the end of file.)
I have noticed for a while that read never actually reads the last line of a file if there is not, at the end of it, a "newline" character. This is understandable if one consider that, as long as there is not a "newline" character in a file, it is as if it contained 0 line (which is quite difficult to admit !). See, for example, the following:
$ echo 'foo' > bar ; wc -l bar
1 bar
But...
$ echo -n 'bar' > foo ; wc -l foo
0 foo
The question is then: how can I handle such situations when using read to process files which have not been created or modified by myself, and about which I don't know if they actually end up with a "newline" character ?
read does, in fact, read an unterminated line into the assigned var ($REPLY by default). It also returns false on such a line, which just means ‘end of file’; directly using its return value in the classic while loop thus skips that one last line. If you change the loop logic slightly, you can process non-new line terminated files correctly, without need for prior sanitisation, with read:
while read -r || [[ -n "$REPLY" ]]; do
# your processing of $REPLY here
done < "/path/to/file"
Note this is much faster than solutions relying on externals.
Hat tip to Gordon Davisson for improving the loop logic.
POSIX requires any line in a file have a newline character at the end to denote it is a line. But this site offers a solution to exactly the scenario you are describing. Final product is this chunklet.
newline='
'
lastline=$(tail -n 1 file; echo x); lastline=${lastline%x}
[ "${lastline#"${lastline%?}"}" != "$newline" ] && echo >> file
# Now file is sane; do our normal processing here...
If you must use read, try this:
awk '{ print $0}' foo | while read line; do
echo the line is $line
done
as awk seems to recognize lines even without the newline char
This is more or less a combination of the answers given so far.
It does not modify the files in place.
(cat file; tail -c1 file | grep -qx . && echo) | while read line
do
...
done