I am trying to automate the set up of site creation for our in-house development server.
Currently, this consists of creating a system user, mysql user, database, and apache config. I know how I can do everything in a single bash file, but I wanted to ask if there was a way to more cleanly generate the apache config.
Essentially what I want to do is generate a conf file based on a template, similar to using printf. I could certainly use printf, but I thought there might be a cleaner way, using sed or awk.
The reason I don't just want to use printf is because the apache config is about 20 lines long, and will take up most of the bash script, as well as make it harder to read.
Any help is appreciated.
Choose a way of marking parameters. One possibility is :parameter:, but any similar pair of markers that won't be confused with legitimate text for the template file(s) is good.
Write a sed script (in sed, awk, perl, ...) similar to the following:
sed -e "s/:param1:/$param1/g" \
-e "s/:param2:/$param2/g" \
-e "s/:param3:/$param3/g" \
httpd.conf.template > $HTTPDHOME/etc/httpd.conf
If you get to a point where you need sometimes to edit something and sometimes don't, you may find it easier to create the relevant sed commands in a command file and then execute that:
{
echo "s/:param1:/$param1/g"
echo "s/:param2:/$param2/g"
echo "s/:param3:/$param3/g"
if [ "$somevariable" = "somevalue" ]
then echo "s/normaldefault/somethingspecial/g"
fi
} >/tmp/sed.$$
sed -f /tmp/sed.$$ httpd.conf.template > $HTTPDHOME/etc/httpd.conf
Note that you should use a trap to ensure the temporary doesn't outlive its usefulness:
tmp=/tmp/sed.$$ # Consider using more secure alternative schemes
trap "rm -f $tmp; exit 1" 0 1 2 3 13 15 # aka EXIT HUP INT QUIT PIPE TERM
...code above...
rm -f $tmp
trap 0
This ensures that your temporary file is removed when the script exits for most plausible signals. You can preserve a non-zero exit status from previous commands and use exit $exit_status after the trap 0 command.
I'm surprised nobody mentioned here documents. This is probably not what the OP wants, but certainly a way to improve legibility of the script you started out with. Just take care to escape or parametrize away any constructs which the shell will perform substitutions on.
#!/bin/sh
# For example's sake, a weird value
# This is in single quotes, to prevent substitution
literal='$%"?*=`!!'
user=me
cat <<HERE >httpd.conf
# Not a valid httpd.conf
User=${user}
Uninterpolated=${literal}
Escaped=\$dollar
HERE
In this context I would recommend ${variable} over the equivalent $variable for clarity and to avoid any possible ambiguity.
Use sed like for example
sed s/%foo%/$foo/g template.conf > $newdir/httpd.conf
Related
I have a situation where a Bash script runs and parses a user-supplied JSON file using jq. Since it's supplied by the user, it's possible for them to include values in the JSON to perform an injection attack.
I'd like to know if there's a way to overcome this. Please note, the setup of: 'my script parsing a user-supplied JSON file' cannot be changed, as it's out of my control. Only thing I can control is the Bash script.
I've tried using jq with and without the -r flag, but in each case, I was successfully able to inject.
Here's what the Bash script looks like at the moment:
#!/bin/bash
set -e
eval "INCLUDES=($(cat user-supplied.json | jq '.Include[]'))"
CMD="echo Includes are: "
for e in "${INCLUDES[#]}"; do
CMD="$CMD\\\"$e\\\" "
done
eval "$CMD"
And here is an example of a sample user-supplied.json file that demonstrates an injection attack:
{
"Include": [
"\\\";ls -al;echo\\\""
]
}
The above JSON file results in the output:
Includes are: ""
, followed by a directory listing (an actual attack would probably be something far more malicious).
What I'd like instead is something like the following to be outputted:
Includes are: "\\\";ls -al;echo\\\""
Edit 1
I used echo as an example command in the script, which probably wasn’t the best example, as then the solution is simply not using eval.
However the actual command that will be needed is dotnet test, and each array item from Includes needs to be passed as an option using /p:<Includes item>. What I was hoping for was a way to globally neutralise injection regardless of the command, but perhaps that’s not possible, ie, the technique you go for relies heavily on the actual command.
You don't need to use eval for dotnet test either. Many bash extensions not present in POSIX sh exist specifically to make eval usage unnecessary; if you think you need eval for something, you should provide enough details to let us explain why it isn't actually required. :)
#!/usr/bin/env bash
# ^^^^- Syntax below is bash-only; the shell *must* be bash, not /bin/sh
include_args=( )
IFS=$'\n' read -r -d '' -a includes < <(jq -r '.Include[]' user-supplied.json && printf '\0')
for include in "${includes[#]}"; do
include_args+=( "/p:$include" )
done
dotnet test "${include_args[#]}"
To speak a bit to what's going on:
IFS=$'\n' read -r -d '' -a arrayname reads up to the next NUL character in stdin (-d specifies a single character to stop at; since C strings are NUL-terminated, the first character in an empty string is a NUL byte), splits on newlines, and puts the result into arrayname.
The shorter way to write this in bash 4.0 or later is readarray -t arrayname, but that doesn't have the advantage of letting you detect whether the program generating the input failed: Because we have the && printf '\0' attached to the jq code, the NUL terminator this read expects is only present if jq succeeds, thus causing the read's exit status to reflect success only if jq reported success as well.
< <(...) is redirecting stdin from a process substitution, which is replaced with a filename which, when read from, returns the output of running the code ....
The reason we can set include_args+=( "/p:$include" ) and have it be exactly the same as include_args+=( /p:"$include" ) is that the quotes are read by the shell itself and used to determine where to perform string-splitting and globbing; they're not persisted in the generated content (and thus later passed to dotnet test).
Some other useful references:
BashFAQ #50: I'm trying to put a command in a variable, but the complex cases always fail! -- explains in depth why you can't store commands in strings without using eval, and describes better practices to use instead (storing commands in functions; storing commands in arrays; etc).
BashFAQ #48: Eval command and security issues -- Goes into more detail on why eval is widely frowned on.
You don't need eval at all.
INCLUDES=( $(jq '.Include[]' user-supplied.json) )
echo "Includes are: "
for e in "${INCLUDES[#]}"; do
echo "$e"
done
The worst that can happen is that the unquoted command substitution may perform word-splitting or pathname expansion where you don't want it (which is a problem in your original as well), but there's no possibility for arbitrary command execution.
I'm trying to use enscript to print PDFs from Mutt, and hitting character encoding issues. One way around them seems to be to just use sed to replace the problem characters: sed -ir 's/[“”]/"/g' {input}
My test input file is this:
“very dirty”
we’re
I'm hoping to get "very dirty" and we're but instead I'm still getting
â\200\234very dirtyâ\200\235
weâ\200\231re
I found a nice little post on printing to PDFs from Mutt that I used as a starting point. I have a bash script that I point to from my .muttrc with set print_command="$HOME/.mutt/print.sh" -- the script currently reads about like this:
#!/bin/bash
input="$1" pdir="$HOME/Desktop" open_pdf=evince
# Straighten out curly quotes
sed -ir 's/[“”]/"/g' $input
sed -ir "s/[’]/'/g" $input
tmpfile="`mktemp $pdir/mutt_XXXXXXXX.pdf`"
enscript --font=Courier8 $input -2r --word-wrap --fancy-header=mutt -p - 2>/dev/null | ps2pdf - $tmpfile
$open_pdf $tmpfile >/dev/null 2>&1 &
sleep 1
rm $tmpfile
It does a fine job of creating a PDF (and works fine if you give it a file as an argument) but I can't figure out how to fix the curly quotes.
I've tried a bunch of variations on the sed line:
input=sed -r 's/[“”]/"/g' $input
$input=sed -ir "s/[’]/'/g" $input
Per the suggestion at Can I use sed to manipulate a variable in bash? I also tried input=$(sed -r 's/[“”]/"/g' <<< $input) and I get an error: "Syntax error: redirection unexpected"
But none manages to actually change $input -- what is the correct syntax to change $input with sed?
Note: I accepted an answer that resolved the question I asked, but as you can see from the comments there are a couple of other issues here. enscript is taking in a whole file as a variable, not just the text of the file. So trying to tweak the text inside the file is going to take a few extra steps. I'm still learning.
On Editing Variables In General
BashFAQ #21 is a comprehensive reference on performing search-and-replace operations in bash, including within variables, and is thus recommended reading. On this particular case:
Use the shell's native string manipulation instead; this is far higher performance than forking off a subshell, launching an external process inside it, and reading that external process's output. BashFAQ #100 covers this topic in detail, and is well worth reading.
Depending on your version of bash and configured locale, it might be possible to use a bracket expression (ie. [“”], as your original code did). However, the most portable thing is to treat “ and ” separately, which will work even without multi-byte character support available.
input='“hello ’cruel’ world”'
input=${input//'“'/'"'}
input=${input//'”'/'"'}
input=${input//'’'/"'"}
printf '%s\n' "$input"
...correctly outputs:
"hello 'cruel' world"
On Using sed
To provide a literal answer -- you almost had a working sed-based approach in your question.
input=$(sed -r 's/[“”]/"/g' <<<"$input")
...adds the missing syntactic double quotes around the parameter expansion of $input, ensuring that it's treated as a single token regardless of how it might be string-split or glob-expanded.
But All That May Not Help...
The below is mentioned because your test script is manipulating content passed on the command line; if that's not the case in production, you can probably disregard the below.
If your script is invoked as ./yourscript “hello * ’cruel’ * world”, then information about exactly what the user entered is lost before the script is started, and nothing you can do here will fix that.
This is because $1, in that scenario, will only contain “hello; ’cruel’ and world” are in their own argv locations, and the *s will have been replaced with lists of files in the current directory (each such file substituted as a separate argument) before the script was even started. Because the shell responsible for parsing the user's command line (which is not the same shell running your script!) did not recognize the quotes as valid at the time when it ran this parsing, by the time the script is running, there's nothing you can do to recover the original data.
Abstract: The way to use sed to change a variable is explored, but what you really need is a way to use and edit a file. It is covered ahead.
Sed
The (two) sed line(s) could be solved with this (note that -i is not used, it is not a file but a value):
input='“very dirty”
we’re'
sed 's/[“”]/\"/g;s/’/'\''/g' <<<"$input"
But it should be faster (for small strings) to use the internals of the shell:
input='“very dirty”
we’re'
input=${input//[“”]/\"}
input=${input//[’]/\'}
printf '%s\n' "$input"
$1
But there is an underlying problem with your script, you are trying to clean an input received from the command line. You are using $1 as the source of the string. Once somebody writes:
./script “very dirty”
we’re
That input is lost. It is broken into shell's tokens and "$1" will be “very only.
But I do not believe that is what you really have.
file
However, you are also saying that the input comes from a file. If that is the case, then read it in with:
input="$(<infile)" # not $1
sed 's/[“”]/\"/g;s/’/'\''/g' <<<"$input"
Or, if you don't mind to edit (change) the file, do this instead:
sed -i 's/[“”]/\"/g;s/’/'\''/g' infile
input="$(<infile)"
Or, if you are clear and certain that what is being given to the script is a filename, like:
./script infile
You can use:
infile="$1"
sed -i 's/[“”]/\"/g;s/’/'\''/g' "$infile"
input="$(<"$infile")"
Other comments:
Then:
Quote your variables.
Do not use the very old `…` syntax, use $(…) instead.
Do not use variables in UPPER case, those are reserved for environment variables.
And (unless you actually meant sh) use a shebang (first line) that targets bash.
The command enscript most definitively requires a file, not a variable.
Maybe you should use evince to open the PS file, there is no need of the step to make a pdf, unless you know you really need it.
I believe that is better use a file to store the output of enscript and ps2pdf.
Do not hide the errors printed by the commands until everything is working as desired, then, just call the script as:
./script infile 2>/dev/null
Or as required to make it less verbose.
Final script.
If you call the script with the name of the file that enscript is going to use, something like:
./script infile
Then, the whole script will look like this (runs both in bash or sh):
#!/usr/bin/env bash
Usage(){ echo "$0; This script require a source file"; exit 1; }
[ $# -lt 1 ] && Usage
[ ! -e $1 ] && Usage
infile="$1"
pdir="$HOME/Desktop"
open_pdf=evince
# Straighten out curly quotes
sed -i 's/[“”]/\"/g;s/’/'\''/g' "$infile"
tmpfile="$(mktemp "$pdir"/mutt_XXXXXXXX.pdf)"
outfile="${tmpfile%.*}.ps"
enscript --font=Courier10 "$infile" -2r \
--word-wrap --fancy-header=mutt -p "$outfile"
ps2pdf "$outfile" "$tmpfile"
"$open_pdf" "$tmpfile" >/dev/null 2>&1 &
sleep 5
rm "$tmpfile" "$outfile"
This is the bash script.
Counter.sh:
#!/bin/bash
rm -rf home/pi/temp.mp3
cd /home/pi/
now=$(date +"%d-%b-%Y")
count="countshift1.sh"
mkdir $(date '+%d-%b-%Y')
On row 5 of this script, the count variable... I just want to know how to use AWK to change the integer 1 (the 18th character, thanks for the response) into a 3 and then save the Counter.sh file.
This is basically http://mywiki.wooledge.org/BashFAQ/050 -- assuming your script actually does something with $count somewhere further down, you should probably refactor that to avoid this antipattern. See the linked FAQ for much more on this topic.
Having said that, it's not hard to do what you are asking here without making changes to live code. Consider something like
awk 'END { print 5 }' /dev/null > file
in a cron job or similar (using Awk just because your question asks for it, not because it's the best tool for this job) and then in your main script, using that file;
read index <file
count="countshift$index.sh"
While this superficially removes the requirement to change the script on the fly (which is a big win) you still have another pesky problem (code in a variable!), and you should probably find a better way to solve it.
I don't think awk is the ideal tool for that. There are many ways to do it.
I would use Perl.
perl -pi -e 's/countshift1/countshift3/' Counter.sh
I'm working in a project that uses a massive amount of shellscripts for all kind of purposes, and the performance and portability are important. Some of these scripts are using configuration files, that have the following format:
VARIABLE1="value"
VARIABLE2="several words, several values"
VARIABLE3="a,list,of,words"
Then, to use these variables we just need to tdo the following:
#!/bin/sh
. /path/to/the/configuration.file
echo "Var 1 is: $VARIABLE1"
echo "Var 2 is: $VARIABLE2"
echo "Var 3 is: $VARIABLE3"
Simple, right?
Not so much. The fact is that while we can protect the scripts against modification with a simple chown root file.sh; chmod 0711 file.sh, the configuration files must be writable, and then we find out that nasty things like this can happen:
VARIABLE1="value"; rm requiredfile.data
VARIABLE2="you dont want to see this: `rm anotherimportantfile.data`
rm thelastrequiredfile.bin
So when the configuration file is invoked, the instructions inserted in it will be executed with the privileges of any user that actually invokes it.
I know what I'm asking is tricky, but I would like to be able to filter all dangerous syntax that can lead to unauthorized code execution.
What I've done so far:
FILTER='
/^$/d # Delete empty lines
/^#/d # Delete comments
/^[A-Z0-9_]\+=.\+$/{ # Select assignments
/`/p # alert with `
/\$/p # alert with $
/\\/p # alert with \
/;/p # alert with ;
d # Accept the rest
}
'
C=`sed -e "$FILTER" $1 | wc -l` 2>/dev/null
if test $C -gt 0; then
echo "#ERR Suspicious strings in configuration file"
fi
What am I missing? Any improvements?
PS: I know that it could be possible to read safely each variable with a grep/cut combination, but it's out of the question for performance issues.
An often cited security paradigm is that you should enumerate the permitted patterns, not try to enumerate all the possible disallowed patterns.
If you restrict yourself to variables being assigned values that are always single-quoted strings, the only pattern you have to allow is
^[A-Za-z_][A-Za-z0-9_]*='[^']*'[\t ]*$
The trailing whitespace isn't strictly necessary (and if you want to be nice, you could allow for leading whitespace as well).
Single quotes inhibit all shell metacharacters; any string in single quotes is taken verbatim.
Allowing double quotes or unquoted strings is simply inviting trouble. The transition to single quotes might be a bit of a hassle, but if you came here for security advice, that's what you'll get.
Incidentally, you can simply use grep to look for violations:
if grep -v "^[A-Za-z_][A-Za-z0-9_]*='[^']*'[\t ]*$" configfile /dev/null >&2; then
echo "$0: Invalid lines in configfile -- aborting" >&2
exit 2
fi
. configfile
:
:
As a user, I would appreciate the diagnostic message to include the violations. This also avoids the cumbersome detour via wc -l.
forget the filter. My understand for this question is, you shouldn't give the write/read permission on config file directly to someone. you should only assign sudo permission to one group, add related users into that group.
I have written a small bash script called "isinFile.sh" for checking if the first term given to the script can be found in the file "file.txt":
#!/bin/bash
FILE="file.txt"
if [ `grep -w "$1" $FILE` ]; then
echo "true"
else
echo "false"
fi
However, running the script like
> ./isinFile.sh -x
breaks the script, since -x is interpreted by grep as an option.
So I improved my script
#!/bin/bash
FILE="file.txt"
if [ `grep -w -- "$1" $FILE` ]; then
echo "true"
else
echo "false"
fi
using -- as an argument to grep. Now running
> ./isinFile.sh -x
false
works. But is using -- the correct and only way to prevent code/option injection in bash scripts? I have not seen it in the wild, only found it mentioned in ABASH: Finding Bugs in Bash Scripts.
grep -w -- ...
prevents that interpretation in what follows --
EDIT
(I did not read the last part sorry). Yes, it is the only way. The other way is to avoid it as first part of the search; e.g. ".{0}-x" works too but it is odd., so e.g.
grep -w ".{0}$1" ...
should work too.
There's actually another code injection (or whatever you want to call it) bug in this script: it simply hands the output of grep to the [ (aka test) command, and assumes that'll return true if it's not empty. But if the output is more than one "word" long, [ will treat it as an expression and try to evaluate it. For example, suppose the file contains the line 0 -eq 2 and you search for "0" -- [ will decide that 0 is not equal to 2, and the script will print false despite the fact that it found a match.
The best way to fix this is to use Ignacio Vazquez-Abrams' suggestion (as clarified by Dennis Williamson) -- this completely avoids the parsing problem, and is also faster (since -q makes grep stop searching at the first match). If that option weren't available, another method would be to protect the output with double-quotes: if [ "$(grep -w -- "$1" "$FILE")" ]; then (note that I also used $() instead of backquotes 'cause I find them much easier to read, and quotes around $FILE just in case it contains anything funny, like whitespace).
Though not applicable in this particular case, another technique can be used to prevent filenames that start with hyphens from being interpreted as options:
rm ./-x
or
rm /path/to/-x