I have some bash commands that I have successfully piped together:
$> foo --color=RED | grep -Eo '(v[1-9])'
$> v1
Assume foo is an alias that prints out many things and I want to grab a version number and pipe that to another command, bar that gets an id. So I add this and it works:
$> foo --color=RED | grep -Eo '(v[1-9])' | \
awk '{print "bar --version="$1" --color=RED"}' | xargs -0 bash -c
$> ID: 1234
Great. Now, I'd like to create an entry in my .aliases file so I can just run this like so:
$> wombat RED
Problems: I cannot get this to work
alias wombat='function _w() {
COLOR=$1; # cache the color
foo --color=$COLOR | grep -Eo "(v[1-9])" | \
awk '{print "bar --version=$1 --color=$COLOR"}' | xargs -0 bash -c;
};_w'
The problem seems to be with how I am escaping (or not escaping) around the awk command. Note: In the awk command I need to reference both the version number and the color that I passed to the alias.
I have tried many variations but cannot seem to get it right. Can anyone assist?
The immediate problem is that quotes don't nest -- you can't nest a single-quoted string (the awk script) inside a single-quoted string (the alias definition). There are ways to get this to work, but it's much simpler to just skip the alias part entirely. Making an alias that defines a function and then immediately executes it is pointless; just define the function once, use it normally. Like this:
wombat() {
local color="$1" # cache the color
foo --color="$color" | grep -Eo "(v[1-9])" | \
awk -v color="$color" '{print "bar --version=" $1 " --color=" color}' | \
xargs -0 bash -c;
}
Note that I also made several changes to how the color name is handled: I used a lowercase shell variable name (there are a bunch of all-caps names with special meanings, so using lower- or mixed-case names for your own stuff is safer), I made that shell variable local, and I passed it in to awk as a variable rather than trying to embed it literally in the awk script. Finally, in the awk script, I put the references to $1 and the color variable outside the quoted strings, so they'll be expanded to their values rather than used literally.
Oh, and I used the POSIX-standard syntax for function definitions, which uses () instead of the function keyword to signal that this is a function definition.
Since I don't actually have the foo or bar programs, I have not actually tested this. But as far as I can see it should work.
Turn on Bash shell debugging so you can see how your aliases and function are interpreted before they get executed
http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_03.html
There are a couple of ways to turn on debugging. One is to start a new Bash session in debug mode, bash -x. Another is to use enable debugging in an existing Bash session, set -x
Related
I'm trying to export some environment variables for use by a TomCat process.
There's a few ways to do this (I know how to solve the overall problem), but it bugged me that I didn't know how to do this particular shell task.
Tomcat recommends that all your environment customizations should be exported by "$CATALINA_HOME/bin/setenv.sh".
This whole thing is gonna be stuffed into a Docker container, so the only parameterizability will be via Docker env variables (let's assume for this task that I don't want to use volume mounts or create setenv.sh during the build process).
First, observe that docker run -e can be used to pass environment into the container:
🍔 docker run -eMY_VAR=SUP alpine env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=a528b6fc264b
MY_VAR=SUP
no_proxy=*.local, 169.254/16
HOME=/root
If we wanted to copy all of that env into setenv.sh, it's as simple as:
SETENV="/usr/local/tomcat/bin/setenv.sh"
echo '#!/bin/sh' > "$SETENV"
echo 'export -p' >> "$SETENV"
env >> "$SETENV"
But copying everything somewhat defeats the point of setenv.sh -- which is, to give your tomcat process a clean environment, with only intentional customizations.
So, we can agree on a convention for "which env vars are ones that we want to pass through to setenv.sh". Everything prefixed with MY_.
And now we get to an interesting shell problem.
env | grep '^MY_' | sed 's/^MY_/EXPORT /'
This gets us pretty close. Output looks like:
🍔 docker run -e MY_VAR=hey alpine sh -c "env | grep '^MY_' | sed 's/^MY_/EXPORT /'"
EXPORT VAR=hey
So, we've selected from the env command: only env vars prefixed with MY_. And we can redirect that output to setenv.sh.
Why do I say "pretty close"? Looks like we're done, right?
Try this for size:
🍔 docker run -e MY_VAR='multi
quote> line
quote> string' alpine sh -c "env | grep '^MY_' | sed 's/^MY_/EXPORT /'"
EXPORT VAR=multi
The script only worked for a simple subset of possibilities. i.e. we only managed to export the first line of our multi-line string.
For your convenience: env output for multi-line strings looks like this:
🍔 docker run -e MY_VAR='multi
line
string' alpine env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=0d0afaac6bec
MY_VAR=multi
line
string
no_proxy=*.local, 169.254/16
HOME=/root
I hesitate to try and tackle this using awk; there may be further string escaping complications that I have not considered.
I wonder whether there's a better way altogether to select & serialize a subset of exported environment?
EDIT: I negligently tagged this as a bash question, when really my intention was to pose an sh question. Specifically my intention is to get something that will work with no dependencies other than those that come with the alpine docker image. i.e. BusyBox sh, sed, grep, awk, env.
I've retained the bash tag so as not to punish the initial answer that was submitted when this was a bash-only question.
But I will give preference to an sh-compatible answer, and in particular to one that works with just the BusyBox UNIX utils.
So you need several things:
Enumerate the environment variables and select a subset.
For each selected environment variable, emit sh code that sets the variable to the desired value.
You can use export -p if you want to export all variables in a form that can be read back in, but parsing it to select only certain variables is harder. One way to make use of export -p is to unset the other variables. This only works if none of the environment variables is read-only, but you can work around that by running a separate shell instance (as opposed to a subshell).
To gather the list of variables to unset, you only need to get a superset of the list of all environment variables, and remove the ones you want to keep. You can easily do that by filtering the env output. I do that with a simple grep, you may want to use more complex code if your criteria for inclusion are more complex than “begins with a specific prefix”.
The occasional false positive due to a variable containing a newline followed by a valid variable name and an equal sign will only lead to calling unset on a non-existent variable, which does nothing. The desired variables are removed from the exclusion list, so the final output will never omit a desired variable.
excluded=$(env | LC_ALL=C sed -n 's/^\([A-Z_a-z][0-9A-Z_a-z]*\)=.*/\1/p' |
grep -v 'MY_')
sh -c 'unset $1; export -p' sh "$excluded" >setenv.sh
Dash prints an extra export PATH (with no value) if PATH was in the environment when it was invoked. If that bothers you, change sh -c … to (unset PATH; sh -c …).
Assuming GNU grep:
grep --null '^MY_' </proc/self/environ
...will emit your environment variables in NUL-delimited form (newlines intact).
Similarly, if you have bash:
while IFS= read -r -d '' vardef; do
[[ $vardef = MY_* ]] && printf '%s\0' "$vardef"
done </proc/self/environ
Note that if these variables were set in the same shell session, you may need to create a subprocess for /proc/self/environ to be updated:
(while IFS= read -r -d '' vardef; do
[[ $vardef = MY_* ]] && printf '%s\0' "$vardef"
done </proc/self/environ)
alpine image doesn't ship with bash.
You can use this script to extract all MY_* variables including newline variables:
docker run -e MY_FOO=bar -e MY_VAR="multi' export MY_INJECTED='val" -e MY_VAR2=$'multi
0MY_line=val
string' alpine sh -c "awk -v RS='\06' -F= '/^MY_/{k=\$1; sub(/^[^=]+=/, \"\");
gsub(/\047/, \"\047\\\\\\047\047\"); printf \"export %s=\047%s\047\n\", k, \$0
}' /proc/self/environ"
This will output:
export MY_FOO='bar'
export MY_VAR='multi'\'' export MY_INJECTED='\''val'
export MY_VAR2='multi
0MY_line=val
string'
Here is how awk works:
-v RS='\6': sets record separator as \6 works for nul byte as well (assuming you don't have \6 in value)
-F=: sets field separator as =
/^MY_/: Only process records starting with MY_
store variable name or $1 in variable k
Using sub function get part after = in $0
Using print format output so that it can be used in $CATALINA_HOME/bin/setenv.sh file.
\047 is for printing single quote
what about
declare -p ${!MY_*}
and
declare -p ${!MY_*} | sed -r 's/^declare (-[^ ]*)* MY_/export /'
or
declare -p ${!MY_*} | sed 's/^declare \(-[^ ]*\)* MY_/export /'
EDIT posix compliant version :
some env or printenv accept -0 option to end each output line by \0 rather than a newline. Thus
env -0 | perl -ne 'BEGIN{$/="\0";$\="\n";$q="\047"}next unless /^MY_/;chomp;s/$q/$q\\$q$q/;s/=/=$q/;s/$/$q/;print'
How it works
$/ : input record separator
$\ : output record separator
$q : variable to store single quote (\047) because of surrounding single quotes in command
next : to filter "MY_" variables
chomp : removes the input separator
s/// : quote substitution
EDIT: variation of perl version in posix shell
env -0 | xargs -0 sh -c 'for entry; do [[ $entry = MY_* ]] || continue; printf "%s=\047%s\047\n" "${entry%%=*}" "$(echo "${entry#*=}" | sed '\''s/\x27/\x27\\\x27\x27/g'\'' )"; done' -
I am trying to create a file using the following script (see below). While the script runs without errors (at least according to shellcheck), I cannot get the resulting file to have the correct name.
#!/bin/bash
# Set some variables
export site_path=~/Documents/Blog
drafts_path=~/Documents/Blog/_drafts
title="$title"
# Create the filename
title=$("$title" | "awk {print tolower($0)}")
filename="$title.markdown"
file_path="$drafts_path/$filename"
echo "File path: $file_path"
# Create the file, Add metadata fields
cat >"$file_path" <<EOL
---
title: \"$title\"
layout:
tags:
---
EOL
# Open the file in BBEdit
bbedit "$file_path"
exit 0
Very new to bash, so I'm not quite sure what I'm doing wrong...
The most glaring error is this:
title=$("$title" | "awk {print tolower($0)}")
It's wrong for several reasons:
This pipeline runs "$title" as a command -- meaning that it looks for a command named with the title of your blog post to run -- and pipes the output of that command (a command that presumably won't exist) to awk.
Using double-quotes around the entire awk command means you're looking for a command named something like /usr/bin/awk {print tolower(bash-)} (if $0 evaluates to bash-, which it will in an interactive interpreter; behavior will differ elsewhere).
Using double-quotes rather than single-quotes to protect your awk script means that the $0 gets evaluated to the shell rather than by awk.
A better alternative might look like:
title=$(awk '{print tolower($0)}' <<<"$title")
...or, to use simpler tools:
title=$(tr '[:upper:]' '[:lower:]' <<<"$title")
...or, to use bash 4.x built-in functionality:
title=${title,,}
Of course, all that assumes that title is set to start with. If you aren't passing it through your environment, you might want something like title=$1 rather than title="$title" earlier in your script.
I'm working with Mac OS X's pbpaste command, which returns the clipboard's contents. I'd like to create a shell script that executes each line returned by pbpaste as a separate bash command. For example, let's say that the clipboard's contents consists of the following lines of text:
echo 1234 >~/a.txt
echo 5678 >~/b.txt
I would like a shell script that executes each of those lines, creating the two files a.txt and b.txt in my home folder. After a fair amount of searching and trial and error, I've gotten to the point where I'm able to assign individual lines of text to a variable in a while loop with the following construct:
pbpaste | egrep -o [^$]+ | while read l; do echo $l; done
which sends the following to standard out, as expected:
echo 1234 >~/a.txt
echo 5678 >~/b.txt
Instead of simply echoing each line of text, I then try to execute them with the following construct:
pbpaste | egrep -o [^$]+ | while read l; do $l; done
I thought that this would execute each line (thus creating two text files a.txt and b.txt in my home folder). Instead, the first term (echo) seems to be interpreted as the command, and the remaining terms (nnnn >~/...) seem to get lumped together as if they were a single parameter, resulting in the following being sent to standard out without any files being created:
1234 >~/a.txt
5678 >~/b.txt
I would be grateful for any help in understanding why my construct isn't working and what changes might get it to work.
[…] the remaining terms (nnnn >~/...) seem to get lumped together as if they were a single parameter, […]
Not exactly. The line actually gets split on whitespace (or whatever $IFS specifies), but the problem is that the redirection operator > cannot be taken from a shell variable. For example, this snippet:
gt='>'
echo $gt foo.txt
will print > foo.txt, rather than printing a newline to foo.txt.
And you'll have similar problems with various other shell metacharacters, such as quotation marks.
What you need is the eval builtin, which takes a string, parses it as a shell command, and runs it:
pbpaste | egrep -o [^$]+ | while IFS= read -r LINE; do eval "$LINE"; done
(The IFS= and -r and the double-quotes around $LINE are all to prevent any other processing besides the processing performed by eval, so that e.g. whitespace inside quotation marks will be preserved.)
Another possibility, depending on the details of what you need, is simply to pipe the commands into a new instance of Bash:
pbpaste | egrep -o [^$]+ | bash
Edited to add: For that matter, it occurs to me that you can pass everything to eval in a single batch; just as you can (per your comment) write pbpaste | bash, you can also write eval "$(pbpaste)". That will support multiline while-loops and so on, while still running in the current shell (useful if you want it to be able to reference shell parameters, to set environment variables, etc., etc.).
I'm trying to write a bash script which would locate a single file in the current directory. The file will be used later but I don't need help there. I tried using ls and grep but it doesn't work, I'm a newbie using bash.
#!/bin/sh
#Here I need smt like
#trFile = ls | grep myString (but I get file not found error)
echo $trFile
Use shell wildcards, as in
ls *${pattern}*
And, to store the result in a variable, put it inside a $() structure (you can also use deprecated backticks if you like using deprecated functionality that doesn't nest well)
var=$( ls *${pattern}* )
Or, put your ls | grep in there (but that's bad practice, IMHO):
var=$( ls | grep -- "$pattern" )
#!/bin/sh
#
trfile=$( ls | grep myString )
echo $trfile
The $( xxx ) causes the commands within to be executed and the output returned.
I believe you are looking for something like this:
#!/bin/sh
trFile=`ls | grep "$myString"`
In order to run a command and redirect/store its output, you need put the command between backticks. The variable that will store the output, equal sign and the backtick need to be together, as in my example.
Hope this helps.
Try this, if I guess what you're trying to do is, get the capture of the filename from grepping via the output of the ls into a shell variable, try this:
#!/bin/sh
trFile=`ls | grep "name_of_file"`
echo $trFile
Notice the usage of the back-tick operator surrounding the command, what-ever is the output, in this case, will get captured.
using output of ls will bite when you least expect. Better use Globbing.
http://tldp.org/LDP/abs/html/globbingref.html
I noticed today Bash printf has a -v option
-v var assign the output to shell variable VAR rather than
display it on the standard output
If I invoke like this it works
$ printf -v var "Hello world"
$ printf "$var"
Hello world
Coming from a pipe it does not work
$ grep "Hello world" test.txt | xargs printf -v var
-vprintf: warning: ignoring excess arguments, starting with `var'
$ grep "Hello world" test.txt | xargs printf -v var "%s"
-vprintf: warning: ignoring excess arguments, starting with `var'
xargs will invoke /usr/bin/printf (or wherever that binary is installed on your system). It will not invoke bash's builtin function. And only a builtin (or sourcing a script or similar) can modify the shell's environment.
Even if it could call bash's builtin, the xargs in your example runs in a subsell. The subshell cannot modify it's parent's environment anyway. So what you're trying cannot work.
A few options I see if I understand your sample correctly; sample data:
$ cat input
abc other stuff
def ignored
cba more stuff
Simple variable (a bit tricky depending on what exactly you want):
$ var=$(grep a input)
$ echo $var
abc other stuff cba more stuff
$ echo "$var"
abc other stuff
cba more stuff
With an array if you want individual words in the arrays:
$ var=($(grep a input))
$ echo "${var[0]}"-"${var[1]}"
abc-other
Or if you want the whole lines in each array element:
$ IFS=$'\n' var=($(grep a input)) ; unset IFS
$ echo "${var[0]}"-"${var[1]}"
abc other stuff-cba more stuff
There are two printf's - one is a shell bultin and this is invoked if you just run printf and the other is a regular binary, usually /usr/bin/printf. The latter doesn't take a -v argument, hence the error message. Since printf is an argument to xargs here, the binary is run, not the shell bulitin. Additionally, since it's at the receiving end of a pipeline, it is run as a subprocess. Variables can only be inherited from parent to child process but not the other way around, so even if the printf binary could modify the environment, the change wouldn't be visible to the parent process. So there are two reasons why your command cannot work. But you can always do var=$(something | bash -c 'some operation using builtin printf').
Mat gives an excellent explanation of what's going on and why.
If you want to iterate over the output of a command and set a variable to successive values using Bash's sprintf-style printf feature (-v), you can do it like this:
grep "Hello world" test.txt | xargs bash -c 'printf -v var "%-25s" "$#"; do_something_with_formatted "$var"' _ {} \;