This question already has answers here:
How to use shell variables in perl command call in a bash shell script?
(6 answers)
Closed 6 years ago.
I am creating a kind-of alias for fast base64 encoding of strings. For it I have created following function and added it to my .bash_profile file:
# My functions
function b64() {
perl -MMIME::Base64 -e 'print encode_base64("$1");'
}
The problem is that it encodes the string "$1" itself without processing actual value that I am "giving" to it in request:
$ b64 "test_value"
JDE=
$ echo -n "JDE=" | base64 -d
$1
I have tried using '$1' and "$1", without any quotes, but the problem persists still and it keeps encoding $1 as string and not a value.
Could you please check what am I missing here?
Thanks in advance!
Apart from the obvious quoting problem that prevents the expansion of $1, you shouldn't inject data like so in your program: you should treat data as data!
Now, I'm no Perl expert, but the following should be more robust:
b64() {
perl -MMIME::Base64 -e 'print encode_base64($ARGV[0]);' -- "$1"
}
You are using the wrong kind of quotes. You can debug this more easily if you use echo to show what you're executing:
$ b64() { echo 'print encode_base64("$1");'; }
$ b64 foo
print encode_base64("$1");
$ b64() { echo "print encode_base64('$1');"; }
$ b64 foo
print encode_base64('foo');
Other debugging techniques exist - for example printf '%q\n' or set -x.
With this knowledge, you can write your b64 as
b64() { perl -MMIME::Base64 -e "print encode_base64('$1');"; }
This gives me the expected result:
$ b64 foo
Zm9v
$ base64 -d <<<Zm9v
foo
You're missing the fact that single quotes inhibit expansion.
perl -MMIME::Base64 -e 'print encode_base64("'"$1"'");'
Related
This question already has answers here:
uppercase first character in a variable with bash
(17 answers)
Closed 2 years ago.
editAppsDotPy() {
echo 'from django.apps import AppConfig' >> apps.py
echo >> apps.py
echo >> apps.py
echo "class ${APP_NAME}Config(AppConfig):" >> apps.py
echo " name = '${APP_NAME}'" >> apps.py
}
How would you capitalize the variable in the 5th line?
I was trying to do it with ${APP_NAME^} but it returns me an error.
Your function rewritten to work with more various shells:
script.sh:
#!/usr/bin/env sh
capitalize()
{
printf '%s' "$1" | head -c 1 | tr [:lower:] [:upper:]
printf '%s' "$1" | tail -c '+2'
}
editAppsDotPy()
{
cat >> 'app.py' <<EOF
from django.apps import AppConfig
class $(capitalize "$APP_NAME")Config(AppConfig):
name = '$APP_NAME'
EOF
}
APP_NAME='foo'
editAppsDotPy
Demoing:
sh script.sh
cat app.py
Output:
from django.apps import AppConfig
class FooConfig(AppConfig):
name = 'foo'
Assuming that tr is in your path, the more common parameter substitutions can help you too.
Your fifth line could look like the following:
echo "class `tr [:lower:] [:upper:] <<<${APP_NAME:0:1}`${APP_NAME:1}Config(AppConfig):" >> apps.py
I also tested this in zsh 5.8.
If your version of bash is too old to support that extension (Like the OS X version), or you're using a shell like zsh that doesn't support it at all, you have to turn to something else. Personally, I like perl (Which I think OS X comes with):
$ perl -ne 'print ucfirst' <<<"foobar"
Foobar
or for something in the middle of a longer string:
$ foo=bar
$ echo "foo='$(perl -ne 'print ucfirst' <<<"$foo")'"
foo='Bar'
which works in bash and zsh.
I am creating this function to make multiple grep's over every line of a file. I run it as following:
cat file.txt | agrep string1 string2 ... stringN
function agrep () {
for a in $#; do
cmd+=" | grep '$a'";
done ;
while read line ; do
eval "echo "\'"$line"\'" $cmd";
done;
}
The idea is to print every line that contains all the strings: string1, string2, ..., stringN. This already works but I want to avoid the usage of the for to construct the expression:
| grep string1 | grep string2 ... | stringN
And if it's possible, also the usage of eval. I tried to make some expansion as follows:
echo "| grep $"{1..3}
And I get:
| grep $1 | grep $2 | grep $3
This is almost what I want but the problem is that when I try:
echo "| grep $"{1..$#}
The expansion doesn't occur because bash cant expand {1..$#} due to the $#. It just works with numbers. I would like to construct some expansion that works in order to avoid the usage of the for in the agrep function.
agrep () {
if [ $# = 0 ]; then
cat
else
pattern="$1"
shift
grep -e "$pattern" | agrep "$#"
fi
}
Instead of running each multiple greps on each line, just get all the lines that match string1, then pipe that to grep for string2, etc. One way to do this is make agrep recursive.
agrep () {
if (( $# == 0 )); then
cat # With no arguments, just output everything
else
grep "$1" | agrep "${#:2}"
fi
}
It's not the most efficient solution, but it's simple.
(Be sure to note Rob Mayoff's answer, which is the POSIX-compliant version of this.)
awk to the rescue!
you can avoid multiple grep calls and constructing the command by switching to awk
awk -v pat='string1 string2 string3' 'BEGIN{n=split(pat,p)}
{for(i=1;i<=n;i++) if($0!~p[i]) next}1 ' file
enter your space delimited strings as in the example above.
Not building a string for the command is definitely better (see chepner's and Rob Mayoff's answers). However, just as an example, you can avoid the for by using printf:
agrep () {
cmd=$(printf ' | grep %q' "$#")
sh -c "cat $cmd"
}
Using printf also helps somewhat with special characters in the patterns. From help printf:
In addition to the standard format specifications described in printf(1),
printf interprets:
%b expand backslash escape sequences in the corresponding argument
%q quote the argument in a way that can be reused as shell input
%(fmt)T output the date-time string resulting from using FMT as a format
string for strftime(3)
Since the aim of %q is providing output suitable for shell input, this should be safe.
Also: You almost always want to use "$#" with the quotes, not just plain $#.
I need to substitute a unique string in a json file: {FILES} by a bash variable that contains thousands of paths: ${FILES}
sed -i "s|{FILES}|$FILES|" ./myFile.json
What would be the most elegant way to achieve that ? The content of ${FILES} is a result of an "aws s3" command. The content would look like :
FILES="/file1.ipk, /file2.ipk, /subfolder1/file3.ipk, /subfolder2/file4.ipk, ..."
I can't think of a solution where xargs would help me.
The safest way is probably to let Bash itself expand the variable. You can create a Bash script containing a here document with the full contents of myFile.json, with the placeholder {FILES} replaced by a reference to the variable $FILES (not the contents itself). Execution of this script would generate the output you seek.
For example, if myFile.json would contain:
{foo: 1, bar: "{FILES}"}
then the script should be:
#!/bin/bash
cat << EOF
{foo: 1, bar: "$FILES"}
EOF
You can generate the script with a single sed command:
sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json
Notice sed is doing two replacements; the first one (s/\$/\\$/g) to escape any dollar signs that might occur within the JSON data (replace every $ by \$). The second replaces {FILES} by $FILES; the literal text $FILES, not the contents of the variable.
Now we can combine everything into a single Bash one-liner that generates the script and immediately executes it by piping it to Bash:
sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json | /bin/bash
Or even better, execute the script without spawning a subshell (useful if $FILES is set without export):
sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json | source /dev/stdin
Output:
{foo: 1, bar: "/file1.ipk, /file2.ipk, /subfolder1/file3.ipk, /subfolder2/file4.ipk, ..."}
Maybe perl would have fewer limitations?
perl -pi -e "s#{FILES}#${FILES}#" ./myFile.json
It's a little gross, but you can do it all within shell...
while read l
do
if ! echo "$l" | grep -q '{DATA}'
then
echo "$l"
else
echo "$l" | sed 's/{DATA}.*$//'
echo "$FILES"
echo "$l" | sed 's/^.*{DATA}//'
fi
done <./myfile.json >newfile.json
#mv newfile.json myfile.json
Obviously I'd leave the final line commented until you were confident it worked...
Maybe just don't do it? Can you just :
echo "var f = " > myFile2.json
echo $FILES >> myFile2.json
And reference myFile2.json from within your other json file? (You should put the global f variable into a namespace if this works for you.)
Instead of putting all those variables in an environment variable, put them in a file. Then read that file in perl:
foo.pl:
open X, "$ARGV[0]" or die "couldn't open";
shift;
$foo = <X>;
while (<>) {
s/world/$foo/;
print;
}
Command to run:
aws s3 ... >/tmp/myfile.$$
perl foo.pl /tmp/myfile.$$ <myFile.json >newFile.json
Hopefully that will bypass the limitations of the environment variable space and the argument length by pulling all the processing within perl itself.
I have a string like this:
SOMETHING='abc.abc.abc'
How can I extract the content of it ( abc.abc.abc ), inside the single-quotes?
$ awk -F"=\047|\047" '/SOMETHING/{print $(NF-1)}' file
abc.abc.abc
str="SOMETHING='abc.abc.abc'"
substr=$(echo "$str" | cut -d "'" -f 2)
With bash, you could write
substr=$(cut -d "'" -f 2 <<< "$str")
Or, shell only:
IFS="'"
set -- $str
substr=$2
Or use an array
IFS="'"
fields=($str)
substr=${fields[1]}
You should be able to already extract it. Example:
$ cat test.sh
#!/bin/bash
SOMETHING='abc.abc.abc'
echo $SOMETHING
$ ./test.sh
abc.abc.abc
You have tagged this question as shell. So I guess you're able to use regular expressions in grep. Or perhaps you could do something with javascript regular expressions.
this regex would find it
abc\.abc\.abc
my question seems to be general, but i can't find any answers.
In sed command, how can you replace the substitution pattern by a value returned by a simple bash function.
For instance, I created the following function :
function parseDates(){
#Some process here with $1 (the pattern found)
return "dateParsed;
}
and the folowing sed command :
myCatFile=`sed -e "s/[0-3][0-9]\/[0-1][0-9]\/[0-9][0-9]/& parseDates &\}/p" myfile`
I found that the caracter '&' represents the current pattern found, i'd like it to be passed to my bash function and the whole pattern to be substituted by the pattern found +dateParsed.
Does anybody have an idea ?
Thanks
you can use the "e" option in sed command like this:
cat t.sh
myecho() {
echo ">>hello,$1<<"
}
export -f myecho
sed -e "s/.*/myecho &/e" <<END
ni
END
you can see the result without "e":
cat t.sh
myecho() {
echo ">>hello,$1<<"
}
export -f myecho
sed -e "s/.*/myecho &/" <<END
ni
END
Agree with Glenn Jackman.
If you want to use bash function in sed, something like this :
sed -rn 's/^([[:digit:].]+)/`date -d #&`/p' file |
while read -r line; do
eval echo "$line"
done
My file here begins with a unix timestamp (e.g. 1362407133.936).
Bash function inside sed (maybe for other purposes):
multi_stdin(){ #Makes function accepet variable or stdin (via pipe)
[[ -n "$1" ]] && echo "$*" || cat -
}
sans_accent(){
multi_stdin "$#" | sed '
y/àáâãäåèéêëìíîïòóôõöùúûü/aaaaaaeeeeiiiiooooouuuu/
y/ÀÁÂÃÄÅÈÉÊËÌÍÎÏÒÓÔÕÖÙÚÛÜ/AAAAAAEEEEIIIIOOOOOUUUU/
y/çÇñÑߢÐð£Øø§µÝý¥¹²³ªº/cCnNBcDdLOoSuYyY123ao/
'
}
eval $(echo "Rogério Madureira" | sed -n 's#.*#echo & | sans_accent#p')
or
eval $(echo "Rogério Madureira" | sed -n 's#.*#sans_accent &#p')
Rogerio
And if you need to keep the output into a variable:
VAR=$( eval $(echo "Rogério Madureira" | sed -n 's#.*#echo & | desacentua#p') )
echo "$VAR"
do it step by step. (also you could use an alternate delimiter , such as "|" instead of "/"
function parseDates(){
#Some process here with $1 (the pattern found)
return "dateParsed;
}
value=$(parseDates)
sed -n "s|[0-3][0-9]/[0-1][0-9]/[0-9][0-9]|& $value &|p" myfile
Note the use of double quotes instead of single quotes, so that $value can be interpolated
I'd like to know if there's a way to do this too. However, for this particular problem you don't need it. If you surround the different components of the date with ()s, you can back reference them with \1 \2 etc and reformat however you want.
For instance, let's reverse 03/04/1973:
echo 03/04/1973 | sed -e 's/\([0-9][0-9]\)\/\([0-9][0-9]\)\/\([0-9][0-9][0-9][0-9]\)/\3\/\2\/\1/g'
sed -e 's#[0-3][0-9]/[0-1][0-9]/[0-9][0-9]#& $(parseDates &)#' myfile |
while read -r line; do
eval echo "$line"
done
You can glue together a sed-command by ending a single-quoted section, and reopening it again.
sed -n 's|[0-3][0-9]/[0-1][0-9]/[0-9][0-9]|& '$(parseDates)' &|p' datefile
However, in contrast to other examples, a function in bash can't return strings, only put them out:
function parseDates(){
# Some process here with $1 (the pattern found)
echo dateParsed
}