What I am trying to achieve is pass the Base64 encoded value captured in the sed regex to the base64 and have it decoded.
But the problem is, even though it seems like the correct value is being passed to the function using backreference, base64 complains that the input is invalid.
Following is my script -
#!/bin/bash
decodeBaseSixtyFour() {
echo "$1 is decoded to `echo $1 | base64 -d`"
}
echo Passing direct value ...
echo SGVsbG8gQmFzZTY0Cg== | sed -r "s/(.+)$/$(decodeBaseSixtyFour SGVsbG8gQmFzZTY0Cg==)/"
echo Passing captured value ...
echo SGVsbG8gQmFzZTY0Cg== | sed -r "s/(.+)$/$(decodeBaseSixtyFour \\1)/"
And when ran it produces the following output -
Passing direct value ...
SGVsbG8gQmFzZTY0Cg== is decoded to Hello Base64
Passing captured value ...
base64: invalid input
SGVsbG8gQmFzZTY0Cg== is decoded to
I think the output explains what I mean.
Is it possible to do what I am trying to do? If not, why?
Perl s/// can do what you want, but I don't think what you're asking for is what you need.
$ echo SGVsbG8gQmFzZTY0Cg== | perl -MMIME::Base64 -pe 's/(.+)/decode_base64($1)/e'
Hello Base64
What's actually happening:
echo SGVsbG8gQmFzZTY0Cg== | sed -r "s/(.+)$/$(decodeBaseSixtyFour \\1)/"
Before sed starts reading input, the shell notices the process substitution in the double quoted string
the decodeBaseSixtyFour function is called with the string "\\1"
base64 chokes on the input \1 and emits the error message
the function returns the string "\1 is decoded to "
now the sed script is 's/(.+)$/\1 is decoded to /' which is how you get the last line.
As I commented sed cannot do an equivalent of replace_callback which is esentially what you're trying to do.
Following awk does close to what you're trying to do:
s="My string is SGVsbG8gQmFzZTY0Cg== something"
awk '{for(i=1; i<=NF; i++) if ($i~/==$/) "base64 -D<<<"$i|getline $i}1'<<<"$s"
My string is Hello Base64 something
Related
I strongly doubt about the grep best use in my code and would like to find a better and cleaner coding style for extracting the session ID and security level from my cookie file :
cat mycookie
# Netscape HTTP Cookie File
# https://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.
#HttpOnly_127.0.0.1 FALSE / FALSE 0 PHPSESSID 1hjs18icittvqvpa4tm2lv9b12
#HttpOnly_127.0.0.1 FALSE /mydir/ FALSE 0 security medium
The expected output is the SSID hash :
1hjs18icittvqvpa4tm2lv9b12
Piping grep with tr '\n' '\0' works like a charm in the command line, but generates warnings (warning: command substitution: ignored null byte in input”) at the bash code execution. Here is the related code (with warnings):
ssid=$(grep -Po 'PHPSESSID.*' path/sessionFile | grep -Po '[a-z]|[0-9]' | tr '\n' '\0')
I am using bash 4.4.12 (x86_64-pc-linux-gnu) and could read here this crystal clear explanation :
Bash variables are stored as C strings. C strings are NUL-terminated.
They thus cannot store NULs by definition.
I could see here and there in both cases a coding solution using read:
# read content from stdin into array variable and a scalar variable "suffix"
array=( )
while IFS= read -r -d '' line; do
array+=( "$line" )
done < <(process that generates NUL stream here)
suffix=$line # content after last NUL, if any
# emit recorded content
printf '%s\0' "${array[#]}"; printf '%s' "$suffix"
I don't want to use arrays nor a while loop for this specific case, or others. I found this workaround using sed:
ssid=$(grep -Po 'PHPSESSID.*' path/sessionFile | grep -Po '[a-z]|[0-9]' | tr '\n' '_' | sed -e 's/_//g')
My two questions are :
1) Would it be a better way to substitute tr '\n' '\0', without using read into a while loop ?
2) Would it be a better way to extract properly the SSID and security level ?
Thx
It looks like you're trying to get rid of the newlines in the output from grep, but turning them into nulls doesn't do this. Nulls aren't visible in your terminal, but are still there and (like many other nonprinting characters) will wreak havoc if they get treated as part of your actual data. If you want to get rid of the newlines, just tell tr to delete them for you with ... | tr -d '\n'. But if you're trying to get the PHPSESSID value from a Netscape-format cookie file, there's a much much better way:
ssid=$(awk '($6 == "PHPSESSID") {print $7}' path/sessionFile)
This looks for "PHPSESSID" only in the sixth field (not in e.g. the path or cookie values -- both places it could legally appear), and specifically prints the seventh field of matching lines (not just anything after "PHPSESSID" that happens to be a digit or lowercase letter).
You could also try this, if you don't want to use awk:
ssid=$(grep -P '\bPHPSESSID\b' you_cookies_file)
echo $ssid # for debug only
which outputs something like
#HttpOnly_127.0.0.1 FALSE / FALSE 0 PHPSESSID 1hjs18icittvqvpa4tm2lv9b12
Then with cut(1) extract the relevant field:
echo $ssid |cut -d" " -f7
which outputs
1hjs18icittvqvpa4tm2lv9b12
Of course you should capture the last echo.
UPDATE
If you don't want to use cut, it is possible to emulate it with:
echo $ssid | (read a1 b2 c3 d4 e5 f6 g7; echo $g7)
Demonstration to capture in a variable:
$ field=$(echo $ssid | (read a1 b2 c3 d4 e5 f6 g7; echo $g7))
$ echo $field
1hjs18icittvqvpa4tm2lv9b12
$
Another way is to use positional parameters passing the string to a function which then refers to $7. Perhaps cleaner. Otherwise, you can use an array:
array=($(echo $ssid))
echo ${array[6]} # outputs the 7th field
It should also be possible to use regular expressions and/or string manipulation is bash, but they seem a little more difficult to me.
This question already has answers here:
BASH extract value after string in variable Not file [duplicate]
(2 answers)
Closed last year.
I need to extract a number from the output of a command: cmd. The output is type: 1000
So my question is how to execute the command, store its output in a variable and extract 1000 in a shell script. Also how do you store the extracted string in a variable?
This question has been answered in pieces here before, it would be something like this:
line=$(sed -n '2p' myfile)
echo "$line"
if [ `echo $line || grep 'type: 1000' ` ] then;
echo "It's there!";
fi;
Store output of sed into a variable
String contains in Bash
EDIT: sed is very limited, you would need to use bash, perl or awk for what you need.
This is a typical use case for grep:
output=$(cmd | grep -o '[0-9]\+')
You can write the output of a command or even a pipeline of commands into a shell variable using so called command substitution:
variable=$(cmd);
In comments it appeared that the output of cmd contains more lines than the type : 1000. In this case I would suggest sed:
output=$(cmd | sed -n 's/type : \([0-9]\+\)/\1/p;q')
You tagged your question as sed but your question description does not restrict other tools, so here's a solution using awk.
output = `cmd | awk -F':' '/type: [0-9]+/{print $2}'`
Alternatively, you can use the newer $( ) syntax. Some find the newer syntax preferable and it can be conveniently nested, without the need for escaping backtics.
output = $(cmd | awk -F':' '/type: [0-9]+/{print $2}')
If the output is rigidly restricted to "type: " followed by a number, you can just use cut.
var=$(echo 'type: 1000' | cut -f 2 -d ' ')
Obviously you'll have to pipe the output of your command to cut, I'm using echo as a demo.
In addition, I'd use grep and then cut if the string you are searching is more complex. If we assume there can be all kind of numbers in the text, but only one occurrence of "type: " followed by a number, you can use the command:
>> var=$(echo "hello 12 type: 1000 foo 1001" | grep -oE "type: [0-9]+" | cut -f 2 -d ' ')
>> echo $var
1000
You can use the | operator to send the output of one command to another, like so:
echo " 1\n 2\n 3\n" | grep "2"
This sends the string " 1\n 2\n 3\n" to the grep command, which will search for the line containing 2. It sound like you might want to do something like:
cmd | grep "type"
Here is a plain sed solution that uses a regualar expression to find the number in your string:
cmd | sed 's/^.*type: \([0-9]\+\)/\1/g'
^ means from the start
.* can be any character (also none)
\([0-9]\+\) are numbers (minimum one character)
\1 means it takes the first pattern it finds (and only in this case) and uses it as replacement for the whole string
i am having a problem with (STDIN)
i have written the following bash script to calculate the sha1 hash of a string and output it in upper-case formatted ready for copying into a plist file for example.
here is the script :-
#!/bin/bash
echo 'Enter String :-';
read temp_var1; #Read string
hash=$(echo -n $temp_var1 | openssl sha1); #Copy hash into variable $hash
uphash=$(echo $hash | tr '[:lower:]' '[:upper:]';); #Convert hash to Upper-case
echo 'Below is the UPPERCASE SHA1 Hash of your string';
echo ' </yourHash>'$uphash'</yourHash>'; #Echo formatted string
This whole script works well apart for one key problem.
the output is not what i want it to be.
for example with an input of
Hello, World!
i want to get the following output:-
Below is the UPPERCASE SHA1 Hash of your string
</yourHash>0A0A9F2A6772942557AB5355D76AF442F8F65E01</yourHash>
instead what i am getting is this :-
Below is the UPPERCASE SHA1 Hash of your string
</yourHash>(STDIN)= 0A0A9F2A6772942557AB5355D76AF442F8F65E01</yourHash>
my question is. How do i remove this (STDIN)= lable and the space that follows it?
many thanks in advance for any help you can offer.
MiRAGE
You can strip it off the front using parameter substitution like this:
a="(STDIN)= 0A0A9F2A6772942557AB5355D76AF442F8F65E01"
b=${a#*= } # Strip up to "=" sign
echo $b
0A0A9F2A6772942557AB5355D76AF442F8F65E01
I suspect that there is an option to openssl to stop it generating the "(STDIN)" part, but mine doesn't do that.
If you want to learn more about this sort of thing, have a look here.
Probably not the best solution, but you can use awk to select 2nd column. Like this:
echo -n $temp_var1 | openssl sha1 | awk '{print $2}'
I was successful with this approach (using sed) as well:
hash="(STDIN)= 0A0A9F2A6772942557AB5355D76AF442F8F65E01"
echo "before"
echo $hash
hash=$(echo $hash | sed 's/(stdin)= //g') #remove identified pattern globally
echo "after"
echo $hash
Result
before
(STDIN)= 0A0A9F2A6772942557AB5355D76AF442F8F65E01
after
0A0A9F2A6772942557AB5355D76AF442F8F65E01
How to I concatenate stdin to a string, like this?
echo "input" | COMMAND "string"
and get
inputstring
A bit hacky, but this might be the shortest way to do what you asked in the question (use a pipe to accept stdout from echo "input" as stdin to another process / command:
echo "input" | awk '{print $1"string"}'
Output:
inputstring
What task are you exactly trying to accomplish? More context can get you more direction on a better solution.
Update - responding to comment:
#NoamRoss
The more idiomatic way of doing what you want is then:
echo 'http://dx.doi.org/'"$(pbpaste)"
The $(...) syntax is called command substitution. In short, it executes the commands enclosed in a new subshell, and substitutes the its stdout output to where the $(...) was invoked in the parent shell. So you would get, in effect:
echo 'http://dx.doi.org/'"rsif.2012.0125"
use cat - to read from stdin, and put it in $() to throw away the trailing newline
echo input | COMMAND "$(cat -)string"
However why don't you drop the pipe and grab the output of the left side in a command substitution:
COMMAND "$(echo input)string"
I'm often using pipes, so this tends to be an easy way to prefix and suffix stdin:
echo -n "my standard in" | cat <(echo -n "prefix... ") - <(echo " ...suffix")
prefix... my standard in ...suffix
There are some ways of accomplish this, i personally think the best is:
echo input | while read line; do echo $line string; done
Another can be by substituting "$" (end of line character) with "string" in a sed command:
echo input | sed "s/$/ string/g"
Why i prefer the former? Because it concatenates a string to stdin instantly, for example with the following command:
(echo input_one ;sleep 5; echo input_two ) | while read line; do echo $line string; done
you get immediatly the first output:
input_one string
and then after 5 seconds you get the other echo:
input_two string
On the other hand using "sed" first it performs all the content of the parenthesis and then it gives it to "sed", so the command
(echo input_one ;sleep 5; echo input_two ) | sed "s/$/ string/g"
will output both the lines
input_one string
input_two string
after 5 seconds.
This can be very useful in cases you are performing calls to functions which takes a long time to complete and want to be continuously updated about the output of the function.
You can do it with sed:
seq 5 | sed '$a\6'
seq 5 | sed '$ s/.*/\0 6/'
In your example:
echo input | sed 's/.*/\0string/'
I know this is a few years late, but you can accomplish this with the xargs -J option:
echo "input" | xargs -J "%" echo "%" "string"
And since it is xargs, you can do this on multiple lines of a file at once. If the file 'names' has three lines, like:
Adam
Bob
Charlie
You could do:
cat names | xargs -n 1 -J "%" echo "I like" "%" "because he is nice"
Also works:
seq -w 0 100 | xargs -I {} echo "string "{}
Will generate strings like:
string 000
string 001
string 002
string 003
string 004
...
The command you posted would take the string "input" use it as COMMAND's stdin stream, which would not produce the results you are looking for unless COMMAND first printed out the contents of its stdin and then printed out its command line arguments.
It seems like what you want to do is more close to command substitution.
http://www.gnu.org/software/bash/manual/html_node/Command-Substitution.html#Command-Substitution
With command substitution you can have a commandline like this:
echo input `COMMAND "string"`
This will first evaluate COMMAND with "string" as input, and then expand the results of that commands execution onto a line, replacing what's between the ‘`’ characters.
cat will be my choice: ls | cat - <(echo new line)
With perl
echo "input" | perl -ne 'print "prefix $_"'
Output:
prefix input
A solution using sd (basically a modern sed; much easier to use IMO):
# replace '$' (end of string marker) with 'Ipsum'
# the `e` flag disables multi-line matching (treats all lines as one)
$ echo "Lorem" | sd --flags e '$' 'Ipsum'
Lorem
Ipsum#no new line here
You might observe that Ipsum appears on a new line, and the output is missing a \n. The reason is echo's output ends in a \n, and you didn't tell sd to add a new \n. sd is technically correct because it's doing exactly what you are asking it to do and nothing else.
However this may not be what you want, so instead you can do this:
# replace '\n$' (new line, immediately followed by end of string) by 'Ipsum\n'
# don't forget to re-add the `\n` that you removed (if you want it)
$ echo "Lorem" | sd --flags e '\n$' 'Ipsum\n'
LoremIpsum
If you have a multi-line string, but you want to append to the end of each individual line:
$ ls
foo bar baz
$ ls | sd '\n' '/file\n'
bar/file
baz/file
foo/file
I want to prepend my sql script with "set" statement before running it.
So I echo the "set" instruction, then pipe it to cat. Command cat takes two parameters : STDIN marked as "-" and my sql file, cat joins both of them to one output. Next I pass the result to mysql command to run it as a script.
echo "set #ZERO_PRODUCTS_DISPLAY='$ZERO_PRODUCTS_DISPLAY';" | cat - sql/test_parameter.sql | mysql
p.s. mysql login and password stored in .my.cnf file
Here is a sample string . I would like to get the output from this in the specified format.
String:
/vob/TEST/.##/main/ch_vobsweb/1/VOBSWeb/main/ch_vobsweb/4/VobsWebUI/main/ch_vobsweb/2/VaultWeb/main/ch_vobsweb/2/func.js
filename;path to file
func.js;VOBSWeb/VosWebUI/VaultWeb/func.js
The filename is listed at the end of the whole string , and it's path is supposed to be stripped using the characters after each numeric value (eg. /1/VOBSWeb/ and then /4/VobsWebUI and then /2/vaultWeb)
one way
$ string="/vob/TEST/.##/main/ch_vobsweb/1/VOBSWeb/main/ch_vobsweb/4/VobsWebUI/main/ch_vobsweb/2/VaultWeb/main/ch_vobsweb/2/func.js"
$ path=$(echo "$string" | sed "s|\/[0-9]\/|\n|g"|sed 's|\/.*||' | tr "\n" "/"|sed 's/\/$//')
$ echo ${path##*/}
func.js
$ echo ${path%\/*}
/VOBSWeb/VobsWebUI/VaultWeb