Is there a way to send the back references of the SED s/// command to pipes? In order to get an entry from the text, change it, and then write it back. I found that a substitution inside of SED works:
$ echo 'Test ....' | sed 's/Test/'$( echo "<\0>" )'/'
<Test> ....
But the first pipe does not:
$ echo 'Test ....' | sed 's/Test/'$( echo "<\0>" | tr 's' 'x' )'/'
<Test> ....
What is the reason? Additionally, I can't understand why this works at all. I thought that $() substitution should be processed before sed (all the more as I broke the quotes).
And how can I insert one s/// command into another using sed? I use bash.
The tr command is operating on the text "<\0>", not on "<Test>". The back reference isn't expanded in sed until after the pipeline completes. Your second example is equivalent to
foo=$( echo "<\0>" | tr 's' 'x' )
echo 'Test ....' | sed 's/Test/'$foo'/'
It's a little easier to see here that tr has no way of seeing "Test" in its input.
You can achieve the effect you're after with GNU sed and the e flag:
echo 'Test ....' | sed 's/Test.*/echo "<\0>" | tr s x/e'
Output:
<Text ....>
Related
It is simple - I have a data stream with IPv4 addresses encoded into hexadecimal representation like for example 0c22384e which stands for 12.34.56.78.
I figured out sed command with substitution of captured octets into decimal numbers separated by dot.
echo "0c22384e" | sed -E 's/([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2})/printf "%d.%d.%d.%d" 0x\1 0x\2 0x\3 0x\4/eg'
This works with a single number BUT as soon I add some text that is not supposed to be matched, it is also passed for the execution - via printf in this case.
How can I preserve the unmatched part of the line without being passed for the execution?
With only one address in a line you could use
echo "Something 0c22384e more" |
sed -r 's/(.*)([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2})(.*)/"\1" 0x\2 0x\3 0x\4 0x\5 "\6"/' |
xargs -n6 printf '%s%d.%d.%d.%d%s\n'
EDIT:
Replaced solution for one line and more addresses
with solution for more lines (assuming no '\r' in the stream):
echo "Something 0c22384e more 0c22385e
Second line: 0c22386e and 0c223870
Third line: 0c22388e and 0c223890
4th line: 0c2238ae and 0c2238b0" |
sed 's/$/\r/' |
sed -r 's/[0-9a-f]{8}/\n&\n/g' |
sed -r 's/([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2})/printf '%d.%d.%d.%d' 0x\1 0x\2 0x\3 0x\4/e' |
tr -d '\n' |
tr '\r' '\n'
I have a variable which contains key/values separated by space:
echo $PROPERTY
server_geo=BOS db.jdbc_url=jdbc\:mysql\://mysql-test.com\:3306/db02 db.name=db02 db.hostname=/mysql-test.com datasource.class.xa=com.mysql.jdbc.jdbc2.optional.MysqlXADataSource server_uid=BOS_mysql57 hibernate33.dialect=org.hibernate.dialect.MySQL5InnoDBDialect hibernate.connection.username=db02 server_labels=mysql57,mysql5,mysql db.jdbc_class=com.mysql.jdbc.Driver db.schema=db02 hibernate.connection.driver_class=com.mysql.jdbc.Driver uuid=a19ua19 db.primary_label=mysql57 db.port=3306 server_label_primary=mysql57 hibernate.dialect=org.hibernate.dialect.MySQL5InnoDBDialect
I'd need to extract the values of the single keys, for example db.jdbc_url.
Using one code snippet I've found:
echo $PROPERTY | sed -e 's/ db.jdbc_url=\(\S*\).*/\1/g'
but that returns also other properties found before my key.
Any help how to fix it ?
Thanks
If db.name always follow db.jdbc_url, then use grep lookaround,
$ echo "${PROPERTY}" | grep -oP '(?<=db.jdbc_url=).*(?=db.name)'
jdbc\:mysql\://mysql-test.com\:3306/db02
or add the VAR to an array,
$ myarr=($(echo $PROPERTY))
$ echo "${myarr[1]}" | grep -oP '(?<=db.jdbc_url=).*(?=$)'
jdbc\:mysql\://mysql-test.com\:3306/db02
This is caused because you are using the substitute command (sed s/.../.../), so any text before your regex is kept as is. Using .* before db\.jdbc_url along with the begin (^) / end ($) of string marks makes you match the whole content of the variable.
In order to be totaly safe, your regex should be :
sed -e 's/^.*db\.jdbc_url=\(\S*\).*$/\1/g'
You can use grep for this, like so:
echo $PROPERTY | grep -oE "db.jdbc_url=\S+" | cut -d'=' -f2
The regex is very close to the one you used with sed.
The -o option is used to print the matched parts of the matching line.
Edit: if you want only the value, cut on the '='
Edit 2: egrep say it is deprecated, so use grep -oE instead, same result. Just to cover all bases :-)
I have a string like this :
test:blabla
And with sed, I want to replace what's after the ':' with something else.
I can manage to replace a word, but not the one after the ':'. I searched on the internet for the answer, but I didn't found anything.
Any help ?
Use: sed 's/:.*/:replaceword/'
$ echo test:blabla | sed 's/:.*/:replaceword/'
test:replaceword
Or for the situation test test:blabla test where you only want to replace the word following : use sed 's/:[^ ]*/:replaceword/':
$ echo "test test:blabla test" | sed 's/:[^ ]*/:replaceword/'
test test:replaceword test
# Use the g flag for multiple matches on a line
$ echo "test test:blabla test test:blah2" | sed 's/:[^ ]*/:replaceword/g'
test test:replaceword test test:replaceword
> echo $SER2
test:blabla
> echo $SER2 | sed 's/\([^:]*:\).*/\1replace/g'
test:replace
>
I want to get the string between <sometag param=' and '>
I tried to use the method from Get any string between 2 string and assign a variable in bash to get the "x":
echo "<sometag param='x'><irrelevant stuff='nonsense'>" | tr "'" _ | sed -n 's/.*<sometag param=_\(.*\)_>.*/\1/p'
The problem (apart from low efficiency because I just cannot manage to escape the apostrophe correctly for sed) is that sed matches the maximum, i.e. the output is:
x_><irrelevant stuff=_nonsense
but the correct output would be the minimum-match, in this example just "x"
Thanks for your help
You are probably looking for something like this:
sed -n "s/.*<sometag param='\([^']*\)'>.*/\1/p"
Test:
echo "<sometag param='x'><irrelevant stuff='nonsense'>" | sed -n "s/.*<sometag param='\([^']*\)'>.*/\1/p"
Results:
x
Explanation:
Instead of a greedy capture, use a non-greedy capture like: [^']* which means match anything except ' any number of times. To make the pattern stick, this is followed by: '>.
You can also use double quotes so that you don't need to escape the single quotes. If you wanted to escape the single quotes, you'd do this:
-
... | sed -n 's/.*<sometag param='\''\([^'\'']*\)'\''>.*/\1/p'
Notice how that the single quotes aren't really escaped. The sed expression is stopped, an escaped single quote is inserted and the sed expression is re-opened. Think of it like a four character escape sequence.
Personally, I'd use GNU grep. It would make for a slightly shorter solution. Run like:
... | grep -oP "(?<=<sometag param=').*?(?='>)"
Test:
echo "<sometag param='x'><irrelevant stuff='nonsense'>" | grep -oP "(?<=<sometag param=').*?(?='>)"
Results:
x
You don't have to assemble regexes in those cases, you can just use ' as the field separator
in="<sometag param='x'><irrelevant stuff='nonsense'>"
IFS="'" read x whatiwant y <<< "$in" # bash
echo "$whatiwant"
awk -F\' '{print $2}' <<< "$in" # awk
How to I concatenate stdin to a string, like this?
echo "input" | COMMAND "string"
and get
inputstring
A bit hacky, but this might be the shortest way to do what you asked in the question (use a pipe to accept stdout from echo "input" as stdin to another process / command:
echo "input" | awk '{print $1"string"}'
Output:
inputstring
What task are you exactly trying to accomplish? More context can get you more direction on a better solution.
Update - responding to comment:
#NoamRoss
The more idiomatic way of doing what you want is then:
echo 'http://dx.doi.org/'"$(pbpaste)"
The $(...) syntax is called command substitution. In short, it executes the commands enclosed in a new subshell, and substitutes the its stdout output to where the $(...) was invoked in the parent shell. So you would get, in effect:
echo 'http://dx.doi.org/'"rsif.2012.0125"
use cat - to read from stdin, and put it in $() to throw away the trailing newline
echo input | COMMAND "$(cat -)string"
However why don't you drop the pipe and grab the output of the left side in a command substitution:
COMMAND "$(echo input)string"
I'm often using pipes, so this tends to be an easy way to prefix and suffix stdin:
echo -n "my standard in" | cat <(echo -n "prefix... ") - <(echo " ...suffix")
prefix... my standard in ...suffix
There are some ways of accomplish this, i personally think the best is:
echo input | while read line; do echo $line string; done
Another can be by substituting "$" (end of line character) with "string" in a sed command:
echo input | sed "s/$/ string/g"
Why i prefer the former? Because it concatenates a string to stdin instantly, for example with the following command:
(echo input_one ;sleep 5; echo input_two ) | while read line; do echo $line string; done
you get immediatly the first output:
input_one string
and then after 5 seconds you get the other echo:
input_two string
On the other hand using "sed" first it performs all the content of the parenthesis and then it gives it to "sed", so the command
(echo input_one ;sleep 5; echo input_two ) | sed "s/$/ string/g"
will output both the lines
input_one string
input_two string
after 5 seconds.
This can be very useful in cases you are performing calls to functions which takes a long time to complete and want to be continuously updated about the output of the function.
You can do it with sed:
seq 5 | sed '$a\6'
seq 5 | sed '$ s/.*/\0 6/'
In your example:
echo input | sed 's/.*/\0string/'
I know this is a few years late, but you can accomplish this with the xargs -J option:
echo "input" | xargs -J "%" echo "%" "string"
And since it is xargs, you can do this on multiple lines of a file at once. If the file 'names' has three lines, like:
Adam
Bob
Charlie
You could do:
cat names | xargs -n 1 -J "%" echo "I like" "%" "because he is nice"
Also works:
seq -w 0 100 | xargs -I {} echo "string "{}
Will generate strings like:
string 000
string 001
string 002
string 003
string 004
...
The command you posted would take the string "input" use it as COMMAND's stdin stream, which would not produce the results you are looking for unless COMMAND first printed out the contents of its stdin and then printed out its command line arguments.
It seems like what you want to do is more close to command substitution.
http://www.gnu.org/software/bash/manual/html_node/Command-Substitution.html#Command-Substitution
With command substitution you can have a commandline like this:
echo input `COMMAND "string"`
This will first evaluate COMMAND with "string" as input, and then expand the results of that commands execution onto a line, replacing what's between the ‘`’ characters.
cat will be my choice: ls | cat - <(echo new line)
With perl
echo "input" | perl -ne 'print "prefix $_"'
Output:
prefix input
A solution using sd (basically a modern sed; much easier to use IMO):
# replace '$' (end of string marker) with 'Ipsum'
# the `e` flag disables multi-line matching (treats all lines as one)
$ echo "Lorem" | sd --flags e '$' 'Ipsum'
Lorem
Ipsum#no new line here
You might observe that Ipsum appears on a new line, and the output is missing a \n. The reason is echo's output ends in a \n, and you didn't tell sd to add a new \n. sd is technically correct because it's doing exactly what you are asking it to do and nothing else.
However this may not be what you want, so instead you can do this:
# replace '\n$' (new line, immediately followed by end of string) by 'Ipsum\n'
# don't forget to re-add the `\n` that you removed (if you want it)
$ echo "Lorem" | sd --flags e '\n$' 'Ipsum\n'
LoremIpsum
If you have a multi-line string, but you want to append to the end of each individual line:
$ ls
foo bar baz
$ ls | sd '\n' '/file\n'
bar/file
baz/file
foo/file
I want to prepend my sql script with "set" statement before running it.
So I echo the "set" instruction, then pipe it to cat. Command cat takes two parameters : STDIN marked as "-" and my sql file, cat joins both of them to one output. Next I pass the result to mysql command to run it as a script.
echo "set #ZERO_PRODUCTS_DISPLAY='$ZERO_PRODUCTS_DISPLAY';" | cat - sql/test_parameter.sql | mysql
p.s. mysql login and password stored in .my.cnf file