Hello I am trying to insert the output of command output > sed replace after a particular string in a file as part of user data on machine boot up
[centos#ip-192-168-2-22 scylla]$ sudo sed -i.bak 's/broadcast_rpc_address: : /broadcast_rpc_address:\/$hostname -i | awk '{print $2}'/' /etc/scylla/scylla.yaml
file is currently
broadcast_rpc_address:
replace with
broadcast_rpc_address: (the ip of the machine)
Something like this should work:
sed -i.bak "s/broadcast_rpc_address:/broadcast_rpc_address: $(hostname -i)/" /etc/scylla/scylla.yaml
This will replace broadcast_rpc_address: by broadcast_rpc_address: $(hostname -i). Now, because this string is in double quotes - not single quotes - this tells the shell to interpret some magic sequences inside the string. In particular $(somecommand) means to run somecommand and insert its output into the string. Of course, change "hostname -i" in the command I gave above to anything else you want (it can even be an entire pipeline.
Your original attempt used something that started with $hostname. This syntax, $hostname, doesn't run the command hostname, but rather looks for a variable called hostname, which isn't what you wanted. You need the $(...) syntax instead. Your original attempt also had problems with nested quotes, which don't work.
Related
I am trying to script a way of removing hosts from the hostgroup file in Nagios Core.
The format of the hostgroup file is:
server1,server2,server3,server4
When removing a server, I need to be able to not only remove the server, but also the comma that follows it. So in my example above, if I am removing server2, the file would result as follows
server1,server3,server4
So I have googled and tested the following which works to remove server2 and a comma after it (I don't know what the b is used for exactly)
sed -i 's/\bserver2\b,//g' myfile
What I want to be able to do is to feed a list of hostnames to a small script to remove a bunch of hosts (and their following comma) with something similar to the following. The problem lies in that placing a variable like $x breaks the script so that nothing happens.
#!/bin/ksh
for x in `cat /tmp/list`
do
sed -i 's/\b${x}\b,//g' myfile
done
I think I am very close on a solution here, but could use a little help. Thanks much in advance for your kind assistance.
Using single quotes tells the shell not to replace the ${x} - it turns off variable interpolation if you want to google for it.
https://www.tldp.org/LDP/abs/html/quotingvar.html. So use double quotes around the sed replacement string instead:
while read -r x; do sed -i "s/\b${x},\b//g" myfile; done < /tmp/list
But since the last field won't have a comma after it, might be a good idea to run two sed commands, one looking for \bword,\b and the other for ,word$ - where \b is a word boundary and $ is the end of line.
while read -r x; do sed -i "s/\b${x},\b//g" myfile; sed -i "s/,${x}$//" myfile ; done < /tmp/list
One other possible boundary condition - what if you have just server2 on a line by itself and that's what you're trying to delete? Perhaps add a third sed, but this one will leave a blank line behind which you might want to remove:
while read -r x
do
sed -i "s/\b${x},\b//g" myfile # find and delete word,
sed -i "s/,${x}$//" myfile # find and delete ,word
sed -i "s/^${x}$//" myfile # find word on a line by itself
done < t
This works quite nicely:
#!/bin/bash
IN_FILE=$1
shift; sed -i "s/\bserver[$#],*\b//g" $IN_FILE; sed -i "s/,$//g" $IN_FILE
if you invoke it like ./remove_server.sh myfile "1 4" for your example file containing server1,server2,server3,server4, you get the following output:
server2,server3
A quick explanation of what it does:
shift shifts the arguments down by one (making sure that "myfile" isn't fed into the regex)
First sed removes the server with the numbers supplied as arguments in the string (e.g. "1 4")
Second sed looks for a trailing comma and removes it
The \b matches a word boundary
This is a great resource for learning about and testing regex: https://regex101.com/r/FxmjO5/1. I would recommend you check it out and use it each time you have a regex problem. It's helped me on so many occasions!
An example of this script working in a more general sense:
I tried it out on this file:
# This is some file containing server info:
# Here are some servers:
server2,server3
# And here are more servers:
server7,server9
with ./remove_server.sh myfile "2 9" and got this:
# This is some file containing info:
# Here are some servers:
server3
# And here are more servers:
server7
Pretty sure there is a pure sed solution for this but here is a script.
#!/usr/bin/env bash
hosts=()
while read -r host; do
hosts+=("s/\b$host,\{,1\}\b//g")
done < /tmp/list
opt=$(IFS=';' ; printf '%s' "${hosts[*]};s/,$//")
sed "$opt" myfile
It does not run sed line-by-line, but only one sed invocation. Just in case, say you have to remove 20+ pattern then sed will not run 20+ times too.
Add the -i if you think the output is ok.
Using perl and regex by setting the servers to a regex group in a shell variable:
$ remove="(server1|server4)"
$ perl -p -e "s/(^|,)$remove(?=(,|$))//g;s/^,//" file
server2,server3
Explained:
remove="(server1|server4)" or "server1" or even "server."
"s/(^|,)$remove(?=(,|$))//g" double-quoted to allow shell vars, remove leading comma, expected to be followed by a comma or the end of string
s/^,// file remove leading comma if the first entry was deleted
Use the -i switch for infile editing.
bash script that reads the servers to remove from standard input, one per line, and uses perl to remove them from the hostfile (Passed as the first argument to the script):
#!/usr/bin/env bash
# Usage: removehost.sh hostgroupfile < listfile
mapfile -t -u 0 servers
IFS="|"
export removals="${servers[*]}"
perl -pi -e 's/,?(?:$ENV{removals})\b//g; s/^,//' "$1"
It reads the servers to remove into an array, joins that into a pipe-separated string, and then uses that in the perl regular expression to remove all the servers in a single pass through the file. Slashes and other funky characters (As long as they're not RE metacharacters) won't mess up the parsing of the perl, because it uses the environment variable instead of embedding the string directly. It also uses a word boundry so that removing server2 won't remove that part of server22.
I'm trying to temporarily disable dhcp on all connections in a computer using bash, so I need the process to be reversible. My approach is to comment out lines that contain BOOTPROTO=dhcp, and then insert a line below it with BOOTPROTO=none. I'm not sure of the correct syntax to make sed understand the line number stored in the $insertLine variable.
fileList=$(ls /etc/sysconfig/network-scripts | grep ^ifcfg)
path="/etc/sysconfig/network-scripts/"
for file in $fileList
do
echo "looking for dhcp entry in $file"
if [ $(cat $path$file | grep ^BOOTPROTO=dhcp) ]; then
echo "disabling dhcp in $file"
editLine=$(grep -n ^BOOTPROTO=dhcp /$path$file | cut -d : -f 1 )
#comment out the original dhcp value
sed -i "s/BOOTPROTO=dhcp/#BOOTPROTO=dhcp/g" $path$file
#insert a line below it with value of none.
((insertLine=$editLine+1))
sed "$($insertLine)iBOOTPROTO=none" $path$file
fi
done
Any help using sed or other stream editor greatly appreciated. I'm using RHEL 6.
The sed editor should be able to do the job, without having to to be combine bash, grep, cat, etc. Easier to test, and more reliable.
The whole scripts can be simplified to the below. It performs all operations (substitution and the insert) with a single pass using multiple sed scriptlets.
#! /bin/sh
for file in $(grep -l "^BOOTPROTO=dhcp" /etc/sysconfig/network-scripts/ifcfg*) ; do
sed -i -e "s/BOOTPROTO=dhcp/#BOOTPROTO=dhcp/g" -e "/BOOTPROTO=dhcp/i BOOTPROTO=none" $file
done
As side note consider NOT using path as variable to avoid possible confusion with the 'PATH` environment variable.
Writing it up, your attempt with the following fails:
sed "$($insertLine)iBOOTPROTO=none" $path$file
because:
$($insertLine) encloses $insertLIne in a command substitution which when $insertLIne is evaluated it returns a number which is not a command generating an error.
your call to sed does not include the -i option to edit the file $path$file in place.
You can correct the issues with:
sed -i "${insertLine}i BOOTPROTO=none" $path$file
Which is just sed - i (edit in place) and Ni where N is the number of the line to insert followed by the content to insert and finally what file to insert it in. You add ${..} to insertLine to protect the variable name from the i that follows and then the expression is double-quoted to allow variable expansion.
Let me know if you have any further questions.
(and see dash-o's answer for refactoring the whole thing to simply use sed to make the change without spawning 10 other subshells)
I want the string index.xml to be appended when I see something ending in /feed/ and starting with http://a.b.c
Using the command line I wrote, and works, this
echo "http://a.b.c/blabla/feed/" | sed -e 's#\(http://a.b.c/.*/feed/\)#\1index.xml#g'
I don't know how to transform this code so that it works in a script using -i and a file as parameter.
I tried the following but it works only if the searched string is on a line alone, while I need to transform also strings between other text. What's the correct code?
#!/bin/bash
sed -i 's#\("http://a.b.c/.*/feed/"\)#"\1index.xml"#g' $1
I think
#!/bin/bash
sed -i 's#\(http://a.b.c/.*/feed/\)#\1index.xml#g' "$1"
should work. I only removed the wrong double quotes in the command and added the missing ones around the $1.
If an input like http://aXbXc/ is not supposed to trigger the replacement, then you should also escape the dots.
Part of a shell script that I am creating takes a plain text list of files...
11111.jpg
22222.jpg
33333.jpg
...and appends a user-defined prefix that is stored in a variable to create a list of paths that looks like this:
user/defined/prefix/11111.jpg
user/defined/prefix/22222.jpg
user/defined/prefix/33333.jpg
I am attempting to use sed to add the prefix in this manner:
sed -e 's/^/prefix/' oldFile > newFile.new
The variable is getting assigned correctly:
echo $selectedPrefix
user/defined/prefix
Put no combinations of single quotes, double quotes of whatever seem to get sed to use the ACTUAL value of the variable instead of just the variable name.
sed -e 's/^/$selectedPrefix/' oldFile > newFile.new
Yields:
$selectedPrefix11111.jpg
$selectedPrefix22222.jpg
$selectedPrefix33333.jpg
Help! I'm sure the solution is simple but I feel like I've tried everything....
As mentionned by Cyrus, you need to used " (double quote) instead ' (single quote) if you want the variable replacement because single quoted string are interpreted literally so it doesn't see $selectedPrefix as a variable but as the string value of $selectedPrefic hence what you saw.
Since you are working with paths in you sed, you are correct in assuming that you should use a different separator for your sed comment. I usually prefer using | but ~ would also work.
so basically you could have:
sed -e "s~^~$selectedPrefix~" oldFile > newFile.new
This code would solve your problem:
selectedPrefixEscaped="$(echo "$selectedPrefix" | sed 's/\//\\\//g')" && sed -e "s/^/$selectedPrefixEscaped/" oldFile > newFile.new
Just using a different delimiter on sed would leave you open to problems when (if) the path contains the new delimiter (ex.: /folder/folder#5/file.txt would be problematic if using # as sed delimiter).
While using Sed to search/ insert a config file, I'm greeted by errors. What's causing them, and how can I fix them?
The Heredoc I'm looking to insert can be defined as follows:
read -d '' APPLICATION_ENV_STATE <<'EOF'
defined('APPLICATION_ENV') || define('APPLICATION_ENV',(getenv('APPLICATION_ENV')
? getenv('APPLICATION_ENV') : 'production'));
EOF
While my Sed command uses the variable like this:
sed -i "/\/\/ \*\* MySQL settings \*\* \/\//i$APPLICATION_ENV_STATE" wp-config.php
Which results in:
sed: -e expression #1, char 1: unknown command: `?'
In addition to an extra characters after command error.
However, the following Heredoc works, but results in some less than pretty formatting in my text file:
read -d '' APPLICATION_ENV_STATE <<'EOF'
defined('APPLICATION_ENV') || define('APPLICATION_ENV', (getenv('APPLICATION_ENV') ? getenv('APPLICATION_ENV') : 'production'));
EOF
How do I get the first example to work?
AIUI, it's not the heredoc that's the problem, it's understanding which process is doing what at various times.
In your script that runs the sed command, Bash is substituting the variable before sed even sees it. Being a multi-line string, it would need escaping for sed). From the man page for sed, under the i command:
i \ Insert text, which has each embedded newline preceded by a back-slash.
Personally, I'd recommend using cat or echo if you can (or a scriping language like Python / Ruby / PHP), having broken the template up into atomic elements, so you can simply concatenate the relevant pieces together.
If you do want to continue with the current method though, you'll at least need to replace the newlines with backslashed newlines - try something like:
echo $APPLICATION_ENV_STATE | sed 's/$/\\/'
You're using the wrong tool. The only constructs you should be using in sed are s, g, and p (with -n). Just use awk and avoid all the quoting/escaping nonsense:
$ cat file
foo
// ** MySQL settings ** //
bar
$ awk -v app="defined('APPLICATION_ENV') || define('APPLICATION_ENV',(getenv('APPLICATION_ENV')
? getenv('APPLICATION_ENV') : 'production'));" '
{print} index($0,"// ** MySQL settings ** //"){print app}' file
foo
// ** MySQL settings ** //
defined('APPLICATION_ENV') || define('APPLICATION_ENV',(getenv('APPLICATION_ENV')
? getenv('APPLICATION_ENV') : 'production'));
bar
Notice that you don't need to escape the RE metachars in the string you want to search for because awk can treat a string as a string and you don't need to escape newlines in the string you're adding and you don't need a here doc with a shell variable, etc.
Your read/sed command as written would fail for various character combinations in your search and/or replacement strings - see Is it possible to escape regex metacharacters reliably with sed for how to robustly search/replace "strings" in sed but then just use awk so you don't have to worry about any of it.