Part of a shell script that I am creating takes a plain text list of files...
11111.jpg
22222.jpg
33333.jpg
...and appends a user-defined prefix that is stored in a variable to create a list of paths that looks like this:
user/defined/prefix/11111.jpg
user/defined/prefix/22222.jpg
user/defined/prefix/33333.jpg
I am attempting to use sed to add the prefix in this manner:
sed -e 's/^/prefix/' oldFile > newFile.new
The variable is getting assigned correctly:
echo $selectedPrefix
user/defined/prefix
Put no combinations of single quotes, double quotes of whatever seem to get sed to use the ACTUAL value of the variable instead of just the variable name.
sed -e 's/^/$selectedPrefix/' oldFile > newFile.new
Yields:
$selectedPrefix11111.jpg
$selectedPrefix22222.jpg
$selectedPrefix33333.jpg
Help! I'm sure the solution is simple but I feel like I've tried everything....
As mentionned by Cyrus, you need to used " (double quote) instead ' (single quote) if you want the variable replacement because single quoted string are interpreted literally so it doesn't see $selectedPrefix as a variable but as the string value of $selectedPrefic hence what you saw.
Since you are working with paths in you sed, you are correct in assuming that you should use a different separator for your sed comment. I usually prefer using | but ~ would also work.
so basically you could have:
sed -e "s~^~$selectedPrefix~" oldFile > newFile.new
This code would solve your problem:
selectedPrefixEscaped="$(echo "$selectedPrefix" | sed 's/\//\\\//g')" && sed -e "s/^/$selectedPrefixEscaped/" oldFile > newFile.new
Just using a different delimiter on sed would leave you open to problems when (if) the path contains the new delimiter (ex.: /folder/folder#5/file.txt would be problematic if using # as sed delimiter).
Related
I'm trying to write a script that replaces text in a text file between two patterns "opt/" and "/".
An example of the data in the file is:
daw9udiwa9diuoawdj098awd89a0909w opt/TEXTTOREPLACE/app-data/version.txt
wdalkwhjf8aufwaoif98fawfojaw98f8 opt/TEXTTOREPLACE/app-data/package.txt
awdhaw9d8yawdf8uaw9f8uwafhiuhawf opt/TEXTTOREPLACE/bin/somefile/somefile
wdalkwhjf8aufwaoif98fawfojaw98f8 opt/TEXTTOREPLACE/bin/someapp/somefile
I've looked at using the 'sed' command, but the pattern matching is confusing me.
I have tried:
sed -e 's!\/[^\/]*\/!\/CHANGE TO ME\/!'
This works - but I would like to add in the "opt" at the beginning to minimise errors
So I tried the following with no luck
sed -e 's!opt\/[^opt\/]*\/!opt\/CHANGE TO ME\/!'
I will be using a $VAR to replace text
so for example
VAR=CHANGED
sed -e 's!opt\/[^opt\/]*\/!opt\/$VAR\/!'
output:
daw9udiwa9diuoawdj098awd89a0909w opt/CHANGED/app-data/version.txt
wdalkwhjf8aufwaoif98fawfojaw98f8 opt/CHANGED/app-data/package.txt
awdhaw9d8yawdf8uaw9f8uwafhiuhawf opt/CHANGED/bin/somefile/somefile
wdalkwhjf8aufwaoif98fawfojaw98f8 opt/CHANGED/bin/someapp/somefile
help appreciated.
Thanks
A few issues with the current sed code:
to expand OS variables the sed script must be wrapped in double quotes
the use of ! as a delimiter may cause issues with some shells and/or shell configurations (eg, ! is a common shorthand for accessing the command line history)
escaping the / is only needed when the / also serves as the sed script delimiter (ie, no need to escape / if using a different delimiter)
One sed idea that addresses these issues:
sed -e "s|opt/[^/]*|opt/$VAR|" input.txt
Where:
opt/[^/]* - match on the string opt/ plus all characters that are not a \
opt/$VAR - replace with the string opt/ plus the contents of the OS VAR variable
This generates:
daw9udiwa9diuoawdj098awd89a0909w opt/CHANGED/app-data/version.txt
wdalkwhjf8aufwaoif98fawfojaw98f8 opt/CHANGED/app-data/package.txt
awdhaw9d8yawdf8uaw9f8uwafhiuhawf opt/CHANGED/bin/somefile/somefile
wdalkwhjf8aufwaoif98fawfojaw98f8 opt/CHANGED/bin/someapp/somefile
If you are open to using awk rather than sed, the following may work for you:
$ awk -v rep="CHANGED" -F/ 'BEGIN{OFS="/"} {$2=rep; print}' file1
daw9udiwa9diuoawdj098awd89a0909w opt/CHANGED/app-data/version.txt
wdalkwhjf8aufwaoif98fawfojaw98f8 opt/CHANGED/app-data/package.txt
awdhaw9d8yawdf8uaw9f8uwafhiuhawf opt/CHANGED/bin/somefile/somefile
wdalkwhjf8aufwaoif98fawfojaw98f8 opt/CHANGED/bin/someapp/somefile
Split each line on the forward slash character and replace the second field with your desired replacement text. Then format the output with the output field separator (OFS) set as a forward slash.
This question already has answers here:
Using different delimiters in sed commands and range addresses
(3 answers)
Closed 1 year ago.
I have a Visual Studio project, which is developed locally. Code files have to be deployed to a remote server. The only problem is the URLs they contain, which are hard-coded.
The project contains URLs such as ?page=one. For the link to be valid on the server, it must be /page/one .
I've decided to replace all URLs in my code files with sed before deployment, but I'm stuck on slashes.
I know this is not a pretty solution, but it's simple and would save me a lot of time. The total number of strings I have to replace is fewer than 10. A total number of files which have to be checked is ~30.
An example describing my situation is below:
The command I'm using:
sed -f replace.txt < a.txt > b.txt
replace.txt which contains all the strings:
s/?page=one&/pageone/g
s/?page=two&/pagetwo/g
s/?page=three&/pagethree/g
a.txt:
?page=one&
?page=two&
?page=three&
Content of b.txt after I run my sed command:
pageone
pagetwo
pagethree
What I want b.txt to contain:
/page/one
/page/two
/page/three
The easiest way would be to use a different delimiter in your search/replace lines, e.g.:
s:?page=one&:pageone:g
You can use any character as a delimiter that's not part of either string. Or, you could escape it with a backslash:
s/\//foo/
Which would replace / with foo. You'd want to use the escaped backslash in cases where you don't know what characters might occur in the replacement strings (if they are shell variables, for example).
The s command can use any character as a delimiter; whatever character comes after the s is used. I was brought up to use a #. Like so:
s#?page=one&#/page/one#g
A very useful but lesser-known fact about sed is that the familiar s/foo/bar/ command can use any punctuation, not only slashes. A common alternative is s#foo#bar#, from which it becomes obvious how to solve your problem.
add \ before special characters:
s/\?page=one&/page\/one\//g
etc.
In a system I am developing, the string to be replaced by sed is input text from a user which is stored in a variable and passed to sed.
As noted earlier on this post, if the string contained within the sed command block contains the actual delimiter used by sed - then sed terminates on syntax error. Consider the following example:
This works:
$ VALUE=12345
$ echo "MyVar=%DEF_VALUE%" | sed -e s/%DEF_VALUE%/${VALUE}/g
MyVar=12345
This breaks:
$ VALUE=12345/6
$ echo "MyVar=%DEF_VALUE%" | sed -e s/%DEF_VALUE%/${VALUE}/g
sed: -e expression #1, char 21: unknown option to `s'
Replacing the default delimiter is not a robust solution in my case as I did not want to limit the user from entering specific characters used by sed as the delimiter (e.g. "/").
However, escaping any occurrences of the delimiter in the input string would solve the problem.
Consider the below solution of systematically escaping the delimiter character in the input string before having it parsed by sed.
Such escaping can be implemented as a replacement using sed itself, this replacement is safe even if the input string contains the delimiter - this is since the input string is not part of the sed command block:
$ VALUE=$(echo ${VALUE} | sed -e "s#/#\\\/#g")
$ echo "MyVar=%DEF_VALUE%" | sed -e s/%DEF_VALUE%/${VALUE}/g
MyVar=12345/6
I have converted this to a function to be used by various scripts:
escapeForwardSlashes() {
# Validate parameters
if [ -z "$1" ]
then
echo -e "Error - no parameter specified!"
return 1
fi
# Perform replacement
echo ${1} | sed -e "s#/#\\\/#g"
return 0
}
this line should work for your 3 examples:
sed -r 's#\?(page)=([^&]*)&#/\1/\2#g' a.txt
I used -r to save some escaping .
the line should be generic for your one, two three case. you don't have to do the sub 3 times
test with your example (a.txt):
kent$ echo "?page=one&
?page=two&
?page=three&"|sed -r 's#\?(page)=([^&]*)&#/\1/\2#g'
/page/one
/page/two
/page/three
replace.txt should be
s/?page=/\/page\//g
s/&//g
please see this article
http://netjunky.net/sed-replace-path-with-slash-separators/
Just using | instead of /
Great answer from Anonymous. \ solved my problem when I tried to escape quotes in HTML strings.
So if you use sed to return some HTML templates (on a server), use double backslash instead of single:
var htmlTemplate = "<div style=\\"color:green;\\"></div>";
A simplier alternative is using AWK as on this answer:
awk '$0="prefix"$0' file > new_file
You may use an alternative regex delimiter as a search pattern by backs lashing it:
sed '\,{some_path},d'
For the s command:
sed 's,{some_path},{other_path},'
I’m having the following issue:
I have the csv file with data inside looks like this:
“1,””name””,””surname””,””age””,””city”””
“2,””Peter””,””Parker””,””30””,””NY”””
“3,””marry””,””Jane””,””30””,””NY”””
Is it possible using bash to delete the first and the last double quote from each row and then first and last double quote from each field in it?
To get something like this:
1,”name”,”surname”,”age”,”NY”
3,”marry”,”Jane”,”30”,”NY”
I would be grateful for some hints. Thanks
To get you started:
echo '"1,""name"",""surname"",""age"",""city"""' | sed "s/\"\"/\"/g" | sed 's/^\"\(.*\)\"$/\1/'
OUPUT
You can take that, adjust it to run over a file, line by line (instead of the first echo and output into another file
Presuming your input looks like this:
"1,""name"",""surname"",""age"",""city"""
"2,""Peter"",""Parker"",""30"",""NY"""
"3,""marry"",""Jane"",""30"",""NY"""
Note the actual "'s not the ”” in your code:
You can then sed multiple things and chain them together e.g.
sed -e "s/\"\"\"/\"/g" -e "s/\"\"/\"/g" input.txt
This first replaces the triple quotes """, reducing them to double quotes "" and then reduces them further.
Final output:
"1,"name","surname","age","city"
"2,"Peter","Parker","30","NY"
"3,"marry","Jane","30","NY"
If you have special characters then simply replace them in the code e.g.:
$ cat input.txt
“1,””name””,””surname””,””age””,””city”””
“2,””Peter””,””Parker””,””30””,””NY”””
“3,””marry””,””Jane””,””30””,””NY”””
$ sed -e "s/\”\”\”/\”/g" -e "s/\”\”/\”/g" input.txt
“1,”name”,”surname”,”age”,”city”
“2,”Peter”,”Parker”,”30”,”NY”
“3,”marry”,”Jane”,”30”,”NY”
Though I think this input is a transpose error in your question.
Using sed:
sed 's/^"\(.*\)"$/\1/;s/"\+/"/g' file
The first substitution removes the outer double quote on the whole line.
The second substitution replaces the parameter quote to only one double quote.
I'm completely new to bash scripting so excuse me....
I am trying to combine some html content with a template that contains standard headings, the template has a place-holder "REPLACEME" which I thought I could just find and replace on. The loop simply repeats the operation on all the files in the directory.
REPLACEME="REPLACEME"
for file in *.html
do
TEMPLATE=$(<../template/template.html)
CONTENT=$(<$file)
OUTPUT="${TEMPLATE/"$REPLACEME"/"$CONTENT"}"
echo $OUTPUT > ../compiled/$file
done
This works but the resulting html file has been stripped of new line characters, which makes it look like junk! Can anyone help?
Replace:
echo $OUTPUT > ../compiled/$file
With:
echo "$OUTPUT" > ../compiled/$file
The shell performs word splitting on unquoted variables. With the default value for IFS, this means that all sequences of whitespace, which includes tabs and newlines, are replaced with a single blank. To prevent that, put the variable in double-quotes as shown above.
Using sed you could achieve it like below :
sed -i 's/REPLACEME/new_text/g' /path/to/your/template.html
The -i option in sed is for inplace edit & the g option is for global substitution.
Edit 1:
If you need to use a variable inside sed you can do it this way
var="Sometext";
sed -i "s/REPLACEME/$var/g" /path/to/your/template.html
Mind the double quotes here, it makes the shell expand variables.
If your system supports gnu-awk (gawk) you may achieve the above with
gawk '
{
$0=gensub(/REPLACEME/"NEWTEXT","g",$0)
printf "%s\n", $0
}' < /path/to/your/template.html > newtemplate.html && mv newtemplate.html template.html
I have a line in a autoexec.py file that I want another script to be able to modify:
kodi.executebuiltin("PlayMedia(/path/to/file)")
I was thinking of using sed to override the value within PlayMedia() depending on certain conditions.
Can sed be used to only touch and overwrite the parts between the PlayMedia() brackets and nothing else? Or is further processing/regex needed?
You could use sed. The below code will replace the chars present inside the brackets following PlayMedia string with foo.
sed 's/\b\(PlayMedia\)([^)]*)/\1(foo)/g' file
If the string you want to replace is a path, you must use a different sed delimiter because filepath may contain forward slashes.
sed 's~\b\(PlayMedia\)([^)]*)~\1(foo)~g' file
Example:
$ echo 'kodi.executebuiltin("PlayMedia(/path/to/file)")' | sed 's/\b\(PlayMedia\)([^)]*)/\1(foo)/g'
kodi.executebuiltin("PlayMedia(foo)")