Please help me creating a shell script to search all lines and replace format in a file using sed.
Example - [‘abc,xyz’] to be changed to [‘abc’,’xyz’]
Here you go:
[user#myserver ~]$ echo "['abc,xyz']" | sed "s:,:\',\':g"
['abc','xyz']
Here colon (:) is used as delimiter and backslash (\) has been used as escape character in sed command.
This might work for you (GNU sed):
sed -E ':a;s/(\[('\''[^'\'',]*'\'',)*'\''[^'\'',]*),([^]]*'\''\])/\1'\'','\''\3/;ta' file
Related
How can I use a cat and sed to read data from a file and insert it into another file under known line?
For example I have a file named script1.txt that contains a few hundred lines, one of the line has the value "COMMANDS="commands"
If I wanted use sed to insert a line under it, simply I can use sed as the command bellow.
sed -i '/^COMMANDS=.*/a NEW LINE HERE' script1.txt
But if I want to insert a multi lines and these lines inside a file, and these line changes every a few hours.. how can i do that ?
I tried:
DATA=$(cat data.txt)
sed -i '/^COMMANDS=.*/a '$DATA'' script1.txt
I got the error bellow.
sed: -e expression #1, char 1: unknown command: `"'
Is there a way other than sed to insert the data from file under known line with no issues?
This might work for you (GNU sed):
sed -i '/^COMMANDS=/r dataFile' file
This will append the contents of the file dataFile after the line beginning COMMANDS= and update file
If the data you want to append is multi-line, you might want to replace newlines with \n.
#!/bin/sh
DATA="$(awk '{gsub(/[]\/$*.^&[]/, "\\\\&");printf (FNR>1)?"\\n%s":"%s",$0}END{print ""}' data.txt)"
sed -i -e '/^COMMANDS=.*/a\' -e "$DATA" script1.txt
Here the awk command escapes sed special characters (for basic regular expressions), then prints "%s" for the first line, and "\\n%s" for the others. A newline is printed at the end, but it's somewhat pointless as $() strips it anyway.
The sed command is almost the same but multiple expressions are used which is equivalent to a multi-line sed script (The a text sed alternative syntax can act weirdly with leading spaces/backslashes).
Using bash, how would one replace all unquoted characters from a file?
I have a system that I can't modify that spits out CSV files such as:
code;prop1;prop2;prop3;prop4;prop5;prop6
0,1000,89,"a1,a2,a3",33,,
1,,,"a55,a10",1,1 L,87
2,25,1001,a4,,"1,5 L",
I need this to become, for a new system being added
code;prop1;prop2;prop3;prop4;prop5;prop6
0;1000;89;a1,a2,a3;33;;
1;;;a55,a10;1;1 L;87
2;25;1001;a4;1,5 L;
If the quotes can be removed after this substitution happens in one command it would be nice :) But I prefer clarity to complicated one-liners for future maintenance.
Thank you
With sed:
sed -e 's/,/;/g' -e ':loop; s/\("\)\([^;]*\);\([^"]*"\)/\1\2,\3/; t loop'
Test:
$ sed -e 's/,/;/g' -e ':loop; s/\("\)\([^;]*\);\([^"]*"\)/\1\2,\3/; t loop' yourfile
code;prop1;prop2;prop3;prop4;prop5;prop6
0;1000;89;"a1,a2,a3";33;;
1;;;"a55,a10";1;1 L;87
2;25;1001;a4;;"1,5 L";
You want to use a csv parser. Parsing csv with shell tools is hard (you will encounter regular expressions soon, and they rarely get all cases).
There is one in almost every language. I recommend python.
You can also do this using excel/openoffice variants by opening the file and then saving with ; as the separator.
You can used sed:
echo '0,1000,89,"a1,a2,a3",33,,' | sed -e "s|\"||g"
This will replace " with the empty string (deletes it), and you can pipe another sed to replace the , with ;:
sed -e "s|,|;|g"
$ echo '0,1000,89,"a1,a2,a3",33,,' | sed -e "s|\"||g" | sed -e "s|,|;|g"
>> 0;1000;89;a1;a2;a3;33;;
Note that you can use any separator you want instead of | inside the sed command. For example, you can rewrite the first sed as:
sed -e "s-\"--g"
I came across the following in our legacy bash code
sed -itmp <something> file.txt
Since I did not understand what it does clearly, I tried the following
Here is the content of file.txt before running sed
dummy={my.java.home}
dummy={my_java_home}
I run now
sed -itmp "s#{my.java.home}#${JAVA_HOME}#g" file.txt
After I run this, I get the following
dummy=/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home
dummy=/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home
I can see how sed replaces. But I fail to understand sed replaces both my_java_home and my.java.home in my original file although I asked it to change only my.java.home when issuing sed command above.
Thanks
Because dot in your regex matches any charcter not only the literal dot. So i suggest you to escape the dots present in your regex , so that it matches my.java.home string only.
sed -itmp "s#{my\.java\.home}#${JAVA_HOME}#g" file.txt
And you won't actually need a tmp parameters.
sed -i "s#{my\.java\.home}#${JAVA_HOME}#g" file.txt
I am trying to write a shell script that will replace whatever characters/strings I choose using sed. My first attempt worked with the exception of special characters. I have been trying to use sed to fix the special characters so that they too will be searched for or replaced. I decided to simplify the script for testing purposed, and just deal with a single offending character. However, I am still having problems.
Edited Script
#! /bin/sh
oldString=$1
newString=$2
file=$3
oldStringFixed=$(echo "$oldString" | sed 's/\\/\\\\/g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\[/\\\[/g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\]/\\\]/g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\^/\\\^/g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\*/\\\*/g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\+/\\\+/g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\./\\\./g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\$/\\\$/g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\-/\\\-/g')
sed -e "s/$oldStringFixed/$newString/g" "$file" > newfile.updated
mv newfile.updated "$file"#! /bin/sh
In case it is not clear, I am trying to search through oldString for the [ character, and replace it with an escaped version and assign the results to oldStringFixed (do I need the backticks for this?). The bottom two lines are slightly modified versions of my original script that I believe works correctly.
When I echo the fixed string, nothing is displayed, and sed outputs an error
sed: can't read [: No such file or directory
Can anyone explain what Is wrong with my first sed line?
EDIT:
Thanks to Jite, the script is working better. However, I am still having a problem with replacing single quoted characters with spaces, i.e. ' *'. The new version is above.
I suggest two improvements:
Do not stack calls to sed as you do, instead pack all of them in a single function, as escape_string below.
You can use a fancy delimiter for the sed substitute command to avoid issues linked to / being part of the strings involved.
With these changes, your script looks like:
#! /bin/sh
oldString="$1"
newString="$2"
file="$3"
escape_string()
{
printf '%s' "$1" | sed -e 's/[][\\^*+.$-]/\\\1/g'
}
fancyDelim=$(printf '\001')
oldStringFixed=$(escape_string "$oldString")
sed -e "s$fancyDelim$oldStringFixed$fancyDelim$newString${fancyDelim}g" "$file" \
> newfile.updated
mv newfile.updated "$file"
To replace values containing special characters try using sed with "|" instead of "/"
Eg: sed -i 's|'$original_value'|'$new_value'|g'
where original_value="comprising_special_char_/"
new_value="comprising_new_special_char:"
Change:
oldStringFixed= `sed 's/\[/\[/g' "$oldString"\`
to:
oldStringFixed=$(echo "$oldString" | sed 's/\[/\\\[/g')
Problem 1: Space after =, it's not allowed when assigning shell variables.
Problem 2: sed expects a file as input, not a string. You may pipe it as my solution does though.
Problem 3: You need to escape the backslash first \\, then you need to escape your char \[, totalling \\\[ :)
Side note: I changed `` to $() since the latter is the recommended praxis (due to nesting, another topic).
For me it was just a nightmare trying to get sed to do this for the general case. I gave up and wrote a short Python code to replace sed:
#!/usr/bin/python
# replace.py
import sys
# Replace string in a file (in place)
match=sys.argv[1]
replace=sys.argv[2]
filename=sys.argv[3]
print "Replacing strings in",filename
with open(filename,"r") as f:
data = f.read().replace(match,replace)
with open(filename,"w") as f:
f.write(data)
Which can then be used like:
#!/bin/bash
orig='<somethinghorrible>'
out='<replacement>'
python replace.py "$orig" "$out" myfile.txt
you can use this for replacing " with \" sed 's/\"/\\\"/g' filename
I want to replace VERSION variable with $version_1 and PROG_VERSION with $version_2 within a same file. I used sed for that
sed s/VERSION/$version_1/g | sed s/PROG_VERSION/$version_2/g
The problem is sed also replaces a portion of PROG_VERSION with PROG_(value of $version_1).
How should I prevent sed from making unwanted changes.
The problem is sed also replaces a portion of PROG_VERSION with
PROG_(value of $version_1). How should I prevent sed from making
unwanted changes.
Use word boundaries in sed to prevent unwanted replacements:
sed "s/\bVERSION\b/$version_1/g"
OR:
sed "s/\<VERSION\>/$version_1/g"
For Mac OSX: For some strange reason none of above syntax works and you need to use this weird syntax:
sed "s/[[:<:]]VERSION[[:>:]]/$version_1/g"
I think the easiest would be to switch the order of the commands.
sed "s/PROG_VERSION/$version_2/g; s/VERSION/$version_1/g" file