Inserting contents of a file with special characters using sed - bash

I want to replace a marker (REPLACETHIS) in file1.txt with the entire contents of file2.txt, which will include newlines and special characters.
An example of file2.txt's contents would be
<Location />
Order deny,allow
Deny from all
Allow from 1.2.3.4
Allow from 5.6.7.8
</Location>
My general code, minus handling of special characters, would look something like this:
value=$(</home/name/scripts/file2.txt)
sed -i -e "s|REPLACETHIS|$value|" /home/name/scripts/file1.txt
What's the best way to go about handling this?

In sed, the best option for inserting text from file would be to use the r file command:
sed -i -e '/REPLACETHIS/{r /path/to/file2.txt' -e ';d;}' file1.txt
or, in the expanded form:
sed -i '/REPLACETHIS/ {
r /path/to/file2.txt
d
}' file1.txt
The r file command will read the text from file and insert it into the output stream. To also delete the REPLACETHIS text, we need the delete d command (heads-up: this will delete the complete line containing the text REPLACETHIS; if you need to use a mid-text marker, you could replace the d with s///, as noted by #ghoti).
In the first example, we had to break the sed program in two expressions, the reason being that r command has to end with a newline. The alternative is to write the program in several lines, as in the expanded example.
Also note that BSD sed handles -i option differently from GNU sed. The above will work in GNU, but if you need it for BSD, you should write: -i '' instead of -i.

One doesn't strictly need to use sed for this at all.
file_to_change=/home/name/scripts/file1.txt
value=$(</home/name/scripts/file2.txt)
infile=$(<"$file_to_change")
tempfile=$(mktemp "$file_to_change.XXXXXX")
if printf '%s\n' "${infile//REPLACETHIS/$value}" >"$tempfile"; then
mv -- "$tempfile" "$file_to_change"
else
rm -f -- "$tempfile"
fi
This works even without sed -i (which is a nonstandard, nonportable, and incompatible between common implementations).

Related

Find two string in same line and then replace using sed

I am doing a find and replace using sed in a bash script. I want to search each file for words with files and no. If both the words are present in the same line then replace red with green else do nothing
sed -i -e '/files|no s/red/green' $file
But I am unable to do so. I am not receiving any error and the file doesn't get updated.
What am I doing wrong here or what is the correct way of achieving my result
/files|no/ means to match lines with either files or no, it doesn't require both words on the same line.
To match the words in either order, use /files.*no|no.*files/.
sed -i -r -e '/files.*no|no.*files/s/red/green/' "$file"
Notice that you need another / at the end of the pattern, before s, and the s operation requires / at the end of the replacement.
And you need the -r option to make sed use extended regexp; otherwise you have to use \| instead of just |.
This might work for you (GNU sed):
sed '/files/{/no/s/red/green/}' file
or:
sed '/files/!b;/no/s/red/green/' file
This method allows for easy extension e.g. foo, bar and baz:
sed '/foo/!b;/bar/!b;/baz/!b;s/red/green/' file
or fee, fie, foe and fix:
sed '/fee/!b;/fi/!b;/foe/!b;/fix/!b;s/bacon/cereal/' file
An awk verison
awk '/files/ && /no/ {sub(/red/,"green")} 1' file
/files/ && /no/ files and no have to be on the same line, in any order
sub(/red/,"green") replace red with green. Use gsub(/red/,"green") if there are multiple red
1 always true, do the default action, print the line.

Delete strings with non-Ukrainian characters bash

Using file structure
foo_11: "Марія"
foo_112: "Superman"
FOOTLONG: "Subway"
foo_13: "Юлія"
I want to remove all strings that don't have at least one character from Ukrainian alphabet.
Script:
for i in *.txt;
do
sed '/[^А-ЯЄЇІа-яєїі]+/d' $i >$i.out
mv $i.out $i
done
doesn't do anything. What is wrong?
Using mac bash.
Assuming that your character class defining Ukrainian letters is correct, the following should work:
sed '/[А-ЯЄЇІа-яєїі]/!d' file
[А-ЯЄЇІа-яєїі] matches a Ukrainian letter anywhere on the line.
Note that even the letters that look like ASCII letters A I a i are actually Ukrainian (Cyrillic) letters with Unicode codepoints U+410 U+406 U+430 U+456.
! negates the match, meaning that only lines not containing at least 1 Ukrainian letter match.
d deletes those lines.
To put it all together:
for f in *.txt; do
sed -i '' '/[А-ЯЄЇІа-яєїі]/!d' "$f" # -i '' is BSD Sed syntax; GNU sed takes just -i
done
As for what you've tried:
As #StefanHegny points out in a comment on the question, + isn't supported when sed is not run with -E in order to enable extended regular expressions; without -E, the cumbersome \{1,\} must be used. (\+ is only supported by GNU sed, not by the BSD version of sed that macOS comes with).
However, even the fixed version of your command, sed '/[^А-ЯЄЇІа-яєїі]\{1,\}/d', doesn't do what you want: it deletes all lines that contain at least one non-Ukrainian-letter character, which eliminates all of your input lines, given that they all have ASCII-based field names and contain :.
You should double-quote variable references such as $i to protect them from shell expansions: "$i"
BSD Sed does support in-place updating with -i, but - unlike GNU Sed - it requires that an empty option-argument (indicating that no backup of the input file should be made) be specified as a separate argument: -i ''.
Your write-to-a-temp-file-first-then-replace-the-original approach works too, but it's generally better to use the following idiom: sed ... file > file.tmp && mv file.tmp file. Separating the mv command with && ensures that the original file is only replaced if the sed command succeeded.
That said, that doesn't help with logic errors as in the case at hand: despite outputting nothing, sed reports success in this case.
This code would achieve what you want (if I understood your question correctly):
grep -i "Я\|Є\|Ї\|І" /folder/file >> /tmp/result
The result is stored on /tmp/result
Note: I don't know Ukranian, so I'm sure I did not included all Ukranian characters, please add/delete Ukranian characters you want to match to the construction above.
Note2: this code is case insensitive thanks to grep -i so you only need to add the character once (lowercase or capital).
To put it on your loop it could be:
for i in *.txt;
do
grep -i "Я\|Є\|Ї\|І" "$i" > "$i".out
mv "$i".out "$i"
done
Edit: I edited this answer to make it simpler, and to add a loop to it.

Remove characters in all text files in a directory using sed

I have a lot of text files that are email templates. Many of them, for some reason, have the following line:
Best Regards,œ
That strange character at the end is what I am interested in removing from all of these files with a single command.
I tried:
for f in *
do
sed 's/"Best Regards,œ"/"Best Regards,"/g' $f | tee $f.t && mv $f.t $f
done
This ran through the process but did not actually remove the 'œ' character.
Please let me know what I am doing incorrectly so I can remove this character and maybe other non-alphanumeric characters using regex [:alnum:], perhaps.
I fixed the issue with removing the unwanted character with:
for f in *
do
sed 's/Best\ Regards\,\œ/Best\ Regards\,/g' $f | tee $f.t && mv $f.t $f
done
However, this still does not remove all of the non-alphanumeric characters from each line of each file. The other things I have tried either do not execute or remove the entire line.
I appreciate your help.
If ① you don't want to have to worry about Unicode, UTF-anything, LANG, etc, and ② you are confident that lines that start with the words "Best Regards," and ONLY those lines are the ones you want to affect, you can simply do this:
sed -i .bak '/^Best Regards,.*/s//Best Regards,/' *
Note that this processes all files in the current directory. If you want to do this in subdirectories, you could use find, with all its goodness. For example:
find /path/to/start/ -exec \
sed -i .bak '/^Best Regards,.*/s//Best Regards,/' {} \;
or if your shell is bash, you could use globstar:
shopt -s globstar
for f in **/*; do
sed -i .bak '/^Best Regards,.*/s//Best Regards,/' "$f"
done
Rather than using tee and mv, these solutions use sed's built-in "in-place" option, and creates a .bak file as a result. Consult the documentation for your implementation of sed to learn more about how to use the -i option -- it works a little differently with different seds.
This approach eliminates the need to search for that character in particular, so you won't need to worry about how it's being represented. Beware though, it will also eliminate any other text that follows the search string on the same line.
You don't need the loop. You can pass the results of the glob expression directly to sed and use the -i option for in place editing of files:
sed -i.bak 's/Best Regards,œ/Best Regards,/' *
-i.bak changes the input file in place and creates a backup file with the extension .bak.
Some implementations of sed, for example GNU sed even support -i without an argument other allow an empty string as argument for -i. In that case sed will not keep any backup files and simply change the original file.
With GNU sed:
sed -i 's/Best Regards,œ/Best Regards,/' *
# OR (BSD, MacOS)
sed -i '' 's/Best Regards,œ/Best Regards,/' *

Bash variable in sed command

I need to add an header recursively to several file according to the name of the file.
So I have tried:
for i in *file
do
sed -i '1 i \A;B;C;D;E;F;G;H;${i%??}' a_${i} > header_a_${i}
done
the problem is that the variable reflecting the name of the file does not expand and in the header I have ${i%??} instead of part of the name file (%?? is to remove some ending characters).
Any help would be great.
Use double quotes:
sed '1 i\
A;B;C;D;E;F;G;H;'"${i%??}" a_${i} > header_a_${i}
It doesn't make any sense to use -i and to redirect the output, so I've omitted -i. Also, I've added an escaped newline after the insert command. Some sed do not require the newline, but many do. However, it seems odd to use sed for this. Instead, just do:
for i in *file; do
{ echo "A;B;C;D;E;F;G;H;${i%??}"; cat a_${i}; } > header_a_${i}
done

How to append to specific lines in a flat file using shell script

I have a flat file that contains something like this:
11|30646|654387|020751520
11|23861|876521|018277154
11|30645|765418|016658304
Using shell script, I would like to append a string to certain lines in this file, if those lines contain a specific string.
For example, in the above file, for lines containing 23861, I would like to append a string "Processed" at the end, so that the file becomes:
11|30646|654387|020751520
11|23861|876521|018277154|Processed
11|30645|765418|016658304
I could use sed to append the string to all lines in the file, but how do I do it for specific lines ?
I'd do it this way
sed '/\|23861\|/{s/$/|Something/;}' file
This is similar to Marcelo's answer but doesn't require extended expressions and is, I think, a little cleaner.
First, match lines having 23861 between pipes
/\|23861\|/
Then, on those lines, replace the end-of-line with the string |Something
{s/$/|Something/;}
If you want to do more than one of these you could simply list them
sed '/\|23861\|/{s/$/|Something/;};/\|30645\|/{s/$/|SomethingElse/;}' file
Use the following awk-script:
$ awk '/23861/ { $0=$0 "|Processed" } {print}' input
11|30646|654387|020751520
11|23861|876521|018277154|Processed
11|30645|765418|016658304
or, using sed:
$ sed 's/\(.*23861.*$\)/\1|Processed/' input
11|30646|654387|020751520
11|23861|876521|018277154|Processed
11|30645|765418|016658304
Use the substitution command:
sed -i~ -E 's/(\|23861\|.*)/\1|Processed/' flat.file
(Note: the -i~ performs the substitution in-place. Just leave it out if you don't want to modify the original file.)
You can use the shell
while read -r line
do
case "$line" in
*23681*) line="$line|Processed";;
esac
echo "$line"
done < file > tempo && mv tempo file
sed is just a stream version of ed, which has a similar command set but was designed to edit files in place (allegedly interactively, but you wouldn't want to use it that way unless all you had was one of these). Something like
field_2_value=23861
appended_text='|processed'
line_match_regex="^[^|]*|$field_2_value|"
ed "$file" <<EOF
g/$line_match_regex/s/$/$appended_text/
wq
EOF
should get you there.
Note that the $ in .../s/$/... is not expanded by the shell, as are $line_match_regex and $appended_text, because there's no such thing as $/ - instead it's passed through as-is to ed, which interprets it as text to substitute ($ being regex-speak for "end of line").
The syntax to do the same job in sed, should you ever want to do this to a stream rather than a file in place, is very similar except that you don't need the leading g before the regex address:
sed -e "/$line_match_regex/s/$/$appended_text/" "$input_file" >"$output_file"
You need to be sure that the values you put in field_2_value and appended_text never contain slashes, because ed's g and s commands use those for delimiters.
If they might do, and you're using bash or some other shell that allows ${name//search/replace} parameter expansion syntax, you could fix them up on the fly by substituting \/ for every / during expansion of those variables. Because bash also uses / as a substitution delimiter and also uses \ as a character escape, this ends up looking horrible:
appended_text='|n/a'
ed "$file" <<EOF
g/${line_match_regex//\//\\/}/s/$/${appended_text//\//\\/}/
wq
EOF
but it does work. Nnote that both ed and sed require a trailing / after the replacement text in s/search/replace/ while bash's ${name//search/replace} syntax doesn't.

Resources