Building filename from command outputs results in ambigous redirect error - bash

I want to edit a file and write it to anoter location without .template at the end. To do so, I use two commands in a row which should give me the new file name, but bash doesn't accept it. I heard you should do this with backquotes but I can't get it to work (the first sed is correct don't bother):
sed -E '/^\s*(#.*)?$/d; s/^\s*(\S+)\s+("([^"]*)"|(\S+)).*/s\/\1\/\3\4\/g/' "$file" > `$(basename $file) | sed -E 's/(.*).template$/.translated\1/g')`
I get such errors:
translate: command substitution: line 25: syntax error near unexpected token `)'
translate: command substitution: line 25: `$(basename $file) | sed -E 's/(.*).template$/.translated\1/g')'
translate: line 25: `$(basename $file) | sed -E 's/(.*).template$/.translated\1/g')`: ambiguous redirect
What should the redirect-to-filename part look like?

Part 1: Why It Happens
Let's look at why you get a "ambiguous redirect" error.
When you run:
foo > `$(basename $file) | sed -E 's/(.*).template$/.translated\1/g')`
...that's telling the shell to first run a command:
$(basename $file) | sed -E 's/(.*).template$/.translated\1/g'
...and use its output as the redirection target.
However, $(basename $file) | ... is not the same as basename $file | ...! Let's say your file is named /path/to/foo.template. After that first command substitution happens, it's translated to:
foo.template | sed -E 's/(.*).template$/.translated\1/g'
...which is running a command named foo.template and feeding its output to sed.
Do you have a command named foo.template? If you don't, then that just generates an error on stderr, and nothing at all on stdout. Since stdout is what's fed into sed, that means sed receives no input at all, so it writes no output at all, so you have an empty filename.
And trying to redirect to an empty filename... is ambiguous! There's your bug.
Part 2: Doing It Right
I'm assuming both your sed commands do what you want them to do (meaning that you want foo.template to create a file named .footranslated -- if you don't want to create this hidden file, your second sed operation is very wrong).
file_base=${file##*/}
sed -E '/^\s*(#.*)?$/d; s/^\s*(\S+)\s+("([^"]*)"|(\S+)).*/s\/\1\/\3\4\/g/' "$file" \
>".translated${file_base%.template}"
...or, if your actual intent is to replace the extension .template with an extension .translated, that would instead be:
file_base=${file##*/}
sed -E '/^\s*(#.*)?$/d; s/^\s*(\S+)\s+("([^"]*)"|(\S+)).*/s\/\1\/\3\4\/g/' "$file" \
>"${file_base%.template}.translated"

Related

How to insert a variable content using sed command

I have shell script (.sh) and I'm trying to insert contents of a file onto another file using the following commands but I it's throwing an error "sed: -e expression #1, char 28: unknown option to `s'":
filename="/home/user1/filename.txt"
contents=$(du -sh /var/log/test.log)
hostname > $filename
sed -i "/test_string/ s/$/, $contents" $filename
I can't seem to figure out where the underlying issue is. Can someone please help?
Example:
filename=/home/user1/filename.txt
hostname = server1.mydomain.com
So the content of $filename is server1.mydomain.com after running hostname > $filename.
The output of du -sh /var/log/test.log command is let say 1.3M /var/log/test.log
So running sed -i "/mydomain.com/ s/$/, $contents" $filename should update the content of the following filename to:
server1.mydomain.com, 1.3M /var/log/test.log
But as I mentioned above, it's throwing an error.
try this
sed -i "s#\$#, ${contents}#g"
Demo :
$cat file.txt
server1.mydomain.com
$echo $contents
1.3M /var/log/test.log
$sed -i "s#\$#, ${contents}#g" file.txt
$cat file.txt
server1.mydomain.com, 1.3M /var/log/test.log
$
sed command usage is s#pattern to search#pattern/String replacement#occurence
This is easier done with perl or another language, to avoid the issues with characters in your variable causing sed parse errors:
contents=$(du -sh /var/log/test.log) perl -pi -e '$_ .= ", $ENV{contents}" if /test_string/' "$filename"
Or using ed to edit the file, and avoiding the $contents variable completely:
ed -s "$filename" <<'EOF'
/test_string/a
,<space>
.
r !du -sh /var/log/test.log
.-2,.j
w
EOF
Replace <space> with a literal space.
This cryptic-at-first set of commands first moves to the first line matching the regular expression test_string, then appends a line ,<space> after it, then reads the output of the du command and inserts it in a line after that, and finally joins those three lines into one and writes the modified file back to disk.
(This assumes that the du invocation will only return one line.)

need to clean file via SED or GREP

I have these files
NotRequired.txt (having lines which need to be remove)
Need2CleanSED.txt (big file , need to clean)
Need2CleanGRP.txt (big file , need to clean)
content:
more NotRequired.txt
[abc-xyz_pqr-pe2_123]
[lon-abc-tkt_1202]
[wat-7600-1_414]
[indo-pak_isu-5_761]
I am reading above file and want to remove lines from Need2Clean???.txt, trying via SED and GREP but no success.
myFile="NotRequired.txt"
while IFS= read -r HKline
do
sed -i '/$HKline/d' Need2CleanSED.txt
done < "$myFile"
myFile="NotRequired.txt"
while IFS= read -r HKline
do
grep -vE \"$HKline\" Need2CleanGRP.txt > Need2CleanGRP.txt
done < "$myFile"
Looks as if the Variable and characters [] making some problem.
What you're doing is extremely inefficient and error prone. Just do this:
grep -vF -f NotRequired.txt Need2CleanGRP.txt > tmp &&
mv tmp Need2CleanGRP.txt
Thanks to grep -F the above treats each line of NotRequired.txt as a string rather than a regexp so you don't have to worry about escaping RE metachars like [ and you don't need to wrap it in a shell loop - that one command will remove all undesirable lines in one execution of grep.
Never do command file > file btw as the shell might decide to execute the > file first and so empty file before command gets a chance to read it! Always do command file > tmp && mv tmp file instead.
Your assumption is correct. The [...] construct looks for any characters in that set, so you have to preface ("escape") them with \. The easiest way is to do that in your original file:
sed -i -e 's:\[:\\[:' -e 's:\]:\\]:' "${myFile}"
If you don't like that, you can probably put the sed command in where you're directing the file in:
done < replace.txt|sed -e 's:\[:\\[:' -e 's:\]:\\]:'
Finally, you can use sed on each HKline variable:
HKline=$( echo $HKline | sed -e 's:\[:\\[:' -e 's:\]:\\]:' )
try gnu sed:
sed -Ez 's/\n/\|/g;s!\[!\\[!g;s!\]!\\]!g; s!(.*).!/\1/d!' NotRequired.txt| sed -Ef - Need2CleanSED.txt
Two sed process are chained into one by shell pipe
NotRequired.txt is 'slurped' by sed -z all at once and substituted its \n and [ meta-char with | and \[ respectively of which the 2nd process uses it as regex script for the input file, ie. Need2CleanSED.txt. 1st process output;
/\[abc-xyz_pqr-pe2_123\]|\[lon-abc-tkt_1202\]|\[wat-7600-1_414\]|\[indo-pak_isu-5_761\]/d
add -u ie. unbuffered, option to evade from batch process, sort of direct i/o

Error on sed script - extra characters after command

I've been trying to create a sed script that reads a list of phone numbers and only prints ones that match the following schemes:
+1(212)xxx-xxxx
1(212)xxx-xxxx
I'm an absolute beginner, but I tried to write a sed script that would print this for me using the -n -r flags (the contents of which are as follows):
/\+1\(212\)[0-9]{3}-[0-9]{4}/p
/1\(212\)[0-9]{3}-[0-9]{4}/p
If I run this in sed directly, it works fine (i.e. sed -n -r '/\+1\(212\)[0-9]{3}-[0-9]{4}/p' sample.txt prints matching lines as expected. This does NOT work in the sed script I wrote, instead sed says:
sed: -e expression #1, char 2: extra characters after command
I could not find a good solution, this error seems to have so many causes and none of the answers I found apply easily here.
EDIT: I ran it with sed -n -r script.sed sample.txt
sed can not automatically determine whether you intended a parameter to be a script file or a script string.
To run a sed script from a file, you have to use -f:
$ echo 's/hello/goodbye/g' > demo.sed
$ echo "hello world" | sed -f demo.sed
goodbye world
If you neglect the -f, sed will try to run the filename as a command, and the delete command is not happy to have emo.sed after it:
$ echo "hello world" | sed demo.sed
sed: -e expression #1, char 2: extra characters after command
Of the various unix tools out there, two use BRE as their default regex dialect. Those two tools are sed and grep.
In most operating systems, you can use egrep or grep -E to tell that tool to use ERE as its dialect. A smaller (but still significant) number of sed implementations will accept a -E option to use ERE.
In BRE mode, however, you can still create atoms with brackets. And you do it by escaping parentheses. That's why your initial expression is failing -- the parentheses are NOT special by default in BRE, but you're MAKING THEM SPECIAL by preceding the characters with backslashes.
The other thing to keep in mind is that if you want sed to execute a script from a command line argument, you should use the -e option.
So:
$ cat ph.txt
+1(212)xxx-xxxx
1(212)xxx-xxxx
212-xxx-xxxx
$ grep '^+\{0,1\}1([0-9]\{3\})' ph.txt
+1(212)xxx-xxxx
1(212)xxx-xxxx
$ egrep '^[+]?1\([0-9]{3}\)' ph.txt
+1(212)xxx-xxxx
1(212)xxx-xxxx
$ sed -n -e '/^+\{0,1\}1([0-9]\{3\})/p' ph.txt
+1(212)xxx-xxxx
1(212)xxx-xxxx
$ sed -E -n -e '/^[+]?1\([0-9]{3}\)/p' ph.txt
+1(212)xxx-xxxx
1(212)xxx-xxxx
Depending on your OS, you may be able to get a full list of how this works from man re_format.

Unix shell scripting, need assign the text files values to the sed command

i was trying to add the lines from the text file to the sed command
observered_list.txt
Uncaught SlingException
cannot render resource
IncludeTag Error
Recursive invocation
Reference component error
i need it to be coded like the following
sed '/Uncaught SlingException\|cannot render resource\|IncludeTag Error\|Recursive invocation\|Reference component error/ d'
help me to do this.
I would suggest you create a sed script and delete each pattern consecutively:
while read -r pattern; do
printf "/%s/ d;\n" "$pattern"
done < observered_list.txt >> remove_patterns.sed
# now invoke sed on the file you want to modify
sed -f remove_patterns.sed file_to_clean
Alternatively you could construct the sed command like this:
pattern=
while read -r line; do
pattern=$pattern'\|'$line
done < observered_list.txt
# strip of first and last \|
pattern=${pattern#\\\|}
pattern=${pattern%\\\|}
printf "sed '/%s/ d'\n" "$pattern"
# you still need to invoke the command, it's just printed
You can use grep for that:
grep -vFf /file/with/patterns.txt /file/to/process.txt
Explanation:
-v excludes lines of process.txt which match one of the patterns from output
-F treats patterns in patterns.txt as fixed strings instead of regexes (looks like this is desired here)
-f reads patterns from patterns.txt
Check man grep for further information.

SED bad substitution error

Here's my problem, I have written the following line of code to format properly a list of files found recursively in a directory.
find * | sed -e '/\(.*\..*\)/ !d' | sed -e "s/^.*/\${File} \${INST\_FILES} &/" | sed -e "s/\( \)\([a-zA-Z0-9]*\/\)/\/\2/" | sed -e "s/\(\/\)\([a-zA-Z0-9\_\-\(\)\{\}\$]*\.[a-zA-Z0-9]*\)/ \2/"
The second step is to write the output of this command in a script. While the code above has the expected behavior, the problem occurs when I try to store its output to a variable, I get a bad substitution error from the first sed command in the line.
#!/bin/bash
nsisscript=myscript.sh
FILES=*
for f in $(find $FILES); do
v=`echo $f | sed -e '/\(.*\..*\)/ !d' | sed -e "s/^.*/\${File} \${INST\_FILES} &/" | sed -e "s/\( \)\([a-zA-Z0-9]*\/\)/\/\2/" | sed -e "s/\(\/\)\([a-zA-Z0-9\_\-\(\)\{\}\$]*\.[a-zA-Z0-9]*\)/ \2/"`
sed -i.backup -e "s/\;Insert files here/$v\\n&/" $nsisscript
done
Could you please help me understand what the difference is between the two cases and why I get this error ?
Thanks in advance!
Well my guess was that your escaping of underscore in INST_FILES is strange as underscore is not a special character in shell nor in sed. The error disappear when you delete the '\' before '_'
my 2 cents
Parsing inside of backquote-style command substitution is a bit weird -- it requires an extra level of escaping (i.e. backslashes) to control when expansions take place. Ugly solution: add more backslashes. Better solution: use $() instead of backquotes -- it does the same thing, but without the weird parsing and escaping issues.
BTW, your script seems to have some other issues. First, I don't know about the sed on your system, but the versions I'm familiar with don't interpret \n in the substitution as a newline (which I presume you want), but as a literal n character. One solution is to include a literal newline in the substitution (preceded by a backslash).
Also, the loop executes for each found file, but for files that don't have a period in the name, the first sed command removes them, $v is empty, and you add a blank line to myscript.sh. You should either put the filtering sed call in the for statement, or add it as a filter to the find command.
#!/bin/bash
nsisscript=myscript.sh
nl=$'\n'
FILES=*
for f in $(find $FILES -name "*.*"); do
v=$(echo $f | sed -e "s/^.*/\${File} \${INST\_FILES} &/" | sed -e "s/\( \)\([a-zA-Z0-9]*\/\)/\/\2/" | sed -e "s/\(\/\)\([a-zA-Z0-9\_\-\(\)\{\}\$]*\.[a-zA-Z0-9]*\)/ \2/")
sed -i.backup -e "s/\;Insert files here/$v\\$nl&/" $nsisscript
done

Resources