SED-POSIX-specified -E option inside script - bash

I have a tricky question.
I have this .sed script where I need to do some stuff and then execute all the lines of the script with "sed -f script.sed filename.csv"
Inside the script I have used a regexp expression and if when executing the script I use -E, the script works. (sed -E -f script.sed filename.csv)
BUT, the thing is, I CANNOT used it, so in the terminal I need to execute the script like this "sed -f script.sed filename.csv"
Then, the question is, Is there a human possible way to make a regexp expression WORK inside a SCRIPT without using -E in the terminal? Can it be included INSIDE the SCRIPT somehow?
I will write down the script to give more context:
s/ESP/Spain/g #this line changes ESP for Spain in all rows
s/DEN/Denmark/g #this line changes DEN for Denmark in all rows
s/NED/Netherlands/g #this line changes NED for Netherlands in all rows
s/^([^,]*,)([^,]+)((,[^,]*){5},(shooting|judo),)/\1\U\2\L\3/ #This line, changes to UPPER CASE all names of athletes that have as sport either judo or shooting. This is the line that does not work without the -E in the terminal.
/[0],[0],[0],$/d #This line deletes all rows where the last 3 columns have these values
1 c id,name,nationality,sex,date_of_birth,height,weight,sport,gold,silver,bronze,info #this line makes the header not upper case
If you need some data to work with, there you have it:
id,name,nationality,sex,date_of_birth,height,weight,sport,gold,silver,bronze,info
388896171,Abdelraouf Benguit,ALG,male,1985-07-03,1.83,90,judo,0,0,0,
285603057,Abderrahmane Mansouri,ALG,male,1995-01-13,1.72,66,cycling,0,0,0,
545134894,Abderrahmane Meziane,ALG,male,1994-03-07,1.68,62,football,0,0,0,
969824503,Abdullah Alrashidi,IOA,male,1963-08-21,1.83,84,shooting,0,0,1,
897549624,Abdullah Hel Baki,BAN,male,1989-08-01,,,shooting,0,0,0,
153457,Abdullahi Shehu,NGR,male,1993-03-12,1.70,,football,0,0,1,
This is part of a file named athletesv2.csv
Please don't ask why I can't use it, but I'm forbiden from using this option.
BR,

You can use the first three commands as is:
s/ESP/Spain/g #this line changes ESP for Spain in all rows
s/DEN/Denmark/g #this line changes DEN for Denmark in all rows
s/NED/Netherlands/g #this line changes NED for Netherlands in all rows
The fourth command should be "converted" to POSIX BRE:
s/^\([^,]*,\)\([^,]\+\)\(\(,[^,]*\)\{5\},\(shooting\|judo\),\)/\1\U\2\L\3/
Note that this syntax is only supported by GNU sed (the \| as an alternation operator and \+ as one or more quantifier are GNU extensions).

Related

How to see syntax errors reported with actual line numbers in the parent script when Perl is embedded within shell script?

For no justifiable reason at all, I have a pretty substantial Perl script embedded within a Bash function that is being invoked within an autoenv .env file.
It looks something like this:
perl='
$inverse = "\e[7m";
$invoff = "\e[27m";
$bold = "\e[1m";
⋮
'
perl -e "$perl" "$inputfile"
I understand that standalone Perl scripts and the PATH variable are a thing, and I understand that Term::ANSIColor is a thing. This is not about that.
My question is, if there's a syntax error in the embedded Perl code, how can I get Perl to report the actual line number within the parent shell script?
For example, say the perl= assignment occurs on line 120 within that file, but there's a syntax error on the 65th line of actual Perl code. I get this:
syntax error at -e line 65, near "s/(#.*)$/$comment\1$endcomment/"
Execution of -e aborted due to compilation errors.
…but I want to see this (the actual line number in the parent script) instead:
syntax error at -e line 185, near "s/(#.*)$/$comment\1$endcomment/"
Things I've tried (that didn't work):
assigning to __LINE__
don't even know why I thought that would work; it's not a variable, it's a constant, and you get an error stating the same
assigning to $. ($INPUT_LINE_NUMBER with use English)
I was pretty sure this wasn't going to work anyway, because this is like NR in Awk, and this clearly isn't what this is for
As described in perlsyn, you can use the following directive to set the line number and (optionally) the file name of the subsequent line:
#line 42 "file.pl"
This means that you could use
#!/bin/sh
perl="#line 4 \"$0\""'
warn("test");
'
perl -e "$perl"
Output:
$ ./a.sh
test at ./a.sh line 4.
There's no clean way to avoid hardcoding the line number when using sh, but it is possible.
#!/bin/sh
script_start=$( perl -ne'if (/^perl=/) { print $.+1; last }' -- "$0" )
perl="#line $script_start \"$0\""'
warn("test");
'
perl -e "$perl"
On the other hand, bash provides the current line number.
#!/bin/bash
script_start=$(( LINENO + 2 ))
perl="#line $script_start \"$0\""'
warn("test");
'
perl -e "$perl"
There is this useful tidbit in the perlrun man page, under the section for -x, which "tells Perl that the program is embedded in a larger chunk of unrelated text, such as in a mail message."
All references to line numbers by the program (warnings, errors, ...) will treat the #! line as the first line. Thus a warning on the 2nd line of the program, which is on the 100th line in the file will be reported as line 2, not as line 100. This can be overridden by using the #line directive. (See Plain Old Comments (Not!) in perlsyn)
Based on the bolded statement, adding #line NNN (where NNN is the actual line number of the parent script where that directive appears) achieves the desired effect:
perl='#line 120
$inverse = "\e[7m";
$invoff = "\e[27m";
$bold = "\e[1m";
⋮
'
⋮

Substitution of substring doesn't work in bash (tried sed, ${a/b/c/})

Before to write, of course I read many other similar cases. Example I used #!/bin/bash instead of #!/bin/sh
I have a very simple script that reads lines from a template file and wants to replace some keywords with real data. Example the string <NAME> will be replaced with a real name. In the example I want to replace it with the word Giuseppe. I tried 2 solutions but they don't work.
#!/bin/bash
#read the template and change variable information
while read LINE
do
sed 'LINE/<NAME>/Giuseppe' #error: sed: -e expression #1, char 2: extra characters after command
${LINE/<NAME>/Giuseppe} #error: WORD(*) command not found
done < template_mail.txt
(*) WORD is the first word found in the line
I am sorry if the question is too basic, but I cannot see the error and the error message is not helping.
EDIT1:
The input file should not be changed, i want to use it for every mail. Every time i read it, i will change with a different name according to the receiver.
EDIT2:
Thanks your answers i am closer to the solution. My example was a simplified case, but i want to change also other data. I want to do multiple substitutions to the same string, but BASH allows me only to make one substitution. In all programming languages i used, i was able to substitute from a string, but BASH makes this very difficult for me. The following lines don't work:
CUSTOM_MAIL=$(sed 's/<NAME>/Giuseppe/' template_mail.txt) # from file it's ok
CUSTOM_MAIL=$(sed 's/<VALUE>/30/' CUSTOM_MAIL) # from variable doesn't work
I want to modify CUSTOM_MAIL a few times in order to include a few real informations.
CUSTOM_MAIL=$(sed 's/<VALUE1>/value1/' template_mail.txt)
${CUSTOM_MAIL/'<VALUE2>'/'value2'}
${CUSTOM_MAIL/'<VALUE3>'/'value3'}
${CUSTOM_MAIL/'<VALUE4>'/'value4'}
What's the way?
No need to do the loop manually. sed command itself runs the expression on each line of provided file:
sed 's/<NAME>/Giuseppe/' template_mail.txt > output_file.txt
You might need g modifier if there are more appearances of the <NAME> string on one line: s/<NAME>/Giuseppe/g

Assign BASH variable from file with specific criteria

A config file that the last line contains data that I want to assign everything to the RIGHT of the = sign into a variable that I can display and call later in the script.
Example: /path/to/magic.conf:
foo
bar
ThisOption=foo.bar.address:location.555
What would be the best method in a bash shell script to read the last line of the file and assign everything to the right of the equal sign? In this case, foo.bar.address:location.555.
The last line always has what I want to target and there will only ever be a single = sign in the file that happens to be the last line.
Google and searching here yielded many close but non-relative results with using sed/awk but I couldn't come up with exactly what I'm looking for.
Use sed:
variable=$(sed -n 's/^ThisOption=//p' /path/to/magic.conf)
echo "The option is: $variable")
This works by finding and removing the ThisOption= marker at the start of the line, and printing the result.
IMPORTANT: This method absolutely requires that the file be trusted 100%. As mentioned in the comments, anytime you "eval" code without any sanitization there are grave risks (a la "rm -rf /" magnitude - don't run that...)
Pure, simple bash. (well...using the tail utility :-) )
The advantage of this method, is that it only requires you to know that it will be the last line of the file, it does not require you to know any information about that line (such as what the variable to the left of the = sign will be - information that you'd need in order to use the sed option)
assignment_line=$(tail -n 1 /path/to/magic.conf)
eval ${assignment_line}
var_name=${assignment_line%%=*}
var_to_give_that_value=${!var_name}
Of course, if the var that you want to have the value is the one that is listed on the left side of the "=" in the file then you can skip the last assignment and just use "${!var_name}" wherever you need it.

Understanding 'sed' command

I am currently trying to install GCC-4.1.2 on my machine: Fedora 20.
In the instruction, the first three commands involve using 'sed' commands, for Makefile modification. However, I am having difficulty in using those commands properly for my case. The website link for GCC-4.1.2.
The commands are:
sed -i 's/install_to_$(INSTALL_DEST) //' libiberty/Makefile.in &&
sed -i 's#\./fixinc\.sh#-c true#' gcc/Makefile.in &&
sed -i 's/#have_mktemp_command#/yes/' gcc/gccbug.in &&
I am trying to understand them by reading the 'sed' man page, but it is not so easy to do so. Any help/tip would be appreciated!
First, the shell part: &&. That just chains the commands together, so each subsequent line will only be run if the prior one is run successfully.
sed -i means "run these commands inline on the file", that is, modify the file directly instead of printing the changed contents to STDOUT. Each sed command here (the string) is a substitute command, which we can tell because the command starts with s.
Substitute looks for a piece of text in the file, and then replaces it. So the order is always s/needle/replacement/. See how the first and last lines have those same forward-slashes? That's the traditional delimiter between the command (substitute), the needle to find in the haystack (install_to_$(INSTALL_DEST), and the text to replace it with ().
So, the first one looks for the string and deletes it (the empty replacement). The last one looks for #have_mktemp_command# and replaces it with yes.
The middle one is a bit weird. See how it starts with s# instead of s/? Well, sed will let you use any delimiter you like to separate the needle from the replacement. Since this needle had a / in it (\./fixinc\.sh), it made sense to use a different delimiter than /. It will replace the text ./fixinc.sh with -c true.
Last note: Why does the second needle have \. instead of .? Well, in a Regular Expression like the needle is (but not used in your example), some characters are magical and do magical fairy dust operations. One of those magic characters is .. To avoid the magic, we put a \ in front of it, escaping away from the magic. (The magic is "match any character", and we want a literal period. That's why.)

How do I set new line with a sed using c\?

I am feeling stupid but have tried multiple ways of having new line in my script. I got help from Jonathan with a sed command. It worked great but the formatting is lost and now I can't find a way to make it work.
The code looks like this:
su -c "sed -i '/aStyle.Landscape {/,/}/c\
MImAbstractKeyAreaStyle.Landscape {\
/*** Label Setttings ***/\
label-margin-top: 10.6mm; /* 0.6 */\
label-margin-left-with-secondary: -1; /* not used, labels are centered horizontally */\
secondary-label-separation: 0;\
...
/*** Key Area Geometry ***/\
size: 854 -1;\
}' file.css"
I wanted to substitute a paragraph with another one. But with this command everything are printed on one line. I want it to keep the formatting. My original question is here: How to substitute a paragraph in file?
First try without the su -c. Between your "" quotes, the escaping is different and make it much more complicated.
When you're done, either put the result in a script file and call that with su -c ./script.sh, or adapt the escaping (but that's never gonna be nice).
This might work for you:
echo hello | sed '/.*/c\This first line is no longer "hello"\nand this is the second line\nand this the third line'
This first line is no longer "hello"
and this is the second line
and this the third line
In GNU sed you can embed the newlines \n but please see my amended answer to you last question (using a here-document) for a WYSIWYG answer.

Resources