So I have tried to install gsed via macports but this hasnt solved the issue. I was going to uninstall it to reduce clutter, however, before I do so, how would i fix the error below. It is because of the BSD version of sed Mac OS X is running from what I understand, but none of the fixes I seem to have found are helping.
sed: 1: "/\[staging: production\ ...": command i expects \ followed by text
#!/bin/bash
test="lala\nkjdsh"
sed -i -e '/\[staging: production\]/ i '$test'' ./test.txt
You have this problem because of "\n" in the $test. Try to remove \n from it.
POSIX standard sed only accepts \n as part of a search pattern. OS X uses the FreeBSD sed, which is strictly POSIX compliant
So if you need a newline in a variable you need to write something like:
$ test="lala\
> kjdsh"
You can also solve the task with perl:
$ test="lala\nkjdsh"
$ perl -n -i -e 'print "'"$test"'\n" if /\[staging: production\]/; print;' ./test.txt
Example:
$ echo '[staging: production]' > /tmp/test.txt
$ test="lala\nkjdsh"
$ perl -n -i -e 'print "'"$test"'\n" if /\[staging: production\]/; print;' ./test.txt
$ cat ./test.txt
lala
kjdsh
[staging: production]
If the test variable does not contain a line containing only . you can use ed to edit the file:
printf '%s\n' '/\[staging: production\]/i' "$test" . w | ed -s ./test.txt
See http://wiki.bash-hackers.org/howto/edit-ed for more about ed.
EDIT: Oh and I missed that you actually had backslash-followed-by-N and not literal newlines in your variable. If you use literal newlines the above should work.
EDIT2: Given the pastebin given in the comments, try:
#!/usr/bin/env bash
#...
ed -s ./test.txt << EOF
/\[staging: production\]/i
; some comment
someStuffHere[] = "XYZ"
someMoreStuff[] = "$someShellVar"
; another comment
.
w
EOF
The . alone on a line ends the insert command, and w is the write command, which actually saves the changes to the file, (like :w in vim)
Related
I want to replace a line in a file with multiple lines. I know I can use \n in the sed replace, but that is rather ugly. I was hoping to HEARDOCs.
So I can do this to replace the line with multiple lines:
$ cat sedtest
DINGO=bingo
$ sed -i -e "s/^DINGO.*$/# added by $(whoami) on $(date)\nDINGO=howdy/" sedtest
$ cat sedtest
# added by user on Sun Feb 3 08:55:44 EST 2019
DINGO=howdy
In the command I want to put the replacement in new lines so it's easier to read/understand. So far I have been using HEREDOCs when I want to add new lines to a file:
CAT << EOF | sudo tee -a file1 file2 file3
line one
line two
line three
EOF
And this has worked well for appending/adding. Is it possible to do something similar but instead use the output as the replacement in sed or is there some other way to do what I'm looking for?
Is this what you're trying to do?
$ awk 'NR==FNR{new=(NR>1?new ORS:"") $0;next} $0=="DINGO=bingo"{$0=new} 1' - file <<!
# added by $(whoami) on $(date)
DINGO=howdy
!
# added by user on Sun, Feb 3, 2019 8:50:41 AM
DINGO=howdy
Note that the above is using literal string operations so it'll work for any characters in the old or new strings unlike your sed script which would fail given /s or any ERE regexp character or capture groups or backreferences or... in the input (see Is it possible to escape regex metacharacters reliably with sed for details).
This might work for you (GNU sed):
cat <<! | sed -i -e '/^DINGO/r /dev/stdin' -e '//d' file
# added by $(whoami) on $(date)
DINGO=howdy
!
This replaces lines starting DINGO with the here-document which is piped to stdin as file within the sed command.
An alternative:
cat <<! | sed -i -e '/^DINGO/e cat /dev/stdin' -e 's/bingo/howdy' file
# added by $(whoami) on $(date)
!
N.B. In the alternative solution the here-doc will only be read once!
I have this line inside a file:
ULNET-PA,client_sgcib,broker_keplersecurities
,KEPLER
I try to get rid of that ^M (carriage return) character so I used:
sed 's/^M//g'
However this does remove everything after ^M:
[root#localhost tmp]# vi test
ULNET-PA,client_sgcib,broker_keplersecurities^M,KEPLER
[root#localhost tmp]# sed 's/^M//g' test
ULNET-PA,client_sgcib,broker_keplersecurities
What I want to obtain is:
[root#localhost tmp]# vi test
ULNET-PA,client_sgcib,broker_keplersecurities,KEPLER
Use tr:
tr -d '^M' < inputfile
(Note that the ^M character can be input using Ctrl+VCtrl+M)
EDIT: As suggested by Glenn Jackman, if you're using bash, you could also say:
tr -d $'\r' < inputfile
still the same line:
sed -i 's/^M//g' file
when you type the command, for ^M you type Ctrl+VCtrl+M
actually if you have already opened the file in vim, you can just in vim do:
:%s/^M//g
same, ^M you type Ctrl-V Ctrl-M
You can simply use dos2unix which is available in most Unix/Linux systems. However I found the following sed command to be better as it removed ^M where dos2unix couldn't:
sed 's/\r//g' < input.txt > output.txt
Hope that helps.
Note: ^M is actually carriage return character which is represented in code as \r
What dos2unix does is most likely equivalent to:
sed 's/\r\n/\n/g' < input.txt > output.txt
It doesn't remove \r when it is not immediately followed by \n and replaces both with just \n. This fails with certain types of files like one I just tested with.
alias dos2unix="sed -i -e 's/'\"\$(printf '\015')\"'//g' "
Usage:
dos2unix file
If Perl is an option:
perl -i -pe 's/\r\n$/\n/g' file
-i makes a .bak version of the input file
\r = carriage return
\n = linefeed
$ = end of line
s/foo/bar/g = globally substitute "foo" with "bar"
In awk:
sub(/\r/,"")
If it is in the end of record, sub(/\r/,"",$NF) should suffice. No need to scan the whole record.
This is the better way to achieve
tr -d '\015' < inputfile_name > outputfile_name
Later rename the file to original file name.
I agree with #twalberg (see accepted answer comments, above), dos2unix on Mac OSX covers this, quoting man dos2unix:
To run in Mac mode use the command-line option "-c mac" or use the
commands "mac2unix" or "unix2mac"
I settled on 'mac2unix', which got rid of my less-cmd-visible '^M' entries, introduced by an Apple 'Messages' transfer of a bash script between 2 Yosemite (OSX 10.10) Macs!
I installed 'dos2unix', trivially, on Mac OSX using the popular Homebrew package installer, I highly recommend it and it's companion command, Cask.
This is clean and simple and it works:
sed -i 's/\r//g' file
where \r of course is the equivalent for ^M.
Simply run the following command:
sed -i -e 's/\r$//' input.file
I verified this as valid in Mac OSX Monterey.
remove any \r :
nawk 'NF+=OFS=_' FS='\r'
gawk 3 ORS= RS='\r'
remove end of line \r :
mawk2 8 RS='\r?\n'
mawk -F'\r$' NF=1
I've been trying to create a sed script that reads a list of phone numbers and only prints ones that match the following schemes:
+1(212)xxx-xxxx
1(212)xxx-xxxx
I'm an absolute beginner, but I tried to write a sed script that would print this for me using the -n -r flags (the contents of which are as follows):
/\+1\(212\)[0-9]{3}-[0-9]{4}/p
/1\(212\)[0-9]{3}-[0-9]{4}/p
If I run this in sed directly, it works fine (i.e. sed -n -r '/\+1\(212\)[0-9]{3}-[0-9]{4}/p' sample.txt prints matching lines as expected. This does NOT work in the sed script I wrote, instead sed says:
sed: -e expression #1, char 2: extra characters after command
I could not find a good solution, this error seems to have so many causes and none of the answers I found apply easily here.
EDIT: I ran it with sed -n -r script.sed sample.txt
sed can not automatically determine whether you intended a parameter to be a script file or a script string.
To run a sed script from a file, you have to use -f:
$ echo 's/hello/goodbye/g' > demo.sed
$ echo "hello world" | sed -f demo.sed
goodbye world
If you neglect the -f, sed will try to run the filename as a command, and the delete command is not happy to have emo.sed after it:
$ echo "hello world" | sed demo.sed
sed: -e expression #1, char 2: extra characters after command
Of the various unix tools out there, two use BRE as their default regex dialect. Those two tools are sed and grep.
In most operating systems, you can use egrep or grep -E to tell that tool to use ERE as its dialect. A smaller (but still significant) number of sed implementations will accept a -E option to use ERE.
In BRE mode, however, you can still create atoms with brackets. And you do it by escaping parentheses. That's why your initial expression is failing -- the parentheses are NOT special by default in BRE, but you're MAKING THEM SPECIAL by preceding the characters with backslashes.
The other thing to keep in mind is that if you want sed to execute a script from a command line argument, you should use the -e option.
So:
$ cat ph.txt
+1(212)xxx-xxxx
1(212)xxx-xxxx
212-xxx-xxxx
$ grep '^+\{0,1\}1([0-9]\{3\})' ph.txt
+1(212)xxx-xxxx
1(212)xxx-xxxx
$ egrep '^[+]?1\([0-9]{3}\)' ph.txt
+1(212)xxx-xxxx
1(212)xxx-xxxx
$ sed -n -e '/^+\{0,1\}1([0-9]\{3\})/p' ph.txt
+1(212)xxx-xxxx
1(212)xxx-xxxx
$ sed -E -n -e '/^[+]?1\([0-9]{3}\)/p' ph.txt
+1(212)xxx-xxxx
1(212)xxx-xxxx
Depending on your OS, you may be able to get a full list of how this works from man re_format.
I want to turn this input_variable = 1
into input_variable = 01
From previous posts here I tried this but didn't work:
sed -e "s/\0" <<< "$input_variable"
I get:
Syntax error: redirection unexpected
What do I do wrong?
Thanks!
EDIT
Thanks to Benjamin I found a workaround (I would still like to know why the sed didn't work):
new_variable="0$input_variable"
While it can be done with sed, simple assignment in your script can do exactly what you want done. For example, if you have input_variable=1 and want input_variable=01, you can simply add a leading 0 by assignment:
input_variable="0${input_variable}"
or for additional types of numeric formatting you can use the printf -v option and take advantage of the format-specifiers provided by the printf function. For example:
printf -v input_variable "%02d" $input_variable
will zero-pad input_variable to a length of 2 (or any width you specify with the field-width modifier). You can also just add the leading zero regardless of the width with:
printf -v input_variable "0%s" $input_variable
sed is an excellent tool, but it isn't really the correct tool for this job.
You don't close the substitution command. Each substitution command must contain 3 delimiters
sed -e 's/pattern/replacement/' <<< 'text' # 3 backslashes
What you want to do could be done with:
sed -e 's/.*/0&/' <<< $input_variable
EDIT:
You are probably using Ubuntu and stumbled upon dash also known as the Almquist shell, which does not have the <<< redirection operator. The following would be a POSIX-compliant alternative, which works with dash as well:
sed -e 's/.*/0&/' <<~
$input_variable
~
And also this:
echo $input_variable | sed -e 's/.*/0&/'
To have the variable take on the new value, do this:
input_variable=$(echo $input_variable | sed -e 's/.*/0&/')
That's however not how you would write the shell script. Shell scripts usually give out some textual output, rather than setting external variables:
So, the script, let's call it append_zero.sh:
#!/bin/sh
echo $1 | sed 's/.*/0&/'
and you would execute it like this:
$ input_variable=1
$ input_variable=$(append_zero.sh input_variable)
$ echo $input_variable
01
This way you have a working shell script that you can reuse with any Unix system that has a POSIX compliant /bin/sh
I have a simple text file ./version containing a version number. Unfortunately, sometimes the version number in the file is followed by whitespaces and newlines like
1.1.3[space][space][newline]
[newline]
[newline]
What is the best, easiest and shortest way to extract the version number into a bash variable without the trailing spaces and newlines? I tried
var=`cat ./version | tr -d ' '`
which works for the whitespaces but when appending a tr -d '\n' it does not work.
Thanks,
Chris
$ echo -e "1.1.1 \n\n" > ./version
$ read var < ./version
$ echo -n "$var" | od -a
0000000 1 . 1 . 1
0000005
Pure Bash, no other process:
echo -e "1.2.3 \n\n" > .version
version=$(<.version)
version=${version// /}
echo "'$version'"
result: '1.2.3'
I still do not know why, but after deleting and recreating the version file this worked:
var=`cat ./version | tr -d ' ' | tr -d '\n'`
I'm confused... what can you do different when creating a text file. However, it works now.
I like the pure bash version from fgm's answer.
I provide this one-line perl command to remove also other characters if any:
perl -pe '($_)=/([0-9]+([.][0-9]+)+)/'
The extracted version number is trimmed/stripped (no newline or carriage return symbols):
$> V=$( bash --version | perl -pe '($_)=/([0-9]+([.][0-9]+)+)/' )
$> echo "The bash version is '$V'"
The bash version is '4.2.45'
I provide more explanation and give other more sophisticated (but still short) one-line perl commands in my other answer.