Let's say I have a file test.txt with contents:
+-foo.bar:2.4
| bar.foo:1.1:test
\| hello.goobye:3.3.3
\|+- baz.yeah:4
I want to use the tr command to delete all instances of the following set of characters:
{' ', '+', '-', '|', '\'}
Done some pretty extensive research on this but found no clear/concise answers.
This is the command that works:
input:
cat test.txt | tr -d "[:blank:]|\\\+-"
output:
foo.bar:2.4
bar.foo:1.1:test
hello.goobye:3.3.3
baz.yeah:4
I experimented with many combinations of that set and I found out that the '-' was being treated as a range indicator (like... [a-z]) and therefore must be put at the end. But I have two main questions:
1) Why must the backslash be double escaped in order to be included in the set?
2) Why does putting the '|' at the end of the set string cause the tr program to delete everything in the file except for trailing new line characters?
Like this:
tr -d '\-|\\+[:blank:] ' < file
You have to escape the - because it is used for denoting ranges of characters like:
tr -d '1-5'
and must therefore being escaped if you mean a literal hyphen. You can also put it at the end. (learned that, thanks! :) )
Furthermore the \ must be escaped when you mean a literal \ because it has a special meaning needed for escape sequences.
The remaining characters must not being escaped.
Why must the \ being doubly escaped in your example?
It's because you are using a "" (double quoted) string to quote the char set. A double quoted string will be interpreted by the shell, a \\ in a double quoted string means a literal \. Try:
echo "\+"
echo "\\+"
echo "\\\+"
To avoid to doubly escape the \ you can just use single quotes as in my example above.
Why does putting the '|' at the end of the set string cause the tr program to delete everything in the file except for trailing new line characters?
Following CharlesDuffy's comment having the | at the end means also that you had the unescaped - not at the end, which means it was describing a range of characters where the actual range depends on the position you had it in the set.
another approach is to define the allowed chars
$ tr -cd '[:alnum:]:.\n' <file
foo.bar:2.4
bar.foo:1.1:test
hello.goobye:3.3.3
baz.yeah:4
or, perhaps delete all the prefix non-word chars
$ sed -E 's/\W+//' file
Related
How can I escape double quotes inside a double string in Bash?
For example, in my shell script
#!/bin/bash
dbload="load data local infile \"'gfpoint.csv'\" into table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY \"'\n'\" IGNORE 1 LINES"
I can't get the ENCLOSED BY '\"' with double quote to escape correctly. I can't use single quotes for my variable, because I want to use variable $dbtable.
Use a backslash:
echo "\"" # Prints one " character.
A simple example of escaping quotes in the shell:
$ echo 'abc'\''abc'
abc'abc
$ echo "abc"\""abc"
abc"abc
It's done by finishing an already-opened one ('), placing the escaped one (\'), and then opening another one (').
Alternatively:
$ echo 'abc'"'"'abc'
abc'abc
$ echo "abc"'"'"abc"
abc"abc
It's done by finishing already opened one ('), placing a quote in another quote ("'"), and then opening another one (').
More examples: Escaping single-quotes within single-quoted strings
Keep in mind that you can avoid escaping by using ASCII codes of the characters you need to echo.
Example:
echo -e "This is \x22\x27\x22\x27\x22text\x22\x27\x22\x27\x22"
This is "'"'"text"'"'"
\x22 is the ASCII code (in hex) for double quotes and \x27 for single quotes. Similarly you can echo any character.
I suppose if we try to echo the above string with backslashes, we will need a messy two rows backslashed echo... :)
For variable assignment this is the equivalent:
a=$'This is \x22text\x22'
echo "$a"
# Output:
This is "text"
If the variable is already set by another program, you can still apply double/single quotes with sed or similar tools.
Example:
b="Just another text here"
echo "$b"
Just another text here
sed 's/text/"'\0'"/' <<<"$b" #\0 is a special sed operator
Just another "0" here #this is not what i wanted to be
sed 's/text/\x22\x27\0\x27\x22/' <<<"$b"
Just another "'text'" here #now we are talking. You would normally need a dozen of backslashes to achieve the same result in the normal way.
Bash allows you to place strings adjacently, and they'll just end up being glued together.
So this:
echo "Hello"', world!'
produces
Hello, world!
The trick is to alternate between single and double-quoted strings as required. Unfortunately, it quickly gets very messy. For example:
echo "I like to use" '"double quotes"' "sometimes"
produces
I like to use "double quotes" sometimes
In your example, I would do it something like this:
dbtable=example
dbload='load data local infile "'"'gfpoint.csv'"'" into '"table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '"'"'"' LINES "'TERMINATED BY "'"'\n'"'" IGNORE 1 LINES'
echo $dbload
which produces the following output:
load data local infile "'gfpoint.csv'" into table example FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY "'\n'" IGNORE 1 LINES
It's difficult to see what's going on here, but I can annotate it using Unicode quotes. The following won't work in Bash – it's just for illustration:
dbload=‘load data local infile "’“'gfpoint.csv'”‘" into ’“table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '”‘"’“' LINES ”‘TERMINATED BY "’“'\n'”‘" IGNORE 1 LINES’
The quotes like “ ‘ ’ ” in the above will be interpreted by bash. The quotes like " ' will end up in the resulting variable.
If I give the same treatment to the earlier example, it looks like this:
echo “I like to use” ‘"double quotes"’ “sometimes”
Store the double quote character in a variable:
dqt='"'
echo "Double quotes ${dqt}X${dqt} inside a double quoted string"
Output:
Double quotes "X" inside a double quoted string
Check out printf...
#!/bin/bash
mystr="say \"hi\""
Without using printf
echo -e $mystr
Output: say "hi"
Using printf
echo -e $(printf '%q' $mystr)
Output: say \"hi\"
Make use of $"string".
In this example, it would be,
dbload=$"load data local infile \"'gfpoint.csv'\" into table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY \"'\n'\" IGNORE 1 LINES"
Note (from the man page):
A double-quoted string preceded by a dollar sign ($"string") will cause the string to be translated according to the current locale. If the current locale is C or POSIX, the dollar sign is ignored. If the string is translated and replaced, the replacement is double-quoted.
For use with variables that might contain spaces in you Bash script, use triple quotes inside the main quote, e.g.:
[ "$(date -r """$touchfile""" +%Y%m%d)" -eq "$(date +%Y%m%d)" ]
Add "\" before double quote to escape it, instead of \
#! /bin/csh -f
set dbtable = balabala
set dbload = "load data local infile "\""'gfpoint.csv'"\"" into table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '"\""' LINES TERMINATED BY "\""'\n'"\"" IGNORE 1 LINES"
echo $dbload
# load data local infile "'gfpoint.csv'" into table balabala FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY "''" IGNORE 1 LINES
I want to use backticks in ruby for a programm call.
The parameter is a String variable containing one or more backticks, i.e.
"&E?##A`?". The following command yields a new label as its return value:
echo "&E?##A\`?" | nauty-labelg 2>/dev/null
From a ruby program I can call it as follows and get the correct result:
new_label = `echo "&E?##A\\\`?" | nauty-labelg 2>/dev/null`
I want to achieve the same using a variable for the label.
So I have to insert three slashes into my variable label = "&E?##A`?" in order to escape the backtick. The following seems to work, though it is not very elegant:
escaped_label = label.gsub(/`/, '\\\`').gsub(/`/, '\\\`').gsub(/`/, '\\\`')
But the new variable cannot be used in the program call:
new_label = `echo "#{escaped_label}" | nauty-labelg 2>/dev/null`
In this case I do not get an answer from nauty-labelg.
So I have to insert three slashes into my variable label = "&E?##A`?" in order to escape the backtick.
No, you only need to add one backslash for the output. To escape the ` special bash character. The other other two are only for representation proposes, otherwise it isn't valid Ruby code.
new_label = `echo "&E?##A\\\`?" | nauty-labelg 2>/dev/null`
The first backslash will escape the second one (outputting one single backslash). The third backslash escapes the ` character (outputting one single `).
You should only add backslashes before characters that have a special meaning within double quoted bash context. These special characters are: $, `, \ and \n. Those can be escaped with the following code:
def escape_bash_string(string)
string.gsub(/([$`"\\\n])/, '\\\\\1')
end
For label = "&E?##A`?" only the ` should be escaped.
escaped_string = escape_bash_string("&E?##A\`?")
puts escaped_string
# &E?##A\`?
I am trying to change the following string
FROM java_jre_8#sha256:92f22331226b9b3c43a15eeeb304dd7
to
FROM docker-registry.service.consul:5000/java_jre_8#sha256:92f22331226b9b3c43a15eeeb304dd7
but am having difficult with sed as a result of / character
This is for a build server.
There are two ways of doing this. The first is to escape each / in the string you're replacing:
sed 's/from/to with \/ ... /'
The other, more simple way is to use a delimiter other than /. While most sed examples use / as a delimiter, you can use any character:
sed 's|from|to with / ...|'
Here, the | is the first character following s, and therefore sed knows to use this as a delimiter.
You can use # as the delimiter as it doesn't appear in your string (you can still use / but then you'll have to quote the actual /s that are part of the string).
sed "s#FROM java_jre_8##FROM docker-registry.service.consul:5000/java_jre_8##'
Example:
$ echo "FROM java_jre_8#sha256:92f22331226b9b3c43a15eeeb304dd7" | sed "s#FROM java_jre_8##FROM docker-registry.service.consul:5000/java_jre_8##"
FROM docker-registry.service.consul:5000/java_jre_8#sha256:92f22331226b9b3c43a15eeeb304dd7
How do I add an escape character "\" before each ", " in a file? (that's a comma followed by a space).
P.S. Those quotes are for formatting purposes and are not presented (nor should appear in the file).
see this example, if it is helpful for you:
kent$ echo "foo,bar,blah"|sed 's/,/\\,/g'
foo\,bar\,blah
if you want to replace the comma, only if there is a space followed, do this:
kent$ echo "foo, bar, blah,skipme"|sed 's/, /\\&/g'
foo\, bar\, blah,skipme
For each character
put wanted character to escape in a class [ ,] (space and ,in this case`)
echo 'Your, string or data' | sed 's/[, ]/\\&/g'
there is special rules for ], - and \ (see RegEx class documentation for this)
For specifing pattern
echo 'Your, string or data' | sed 's/, /\\&/g'
How can I escape double quotes inside a double string in Bash?
For example, in my shell script
#!/bin/bash
dbload="load data local infile \"'gfpoint.csv'\" into table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY \"'\n'\" IGNORE 1 LINES"
I can't get the ENCLOSED BY '\"' with double quote to escape correctly. I can't use single quotes for my variable, because I want to use variable $dbtable.
Use a backslash:
echo "\"" # Prints one " character.
A simple example of escaping quotes in the shell:
$ echo 'abc'\''abc'
abc'abc
$ echo "abc"\""abc"
abc"abc
It's done by finishing an already-opened one ('), placing the escaped one (\'), and then opening another one (').
Alternatively:
$ echo 'abc'"'"'abc'
abc'abc
$ echo "abc"'"'"abc"
abc"abc
It's done by finishing already opened one ('), placing a quote in another quote ("'"), and then opening another one (').
More examples: Escaping single-quotes within single-quoted strings
Keep in mind that you can avoid escaping by using ASCII codes of the characters you need to echo.
Example:
echo -e "This is \x22\x27\x22\x27\x22text\x22\x27\x22\x27\x22"
This is "'"'"text"'"'"
\x22 is the ASCII code (in hex) for double quotes and \x27 for single quotes. Similarly you can echo any character.
I suppose if we try to echo the above string with backslashes, we will need a messy two rows backslashed echo... :)
For variable assignment this is the equivalent:
a=$'This is \x22text\x22'
echo "$a"
# Output:
This is "text"
If the variable is already set by another program, you can still apply double/single quotes with sed or similar tools.
Example:
b="Just another text here"
echo "$b"
Just another text here
sed 's/text/"'\0'"/' <<<"$b" #\0 is a special sed operator
Just another "0" here #this is not what i wanted to be
sed 's/text/\x22\x27\0\x27\x22/' <<<"$b"
Just another "'text'" here #now we are talking. You would normally need a dozen of backslashes to achieve the same result in the normal way.
Bash allows you to place strings adjacently, and they'll just end up being glued together.
So this:
echo "Hello"', world!'
produces
Hello, world!
The trick is to alternate between single and double-quoted strings as required. Unfortunately, it quickly gets very messy. For example:
echo "I like to use" '"double quotes"' "sometimes"
produces
I like to use "double quotes" sometimes
In your example, I would do it something like this:
dbtable=example
dbload='load data local infile "'"'gfpoint.csv'"'" into '"table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '"'"'"' LINES "'TERMINATED BY "'"'\n'"'" IGNORE 1 LINES'
echo $dbload
which produces the following output:
load data local infile "'gfpoint.csv'" into table example FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY "'\n'" IGNORE 1 LINES
It's difficult to see what's going on here, but I can annotate it using Unicode quotes. The following won't work in Bash – it's just for illustration:
dbload=‘load data local infile "’“'gfpoint.csv'”‘" into ’“table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '”‘"’“' LINES ”‘TERMINATED BY "’“'\n'”‘" IGNORE 1 LINES’
The quotes like “ ‘ ’ ” in the above will be interpreted by bash. The quotes like " ' will end up in the resulting variable.
If I give the same treatment to the earlier example, it looks like this:
echo “I like to use” ‘"double quotes"’ “sometimes”
Store the double quote character in a variable:
dqt='"'
echo "Double quotes ${dqt}X${dqt} inside a double quoted string"
Output:
Double quotes "X" inside a double quoted string
Check out printf...
#!/bin/bash
mystr="say \"hi\""
Without using printf
echo -e $mystr
Output: say "hi"
Using printf
echo -e $(printf '%q' $mystr)
Output: say \"hi\"
Make use of $"string".
In this example, it would be,
dbload=$"load data local infile \"'gfpoint.csv'\" into table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY \"'\n'\" IGNORE 1 LINES"
Note (from the man page):
A double-quoted string preceded by a dollar sign ($"string") will cause the string to be translated according to the current locale. If the current locale is C or POSIX, the dollar sign is ignored. If the string is translated and replaced, the replacement is double-quoted.
For use with variables that might contain spaces in you Bash script, use triple quotes inside the main quote, e.g.:
[ "$(date -r """$touchfile""" +%Y%m%d)" -eq "$(date +%Y%m%d)" ]
Add "\" before double quote to escape it, instead of \
#! /bin/csh -f
set dbtable = balabala
set dbload = "load data local infile "\""'gfpoint.csv'"\"" into table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '"\""' LINES TERMINATED BY "\""'\n'"\"" IGNORE 1 LINES"
echo $dbload
# load data local infile "'gfpoint.csv'" into table balabala FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY "''" IGNORE 1 LINES