I am making a grammar for bash scripts. I am facing a problem while tokenising the "," symbol. The following grammar tokenises it as <BLOB> while I expect it to be tokenised as <OTHER>.
grammar newgram;
code : KEY (BLOB)+ (EOF | '\n')+;
KEY : 'wget';
BLOB : [a-zA-Z0-9#!$^%*&+-.]+?;
OTHER : .;
However, if I make BLOB to be [a-zA-Z0-9#!$^%*&+.-]+?;, then it is tokenised as <OTHER>.
I cannot understand why is it happening like this.
In the former case, the characters : and / are also tokenised as <OTHER>, so I do not see a reason for ,, to be marked <BLOB>.
Input I am tokenising, wget -o --quiet https,://www.google.com
The output I am receiving with the mentioned grammar,
[#0,0:3='wget',<'wget'>,1:0]
[#1,4:4=' ',<OTHER>,1:4]
[#2,5:5='-',<BLOB>,1:5]
[#3,6:6='o',<BLOB>,1:6]
[#4,7:7=' ',<OTHER>,1:7]
[#5,8:8='-',<BLOB>,1:8]
[#6,9:9='-',<BLOB>,1:9]
[#7,10:10='q',<BLOB>,1:10]
[#8,11:11='u',<BLOB>,1:11]
[#9,12:12='i',<BLOB>,1:12]
[#10,13:13='e',<BLOB>,1:13]
[#11,14:14='t',<BLOB>,1:14]
[#12,15:15=' ',<OTHER>,1:15]
[#13,16:16='h',<BLOB>,1:16]
[#14,17:17='t',<BLOB>,1:17]
[#15,18:18='t',<BLOB>,1:18]
[#16,19:19='p',<BLOB>,1:19]
[#17,20:20='s',<BLOB>,1:20]
[#18,21:21=',',<BLOB>,1:21]
[#19,22:22=':',<OTHER>,1:22]
[#20,23:23='/',<OTHER>,1:23]
[#21,24:24='/',<OTHER>,1:24]
[#22,25:25='w',<BLOB>,1:25]
[#23,26:26='w',<BLOB>,1:26]
[#24,27:27='w',<BLOB>,1:27]
[#25,28:28='.',<BLOB>,1:28]
[#26,29:29='g',<BLOB>,1:29]
[#27,30:30='o',<BLOB>,1:30]
[#28,31:31='o',<BLOB>,1:31]
[#29,32:32='g',<BLOB>,1:32]
[#30,33:33='l',<BLOB>,1:33]
[#31,34:34='e',<BLOB>,1:34]
[#32,35:35='.',<BLOB>,1:35]
[#33,36:36='c',<BLOB>,1:36]
[#34,37:37='o',<BLOB>,1:37]
[#35,38:38='m',<BLOB>,1:38]
[#36,39:39='\n',<'
'>,1:39]
[#37,40:39='<EOF>',<EOF>,2:0]
line 1:4 extraneous input ' ' expecting BLOB
line 1:7 extraneous input ' ' expecting {<EOF>, '
', BLOB}
line 1:15 extraneous input ' ' expecting {<EOF>, '
', BLOB}
line 1:22 extraneous input ':' expecting {<EOF>, '
', BLOB}
As already mentioned in a comment, the - in +-. inside your character class is interpreted as a range operator. And the , is inside that range. Escape it like this: [a-zA-Z0-9#!$^%*&+\-.]+?
Also, a trailing [ ... ]+? at the end of a lexer rule will always match a single character. So [a-zA-Z0-9#!$^%*&+\-.]+? can just as well be written as [a-zA-Z0-9#!$^%*&+\-.]
How to get a regular expression to replace all characters in a string in perl with *? The string has some utf-8 or iso-8859-1 characters also. I tried with "s/\w/*/g". But it did not replace utf-8 or iso-8859-1 characters.
my $value="hellö";
print "$value\n";
$value =~ s/\w/*/g;
print "after replacing $value\n"; //It prints ****ö.
I expect all characters should be replaced with * i.e hellö should be replaced with *****.
Please note, few special characters like -,_,\,/ etc should be skipped.
If you want to skip just a few characters, you can always do something along the lines of
s/[^, \/\\\-]/*/g;
To replace all the characters in a string? The \w is for matching word characters, but using just a dot should match all characters: s/./*/g
Let's say I have a file test.txt with contents:
+-foo.bar:2.4
| bar.foo:1.1:test
\| hello.goobye:3.3.3
\|+- baz.yeah:4
I want to use the tr command to delete all instances of the following set of characters:
{' ', '+', '-', '|', '\'}
Done some pretty extensive research on this but found no clear/concise answers.
This is the command that works:
input:
cat test.txt | tr -d "[:blank:]|\\\+-"
output:
foo.bar:2.4
bar.foo:1.1:test
hello.goobye:3.3.3
baz.yeah:4
I experimented with many combinations of that set and I found out that the '-' was being treated as a range indicator (like... [a-z]) and therefore must be put at the end. But I have two main questions:
1) Why must the backslash be double escaped in order to be included in the set?
2) Why does putting the '|' at the end of the set string cause the tr program to delete everything in the file except for trailing new line characters?
Like this:
tr -d '\-|\\+[:blank:] ' < file
You have to escape the - because it is used for denoting ranges of characters like:
tr -d '1-5'
and must therefore being escaped if you mean a literal hyphen. You can also put it at the end. (learned that, thanks! :) )
Furthermore the \ must be escaped when you mean a literal \ because it has a special meaning needed for escape sequences.
The remaining characters must not being escaped.
Why must the \ being doubly escaped in your example?
It's because you are using a "" (double quoted) string to quote the char set. A double quoted string will be interpreted by the shell, a \\ in a double quoted string means a literal \. Try:
echo "\+"
echo "\\+"
echo "\\\+"
To avoid to doubly escape the \ you can just use single quotes as in my example above.
Why does putting the '|' at the end of the set string cause the tr program to delete everything in the file except for trailing new line characters?
Following CharlesDuffy's comment having the | at the end means also that you had the unescaped - not at the end, which means it was describing a range of characters where the actual range depends on the position you had it in the set.
another approach is to define the allowed chars
$ tr -cd '[:alnum:]:.\n' <file
foo.bar:2.4
bar.foo:1.1:test
hello.goobye:3.3.3
baz.yeah:4
or, perhaps delete all the prefix non-word chars
$ sed -E 's/\W+//' file
I'm trying to do a regex with lookbehind that changes \n to but not if it's a \\n.
My closest attempt has no effect:
text.gsub /(?<!\\)\n/, ''
Unfortunately, no number of backslashes in the lookbehind seem to fix the problem. How can I address this?
You need to double the backslash before the n in the regex, otherwise it's looking for a newline instead of a literal backslash followed by n:
irb(main):001:0> puts "hello\\nthere\\\\n".gsub(/(?<!\\)\\n/, ' ')
hello there\\n
You don't need anything special. "\n" is a single character. It does not include a "\" or "n" character.
text.gsub(/\n/, "")
But instead of that, you should do:
text.gsub("\n", "")
or
text.tr("\n", "")
But I would do:
text.tr($/, "")
I have a string that looks like this.
mystring="The Body of a\r\n\t\t\t\tSpider"
I want to replace all the \r, \n, \t etc with a whitespace.
The code I wrote for this is :
mystring.gsub(/\\./, " ")
But this isn't doing anything to the string.
Help.
\r, \n and \t are escape sequences representing carriage return, line feed and tab. Although they are written as two characters, they are interpreted as a single character:
"\r\n\t".codepoints #=> [13, 10, 9]
Because it is such a common requirement, there's a shortcut \s to match all whitespace characters:
mystring.gsub(/\s/, ' ')
#=> "The Body of a Spider"
Or \s+ to match multiple whitespace characters:
mystring.gsub(/\s+/, ' ')
#=> "The Body of a Spider"
/\s/ is equivalent to /[ \t\r\n\f]/
String#tr is designed for stream symbol substitution. It appears to be a bit quickier, than String#gsub:
mystring.tr "\r", ' '
It hasan insplace version also (this will replace all carriage returns, line feed and spaces with space):
mystring.tr! "\s\r\n\t\f", ' '
Stefen's Answer is really very Cool as always comeup with very short and clean solutions. But here what I tried to remove all special characters. [Posted as just optional solution] ;)
> a = "The Body of a\r\n\t\t\t\tSpider"
=> "The Body of a\r\n\t\t\t\tSpider"
> a.gsub(/[^0-9A-Za-z]/, ' ')
=> "The Body of a Spider"
you can use strip , then add a space to your string
mystring.strip . " "
If you literally has \r\n\t in your string:
mystring="The Body of a\r\n\t\t\t\tSpider"
mystring.split(/[\r\t\n]/)