The DNS entries i am trying to validate are quite long. Here's an example of what the structure might look like:
qwer-0123a4bcd567890e1-uuuuu3xx.qwer-gfd-1e098765dcb4a3210.ps-sdlk-6.qwer.domain.com
These entries can be thought of as three distinct parts:
qwer-0123a4bcd567890e1-uuuuu3xx.qwer-gfd-1e098765dcb4a3210.
Always starts with qwer-
Followed by 17 alphanumerics, a -, 8 more alphanumerics
Followed by qwer-gfd-
Followed by 17 more alphanumerics and a .
ps-sdlk-6
Always starts with ps-sdlk-
Followed by either one or two alphanumeric. In this case it could be ps-sdlk-6 or something like ps-sdlk-6e
.qwer.domain.com
The domain target always ends with .qwer.domain.com
I've been hacking together a regex and came up with this monstrosity:
qwer-[\w]{17}-[\w]{8}.qwer-gfd-[\w]{17}.(.*)(qwer.domain.com)
That solution is pretty hideous and it returns multiple match groups which doesn't give me much confidence in the accuracy. I'm using ruby 2.5 but non std lib stuff is difficult to import in this case.
Is there a more sensible and complete/accurate regex to confirm the validity of these dns targets? Is there a better way to do this without regex?
Considering the complexity of testing longish regular expressions, and also the possibility--if not the probability--that changes will be needed in future, I would be inclined to split the string on hyphens and test each string in the resulting array.
PIECES = [['qwer'],
['0123a4bcd567890e1'.size],
['uuuuu3xx'.size, '.qwer'],
['gfd'],
['1e098765dcb4a3210'.size, '.ps'],
['sdlk'],
[[1, 2], '.qwer.domain.com']].
map do |a|
Regexp.new(
a.each_with_object('\A') do |o,s|
case o
when String
s << o.gsub('.', '\.')
when Integer
s << "\\p{Alnum}{#{o}}"
else # array
s << "\\p{Alnum}{#{o.first},#{o.last}}"
end
end << '\z')
end
#=> [/\Aqwer\z/, /\A\p{Alnum}{17}\z/, /\A\p{Alnum}{8}\.qwer\z/,
# /\Agfd\z/, /\A\p{Alnum}{17}\.ps\z/, /\Asdlk\z/,
# /\A\p{Alnum}{1,2}\.qwer\.domain\.com\z/]
Notice that I've used single quotes in most places to be able to write, for example, '\A' rather than "\\A". However, double quotes are needed for the two lines where interpolation is performed (#{o}). I've also used strings from the example to determine the lengths of various runs of alphanumeric characters and have escaped periods and added anchors in simple code. I did that to both reduce the chance of counting errors and help readers of the code understand what is being done. Though the elements of PIECES (regular expressions) are here being used to test the string used to construct PIECES that is of course irrelevant, assuming, as we must, that all strings to be tested will have the same pattern.
def valid?(str)
arr = str.split('-')
return false unless arr.size == PIECES.size
arr.zip(PIECES).all? { |s,r| s.match? r }
end
If Enumerable#all?'s block returns false all? immediately returns false. This is sometimes referred to as short-circuiting behaviour.
For the string given in the example, str,
valid?(str)
#=> true
Note the following intermediate calculation.
str.split('-').zip(PIECES)
#=> [["qwer", /\Aqwer\z/],
# ["0123a4bcd567890e1", /\A\p{Alnum}{17}\z/],
# ["uuuuu3xx.qwer", /\A\p{Alnum}{8}\.qwer\z/],
# ["gfd", /\Agfd\z/],
# ["1e098765dcb4a3210.ps", /\A\p{Alnum}{17}\.ps\z/],
# ["sdlk", /\Asdlk\z/],
# ["6.qwer.domain.com", /\A\p{Alnum}{1,2}\.qwer\.domain\.com\z/]]
This may seem overkill (and I'm not certain it isn't), but it does facilitate debugging and testing and if, in future, the string pattern changes (within limits) it should be relatively easy to modify the matching test (by changing the array of arrays above from which PIECES is derived).
I think given your input data you have no choice but an ugly regex e.g.
^qwer-\w{17}-\w{8}\.qwer-gfd-\w{17}\.ps-sdlk-\w{1,2}\.qwer\.domain\.com$
Note that I have used \w as you did, however \w also matches _ as well as alphanumeric characters, so you may want to replace it with [A-Za-z0-9]. Also, . will match any character, so to specifically match a . you need \. in your regex.
Demo on regex101.com
Related
As an exercise, I'm working on an accent translation dictionary. My dictionary is contained in a hash, and I'm thinking of using #gsub! to run inputted strings through the translator.
I'm wondering if there's any way to make the substitutions case-sensitive. For example, I want "didja" to translate to "did you" and "Didja" to translate to "Did you", but I don't want to have to create multiple dictionary entries to deal with case.
I know I can use regex syntax to find strings to replace case-insensitively, with str.gsub!(/#{x}/i,dictionary[x]) where x is a variable. The problem is that this replaces "Didja" with "did you", rather than matching the original case.
Is there any way to make it match the original case?
Suppose we have:
a method to_key that converts a string str to a key in a hash DICTIONARY; and
a method transform that converts the pair [str, DICTIONARY[to_key(str)]] to the replacement for str.
Then str is to be replaced with:
transform(str, DICTIONARY[to_key(str)]])
Without lose of generality, I think we can assume that DICTIONARY's keys and values are all of the same case (say, lower case) and that to_key is simply:
def to_key(str)
str.downcase
end
So all that is necessary is to define the method transform. However, the specification provided does not imply a unique mapping. We therefore must decide what transform should do.
For example, suppose the rule is simply that, if the first character of str and the first character of the dictionary value are both letters, the latter is to be converted to upper case if the former is upper case. Then:
def transform(str, dict_value)
(str[0] =~ /[A-Z]/) ? dict_value.capitalize : dict_value
end
(I originally had dict_value[0] = dict_value[0].upcase if..., but came to my senses after reading #sawa's answer.)
Note that if DICTIONARY['cat'] => 'dog', 'Cat' will be converted to 'Dog'.
One might think that another possibility is that all characters of str that are letters should maintain their case. This is problematic, however, as the dictionary mapping may (without further specification) remove letters, and it may not be clear from DICTIONARY[str] which letters of str were removed, some of which may be lower case and others upper case.
It is not clear what capitalization patterns you have in mind. I assume that you only need to deal with words that are all low case or all low case except the first letter.
str.gsub!(/#{x}/i){|x| x.downcase! ? dictionary[x].capitalize : dictionary[x]}
I don't think this is possible since in this scenario you need to specify the exact string that must take place of the replaced string.
With that in mind, this is the best I can suggest:
subs = {'didja' => 'did you'}
subs.clone.each{ |k, v| subs[k.capitalize] = v.capitalize }
# if you want to replace all occurrences i.e. even substrings:
regex = /#{subs.keys.join('|')}/
# if you want to remove complete words only: (as the Tin man points out)
regex = /\b(?:#{subs.keys.join('|')})\b/ # \b checks for word-boundaries
"didja Didja".gsub(regex, subs)
Update:
Because in your example, the case-sensitive character isn't to be replaced by another value, you could use this:
regex = /(?<=(d))idja/i # again, keep in mind the substrings
"didja Didja".gsub(regex, "id you")
I'm trying to make a simple Ruby regex to detect a JavaScript Declaration, but it fails.
Regex:
lines.each do |line|
unminifiedvar = /var [0-9a-zA-Z] = [0-9];/.match(line)
next if unminifiedvar == nil #no variable declarations on the line
#...
end
Testing Line:
var testvariable10 = 9;
A variable name can have more than one character, so you need a + after the character-set [...]. (Also, JS variable names can contain other characters besides alphanumerics.) A numeric literal can have more than one character, so you want a + on the RHS too.
More importantly, though, there are lots of other bits of flexibility that you'll find more painful to process with a regular expression. For instance, consider var x = 1+2+3; or var myString = "foo bar baz";. A variable declaration may span several lines. It need not end with a semicolon. It may have comments in the middle of it. And so on. Regular expressions are not really the right tool for this job.
Of course, it may happen that you're parsing code from a particular source with a very special structure and can guarantee that every declaration has the particular form you're looking for. In that case, go ahead, but if there's any danger that the nature of the code you're processing might change then you're going to be facing a painful problem that really isn't designed to be solved with regular expressions.
[EDITED about a day after writing, to fix a mistake kindly pointed out by "the Tin Man".]
You forgot the +, as in, more than one character for the variable name.
var [0-9a-zA-Z]+ = [0-9];
You may also want to add a + after the [0-9]. That way it can match multiple digits.
var [0-9a-zA-Z]+ = [0-9]+;
http://rubular.com/r/kPlNcGRaHA
Try /var [0-9a-zA-Z]+ = \d+;/
Without the +, [0-9a-zA-Z] will only match a single alphanumeric character. With +, it can match 1 or more alphanumeric characters.
By the way, to make it more robust, you may want to make it match any number of spaces between the tokens, not just exactly one space each. You may also want to make the semicolon at the end optional (because Javascript syntax doesn't require a semicolon). You might also want to make it always match against the whole line, not just a part of the line. That would be:
/\Avar\s+[0-9a-zA-Z]+\s*=\s*\d+;?\Z/
(There is a way to write [0-9a-zA-Z] more concisely, but it has slipped my memory; if someone else knows, feel free to edit this answer.)
At the moment I have a regular expression that looks like this:
^(cat|dog|bird){1}(cat|dog|bird)?(cat|dog|bird)?$
It matches at least 1, and at most 3 instances of a long list of words and makes the matching words for each group available via the corresponding variable.
Is there a way to revise this so that I can return the result for each word in the string without specifying the number of groups beforehand?
^(cat|dog|bird)+$
works but only returns the last match separately , because there is only one group.
OK, so I found a solution to this.
It doesn't look like it is possible to create an unknown number of groups, so I went digging for another way of achieving the desired outcome: To be able to tell if a string was made up of words in a given list; and to match the longest words possible in each position.
I have been reading Mastering Regular Expressions by Jeffrey E. F. Friedl and it shed some light on things for me. It turns out that NFA based Regexp engines (like the one used in Ruby) are sequential as well as lazy/greedy. This means that you can dictate how a pattern is matched using the order in which you give it choices. This explains why scan was returning variable results, it was looking for the first word in the list that matched the criteria and then moved on to the next match. By design it was not looking for the longest match, but the first one. So in order to rectify this all I needed to do was reorder the array of words used to generate the regular expression from alphabetical order, to length order (longest to shortest).
array = %w[ as ascarid car id ]
list = array.sort_by {|word| -word.length }
regexp = Regexp.union(list)
Now the first match found by scan will be the longest word available. It is also pretty simple to tell if a string contains only words in the list using scan:
if "ascarid".scan(regexp).join.length == word.length
return true
else
return false
end
Thanks to everyone that posted in response to this question, I hope that this will help others in the future.
You could do it in two steps:
Use /^(cat|dog|bird)+$/ (or better /\A(cat|dog|bird)+\z/) to make sure it matches.
Then string.scan(/cat|dog|bird/) to get the pieces.
You could also use split and a Set to do both at once. Suppose you have your words in the array a and your string in s, then:
words = Set.new(a)
re = /(#{a.map{|w| Regexp.quote(w)}.join('|')})/
parts = s.split(re).reject(&:empty?)
if(parts.any? {|w| !words.include?(w) })
# 's' didn't match what you expected so throw a
# hissy fit, format the hard drive, set fire to
# the backups, or whatever is appropriate.
else
# Everything you were looking for is in 'parts'
# so you can check the length (if you care about
# how many matches there were) or something useful
# and productive.
end
When you use split with a pattern that contains groups then
the respective matches will be returned in the array as well.
In this case, the split will hand us something like ["", "cat", "", "dog"] and the empty strings will only occur between the separators that we're looking for and so we can reject them and pretend they don't exist. This may be an unexpected use of split since we're more interested in the delimiters more than what is being delimited (except to make sure that nothing is being delimited) but it gets the job done.
Based on your comments, it looks like you want an ordered alternation so that (ascarid|car|as|id) would try to match from left to right. I can't find anything in the Ruby Oniguruma (the Ruby 1.9 regex engine) docs that says that | is ordered or unordered; Perl's alternation appears to be specified (or at least strongly implied) to be ordered and Ruby's certainly behaves as though it is ordered:
>> 'pancakes' =~ /(pan|pancakes)/; puts $1
pan
So you could sort your words from longest to shortest when building your regex:
re = /(#{a.sort_by{|w| -w.length}.map{|w| Regexp.quote(w)}.join('|')})/
and hope that Oniguruma really will match alternations from left to right. AFAIK, Ruby's regexes will be eager because they support backreferences and lazy/non-greedy matching so this approach should be safe.
Or you could be properly paranoid and parse it in steps; first you'd make sure your string looks like what you want:
if(s !~ /\A(#{a.map{|w| Regexp.quote(w)}.join('|')})+\z/)
# Bail out and complain that 's' doesn't look right
end
The group your words by length:
by_length = a.group_by(&:length)
and scan for the groups from the longest words to the shortest words:
# This loses the order of the substrings within 's'...
matches = [ ]
by_length.keys.sort_by { |k| -k }.each do |group|
re = /(#{a.map{|w| Regexp.quote(w)}.join('|')})/
s.gsub!(re) { |w| matches.push(w); '' }
end
# 's' should now be empty and the matched substrings will be
# in 'matches'
There is still room for possible overlaps in these approaches but at least you'd be extracting the longest matches.
If you need to repeat parts of a regex, one option is to store the repeated part in a variable and just reference that, for example:
r = "(cat|dog|bird)"
str.match(/#{r}#{r}?#{r}?/)
You can do it with .Net regular expressions. If I write the following in PowerShell
$pat = [regex] "^(cat|dog|bird)+$"
$m = $pat.match('birddogcatbird')
$m.groups[1].captures | %{$_.value}
then I get
bird
dog
cat
bird
when I run it. I know even less about IronRuby than I do about PowerShell, but perhaps this means you can do it in IronRuby as well.
I am trying to use regular expressions to match some text.
The following pattern is what I am trying to gather.
#Identifier('VariableA', 'VariableB', 'VariableX', ..., 'VariableZ')
I would like to grab a dynamic number of variables rather than a fixed set of two or three.
Is there any way to do this? I have an existing Regular Expression:
\#(\w+)\W+(\w+)\W+(\w+)\W+(\w+)
This captures the Identifier and up to three variables.
Edit: Is it just me, or are regular expressions not as powerful as I'm making them out to be?
You want to use scan for this sort of thing. The basic pattern would be this:
s.scan(/\w+/)
That would give you an array of all the contiguous sequences for word characters:
>> "#Identifier('VariableA', 'VariableB', 'VariableX', 'VariableZ')".scan(/\w+/)
=> ["Identifier", "VariableA", "VariableB", "VariableX", "VariableZ"]
You say you might have multiple instances of your pattern with arbitrary stuff surrounding them. You can deal with that with nested scans:
s.scan(/#(\w+)\(([^)]+?)\)/).map { |m| [ m.first, m.last.scan(/\w+/) ] }
That will give you an array of arrays, each inner array will have the "Identifier" part as the first element and that "Variable" parts as an array in the second element. For example:
>> s = "pancakes #Identifier('VariableA', 'VariableB', 'VariableX', 'VariableZ') pancakes #Pancakes('one','two','three') eggs"
>> s.scan(/#(\w+)\(([^)]+?)\)/).map { |m| [ m.first, m.last.scan(/\w+/) ] }
=> [["Identifier", ["VariableA", "VariableB", "VariableX", "VariableZ"]], ["Pancakes", ["one", "two", "three"]]]
If you might be facing escaped quotes inside your "Variable" bits then you'll need something more complex.
Some notes on the expression:
# # A literal "#".
( # Open a group
\w+ # One more more ("+") word characters ("\w").
) # Close the group.
\( # A literal "(", parentheses are used for group so we escape it.
( # Open a group.
[ # Open a character class.
^) # The "^" at the beginning of a [] means "not", the ")" isn't escaped because it doesn't have any special meaning inside a character class.
] # Close a character class.
+? # One more of the preceding pattern but don't be greedy.
) # Close the group.
\) # A literal ")".
You don't really need [^)]+? here, just [^)]+ would do but I use the non-greedy forms by habit because that's usually what I mean. The grouping is used to separate the #Identifier and Variable parts so that we can easily get the desired nested array output.
But alex thinks that you meant you wanted to capture the same thing four times. If you want to capture the same pattern, but different things, then you may want to consider two things:
Iteration. In perl, you can say
while ($variable =~ /regex/g) {
the 'g' stands for 'global', and means that each time the regex is called, it matches the /next/ instance.
The other option is recursion. Write your regex like this:
/(what you want)(.*)/
Then, you have backreference 1 containing the first thing, which you can push to an array, and backreference 2 which you'll then recurse over until it no longer matches.
You may use simply (\w+).
Given the input string
#Identifier('VariableA', 'VariableB', 'VariableX', 'VariableZ')
The results would be:
Identifier
VariableA
VariableB
VariableX
VariableZ
This would work for an arbitrary number of variables.
For future reference, it's easy and fun to play around with regexp ideas on Rubular.
So you are asking if there is a way to capture both the identifier and an arbitrary number of variables. I am afraid that you can only do this with regex engines that support captures. Note here that captures and capturing groups are not the one and the same thing. You want to remember all the "variables". This can't be done with simple capturing groups.
I am unaware whether Ruby supports this or not, but I am sure that .NET and the new PERL 6 support it.
In your case you could use two regexes. One to capture the identifier e.g. ^\s*#(\w+)
and another one to capture all variables e.g. result = subject.scan(/'[^']+'/)
I have the following
address.gsub(/^\d*/, "").gsub(/\d*-?\d*$/, "").gsub(/\# ?\d*/,"")
Can this be done in one gsub? I would like to pass a list of patterns rather then just one pattern - they are all being replaced by the same thing.
You could combine them with an alternation operator (|):
address = '6 66-666 #99 11-23'
address.gsub(/^\d*|\d*-?\d*$|\# ?\d*/, "")
# " 66-666 "
address = 'pancakes 6 66-666 # pancakes #99 11-23'
address.gsub(/^\d*|\d*-?\d*$|\# ?\d*/,"")
# "pancakes 6 66-666 pancakes "
You might want to add little more whitespace cleanup. And you might want to switch to one of:
/\A\d*|\d*-?\d*\z|\# ?\d*/
/\A\d*|\d*-?\d*\Z|\# ?\d*/
depending on what your data really looks like and how you need to handle newlines.
Combining the regexes is a good idea--and relatively simple--but I'd like to recommend some additional changes. To wit:
address.gsub(/^\d+|\d+(?:-\d+)?$|\# *\d+/, "")
Of your original regexes, ^\d* and \d*-?\d*$ will always match, because they don't have to consume any characters. So you're guaranteed to perform two replacements on every line, even if that's just replacing empty strings with empty strings. Of my regexes, ^\d+ doesn't bother to match unless there's at least one digit at the beginning of the line, and \d+(?:-\d+)?$ matches what looks like an integer-or-range expression at the end of the line.
Your third regex, \# ?\d*, will match any # character, and if the # is followed by a space and some digits, it'll take those as well. Judging by your other regexes and my experience with other questions, I suspect you meant to match a # only if it's followed by one or more digits, with optional spaces intervening. That's what my third regex does.
If any of my guesses are wrong, please describe what you were trying to do, and I'll do my best to come up with the right regex. But I really don't think those first two regexes, at least, are what you want.
EDIT (in answer to the comment): When working with regexes, you should always be aware of the distinction between a regex the matches nothing and a regex that doesn't match. You say you're applying the regexes to street addresses. If an address doesn't happen to start with a house number, ^\d* will match nothing--that is, it will report a successful match, said match consisting of the empty string preceding the first character in the address.
That doesn't matter to you, you're just replacing it with another empty string anyway. But why bother doing the replacement at all? If you change the regex to ^\d+, it will report a failed match and no replacement will be performed. The result is the same either way, but the "matches noting" scenario (^\d*) results in a lot of extra work that the "doesn't match" scenario avoids. In a high-throughput situation, that could be a life-saver.
The other two regexes bring additional complications: \d*-?\d*$ could match a hyphen at the end of the string (e.g. "123-", or even "-"); and \# ?\d* could match a hash symbol anywhere in string, not just as part of an apartment/office number. You know your data, so you probably know neither of those problems will ever arise; I'm just making sure you're aware of them. My regex \d+(?:-\d+)?$ deals with the trailing-hyphen issue, and \# *\d+ at least makes sure there are digits after the hash symbol.
I think that if you combine them together in a single gsub() regex, as an alternation,
it changes the context of the starting search position.
Example, each of these lines start at the beginning of the result of the previous
regex substitution.
s/^\d*//g
s/\d*-?\d*$//g
s/\# ?\d*//g
and this
s/^\d*|\d*-?\d*$|\# ?\d*//g
resumes search/replace where the last match left off and could potentially produce a different overall output, especially since a lot of the subexpressions search for similar
if not the same characters, distinguished only by line anchors.
I think your regex's are unique enough in this case, and of course changing the order
changes the result.