I've been trying to figure this out, and i've searched but I'm stuck.
Lets say I have the string www.google.com or http://google.com or just google.com
and I want to extract the string google out of those parameters.
A solution I can think of is first removing the first parameters (www.) then removing the second section of the string (.com) but I know there is a similar more efficient way.
any help would be greatly appreciated!
First, start with a tool designed to work with URLs. Ruby includes URI, and there's also Addressable::URI.
Using these you can strip down a URI into its defined components:
require 'uri'
uri = URI.parse('http://www.ruby-doc.org/stdlib-2.1.1/libdoc/uri/rdoc/URI.html')
uri.host # => "www.ruby-doc.org"
If your string doesn't start with a scheme, you can add one. (Schemes are important.)
url = 'foo.bar.com/some/path'
URI.parse('http://' + url).host
# => "foo.bar.com"
From that point you're going to have a tough time determining what is the true host, versus the domain. A domain can be anything (pretty much) and the host can be the domain name. Possibly you can get a list of domains but, remember that the list is constantly changing.
ICANN has a list of TLDs, as does IANA. Those are ONLY the top-level-domains, not the hosts that sit under them. However, using those lists you can strip the TLD from a host, and at least be a tiny bit closer to where you want to be.
Related
I'm using DMOZ's list of url topics, which contains some urls that have hostnames that contain an underscore.
For example:
608 <ExternalPage about="http://outer_heaven4.tripod.com/index2.htm">
609 <d:Title>The Outer Heaven</d:Title>
610 <d:Description>Information and image gallery of McFarlane's action figures for Trigun, Akira, Tenchi Muyo and other Japanese Sci-Fi animations.</d:Description>
611 <topic>Top/Arts/Animation/Anime/Collectibles/Models_and_Figures/Action_Figures</topic>
612 </ExternalPage>
While this url will work in a web browser (or, at least, it does in mine :p), it's not legal according to the standard:
a hostname may not contain other characters, such as the underscore character (_),
which causes errors when trying to parse such URL with URI.parse:
[2] pry(main)> require 'uri'
=> true
[3] pry(main)> URI.parse "http://outer_heaven4.tripod.com/index2.htm"
URI::InvalidURIError: the scheme http does not accept registry part: outer_heaven4.tripod.com (or bad hostname?)
from ~/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/uri/generic.rb:213:in `initialize'
Is there an alternative to URI.parse I can use that has lower strictness without just rolling my own?
Try Addressable::URI. It follows the RFCs more closely than URI and is very flexible.
require 'addressable/uri'
uri = Addressable::URI.parse('http://outer_heaven4.tripod.com/index2.htm')
uri.host
=> "outer_heaven4.tripod.com"
I've used it for some projects and have been happy with it. URI is getting a bit... rusty and is in need of TLC. Other's have commented on it too:
http://www.cloudspace.com/blog/2009/05/26/replacing-rubys-uri-with-addressable/
There was quite a discussion about URI's state several years ago among the Ruby developers. I can't find the link to it right now, but there was a recommendation that Addressable::URI be used as a replacement. I don't know if someone stepped up to take over URI development, or where things stand right now. In my own code I continue to use URI for simple things and switch to Addressable::URI when URI proves to do the wrong thing for me.
How do I escape a URL as needed, without double escaping?
Is there a Ruby library that already does this? I wonder what algorithm WebKit or Chrome uses.
Two examples:
This URL is not valid, since the % is not escaped: http://x.co/op&k=21%. If you type it into the URL bar, it knows to escape it. (It is escaping the '%' behind the scenes, right?)
If you type http://localhost:3000/?s=hello%20world into a browser, it knows to not escape %20 again.
I want to reuse great code that has already worked the edge cases that browsers have to handle. I don't mind calling an external library if necessary.
Update: Yes, I know about URI.parse. No need to show me the syntax. My question is harder than that.
So far, the winners are:
Addressable::URI#normalize: "Returns a normalized URI object. NOTE: This method does not attempt to fully conform to specifications. It exists largely to correct other people’s failures to read the specifications, and also to deal with caching issues since several different URIs may represent the same resource and should not be cached multiple times."
Addressable::URI.heuristic_parse: "Converts an input to a URI. The input does not have to be a valid URI -- the method will use heuristics to guess what URI was intended. This is not standards-compliant, merely user-friendly."
Knowing whether you need to encode or decode multiple times is up to you. You're the programmer and need to be aware of what state the URL is in as you massage it.
A browser can assume that a % not followed by a numeric-value is bare, and should be escaped. See "Uniform Resource Identifier (URI): Generic Syntax" for more information.
You can use Ruby's built-in URI, or the Addressable::URI gems to encode/decode.
require 'uri'
uri = URI.parse('http://x.co/op')
uri.query = URI.encode_www_form('k' => '21%')
puts uri.to_s # => http://x.co/op?k=21%25
or:
require 'addressable/uri'
uri = Addressable::URI.parse('http://x.co/op')
uri.query_values = {'k' => '21%'}
puts uri.to_s # => "http://x.co/op?k=21%25"
I'm having a problem getting my RegEx to work with my Ruby script.
Here is what I'm trying to match:
http://my.test.website.com/{GUID}/{GUID}/
Here is the RegEx that I've tested and should be matching the string as shown above:
/([-a-zA-Z0-9#:%_\+.~#?&\/\/=]{2,256}\.[a-z]{2,4}\b(\/[-a-zA-Z0-9#:%_\+.~#?&\/\/=]*)([\/\/[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\/\/[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\/\/])*?\/)/
3 capturing groups:
group 1: ([-a-zA-Z0-9#:%_\+.~#?&\/\/=]{2,256}\.[a-z]{2,4}\b(\/[-a-zA-Z0-9#:%_\+.~#?&\/\/=]*)([\/\/[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\/\/[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\/\/])*?\/)
group 2: (\/[-a-zA-Z0-9#:%_\+.~#?&\/\/=]*)
group 3: ([\/\/[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\/\/[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\/\/])
Ruby is giving me an error when trying to validate a match against this regex:
empty range in char class: (My RegEx goes here) (SyntaxError)
I appreciate any thoughts or suggestions on this.
You could simplify things a bit by using URI to deal parsing the URL, \h in the regex, and scan to pull out the GUIDs:
uri = URI.parse(your_url)
path = uri.path
guids = path.scan(/\h{8}-\h{4}-\h{4}-\h{4}-\h{12}/)
If you need any of the non-path components of the URL the you can easily pull them out of uri.
You might need to tighten things up a bit depending on your data or it might be sufficient to check that guids has two elements.
You have several errors in your RegEx. I am very sleepy now, so I'll just give you a hint instead of a solution:
...[\/\/[0-9a-fA-F]....
the first [ does not belong there. Also, having \/\/ inside [] is unnecessary - you only need each character once inside []. Also,
...[-a-zA-Z0-9#:%_\+.~#?&\/\/=]{2,256}...
is greedy, and includes a period - indeed, includes all chars (AFAICS) that can come after it, effectively swallowing the whole string (when you get rid of other bugs). Consider {2,256}? instead.
I am trying to construct a regex to extract a domain given a url.
for:
http://www.abc.google.com/
http://abc.google.com/
https://www.abc.google.com/
http://abc.google.com/
should give:
abc.google.com
URI.parse('http://www.abc.google.com/').host
#=> "www.abc.google.com"
Not a regex, but probably more robust then anything we come up with here.
URI.parse('http://www.abc.google.com/').host.gsub(/^www\./, '')
If you want to remove the www. as well this will work without raising any errors if the www. is not there.
Don't know much about ruby but this regex pattern gives you the last 3 parts of the url excluding the trailing slash with a minumum of 2 characters per part.
([\w-]{2,}\.[\w-]{2,}\.[\w-]{2,})/$
you may be able to use the domain_name gem for this kind of work. From the README:
require "domain_name"
host = DomainName("a.b.example.co.uk")
host.domain #=> "example.co.uk"
Your question is a little bit vague. Can you give a precise specification of what it is exactly that you want to do? (Preferable with a testsuite.) Right now, all your question says is that you want a method that always returns 'abc.google.com'. That's easy:
def extract_domain
return 'abc.google.com'
end
But that's probably not what you meant …
Also, you say that you need a Regexp. Why? What's wrong with, for example, using the URI class? After all, parsing and manipulating URIs is exactly what it was made for!
require 'uri'
URI.parse('https://abc.google.com/').host # => 'abc.google.com'
And lastly, you say you are "trying to extract a domain", but you never specify what you mean by "domain". It looks you are sometimes meaning the FQDN and sometimes randomly dropping parts of the FQDN, but according to what rules? For example, for the FQDN abc.google.com, the domain name is google.com and the host name is abc, but you want it to return abc.google.com which is not just the domain name but the full FQDN. Why?
Is there an api in windows that retrieves the server name from a UNC path ? (\\server\share)
Or do i need to make my own ?
I found PathStripToRoot but it doesn't do the trick.
I don't know of a Win32 API for parsing a UNC path; however you should check for:
\\computername\share
\\?\UNC\computername\share (people use this to access long paths > 260 chars)
You can optionally also handle this case: smb://computername/share and this case hostname:/directorypath/resource
Read here for more information
This is untested, but maybe a combination of PathIsUNC() and PathFindNextComponent() would do the trick.
I don't know if there is a specific API for this, I would just implement the simple string handling on my own (skip past "\\" or return null, look for next \ or end of string and return that substring) possibly calling PathIsUNC() first
If you'll be receiving the data as plain text you should be able to parse it with a simple regex, not sure what language you use but I tend to use perk for quick searches like this. Supposing you have a large document containing multiple lines containing one path per line you can search on \\'s I.e
m/\\\\([0-9][0-9][0-9]\.(repeat 3 times, of course not recalling ip address requirements you might need to modify the first one for sure) then\\)? To make it optional and include the trailing slash, and finally (.*)\\/ig it's rough but should do the trick, and the path name should be in $2 for use!
I hope that was clear enough!