https://www.iana.org/domains/arpa
I can get following output using the xpath '//table[#id="arpa-table"]/tbody/tr/join((td[1], normalize-space(td[2])), x:cps(9))' with xidel. But I want to put things like RFC 3172 in a 3rd column and /go/rfc3172 in a forth column. Does anybody let me know how you do it?
arpa▸ Reserved exclusively to support operationally-critical infrastructural identifier spaces as advised by the Internet Architecture Board RFC 3172¬
as112.arpa▸ For sinking DNS traffic for reverse IP address lookups and other applications RFC 7535¬
e164.arpa▸ For mapping E.164 numbers to Internet URIs RFC 6116¬
home.arpa▸ For non-unique use in residential home networks RFC 8375¬
in-addr-servers.arpa▸ For hosting authoritative name servers for the in-addr.arpa domain RFC 5855¬
in-addr.arpa▸ For mapping IPv4 addresses to Internet domain names RFC 1035¬
ip6-servers.arpa▸ For hosting authoritative name servers for the ip6.arpa domain RFC 5855¬
ip6.arpa▸ For mapping IPv6 addresses to Internet domain names RFC 3152¬
ipv4only.arpa▸ For detecting the presence of DNS64 and for learning the IPv6 prefix used for protocol translation RFC 7050¬
iris.arpa▸ For locating Internet Registry Information Services RFC 4698¬
uri.arpa▸ For resolving Uniform Resource Identifiers according to the Dynamic Delegation Discovery System RFC 3405 RFC 8958¬
urn.arpa▸ For resolving Uniform Resource Names according to the Dynamic Delegation Discovery System RFC 3405¬
The first row should be something like
arpa▸ Reserved exclusively to support operationally-critical infrastructural identifier spaces as advised by the Internet Architecture Board▸ RFC 3172¬
By default xidel prints the node/element its string-value (string()). It's "the concatenation of the string-values of all its descendant text nodes", as E. Lenz puts it:
$ xidel -s https://www.iana.org/domains/arpa -e '
//table[#id="arpa-table"]/tbody/tr[1]/td[2] ! (position(),.)
'
#or
$ xidel -s https://www.iana.org/domains/arpa -e '
//table[#id="arpa-table"]/tbody/tr[1]/td[2]/string() ! (position(),.)
'
1
Reserved exclusively to support operationally-critical infrastructural identifier spaces as advised by the Internet Architecture Board
RFC 3172
As you can see, 1 item/node.
That's why normalize-space(td[2]) returns Reserved exclusively [...] RFC 3172.
With text() on the other hand you'll get the node/element its direct text-nodes:
$ xidel -s https://www.iana.org/domains/arpa -e '
//table[#id="arpa-table"]/tbody/tr[1]/td[2]/text() ! (position(),.)
'
1
Reserved exclusively to support operationally-critical infrastructural identifier spaces as advised by the Internet Architecture Board
2
3
Or all of its descendant text-nodes:
$ xidel -s https://www.iana.org/domains/arpa -e '
//table[#id="arpa-table"]/tbody/tr[1]/td[2]//text() ! (position(),.)
'
1
Reserved exclusively to support operationally-critical infrastructural identifier spaces as advised by the Internet Architecture Board
2
3
RFC 3172
4
As you can see, 3 and 4 different items/nodes.
To get the 1st text-node, simply td[2]/text()[1] would do, but normalize-space(td[2]/text()) and even normalize-space(td[2]//text()) would work too.
So I am attempting to solve a parsing issue I am faced with...
I have a log file which contains pseudorandom strings at the start of each line. They can be anything but will always be the same length, which is the first data set I wish to keep (E.G start-of-log-001 - ) made up of 19 characters, including the space.
start-of-log-001 - bacon bacon email1#email.com bacon metadata123 bacon 123 bacon
start-of-log-002 - bacon bacon metadata123 bacon email2#email.com bacon 123 bacon
start-of-log-010 - bacon bacon metadata123 bacon bacon bacon 123 bacon email3#email.com
The second part of the data set I want to extract is the email addresses, but that is proving a little more tricky because they can be anywhere within the rest of the string. My aim is to keep each lines 'start-of-file' value and its associated email on the same line as they relate to each other, but to remove anything else. I am currently using the following Regex (I understand there is a small possibility it won't catch every email).
[[:alnum:]+\.\_\-]*#[[:alnum:]+\.\_\-]
So that regex should pull out the emails but I need a way to run through each of these lines, parse them to keep the first 19 characters, and then remove all data apart from the email for that line, so it will result in the following.
start-of-log-001 - email1#email.com
start-of-log-002 - email2#email.com
start-of-log-010 - email3#email.com
I have looked into regex positive lookbehinds and sed strings but cannot get anything to work thus far.
Any help would be much appreciated.
The specification of what a makes up a valid email address is quite complex (see RFC 822), so instead of trying to do that, we can just say "the email address is the longest match of non-blanks around the # sign".
This command extracts the first 19 characters and then the aforementioned longest match of non-blanks:
$ sed -E 's/^(.{19}).*[[:blank:]]([^[:blank:]]*#[^[:blank:]]*).*/\1\2/' infile
start-of-log-001 - email1#email.com
start-of-log-002 - email2#email.com
start-of-log-010 - email3#email.com
-E (extended regular expressions) is just for convenience; without it, we have to escape () and {} to \(\) and \{\}.
^(.{19}) matches and captures the first 19 characters per line.
[[:blank:]]([^[:blank:]]*#[^[:blank:]]*) is a blank, then the # sign surrounded by non-blanks. The first blank is required because the greedy matching would otherwise match .* up to the # sign.
The substitution then just places the two captured groups next to each other.
Side note: within bracket expressions, nothing has to be escaped, so yours could just be [[:alnum:]+._-].
sed -r 's/^(.{19}).*? ([^#]+#[^ ]*).*/\1\2/'
RFC 3986 specifies that the host component of a URI is 'case insensitive'. However, it doesn't specify what 'case insensitive' means in terms of UCS or UTF-8 characters.
Examples given in the RFC (e.g. "<HTTP://www.EXAMPLE.com/> is equivalent to <http://www.example.com/>") allow us to infer that 'case insensitive' means at least that the characters A-Z are considered equivalent to the character 32 ahead of them in the UTF-8 character set, i.e. a-z. However, no mention is made of how characters outside this range should be treated. So, given an non-encoded, non-normalised registered name of www.OLÉ.com, I see three potential forms of normalisation permissible by the RFC:
Lower case to www.olé.com then percent encode to www.ol%E9.com
Lower case only A-Z characters to www.olÉ.com and then percent encode to www.ol%C9.com
Percent encode to www.OL%C9.com, and then lower case the non-percent encoded parts to www.ol%C9.com, producing the same result as 2.
So the question is: Which is correct? If it's case 1., what defines which characters are considered upper case, and which are considered lower case (and which characters don't have a case)?
Hostnames resolved by DNS are always lowercase.
It is not possible to have UTF-8 characters in DNS hostnames (RFC 1123), however, a workaround has been put in place with "internationalized domain names". This workaround is commonly known as punycode.
Punycode enables non ASCII characters to be represented by ASCII characters.
non-ASCII characters are represented by ASCII characters that are allowed in host name labels (letters, digits, and hyphens).
-- https://www.ietf.org/rfc/rfc3492.txt
As for the example that you have provided in your question (www.olé.com), the domain name that would be resolved is not www.ol%E9.com.
If you are getting percentage signs in your domain name, it means that you have URL-encoded the hostname, and that is not correct, at least not for resolving.
For example, it will work correctly to have an a tag that looks like this:
Click Here
However, the DNS server will not resolve www.ol%C3%A9.com, but rather, the converted domain name as punycode:
Example
www.ol%C3%A9.com
becomes
www.olé.com
which in punycode translates to:
www.xn--ol-cja.com
Web browsers will generally convert uppercase characters to the lowercase version. For example, both www.olé.com and www.olÉ.com translate to the same DNS hostname (www.xn--ol-cja.com), because www.olÉ.com was lowercased to www.olé.com.
I recommend two tools to check IDN domain names to see what a domain name looks like once it goes through the punycode translation:
Verisign's IDN Conversion Tool (http://mct.verisign-grs.com/)
Punycoder Punycode to Text/Unicode https://www.punycoder.com/
Verisign's IDN tool is much stricter. Try both tools with www.olÉ.com as the input to see what I mean.
The rules for IDNA (Internationalized Domain Names for Applications) are complicated, but there are two main RFC's that are worth a look at:
Internationalized Domain Names for Applications (IDNA): Background, Explanation, and Rationalehttps://www.rfc-editor.org/rfc/rfc5894
The Unicode Code Points and Internationalized Domain Names for Applications https://www.rfc-editor.org/rfc/rfc5892
rfc5894 section 3.1.3 specifies that characters may not be allowed if:
The character is an uppercase form or some other form that is
mapped to another character by Unicode case folding.
I'm specifically looking for the minimum length of the prefix and domain.
I've seen conflicting information and nothing that looks authoritative.
For reference, I found this page which claims that a one character email address is functional:
http://www.cjvandyk.com/blog/Lists/Posts/Post.aspx?ID=176
I tried validating email addresses at Gmail and they expect prefix greater than or equal to 6.
These are obviously way off.
My web framework expects prefix greater than or equal to 2.
The shortest valid email address may consist of only two parts: name and domain.
name#domain
Since both the name and domain may have the length of 1 character, the minimal total length resolves to 3 characters.
well the problem is really the question.. email depends on if it is sent over the internet, or within a closed system (eg intranet). over the internet, I believe x#y.zz is the shortest email possible (e.g. google's G.CN for china would result in the shortest email adress possible, e.g. i#g.cn, which is 6 characters long). on the intranet however, it is an entirely different thing, and i#y would be possible, which is just 3 characters long.
I believe the standard you are looking for is RFC 2822 - Internet Message Format
More specific info on email address restrictions in RFC 3696 - Section 3
To quote the spec:
Contemporary email addresses consist of a "local part" separated from a "domain part" (a fully-qualified domain name) by an at-sign ("#").
So three characters is the shortest.
I originally got this info from Phil Haack's blog post.
Many mail-servers will not accept the email-address if there aren't at least 2 characters before the #.
That doesn't make it an invalid address, but if the servers don't know that, it sure can lead to a lot of problems.
Which characters make a URL invalid?
Are these valid URLs?
example.com/file[/].html
http://example.com/file[/].html
In general URIs as defined by RFC 3986 (see Section 2: Characters) may contain any of the following 84 characters:
ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~:/?#[]#!$&'()*+,;=
Note that this list doesn't state where in the URI these characters may occur.
Any other character needs to be encoded with the percent-encoding (%hh). Each part of the URI has further restrictions about what characters need to be represented by an percent-encoded word.
The '[' and ']' in this example are "unwise" characters but still legal. If the '/' in the []'s is meant to be part of file name then it is invalid since '/' is reserved and should be properly encoded:
http://example.com/file[/].html
To add some clarification and directly address the question above, there are several classes of characters that cause problems for URLs and URIs.
There are some characters that are disallowed and should never appear in a URL/URI, reserved characters (described below), and other characters that may cause problems in some cases, but are marked as "unwise" or "unsafe". Explanations for why the characters are restricted are clearly spelled out in RFC-1738 (URLs) and RFC-2396 (URIs). Note the newer RFC-3986 (update to RFC-1738) defines the construction of what characters are allowed in a given context but the older spec offers a simpler and more general description of which characters are not allowed with the following rules.
Excluded US-ASCII Characters disallowed within the URI syntax:
control = <US-ASCII coded characters 00-1F and 7F hexadecimal>
space = <US-ASCII coded character 20 hexadecimal>
delims = "<" | ">" | "#" | "%" | <">
The character "#" is excluded because it is used to delimit a URI from a fragment identifier. The percent character "%" is excluded because it is used for the encoding of escaped characters. In other words, the "#" and "%" are reserved characters that must be used in a specific context.
List of unwise characters are allowed but may cause problems:
unwise = "{" | "}" | "|" | "\" | "^" | "[" | "]" | "`"
Characters that are reserved within a query component and/or have special meaning within a URI/URL:
reserved = ";" | "/" | "?" | ":" | "#" | "&" | "=" | "+" | "$" | ","
The "reserved" syntax class above refers to those characters that are allowed within a URI, but which may not be allowed within a particular component of the generic URI syntax. Characters in the "reserved" set are not reserved in all contexts. The hostname, for example, can contain an optional username so it could be something like ftp://user#hostname/ where the '#' character has special meaning.
Here is an example of a URL that has invalid and unwise characters (e.g. '$', '[', ']') and should be properly encoded:
http://mw1.google.com/mw-earth-vectordb/kml-samples/gp/seattle/gigapxl/$[level]/r$[y]_c$[x].jpg
Some of the character restrictions for URIs and URLs are programming language-dependent. For example, the '|' (0x7C) character although only marked as "unwise" in the URI spec will throw a URISyntaxException in the Java java.net.URI constructor so a URL like http://api.google.com/q?exp=a|b is not allowed and must be encoded instead as http://api.google.com/q?exp=a%7Cb if using Java with a URI object instance.
Most of the existing answers here are impractical because they totally ignore the real-world usage of addresses like:
https://en.wikipedia.org/wiki/Möbius_strip or
https://zh.wikipedia.org/wiki/Wikipedia:关于中文维基百科/en.
First, a digression into terminology. What are these addresses? Are they valid URLs?
Historically, the answer was "no". According to RFC 3986, from 2005, such addresses are not URIs (and therefore not URLs, since URLs are a type of URIs). Per the terminology of 2005 IETF standards, we should properly call them IRIs (Internationalized Resource Identifiers), as defined in RFC 3987, which are technically not URIs but can be converted to URIs simply by percent-encoding all non-ASCII characters in the IRI.
Per modern spec, the answer is "yes". The WHATWG Living Standard simply classifies everything that would previously be called "URIs" or "IRIs" as "URLs". This aligns the specced terminology with how normal people who haven't read the spec use the word "URL", which was one of the spec's goals.
What characters are allowed under the WHATWG Living Standard?
Per this newer meaning of "URL", what characters are allowed? In many parts of the URL, such as the query string and path, we're allowed to use arbitrary "URL units", which are
URL code points and percent-encoded bytes.
What are "URL code points"?
The URL code points are ASCII alphanumeric, U+0021 (!), U+0024 ($), U+0026 (&), U+0027 ('), U+0028 LEFT PARENTHESIS, U+0029 RIGHT PARENTHESIS, U+002A (*), U+002B (+), U+002C (,), U+002D (-), U+002E (.), U+002F (/), U+003A (:), U+003B (;), U+003D (=), U+003F (?), U+0040 (#), U+005F (_), U+007E (~), and code points in the range U+00A0 to U+10FFFD, inclusive, excluding surrogates and noncharacters.
(Note that the list of "URL code points" doesn't include %, but that %s are allowed in "URL code units" if they're part of a percent-encoding sequence.)
The only place I can spot where the spec permits the use of any character that's not in this set is in the host, where IPv6 addresses are enclosed in [ and ] characters. Everywhere else in the URL, either URL units are allowed or some even more restrictive set of characters.
What characters were allowed under the old RFCs?
For the sake of history, and since it's not explored fully elsewhere in the answers here, let's examine was allowed under the older pair of specs.
First of all, we have two types of RFC 3986 reserved characters:
:/?#[]#, which are part of the generic syntax for a URI defined in RFC 3986
!$&'()*+,;=, which aren't part of the RFC's generic syntax, but are reserved for use as syntactic components of particular URI schemes. For instance, semicolons and commas are used as part of the syntax of data URIs, and & and = are used as part of the ubiquitous ?foo=bar&qux=baz format in query strings (which isn't specified by RFC 3986).
Any of the reserved characters above can be legally used in a URI without encoding, either to serve their syntactic purpose or just as literal characters in data in some places where such use could not be misinterpreted as the character serving its syntactic purpose. (For example, although / has syntactic meaning in a URL, you can use it unencoded in a query string, because it doesn't have meaning in a query string.)
RFC 3986 also specifies some unreserved characters, which can always be used simply to represent data without any encoding:
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-._~
Finally, the % character itself is allowed for percent-encodings.
That leaves only the following ASCII characters that are forbidden from appearing in a URL:
The control characters (chars 0-1F and 7F), including new line, tab, and carriage return.
"<>^`{|}
Every other character from ASCII can legally feature in a URL.
Then RFC 3987 extends that set of unreserved characters with the following unicode character ranges:
%xA0-D7FF / %xF900-FDCF / %xFDF0-FFEF
/ %x10000-1FFFD / %x20000-2FFFD / %x30000-3FFFD
/ %x40000-4FFFD / %x50000-5FFFD / %x60000-6FFFD
/ %x70000-7FFFD / %x80000-8FFFD / %x90000-9FFFD
/ %xA0000-AFFFD / %xB0000-BFFFD / %xC0000-CFFFD
/ %xD0000-DFFFD / %xE1000-EFFFD
These block choices from the old spec seem bizarre and arbitrary given the latest Unicode block definitions; this is probably because the blocks have been added to in the decade since RFC 3987 was written.
Finally, it's perhaps worth noting that simply knowing which characters can legally appear in a URL isn't sufficient to recognise whether some given string is a legal URL or not, since some characters are only legal in particular parts of the URL. For example, the reserved characters [ and ] are legal as part of an IPv6 literal host in a URL like http://[1080::8:800:200C:417A]/foo but aren't legal in any other context, so the OP's example of http://example.com/file[/].html is illegal.
In your supplementary question you asked if www.example.com/file[/].html is a valid URL.
That URL isn't valid because a URL is a type of URI and a valid URI must have a scheme like http: (see RFC 3986).
If you meant to ask if http://www.example.com/file[/].html is a valid URL then the answer is still no because the square bracket characters aren't valid there.
The square bracket characters are reserved for URLs in this format: http://[2001:db8:85a3::8a2e:370:7334]/foo/bar (i.e. an IPv6 literal instead of a host name)
It's worth reading RFC 3986 carefully if you want to understand the issue fully.
All valid characters that can be used in a URI (a URL is a type of URI) are defined in RFC 3986.
All other characters can be used in a URL provided that they are "URL Encoded" first. This involves changing the invalid character for specific "codes" (usually in the form of the percent symbol (%) followed by a hexadecimal number).
This link, HTML URL Encoding Reference, contains a list of the encodings for invalid characters.
Several of Unicode character ranges are valid HTML5, although it might still not be a good idea to use them.
E.g., href docs say http://www.w3.org/TR/html5/links.html#attr-hyperlink-href:
The href attribute on a and area elements must have a value that is a valid URL potentially surrounded by spaces.
Then the definition of "valid URL" points to http://url.spec.whatwg.org/, which says it aims to:
Align RFC 3986 and RFC 3987 with contemporary implementations and obsolete them in the process.
That document defines URL code points as:
ASCII alphanumeric, "!", "$", "&", "'", "(", ")", "*", "+", ",", "-", ".", "/", ":", ";", "=", "?", "#", "_", "~", and code points in the ranges U+00A0 to U+D7FF, U+E000 to U+FDCF, U+FDF0 to U+FFFD, U+10000 to U+1FFFD, U+20000 to U+2FFFD, U+30000 to U+3FFFD, U+40000 to U+4FFFD, U+50000 to U+5FFFD, U+60000 to U+6FFFD, U+70000 to U+7FFFD, U+80000 to U+8FFFD, U+90000 to U+9FFFD, U+A0000 to U+AFFFD, U+B0000 to U+BFFFD, U+C0000 to U+CFFFD, U+D0000 to U+DFFFD, U+E1000 to U+EFFFD, U+F0000 to U+FFFFD, U+100000 to U+10FFFD.
The term "URL code points" is then used in the statement:
If c is not a URL code point and not "%", parse error.
in a several parts of the parsing algorithm, including the schema, authority, relative path, query and fragment states: so basically the entire URL.
Also, the validator http://validator.w3.org/ passes for URLs like "你好", and does not pass for URLs with characters like spaces "a b"
Of course, as mentioned by Stephen C, it is not just about characters but also about context: you have to understand the entire algorithm. But since class "URL code points" is used on key points of the algorithm, it that gives a good idea of what you can use or not.
See also: Unicode characters in URLs
I needed to select characters to split URLs in a string, so I decided to create a list of characters which could not be found in the URL by myself:
>>> allowed = "-_.~!*'();:#&=+$,/?%#[]?#ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"
>>> from string import printable
>>> ''.join(set(printable).difference(set(allowed)))
'`" <\x0b\n\r\x0c\\\t{^}|>'
So, the possible choices are the newline, tab, space, backslash and "<>{}^|. I guess I'll go with the space or newline. :)
I am implementing an old HTTP (0.9, 1.0, 1.1) request and response reader/writer. The request URI is the most problematic place.
You can't just use RFC 1738, 2396 or 3986 as it is. There are many old HTTP clients and servers that allow more characters. So I've made research based on accidentally published web server access logs: "GET URI HTTP/1.0" 200.
I've found that the following non-standard characters are often used in URIs:
\ { } < > | ` ^ "
These characters were described in RFC 1738 as unsafe.
If you want to be compatible with all old HTTP clients and servers - you have to allow these characters in the request URI.
Please read more information about this research in oghttp-request-collector.
This is not really an answer to your question, but validating URLs is really a serious p.i.t.a. You're probably just better off validating the domain name and leave query part of the URL be. That is my experience.
You could also resort to pinging the URL and seeing if it results in a valid response, but that might be too much for such a simple task.
Regular expressions to detect URLs are abundant, google it :)
I can't comment on the above answers, but wanted to emphasize the point (in another answer) that allowed characters aren't allowed everywhere. For example, domain names can't have underscores, so http://test_url.com is invalid.
From the source (emphasis added when needed):
Unsafe:
Characters can be unsafe for a number of reasons. The space character is unsafe because significant spaces may disappear and insignificant spaces may be introduced when URLs are transcribed or typeset or subjected to the treatment of word-processing programs.
The characters "<" and ">" are unsafe because they are used as the
delimiters around URLs in free text; the quote mark (""") is used to
delimit URLs in some systems. The character "#" is unsafe and should
always be encoded because it is used in World Wide Web and in other
systems to delimit a URL from a fragment/anchor identifier that might
follow it. The character "%" is unsafe because it is used for
encodings of other characters. Other characters are unsafe because
gateways and other transport agents are known to sometimes modify such
characters. These characters are "{", "}", "|", "", "^", "~", "[",
"]", and "`".
All unsafe characters must always be encoded within a URL. For
example, the character "#" must be encoded within URLs even in systems
that do not normally deal with fragment or anchor identifiers, so that
if the URL is copied into another system that does use them, it will
not be necessary to change the URL encoding.
Source
I came up with a couple of regular expressions for PHP that will convert URLs in text to anchor tags. (First it converts all www. URLs to http://, and then converts all URLs with https?:// to a href=... HTML links
$string = preg_replace('/(https?:\/\/)([!#$&-;=?\-\[\]_a-z~%]+)/sim', '$2', preg_replace('/(\s)((www\.)([!#$&-;=?\-\[\]_a-z~%]+))/sim', '$1http://$2', $string) );