I'm having a problem doing a request using Typhoeus as my query needs to have quotation marks into it.
If the URl is
url = "http://app.com/method.json?'my_query'"
everything works fine. However, the method I'm trying to run only returns the results I want if the query is the following (i've tested it in browser):
url2 = "http://app.com/method.json?"my_query""
When running
Typhoeus::Request.get(url2)
I get (URI::InvalidURIError)
Escaping quotes with "\" does not work. How can I do this?
Thanks
You should be properly encoding your URI with URI.encode or CGI.escape, doing so will get you proper URLs like this:
http://app.com/method.json?%27my_query%27 # Single quotes
http://app.com/method.json?%22my_query%22 # Double quotes
Try:
require 'uri'
URI.encode('"foo"')
=> "%22foo%22"
Passing json, quotes etc in GET request is tricky. In Ruby 2+ we can use Ruby's URI module's 'escape' method.
> URI.escape('http://app.com/method.json?agent={"account":
{"homePage":"http://demo.my.com","name":"Senior Leadership"}}')
But I suggest use it as POST request and pass it as a message body.
Related
How would one parse a JSON string like this:
"[{\"something\": \"information \"YES\"\", \"next\": \"normal\"}]"
I've used both the json gem and the Oj gem but they both run into errors. I've also tried using eval() on it.
I've also tried using different regexes to target the quotes surrounding YES and replacing them with single quotes but I haven't been successful in figuring one out.
The string you posted isn't valid JSON. This would be the non-stringified JSON:
[{"something":"information \"YES\"","next":"normal"}]
Note that the escaping is still present in the value for something.
If you had this JSON as a string, the double-quote escaping depends on the language you're working in. In Ruby, this is what it looks like:
"[{\"something\":\"information \\\"YES\\\"\",\"next\":\"normal\"}]"
If you use that, you'll be able to parse it just fine:
JSON.parse("[{\"something\":\"information \\\"YES\\\"\",\"next\":\"normal\"}]")
#=> [{"something"=>"information \"YES\"", "next"=>"normal"}]
I have a file containing a bunch of random URLs:
https://google.com
http://fakesite.net/php?id=2
https://example.com/
http://anotherexample.com/pho?admin=2
How can I verify that the URL contains a GET parameter and not a POST using a regular expression?
You mean whether it has query parameters?
require 'uri'
if URI('http://fakesite.net/php').query
...
end
If you insist on using a regexp use this
match = URI.regexp.match('http://fakesite.net/php?id=2')
if match[8]
...
end
You can print the regular expression with
puts URI.regexp
It is looooooong, you best just use the URI class.
I am trying to use wget with a url that includes a "#" sign. No matter what I do to escape the character, it doesn't work. I've used \, ', and ". But none of them work. Does any one have any suggestions?
Thank you!
Send it as %23 if you really mean for it to have a hash. If you're trying to send a fragment, don't bother since the server won't care about it regardless.
maybe put uri around'' ? I believe it works
Are you quoting the url? It shouldn't be a problem if you are.
My guess is you're doing something like:
wget http://foo.com/#!/blah
Instead of:
wget "http://foo.com/#!/blah"
# is the shell script comment character.
I want to match urls in text and replace them with anchor tags, but I want to exclude some terminators just like how Twitter matches urls in tweets.
So far I've got this, but it's obviously not working too well.
(http[s]?\:\/\/[a-zA-Z0-9\-\.]+\.[a-zA-Z]{2,3}(\/\S*)?)
EDIT: Some example urls. In all cases below I only want to match "http://www.example.com"
http://www.example.com.
http://www.example.com:
"http://www.example.com"
http://www.example.com;
http://www.example.com!
[http://www.example.com]
{http://www.example.com}
http://www.example.com*
I looked into this very issue last year and developed a solution that you may want to look at - See: URL Linkification (HTTP/FTP) This link is a test page for the Javascript solution with many examples of difficult-to-linkify URLs.
My regex solution, written for both PHP and Javascript - (but could easily be translated to Ruby) is not simple (but neither is the problem as it turns out.) For more information I would recommend also reading:
The Problem With URLs by Jeff Atwood, and
An Improved Liberal, Accurate Regex Pattern for Matching URLs by John Gruber
The comments following Jeff's blog post are a must read if you want to do this right...
Ruby's URI module has a extract method that is used to parse out URLs from text. Parsing the returned values lets you piggyback on the heuristics in the module to extract the scheme and host information from a URL, avoiding reinventing the wheel.
text = '
http://www.example.com.
http://www.example.com:
"http://www.example.com"
http://www.example.com;
http://www.example.com!
[http://www.example.com]
{http://www.example.com}
http://www.example.com*
http://www.example.com/foo/bar?q=foobar
http://www.example.com:81
'
require 'uri'
puts URI::extract(text).map{ |u| uri = URI.parse(u); "#{ uri.scheme }://#{ uri.host[/(^.+?)\.?$/, 1] }" }
# >> http://www.example.com
# >> http://www.example.com
# >> http://www.example.com
# >> http://www.example.com
# >> http://www.example.com
# >> http://www.example.com
# >> http://www.example.com
# >> http://www.example.com
# >> http://www.example.com
# >> http://www.example.com
The only gotcha, is that a period '.' is a legitimate character in a host name, so URI#host won't strip it. Those get caught in the map statement where the URL is rebuilt. Note that URI is stripping off the path and query information.
A pragmatic and easy understandable solution is:
regex = %r!"(https?://[-.\w]+\.\w{2,6})"!
Some notes:
With %r we can choose the start and end delimiter. In this case I used exclamation mark, since I want to use slash unescaped in the regex.
The optional quantifier (i.e. '?') binds only to the preceding expression, in this case 's'. There's no need to put the 's' in a character class [s]?. It's the same as s?.
Inside the character class [-.\w] we don't need to escape dash and dot in order to make them match dot and dash literally. Dash should be first, however, to not mean range.
\w matches [A-Za-z0-9_] in Ruby. It's not exactly the full definition of URL characters, but combined with dash and dot it may be enough for our needs.
Top domains are between 2 and 6 characters long, e.g. '.se' and '.travel'
I'm not sure what you mean by I want to exclude some terminators but this regex matches only the wanted one in your example.
We want to use the first capture group, e.g. like this:
if input =~ %r!"(https?://[-.\w]+.\w{2,6})"!
match = $~[1]
else
match = ""
end
What about this?
%r|https?://[-\w.]*\w|
I want to let a user of a web app enter a URL and then pass that URL onto curl. I'd rather use curl than Net::HTTP or open-uri. But this poses a security risk. What's the best way to check the URL string and prevent any injection attacks?
I'm thinking of just using a regular expression like this to check for an injection attack:
raise "possible injection attack" if url =~ /[\s']/
and if that check succeeds, then just invoke curl like so
html = `curl '#{url}'`
Is this safe enough?
system("curl", url)
this is safe because parameters are directly passed to the main(argv) of the command rather than being parsed by a command line processor. For example system("echo", "* && echo shenanigan") literally outputs * && echo shenanigan.
Maybe you're better off using a library like libcurl (to avoid having to use shell commands) ?