I am trying to get an image from the response body. Right now this gives me the entire HTML page. I see the tag but cannot specifically retrieve it. Any help would be great!
#Get Request
encoded_response = response.body.force_encoding("UTF-8")
url = URI.parse(encoded_response)
req = Net::HTTP::Get.new(url.to_s)
res = Net::HTTP.start(url.host, url.port) {|http|
http.request(req)
}
puts res.img
For those about to ask, I had to encode the response because I was getting a Bad URI errorIP
Have you looked at a parsing library like Nokogiri?
html = Nokogiri::HTML.parse(response.body.force_encoding("UTF-8"))
image_urls = html.css('img').map { |image_tag| image_tag["src"] }
For "downloading" the image, see here: Download an image from a URL?
Related
On the site "example.net" I have standart api method to take access token (which i can call /Token) and if I make POST request using Fiddler to example.net/Token with parameters in request body
And all is OK. Status code 200 and in the response access token and other info.
But if I do this request from other site using RestSharp - 500 Internal Server Error. I tried to AddParameter, AddBody, AddObject. Make parameters as a JSON string, to change DataFormat, to AddHeader of Content-Type. This is my last version of request.
request = new RestRequest(URL, Method.POST);
//request.AddHeader("Content-Type", ContentType);
string UrlEncoded = "";
//I parse Parameters to format like I use in request body in Fiddler.
if (Parameters.Count != 0)
foreach (var param in Parameters)
UrlEncoded = param.ParamToUrlEncoded(UrlEncoded);
request.AddBody(UrlEncoded);
IRestResponse response = client.Execute(request);
var content = response.Content;
Do i need to set any more attributes on the request or something?
Thank you.
I am working on unity webgl build on our existing IOS running app. All the data about users are saving on Parse. I have implemented Rest API to communicate with Parse. Implemented Get and Post(without passing the method header) they are working fine but when i am trying to update the data using PUT :
string url = "https://api.parse.com/1/"
string ObjectID = "ERd99Q0kmd"
string CallLink = url + "classes/PlayerProfile/" + ObjectID ;
string jsonString = "{\"TotalCoins\":40}";
WWWForm form = new WWWForm();
var headers = form.headers;
headers["X-Parse-Application-Id"] = appID;
headers["X-Parse-REST-API-Key"] = restapikey;
headers["Content-Type"] = "application/json";
headers["Content-Length"] = jsonString.Length.ToString();
var encoding = new System.Text.UTF8Encoding();
WWW www = new WWW(CallLink,encoding.GetBytes(jsonString),headers);
yield return www;
if (www.error != null)
{
Debug.Log( "CallGet:Error:"+www.error);
}
else
{
Debug.Log("CallGet:Success:"+www.text);
}
It gives Bad Request error. I also tried the header "Method" it also give Bad Request but when i tried "X-HTTP-Method-Override" it works in unity editor but still it doesn't working in Browser and getting following Error :
Request header field X-HTTP-Method-Override is not allowed by
Access-Control-Allow-Headers in preflight response.
Please help me out how can i update the data.
I'm trying to find a button on a webpage and click it. Here's a script I tried to make for this:
IfWinExist, Google - Mozilla Firefox
WinActivate
ImageSearch, Foundx, Foundy, 18, 69, 371, 328, C:\users\bob\desktop\google.png
if ErrorLevel
MsgBox, Image not found.
else,
MouseMove
This isn't my actual script obviously, but it's the same commands. I want a script to locate an image on a page, move the mouse to the center of the image, and click. My problem with my script is that I can't save the coordinates of the found image and move the mouse to it.
You need to establish whether the search form makes a POST or GET request. A GET request means that values are passed in the querystring. You can see this with Google. All you need to do is formulate your own querystring to include the search word and make an HttpWebRequest using that. If it is a POST request, you need to make a slightly different type of HttpWebRequest. which passes values in the Form collection rather than the QueryString.
Here's an article that makes basic use of the GET request: http://www.mikesdotnetting.com/Article/49/How-to-read-a-remote-web-page-with-ASP.NET-2.0. A method ofr the Form request is below:
public static string HttpPostRequest(string url, string post)
{
var encoding = new ASCIIEncoding();
byte[] data = encoding.GetBytes(post);
WebRequest request = WebRequest.Create(url);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = data.Length;
Stream stream = request.GetRequestStream();
stream.Write(data, 0, data.Length);
stream.Close();
WebResponse response = request.GetResponse();
String result;
using (var sr = new StreamReader(response.GetResponseStream()))
{
result = sr.ReadToEnd();
sr.Close();
}
return result;
}
answer from "Mikesdotnetting"
in : http://forums.asp.net/t/1495798.aspx/1
I need to extract the HTML of a web page
I'm using HTTPuilder in groovy, making the following get:
def http = new HTTPBuilder('http://www.google.com/search')
http.request(Method.GET) {
requestContentType = ContentType.HTML
response.success = { resp, reader ->
println "resp: " + resp
println "READER: " + reader
}
response.failure = { resp, reader ->
println "Failure"
}
}
The response I get, does not contain the same html I can see when I explore the html source of www.google.com/search. In fact, it's neither an html, and does not contains the same info I can see in the html source of the page.
I've tried setting differents headers (for example, headers.Accept = 'text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8', headers.Accept = 'text/html', seting the user-agent, etc), but the result is the same.
How can I get the html of www.google.com/search (or any web page) using http builder?
Why use httpBuilder? You might instead use
def url = "http://www.google.com/".toURL()
println url.text`
to extract the content of the webpage
Because the httpbuilder will auto parse the result by the content type.
to get the raw html, try to get text from Entity
def htmlResult = http.get(uri: url, contentType: TEXT){ resp->
return resp.getEntity().getContent().getText()
}
I am trying to use Ruby Sinatra to create a simple proxy for a specific web page. I can do it in C#, I just can't seem to work it out for Sinatra, the C# code is below:
<%# WebHandler Language="C#" Class="Map" %>
using System;
using System.Web;
using System.Net;
using System.IO;
public class Map : IHttpHandler {
static void CopyStream(Stream input, Stream output)
{
byte[] buffer = new byte[0x1000];
int read;
while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
output.Write(buffer, 0, read);
}
public void ProcessRequest(HttpContext context)
{
string gmapUri = string.Format("http://maps.google.com/maps/api/staticmap{0}", context.Request.Url.Query);
WebRequest request = WebRequest.Create(gmapUri);
using (WebResponse response = request.GetResponse())
{
context.Response.ContentType = response.ContentType;
Stream responseStream = response.GetResponseStream();
CopyStream(responseStream, context.Response.OutputStream);
}
}
public bool IsReusable {
get {
return false;
}
}
}
The Ruby Sinatra code I have tried is as follows:
require 'rubygems'
require 'sinatra'
get '/mapsproxy/staticmap' do
request.path_info = 'http://maps.google.com/maps/api/staticmap'
pass
end
I am assuming that the Sinatra one does not work (get a 404) as is is only passing the request to pages in the same domain. Any hep would be greatly appreciated.
EDIT:
With the Tin Man's help I've come up with a nice succinct solution, which works well for me:
get '/proxy/path' do
URI.parse(<URI> + request.query_string.gsub("|", "%7C")).read
end
Thanks for all the help.
If you want your Sinatra app to retrieve the URL, you'll need to fire up a HTTP client of some sort:
get '/mapsproxy/staticmap' do
require 'open-uri'
open('http://maps.google.com/maps/api/staticmap').read
end
I think this will work and is about as minimal as you can get.
You could use HTTPClient if you need more tweakability.
Also, I think that Rack can do it. Sinatra is built on top of Rack, but it's been a while since I played at that level.
I still need to find a way to extract the contentType from the response
From the Open-URI docs:
The opened file has several methods for meta information as follows since
it is extended by OpenURI::Meta.
open("http://www.ruby-lang.org/en") {|f|
f.each_line {|line| p line}
p f.base_uri # <URI::HTTP:0x40e6ef2 URL:http://www.ruby-lang.org/en/>
p f.content_type # "text/html"
p f.charset # "iso-8859-1"
p f.content_encoding # []
p f.last_modified # Thu Dec 05 02:45:02 UTC 2002
}
For your purposes something like this should work:
content_type = ''
body = open("http://www.ruby-lang.org/en") {|f|
content_type = f.content_type # "text/html"
f.read
}
I haven't tested that, but I think the return value of the block will be assigned to body. If that doesn't work then try:
content_type = ''
body = ''
open("http://www.ruby-lang.org/en") {|f|
content_type = f.content_type # "text/html"
body = f.read
}
but I think the first will work.
With the help of the Tin Man and TK-421 I've worked out a solution, see the Sinatra route below:
get '/proxy/path' do
require 'open-uri'
uri = URI.parse(<URI>)
getresult = uri.read
halt 200, {'Content-Type' => getresult.content_type}, getresult
end
Just replace the <URI> with the page you require, and you're good to go.
After some more playing this is what I've come up with:
get '/proxy/path' do
URI.parse(<URI> + request.query_string.gsub("|", "%7C")).read
end
As mentioned else where you need to require 'open-uri' at the top of the code. The reason for the gsub is that for some reason the parse fails if they are left in, and my browser doesn't encode them automatically.