I am using finagle as rest client. In ClientBuilder I specify a range of hosts, but the request requires setting a url with host.
How can I avoid specifying host in the request and let finagle choose one ?
Thanks.
val client = ClientBuilder().hosts("host1:81,host2:82").codec(Http()).build()
val request = RequestBuilder()
// .url("http://host1/get") // dont want to specify host
// .url("/get") // MalformedURLException: no protocol
.buildGet()
var resp = client(request) // sent to host specified by url
It looks like you're using finagle-http module. It's not possible to build request without host in URL using RequestBuilder. Still, you can construct Request manually (or create your own RequestBuilder for further use)
I'd recommend however switching to finagle-httpx module (https://github.com/twitter/finagle/tree/develop/finagle-httpx). It's not compatible with finagle-http, but it has lots of API improvements and the ability to create requests without host in URL in among them, for example:
val client = Httpx.client.withTls("my.api")
.newService("host1.my.api:443,host2.my.api:443")
val req = Request("/get")
val rep = client(req)
Related
I'm trying to make a web server in Rust for a simple browser game. I want the server to be able to deliver pages through HTTPS, but also be able to communicate through WebSockets. I'm planning to put this server on Heroku, but since they only allow one port per application I have to make the WebSocket server operate on the same port as the other HTTPS code.
It seems like this is possible with crates like rust-websocket, but that crate uses an outdated version of hyper and seems to be no longer maintained. The crate tokio_tungstenite is much more up to date.
The problem is that both hyper and tungstenite have their own implementation of the HTTP protocol that WebSockets operate over with no way to convert between the two. This means that once an HTTPS request has been parsed by either hyper or tungstenite there is no way to continue the processing by the other part, so you can't really try to connect the WebSocket and match on an error in tungstenite and process it by hyper, nor can you parse the request by hyper and check if it's a WebSocket request and send it over to tungstenite. Is there any way to resolve this problem?
I think it should be possible to do that, the tungstenite and tokio-tungstenite allow you to specify custom headers (there are helpers functions for that, prefixed with hdr), so depending on the hyper version you use, if you can convert a request to some form, when the headers can be extracted, you can pass them to tungstenite.
You might also want to try warp crate, it's built on top of hyper and it uses tungstenite under the hood for the websocket support, so if you want to write your own version of warp, you can take a look at the source code (the source code may contain hints on how to use hyper and tungstenite together).
You can do it, but it's quite fiddly. You'll have to use tokio-tungstenite, do the handshake yourself (check header, set response headers) and spawn a new future on the runtime that will handle the websockets connection. The new future can be created by calling on_upgrade() on the request body with the latest version of hyper, and the connection can then be passed to tokio_tungstenite::WebSocketStream::from_raw_socket to turn it into a websockets connection.
Example handler (note that this doesn't fully check the request headers and assumes we want an upgrade):
fn websocket(req: Request<Body>) -> Result<Response<Body>, &'static str> {
// TODO check other header
let key = match req.headers().typed_get::<headers::SecWebsocketKey>() {
Some(key) => key,
None => return Err("failed to read ws key from headers"),
};
let websocket_future = req
.into_body()
.on_upgrade()
.map_err(|err| eprintln!("Error on upgrade: {}", err))
.and_then(|upgraded| {
let ws_stream = tokio_tungstenite::WebSocketStream::from_raw_socket(
upgraded,
tokio_tungstenite::tungstenite::protocol::Role::Server,
None,
);
let (sink, stream) = ws_stream.split();
sink.send_all(stream)
.map(|_| ())
.map_err(|err| error!("{}", err))
});
hyper::rt::spawn(websocket_future);
let mut upgrade_rsp = Response::builder()
.status(StatusCode::SWITCHING_PROTOCOLS)
.body(Body::empty())
.unwrap();
upgrade_rsp
.headers_mut()
.typed_insert(headers::Upgrade::websocket());
upgrade_rsp
.headers_mut()
.typed_insert(headers::Connection::upgrade());
upgrade_rsp
.headers_mut()
.typed_insert(headers::SecWebsocketAccept::from(key));
Ok(upgrade_rsp)
}
I'm trying to implement http handler for handling file upload by wp background transfers. I've tried this:
var request = new BackgroundTransferRequest(#"http://computerName/test.ashx")
{
Method = "POST",
TransferPreferences = TransferPreferences.None,
UploadLocation = new Uri(#"shared/transfers/testfile.txt", UriKind.RelativeOrAbsolute)
};
In this case my phone always sends Range:0- . ContentLength equals actual source file size. And request stream contains all data... I did not know how to make it sending data partially.
And, I can not find any actual info about how uploading works, what headers it uses and so on. There is no any specification for server!
Sadly, BackgroundTransferRequests do not support range for upload or download. If you don't need to allow transfers when your app is not running, I would suggest writing your own transfer code. Then you can support RANGE and you can control the number of concurrent transfers(and you can get around the 2 transfer limit for the phone) and you don't have to deal with the various file size/network type limitations.
Below is the documentation explaining this from the following link:
http://msdn.microsoft.com/en-us/library/windowsphone/develop/hh202955%28v=vs.105%29.aspx#BKMK_BackgroundTransferPolicies
The Headers property of the BackgroundTransferRequest object is used to set the HTTP headers for a transfer request. The following headers are reserved for use by the system and cannot be used by calling applications. Adding one of the following headers to the Headers collection will cause a NotSupportedException to be thrown when the Add(BackgroundTransferRequest) method is used to queue the transfer request:
•
If-Modified-Since
•
If-None-Match
•
If-Range
•
Range
•
Unless-Modified-Since
I am implementing an API with the following basic structure:
Run serverA
Client connects to serverA and sends data
ServerA processes the data and sends it on to serverB
ServerB replies to serverA
ServerA responds to client's request based on the input received from serverB
So far I have tried two solutions:
1) Create a standard non-twisted TCP connection using httplib to process the request from serverA to serverB. This however effectively blocks the server for the duration of the httplib call.
2) Create a second class inheriting from protocol.Protocol and use
factory = protocol.ClientFactory()
factory.protocol = Authenticate
reactor.connectSSL("localhost",31337,factory, ssl.ClientContextFactory())
to create the connection between serverA and serverB. However when doing this, I don't seem to be able to access the original client-to-serverA connection from within the callbacks of the request class.
What would be the correct way to handle such a setting in Twisted?
The "client-to-serverA" connection is represented by a protocol instance associated with a transport. These are both regular old Python objects, so you can do things like pass them as arguments to functions or class initializers, or set them as attributes on other objects.
For example, if you have ClientToServerAProtocol with a fetchServerBData method which is invoked in response to some bytes being received from the client, you might write it something like this:
class ClientToServerAProtocol(Protocol):
...
def fetchServerBData(self, anArg):
factory = protocol.ClientFactory()
factory.protocol = Authenticate
factory.clientToServerAProtocol = self
reactor.connectSSL("localhost",31337, factory, ssl.ClientContextFactory())
Since ClientFactory sets itself as the factory attribute on any protocol it creates, the Authenticate instance which will result from this will be able to say `self.factory.clientToServerAProtocol and get a reference to that "client-to-serverA" connection.
There are lots of variations on this approach. Here's another one, using the more recently introduced endpoint API:
from twisted.internet.endpoints import SSL4ClientEndpoint
class ClientToServerAProtocol(Protocol):
...
def fetchServerBData(self, anArg):
e = SSL4ClientEndpoint(reactor, "localhost", 31337, ssl.ClientContextFactory())
factory = protocol.Factory()
factory.protocol = Authenticate
connectDeferred = e.connect(factory)
def connected(authProto):
authProto.doSomethingWith(self)
connectDeferred.addCallback(connected)
Same basic idea here - use self to give a reference to the "client-to-serverA" connection you're interested in to the Authenticate protocol. Here, I used a nested function to ''close over'' self. That's just another of the many options you have for getting references into the right part of your program.
I am building an extension with Firefox's ADD ON SDK (v1.9) that will be able to read all HTTP requests / responses and calculate the time they took to load. This includes not only the main frame but any other loading file (sub frame, script, css, image, etc.).
So far, I am able to use the "observer-service" module to listen for:
"http-on-modify-request" when a HTTP request is created.
"http-on-examine-response" when a HTTP response is received
"http-on-examine-cached-response" when a HTTP response is received entirely from cache
"http-on-examine-merged-response" when a HTTP response is received partially from cache
My application follows the following sequence:
A request is created and registered through the observer.
I save the current time and mark it as start_time of the request load.
A response for a request is received and registered through one of the observers.
I save the current time and use the previously saved time to calculate load time of the request.
Problem:
I am not able to link the start and end times of the load since I cannot find a request ID (or other unique value) that will tie the request with the response.
I am currently using the URL of the request / response to tie them together but this is not correct since it will raise a "race condition" if two or more equal urls are loading at the same time. Google Chrome solves this issue by providing unique requestIds, but I have not been able to find a similar functionality on Firefox.
I am aware of two ways to recognize a channel that you receive in this observer. The "old" solution is to use nsIWritablePropertyBag interface to attach data to the channel:
var {Ci} = require("chrome");
var channelId = 0;
...
// Attach channel ID to a channel
if (channel instanceof Ci.nsIWritablePropertyBag)
channel.setProperty("myExtension-channelId", ++channelId);
...
// Read out channel ID for a channel
if (channel instanceof Ci.nsIPropertyBag)
console.log(channel.getProperty("myExtension-channelId"));
The other solution would be using WeakMap API (only works properly starting with Firefox 13):
var channelMap = new WeakMap();
var channelId = 0;
...
// Attach channel ID to a channel
channelMap.set(channel, ++channelId);
...
// Read out channel ID for a channel
console.log(channelMap.get(channel));
I'm not sure whether WeakMap is available in the context of Add-on SDK modules, you might have to "steal" it from a regular JavaScript module:
var {Cu} = require("chrome");
var {WeakMap} = Cu.import("resource://gre/modules/FileUtils.jsm", null);
Obviously, in both cases you can attach more data to the channel than a simple number.
Firebug does what you're thinking of by implementing a central observer for these events:
https://github.com/firebug/firebug/blob/master/extension/modules/firebug-http-observer.js
This might be a good place to start, although eventually Firefox will ship a more complete network monitor / debugger by default. I think I read somewhere that it will be based on Firebug's.
How can I check which platform my app runs, AWS EC2 instance, Azure Role instance and non-cloud system?
now I do that like this:
if(isAzure())
{
//run in Azure role instance
}
else if(isAWS())
{
//run in AWS EC2 instance
}
else
{
//run in the non-cloud system
}
//checked whether it runs in AWS EC2 instance or not.
bool isAWS()
{
string url = "http://instance-data";
try
{
WebRequest req = WebRequest.Create(url);
req.GetResponse();
return true;
}
catch
{
return false;
}
}
but I have one problem when my apps runs in the non-cloud system, like local windows system. It got very slowly while executing isAWS() method. the code 'req.GetResponse()' takes a long time. so I want to know how can I to deal with it? please help me! thanks in advance.
The better way to do this would be to make a request to get instance metadata.
From the AWS Documentation:
To view all categories of instance metadata from within a running
instance, use the following URI:
http://169.254.169.254/latest/meta-data/
On a Linux instance, you can use a tool such as cURL, or use the GET
command, for example:
PROMPT> GET http://169.254.169.254/latest/meta-data/
Here's an example using the Python Boto wrapper:
from boto.utils import get_instance_metadata
m = get_instance_metadata()
if len(m.keys()) > 0:
print "Running on EC2"
else:
print "Not running on EC2"
I think your original idea is pretty good, but no need to make the web request. Simply try to see if the name resolves (in python):
def is_ec2():
import socket
try:
socket.gethostbyname('instance-data.ec2.internal.')
return True
except socket.gaierror:
return False
As you said the WebRequest.Create() call is slow on your desktop so you really need to check the network traffic (using Netmon) to actually determine what took long time. This request, opens connection, connects to target server, downloads the content and then close the connection so it is good to know where this time is taken.
Also if you just want to know if any URL (on Azure, on EC2 or any other web server is live and working fine you can just request to only download headers by using
string URI = "http://www.microsoft.com";
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(URI);
req.Method = WebRequestMethods.Http.Head;
var response = req.GetResponse();
int TotalSize = Int32.Parse(response.Headers["Content-Length"]);
// Now you can parse the headers for 200 OK and know that it is working.
You can also use GET only a range of the data instead of full data to expedite to call:
HttpWebRequest myHttpWebReq =(HttpWebRequest)WebRequest.Create("http://www.contoso.com");
myHttpWebReq.AddRange(-200, ContentLength); // return first 0-200 bytes
//Now you can send the request and then parse date for headers for 200 OK
Any of the above method will be faster to get where your site is running.
On ec2 Ubuntu instances, the file /sys/hypervisor/uuid exists and its first three characters are 'ec2'. I like using this because it doesn't rely on external servers.