Is it possible (for example with C++, but it does not really matter) to create a bridge/proxy application to get the data requested by another application? To be more detailed, I'm talking about a Adobe Air based game. (I want to create a report with stats based on the data acquired, but that is not actually part of this question.)
Rather than simple "boolean" answer please provide some link to example/documentation. Thanks
It would always be possible, and depending on the your target operating system, may require a fair amount of effort, which begs the question - is there a reason you cannot use Fiddler or some packet sniffing software for your target OS?
You can write a proxy by hand, in python can be quite easy. All you have to do is to set localhost as proxy, then forward the request and pass it back to the calling socket.
I've started writing something like this some times ago. The idea was to write a simple replacement for dansguardian.
I've uploaded it on github so you can give it a look if it can help.
I do not remember well (I've started writing it the last year) but maybe with some modification can fit well your requests.
Conceptually, this is your configuration:
app_client -> [app_channel] -> proxy -> [server_channel] -> app_server
Your proxy starts a server socket, the app_client connects to it. This is our app_channel. Now your proxy creates a connection to the app_server. This is your server_channel.
Now start 2 threads, one which reads from the app_channel and writes to the server_channel, the other reads from the server_channel and writes to the app_channel.
This will create a transparent connection to the app_server via your proxy. You can extract the data as you wish. If the data is encrypted though, there's very little you can actually do by way of analysis.
Related
I have search high and low for this answer. Can one code up a Named Pipes server where the connection that a client makes is persistent until you close the applications? This would be in C/C++. Not asking any one to actually do this, as I am capable. To explain in a little more detail if my question is not clear, I want to be able to have the client connect to the server and then be able to pass data back and forth without having to kill the connection at the end of each data transaction and then start a new one again for the next. It seams that in every example I have seen or read, the transaction only lasted for that one data exchange. That seems wasteful and extremely time consuming. Then I want to thread it so I can have up to 8 clients on the same named pipe. If you know of example code that does this, that would be great also. Already read the Microsoft examples, and they seem to be single data exchanges with new connections every time.
My confusion lies with the readfile() and writefile() functions. They need the pipe handle and pointers to the data structures just like a file R/W on the hard drive. Those files can be opened at program start, used, and then finally closed just before you exit your application. There are risks to doing this, but sometimes necessary. So I want my server application to be in control not the clients.
Thanks in advance. I will answer any questions if this is not clear to you.
I am not surprised at your answers. I was really wanting a way to keep the connection open per instance, but since this is not how pipes work, I get it. I have devised a better way to make my applications talk to each other. I originally had one server and many clients, so I turned that around and now have one client and many servers each with a different pipe name. Since my client, that was the server, did most of the initiating of the messages, I can now manage better they way I send and request the data via messages/pipes. The only draw back is not giving the data to all of them at once, but what is a few microseconds amount friends. Please let me know if this will not work as I expect it will, before I spend a lot of time on the code. Thanks. Any suggestions are welcomed.
Problem
I'm making a game where you would provide a piece of code to represent the agent program of an Intelligent Agent (think Robocode and the like), but browser-based. Being an AI/ML guy for the most part, my knowledge of web development was/is pretty lacking, so I'm having a bit of a trouble implementing the whole architecture. Basically, after the upload of text (code), naturally part of the client-side, the backend would be responsible for running the core logics and returning JSON data that would be parsed and used by the client mainly for the drawing part. There isn't really a need for multiplayer support right now.
If I model after Robocode's execution loop, I would need a separate process for each battle that then assigns different agents (user-made or not) to different threads and gives them some execution time for each loop, generating new information to be given to the agents as well as data for drawing the whole scene. I've tried to think of a good way to structure the multiple clients, servers/web servers/processes [...], and came to multiple possible solutions.
Favored solution (as of right now)
Clients communicate with a Node.js server that works kinda like an interface (think websocketd) for unique processes running on the same (server) machine, keeping track of client and process via ID and forwarding the data (via webSockets) accordingly. So an example scenario would be:
Client C1 requests new battle to server S and sends code (not necessarily a single step, I know);
S handles the code (e.g. compiling), executes new battle and starts a connection with it's process P1 (named pipes/FIFO?);
P1 generates JSON, sends to S;
S sees P1 is "connected" to C1, sends data to C1 (steps 3 and 4 will be repeated as long as the battle is active);
Client C2 requests new battle;
Previous steps repeated; C2 is assigned to new process P2;
Client C3 requests "watching" battle under P1 (using a unique URL or a token);
S finds P1's ID, compares to the received one and binds P1 to C3;
This way, the Server forwards received data from forked processes to all clients connected to each specific Battle.
Questions
Regarding this approach:
Is it simple enough? Are there easier or even more elegant ways of doing it? Could scalability be a problem?
Is it secure enough (the whole compiling and running code — likely C++ — on the server)?
Is it fast enough (this one worries me the most for now)? It seems a bit counter intuitive to have a single server dealing with the entire traffic, but as far as I know, if I'd assign all these processes to a separate web server, I would need different ports for each of them, which seems even worse.
Since this is a theoretical and opinion based question... I feel free to throwing the ball in different directions. I'll probably edit the answer as I think things over or read comments.
A process per battle?
sounds expensive. Also, there is the issue of messages going back and forth between processes... might as well be able to send the messages between machines and have a total separation of concerns.
Instead of forking battles, we could have them running on their own, allowing them to crash and reboot and do whatever they feel like without ever causing any of the other battles or our server any harm.
Javascript? Why just one language?
I would consider leveraging an Object Oriented approach or language - at least for the battles, if not for the server as well.
If we are separating the code, we can use different languages. Otherwise I would probably go with Ruby, as it's easy for me, but maybe I'm mistaken and delving deeper into Javascript's prototypes will do.
Oh... foreign code - sanitization is in order.
How safe is the foreign code? should it be in a localized sped language that promises safety of using an existing language interpreter, that might allow the code to mess around with things it really shouldn't...
I would probably write my own "pseudo language" designed for the battles... or (if it was a very local project for me and mine) use Ruby with one of it's a sanitizing gems.
Battles and the web-services might not scale at the same speed.
It seems to me that handling messages - both client->server->battle and battle->server->client - is fairly easy work. However, handling the battle seems more resource intensive.
I'm convincing myself that a separation of concerns is almost unavoidable.
Having a server backend and a different battle backend would allow you to scale the battle handlers up more rapidly and without wasting resources on scaling the web-server before there's any need.
Network disconnections.
Assuming we allow the players to go offline while their agents "fight" in the field ... What happens when we need to send our user "Mitchel", who just reconnected to server X, a message to a battle he left raging on server Y?
Separating concerns would mean that right from the start we have a communication system that is ready to scale, allowing our users to connect to different endpoints and still get their messages.
Summing these up, I would consider this as my workflow:
Http workflow:
Client -> Web Server : requesting agent with identifier and optional battle data (battle data is for creating an agent, omitting battle data will be used for limiting the request to an existing agent if it exists).
This step might be automated based on Client authentication / credentials (i.e. session data / cookie identifier or login process).
if battle data exists in the request (request to make):
Web Server -> Battle instance for : creating agent if it doesn't exist.
if battle data is missing from the request:
Web Server -> Battle Database, to check if agent exists.
Web Server -> Client : response about agent (exists / created vs none)
If Agent exists or created, initiate a Websocket connection after setting up credentials for the connection (session data, a unique cookie identifier or a single-use unique token to be appended to the Websocket request query).
If Agent does't exist, forward client to a web form to fill in data such as agent code, battle type etc'.
Websocket "workflows" (non linear):
Agent has data: Agent message -> (Battle communication manager) -> Web Server -> Client
It's possible to put Redis or a similar DB in there, to allow messages to stack while the user is offline and to allow multiple battle instances and multiple web server instances.
Client updates to Agent: Client message -> (Battle communication manager) -> Web Server -> Agent
I am wondering what would be the best practice for deploying updates to a (MVC) Go web application. Imagine the following scenario :
1) Code and test some changes for my Go Web Application
2) Deploy update without anyone currently using the previous version getting interrupted.
I don't know how to make sure point 2) can be covered - when somebody is sending a request to the server and I rebuild/restart it just in this moment, he gets an error - even if the request just uses a part of the code I did not touch or that is backwards-compatible, or if I just added a new Request-handler.
Maybe I'm missing something trivial or a well-known pattern as I am just in the process of learning go and my previous web applications were ASP.NET- or php-applications where this was no issue as I did not need to restart the webserver on code changes.
It's not just an issue with Go, but in general we can divide the problem into two separate ones:
Making sure current requests do not get terminated and affect user experience.
Making sure there is no down-time in which new requests cannot be handled.
The first one is easier to tackle: You just don't violently kill your server, but tell it to exit, causing a "Drain phase", in which it does not accept new requests and only finishes the currently running requests, and exits. This can be done by listening on signals for example, and entering the app into a special state.
It's not trivial with Go as the default http server doesn't support shutting it down, but you can start a server with a net.Listener, and then keep a reference to it an close it when the time is due.
Now, doing only approach one and then starting the service again will cause new requests not to be accepted while this is going on, and we all know this can take a number of seconds in extreme cases.
So what we need is another instance of the server already running with the new code, the instant the old one is not responding to new requests, right? That can be done in several ways:
Having more than one server, and a load-balancer on top of them, allowing one (or more) server to take the load while we restart another. That's the simplest way, and the way most people do it. If you need N servers to take the load of your users, just keep N+1 and restart one at a time.
Using socket sharing tricks. In Newer Linux kernels, Many processes can listen and accept on the same port. What you do is simply start the new instance and then tell the old one to finish and exit. This way there is no pause. This is done by setting SO_REUSEPORT on the listening socket.
The above can be automated with ready to ship solutions, like Einhorn, that deals with all the details for you, see https://github.com/stripe/einhorn
Another approach is documented in this blog post: http://blog.nella.org/?p=879
I am going to tell the problem that I have to solve and I need some suggestions if i am in the right path.
The problem is:
I need to create a Windows Service application that receive a request and do some action. (Socket communication) This action is to execute a script (maybe in lua or perl).This script models te bussiness rules of the client, querying in Databases, making request in websites and then send a response to the client.
There are 3 mandatory requirements:
The service will receive a lot of request at the same time. So I think to use the worker's thread model.
The service must have a high throughput. I will have many of requests at the same second.
Low Latency: I must response these requests very quickly.
Every request will generate a log entries. I cant write these log entries in the physical disk at same time the scripts execute because the big I/O time. Probably I will make a queue in memory and others threds will consume this queue and write on disk.
In the future, is possible that two woker's thread have to change messages.
I have to make a protocol to this service. I was thinking to use Thrift, but i don't know the overhead involved. Maybe i will make my own protocol.
To write the windows service, i was thinking in Erlang. Is it a good idea?
Does anyone have suggestions/hints to solve this problem? Which is the better language to write this service?
Yes, Erlang is a good choice if you're know it or ready to learn. With Erlang you don't need any worker thread, just implement your server in Erlang style and you'll receive multithreaded solution automatically.
Not sure how to convert Erlang program to Windows service, but probably it's doable.
Writing to the same log file from many threads are suboptimal because requires locking. It's better to have a log-entries queue (lock-free?) and a separate thread (Erlang process?) that writes them to the file. BTW, are you sure that executing external script in another language is much faster than writing a log-record to the file?
It's doubtfully you'll receive much better performance with your own serialization library than Thrift provides for free. Another option is Google Protocol Buffers, somebody claimed that it's faster.
Theoretically (!) it's possible that Erlang solution won't provide you required performance. In this case consider a compilable language, e.g. C++ and asynchronous networking, e.g. Boost.Asio. But be ready that it's much more complicated than Erlang way.
I'm working on a consumer web app that needs to do a long running background process that is tied to each customer request. By long running, I mean anywhere between 1 and 3 minutes.
Here is an example flow. The object/widget doesn't really matter.
Customer comes to the site and specifies object/widget they are looking for.
We search/clean/filter for widgets matching some initial criteria. <-- long running process
Customer further configures more detail about the widget they are looking for.
When the long running process is complete the customer is able to complete the last few steps before conversion.
Steps 3 and 4 aren't really important. I just mention them because we can buy some time while we are doing the long running process.
The environment we are working in is a LAMP stack-- currently using PHP. It doesn't seem like a good design to have the long running process take up an apache thread in mod_php (or fastcgi process). The apache layer of our app should be focused on serving up content and not data processing IMO.
A few questions:
Is our thinking right in that we should separate this "long running" part out of the apache/web app layer?
Is there a standard/typical way to break this out under Linux/Apache/MySQL/PHP (we're open to using a different language for the processing if appropriate)?
Any suggestions on how to go about breaking it out? E.g. do we create a deamon that churns through a FIFO queue?
Edit: Just to clarify, only about 1/4 of the long running process is database centric. We're working on optimizing that part. There is some work that we could potentially do, but we are limited in the amount we can do right now.
Thanks!
Consider providing the search results via AJAX from a web service instead of your application. Presumably you could offload this to another server and let you web application deal with the content as you desire.
Just curious: 1-3 minutes seems like a long time for a lookup query. Have you looked at indexes on the columns you are querying to improve the speed? Or do you need to do some algorithmic process -- perhaps you could perform some of this offline and prepopulate some common searches with hints?
As Jonnii suggested, you can start a child process to carry out background processing. However, this needs to be done with some care:
Make sure that any parameters passed through are escaped correctly
Ensure that more than one copy of the process does not run at once
If several copies of the process run, there's nothing stopping a (not even malicious, just impatient) user from hitting reload on the page which kicks it off, eventually starting so many copies that the machine runs out of ram and grinds to a halt.
So you can use a subprocess, but do it carefully, in a controlled manner, and test it properly.
Another option is to have a daemon permanently running waiting for requests, which processes them and then records the results somewhere (perhaps in a database)
This is the poor man's solution:
exec ("/usr/bin/php long_running_process.php > /dev/null &");
Alternatively you could:
Insert a row into your database with details of the background request, which a daemon can then read and process.
Write a message to a message queue which a daemon then read and processed.
Here's some discussion on the Java version of this problem.
See java: what are the best techniques for communicating with a batch server
Two important things you might do:
Switch to Java and use JMS.
Read up on JMS but use another queue manager. Unix named pipes, for instance, might be an acceptable implementation.
Java servlets can do background processing. You could do something similar to this technology in a web technology with threading support. I don't know about PHP though.
Not a complete answer but I would think using AJAX and passing the 2nd step to something thats faster then PHP (C, C++, C#) then a PHP function pick the results off of some stack most likely just a database.