I have a program that does some networking using Winsock, and one of our requirements right now is to port over our program to Linux. The only thing stopping us from doing this is Winsock.
My question is: How easy can I port this over to a Linux implementation?
Are there any pitfalls I should be aware of, and if I simply include the appropriate header files, what sort of things will I have to be sure to handle?
Thanks for any help!
I'd post code but I can't unfortunately due to legal reasons.
But, our code does use the following:
WSAStartup(..)
WSACleanup(..)
Socket(..)
sendto(..)
recvfrom(..)
ioctlsocket(..)
setsocketopt(..)
Based on that list of functions, things should more or less just work. Add #if _WIN32 around the calls to WSAStartup and WSACleanup (the linux equivalent is to not do anything, the sockets library is initialized automatically).
You also might need some OS-dependent code when setting socket options, some of them are the same, some aren't, and the types might be different.
It will depend if you use any windows specific networking functionality or if you're just using mostly the mostly BSD compatible API.
So, if you're using overlapped I/O and I/O completion ports and other advanced parts of the Winsock API then things will be very difficult to port and if you're just using the BSD compatible stuff then it should be easy to write a thin translation layer or even just have the winsock startup and shutdown stuff inside a windows specific ifdef...
This may help: http://tangentsoft.net/wskfaq/articles/bsd-compatibility.html
The only calls that make porting difficult are the WSA* calls.
WSAStartup() -> nop
WSACleanup() -> nop
Socket/setsockopt -> socket/setsockopt
Under *nix, sockets are blocking by default and it's not necessary or possible to use that weird setsockopt call to fiddle with it.
ioctlsocket -> ioctl
Under *nix we don't like asynchronous sockets much and prefer to use the select() system call.
---- Rest of this answer seems only to apply to Win95 compatible winsock ----
Unfortunately as the original socket() in Winsock was broken in some cases, you probably used WSASocket() and so have to convert those calls.
Without seeing code, it's tough to say how easy it is. But you should be able to replace winsock calls to analogs in sys/socket.h.
Related
The official FastCGI documentation says that stdin is repurposed as a listening socket when a FastCGI module is started. That's great on Linux, where stdin and sockets are all ints, but I don't think it could it work on Windows, where stdin is a FILE*, and a socket is a HANDLE.
Since Windows servers do support FastCGI, someone has either found a way to make them compatible, or redefined the system for that OS. My Google-fu doesn't seem to be up to locating how though. Where can I find documentation on it?
FastCGI defines only the message exchange protocol, but people behind FastCGI also provide one implementation of that protocol for C++. In this implementation your app must use provided FCGX_Request object to rewire three provided FCGX_Stream objects to the usual ones (cin, cout, cerr). But I suspect that you don't have to rewire the streams, and can use them directly. Check out this FastCGI Hello World to see how it's done.
So, your app does not see HANDLE or FILE*. It sees instead fcgi_streambuf, which inherits from std::streambuf. The way the previously mentioned protocol is implemented is just a detail that you're not supposed to be concerned with. The implementation gets hold of a stream of bytes and provides it to the app, and also the other way around.
In OpenMPI, if I follow the call stack of any collective operation (e.g MPI_Reduce) deep enough, I find that it calls a function called send().
After a lot of grepping, I'm not sure where send() is implemented. I suspect that send() may be buried inside of a macro or obscure shim layer of some sort.
Where are the implementation(s) of send() located in the OpenMPI codebase?
I'm looking at OpenMPI v1.8.1, though I suspect that the organization of the sorce tree hasn't changed that much between versions.
send(2) is the BSD socket system call for sending data over network sockets. It is ultimately used by the tcp BTL of Open MPI to perform the actual network transfer from one process to another and its implementation is to be found in the source code of the standard C library and in the OS kernel.
If you are interested in the actual higher-level mechanism that Open MPI uses to transmit messages from one rank to another over TCP/IP networks, then the tcp BTL itself is to be found in $OMPI_SOURCE/ompi/mca/btl/tcp/ (for older Open MPI versions) or in $OMPI_SOURCE/opal/mca/btl/tcp/ (for newer versions).
What I mean is that, when I code a project, I need to communicate with the serial port like COM1, COM2... but sometimes there is no device connected and I also can use the function CreateFile to get the COM port handle.
When I use the WriteFile function to send a string to the COM port the software blocks.
After I dig into the problem I find another function GetCommModemStatus which can get status of the COM port but when I use the usb-rs232 transition line, the second parameter always returns 0.
How can I get the COM port status so that I can check if is there some devices connected to the computer?
If I understand correctly, you want to detect if a device is connected to your COM port and ready to accept packets. If that's the case, you need to check control signals (DTR/DSR and CTS/RTS) before sending data, assuming your device is aware of them and sets the appropriate PIN on your DB-9 or DB-25 connector. Some devices rely on software handshaking instead (XON/XOFF) and do not set control signals. Your best bet would be to consult documentation of your device.
I have been using ComPort Library version 4.10 by Dejan Crnila. It does support both hardware and software handshakings, so you can focus on your own code instead of reinventing the wheel.
As several people have already pointed out, it is not a good idea to try to "re-invent the wheel." Except for "quick and dirty" testing, your code will have to handle the com port in a separate thread and the available solutions all make this much easier.
BUT, if you Really want to do it, I'll give you some pointers.
If you are using "WriteFile" then you have probably already figured out the "CreateFile" part of the procedure and how complicated things can get depends upon what kind of IO you specified in that procedure, Overlapped or not. Overlapped IO is more complicated but does let the OS handle some of the burden.
You mentioned that your call to "WriteFile" hangs. You should look up the "SetCommTimeOuts" function. Setting both the WriteTotalTimeoutMultiplier and WriteTotalTimeoutConstant members of TCommTimeouts to zero will tell the OS to return immediately. You may also need to "SetCommMask" if your target uses handshaking.
What happens next really depends on what your target is supposed to do. The GetCommMask function can be used to check the status of the handshake lines.
Question:
Can I use the multiprocessing module together with gevent on Windows in an efficient way?
Scenario:
I have a gevent based Python application doing asynchronous I/O on Windows. The application is mostly I/O bound, but there are spikes of higher CPU load as well. This application would need to control a console application via its stdin and stdout. I cannot modify this console application and the user will be able to use his own custom one, only the text (line) based communication protocol is fixed.
I have a working implementation using subprocess and threads, but I would rather move the whole subprocess based communication code together with those threads into a separate process to turn the main application back to single-threaded. I plan to use the multiprocessing module for this.
Prior reading:
I have been searching the Web a lot and read some source code, so I know that the multiprocessing module is using a Pipe implementation based on named pipes on Windows. A pair of multiprocessing.queue.Queue objects would be used to communicate with the second Python process. These queues are based on that Pipe implementation, e.g. the IPC would be done via named pipes.
The key question is, whether calling the incoming Queue's get method would block gevent's main loop or not. There's a timeout for that method, so I could make it into a loop with a small timeout, but that's not a good solution, since it would still block gevent for small time periods hurting its low I/O latency.
I'm also open to suggestions on how to circumvent the whole problem of using pipes on Windows, which is known to be hard and sometimes fragile. I'm not sure whether shared memory based IPC is possible on Windows or not. Maybe I could wrap the console application in a way which would allow communicating with the child process using network sockets, which is known to work well with gevent.
Please don't question my primary use case, if possible. Thanks.
The Queue's get method is really blocking. Using it with timeout could potentially solve your problem, but it definitely won't be a cleanest solution and, which is the most important, will introduce extra latency for no good reason. Even if it wasn't blocking, that won't be a good solution either. Just because non-blocking itself is not enough, the good asynchronous call/API should smoothly integrate into the I/O framework in use. Be that gevent for Python, libevent for C or Boost ASIO for C++.
The easiest solution would be to use simple I/O by spawning your console applications and attaching to its console in and out descriptors. There are at two major factors to consider:
It will be extremely easy for your clients to write client applications. They will not have to work with any kind of IPC, socket or other code, which could be very hard thing for many. With this approach, application will just read from stdin and write to stdout.
It will be extremely easy to test console applications using this approach as you can manually start them, enter text into console and see results.
Gevent is a perfect fit for async read/write here.
However, the downside is that you will have to start this application, there will be no support for concurrent communication with it, and there will be no support for communication over network. There is even a good example for starters.
To keep it simple but more flexible, you can use TCP/IP sockets. If both client and server are running on the same machine. Also, a good operating system will use IPC as an underlying implementation, so it will be fast. And, if you are worrying about performance of this case, you probably should not use Python at all and look at other technologies.
Even fancies solution – use ZeroC ICE. It is very modern technology allowing almost seamless inter-process communication. It is a CORBA killer, very easy to use. It is heavily used by many, proven to be fastest in its class and rock stable. The beauty of this solution is that you can seamlessly integrate programs in many different languages, like Python, Java, C++ etc. But this will require some of your time to get familiar with a concept. If you decide to go this way, just spend a day reading trough documentation.
Hope it helps. Good luck!
Your question is already quite old. Nevertheless, I would like to recommend http://gehrcke.de/gipc which -- I believe -- would tackle the outlined challenge in a very straight-forward fashion. Basically, it allows you to integrate multiprocessing-based child processes anywhere in your application (also on Windows). Interaction with Process objects (such as calling join()) is gevent-cooperative. Via its pipe management, it allows for cooperatively blocking inter-process communication. However, on Windows, IPC currently is much less efficient than on POSIX-compliant systems (since non-blocking I/O is imitated through a thread pool). Depending on the IPC messaging volume of your application, this might or might not be of significance.
In my computer, there are two network adapters, connecting to different subnet. As below:
adapter A: 10.20.30.201
adapter B: 10.20.31.201
I want to make all outgoing data of a special process (for example Process A) through adapter A. That is I want to make adapter A as the process's default route.
I know, I can modify route table for some special destination, But what I want to do here is very different. Process A may communicate with many different IP and I don't know in advance.
Winsock2 provides LSP as a way to lay a dll in TCP/IP stack. I'm not familiar with LSP and don't know whether LSP can do what I want to do.
Can anybody give me some suggestion, Thanks.
A quick background on LSP:
An application, which uses Winsock2 API, calls a combination of WSA-prefix functions, eg WSAConnect, WSASocket, WSASend, WSARecv, etc.
If an application still use old winsock functions, these functions are mapped to Winsock2 behind the scene anyway. For instances: send() is mapped to WSASend(), recv() to WSARecv(), etc
WSA-prefix functions will internally call their corresponding WSP-prefix functions provided by LSP. For instances WSASend() calls WSPSend(), WSASocket() call WSPSocket(), etc. In short, WSAWhateverFunction() will calls WSPWhateverFunction(). Their parameters/returns are also the same (Not quite, but kind of).
LSP is a dll with these WSP-prefix functions implemented, eg. modify outbound/inbound traffic, filtering, etc. However an LSP is still a userspace dll. It's as limited as other userspace programs, and has no higher privilege than its host application, eg internet browsers. It has access to same set of system functions that is available to other programs, eg. winsock etc.
Conclusion is if your program can direct out-coming traffic to specific NIC, LSP can do it too. If it can't, neither can LSP. LSP therefore is irrelevant to your problem.