I've been working with a golang application lately that does network i/o using a bunch of protocols- HTTP (TCP), DNS and WHOIS (UDP) as well as a few others. Some make use of third-party APIs
The application is open-source so I would like to make changes allowing me to specify the network interface for the sockets to bind to, allowing me to use different interfaces depending on a runtime flag. The only way around this without writing code would be to modify the system-wide routing table each time I want to utilize a different interface, which isn't a very appealing solution
Before I go an modify every instance where a Dialer is used (or try to create a wrapper that they can all use) is there a golang feature that would allow setting the interface globally once, so that the various Dialer invocations would "Just Work"- and adhere to the interface I specified?
I did some searching and have only found ways to do this when each Dialer is created (using DialerContext.LocalAddr) but given I'm really a C programmer and not a golang programmer, I realize I may be totally missing a golang idiom for doing something like this
Related
Kubernetes has a rapidly evolving API and I am trying to find best practices, recommendations, or really any kind of guidance about how to write Go software that gracefully handles supporting its evolving API and supports multiple versions simultaneously. I am sure I am not the first person to attempt this, but so far I have not found any guidance about Kubernetes specifically, and what I have read about polymorphism in Go has not inspired a great solution yet.
Kubernetes is written in Go and provides Go packages like k8s.io/api/extensions/v1beta1 and k8s.io/api/networking/v1beta1. Kubernetes resources, for example Ingress, are first released in one API group (extensions) and as they become more mature, get moved to another API group (networking) and can also change versions (e.g. go from v1beta1 to plain v1). Kubernetes also provides k8s.io/client-go for interacting with a Kubernetes cluster.
I am an experienced object-oriented (and other types of) programmer, but fairly new to Go and completely new to the Kubernetes packages. What I want to accomplish is a program architecture that allows me to write code once and have it work on any version of the Kubernetes resource, at least as long as the resource contains all the features I care about. In a typical object-oriented environment, I would create a base Ingress class and have all these various versions derive from it, and package up operations so that I could just work on Ingress everywhere. My sense is that Go intends for people to take a different approach, and in any case there are complications because of the client/server aspect.
Client/server and APIs
My Go program is a client of the Kubernetes server. Various version of the server will support various version of the Kubernetes API, and therefor various versions of the Ingress resource. So my first problem is that I have to do something like this to get a list of all the Ingresses:
ingressesExt, err := il.kubeClient.ExtensionsV1beta1().Ingresses(namespace).List(metav1.ListOptions{})
ingressesNet, err := il.kubeClient.NetworkingV1beta1().Ingresses(namespace).List(metav1.ListOptions{})
I have to gracefully handle errors about the API not being supported. Because the return types are different, AFAIK there is no unified interface where I can just make one call and get the results in a single list. It seems like this is the sort of thing someone should have solved and provided a solution for, but so far I have not found anything.
Type conversion
I also have to find some way to merge ingressesExt and ingressesNet into a single usable list, with an eye toward maintainability/extensibility now that Ingress has graduated to NetworkingV1.
Kubernetes utilities
I see that Kubernetes provides a lot of auto-generated code and utilities, but I have not found a lot of documentation about how to use them. For example, Ingress has functions like
DeepCopy
Marshal
XXX_DiscardUnknown
XXX_Merge
XXX_Unmarshal
Maybe I can use these to do the type conversion? Combine marshal, unmarshall, discard, and merge somehow to take the data from on version and import it into another?
Questions
Hopefully you see the issue and understand what I am trying to achieve.
Are there packages from Kubernetes or other open source authors that make some progress in unifying the APIs like I need?
Are any of the Kubernetes auto-generated functions meant for general use (as opposed to internal use) and helpful to my challenge? I have not found documentation for any but DeepCopy.
What is the "Go way" of abstracting out the differences between the various versions of the Ingress object such that I can write the rest of the code to work on any version? Keep in mind that I may need to make another API call for further processing, in which case I would need to know the concrete type of the object and select the right API call. It is not obvious to me that client-go provides any support for such auto-selection of API calls.
I currently have a primitive RPC setup relying on JSON transferred over secured sockets, but I would like to switch to gRPC. Unfortunately I also need access to AF_UNIX on windows (Which Microsoft recently started supporting, but gRPC has not implemented).
Since I have an existing working connection (managed with a different library), my preference would be to just use that in conjunction with GRPC to send/receive commands in place of my JSON parsing, but I am struggling to identify the best way to do that.
I have seen Plugging custom transport into gRPC but this question differs in the following ways (As well as my hope for a more recent answer)
I am wanting to avoid making changes to the core of gRPC. I'd prefer to extend it if possible from within my library, but the answer here implies adding a new transport to gRPC.If I did need to do this at the transport level, is there a mechanism to register it with gRPC after the core has been built?
I am unsure if I need to define this as a full custom transport, since I do already have an existing connection established and ready. I have seen some things that imply I could simply extend Channel, but I might be wrong.
I need to be able to support Windows, or at least modern versions of it (Which means that the from_fd options gRPC provides are not available since they are currently only implemented for POSIX)
Has anyone solved similar problems with gRPC?
I may have figured out my own answer. I seem to have been overly focused on gRPC, when the service definition component of Protobuf is not dependent on that.
How can i write my own RPC Implementation for Protocol Buffers utilizing ZeroMQ is very similar to my use case, with https://developers.google.com/protocol-buffers/docs/proto#services seeming to resolve my issue (And this also explains why I seem to have been mixing up the different kinds of "Channels" involved
I welcome any improvements/suggestions, and hope that maybe this can be found in future searches by people that had the same confusion.
I need to do something relatively simple, and I don't really want to install a MOM like RabittMQ etc.
There are several programs that "register" with a central
"service" server through TCP. The only function of the server is to
call back all the registered clients when they all in turn say
"DONE". So it is a kind of "join" (edit: Barrier) for distributed client processes.
When all clients say "DONE" (they can be done at totally different times), the central server messages
them all saying "ALL-COMPLETE". The clients "block" until asynchronously called back.
So this is a kind of distributed asynchronous Observer Pattern. The server has to keep track of where the clients are somehow. It is ok for the client to pass its IP address to the server etc. It is constructable with things like Boost::Signal, BOOST::Asio, BOOST::Dataflow etc, but I don't want to reinvent the wheel if something simple already exists. I got very close with ZeroMQ, but non of their patterns support this use-case very well, AFAIK.
Is there a very simple system that does this? Notice that the server can be written in any language. I just need C++ bindings for the clients.
After much searching, I used this library
https://github.com/actor-framework
It turns out that doing this with this framework is relatively straightforward. The only real "impediment" to using it is that the library seems to have gotten an API transition recently and the documentation .pdf file has not completely caught up with the source. No biggie since the example programs and the source (.hpp) files get you over this hump. However, they need to bring the docs in sync with the source. In addition, IMO they need to provide more interesting examples on how to use c++ Actors for extreme performance. For my case it is not needed, but the idea of actors (shared nothing) in this use-case is one of the reasons people use it instead shared memory communication when using threads.
Also, getting used to the syntax that the library enforces (get used to lambdas!) if one is not used to state of the art c++11 programs it can be a bit of a mind-twister at first. Then, the triviality of remembering all the clients that registered with the server was the only other caveat.
STRONGLY RECOMMENDED.
My company is looking at implementing a new VPN solution, but require that the connection be maintained programatically by our software. The VPN solution consists of a background service that seems to manage the physical connection and a command line/GUI utilty that initiates the request to connect/disconnect. I am looking for a way to "spy" on the API calls between the front-end utilty and back-end service so that our software can make the same calls to the service. Are there any recommended software solutions or methods to do this?
Typically, communications between a front-end application and back-end service are done through some form of IPC (sockets, named pipes, etc.) or through custom messages sent through the Service Control Manager. You'll probably need to find out which method this solution uses, and work from there - though if it's encrypted communication over a socket, this could be difficult.
Like Harper Shelby said, it could be very difficult, but you may start with filemon, which can tell you when certain processes create or write to files, regmon, which can do the same for registry writes and reads, and wireshark to monitor the network traffic. This can get you some data, but even with the data, it may be too difficult to interpret in a manner that would allow you to make the same calls.
I don't understand why you want to replace the utility, instead of simply running the utility from your application.
Anyway, you can run "dumpbin /imports whatevertheutilitynameis.exe" to see the static list of API function names to which the utility is linked; this doesn't show the sequence in which they're called, nor the parameter values.
You can then use a system debugger (e.g. Winice or whatever its more modern equivalent might be) to set breakpoints on these API, so that you break into the debugger (and can then inspect parameter values) when the utility invokes these APIs.
You might be able to glean some information using tools such as Spy++ to look at Windows messages. Debugging/tracing tools (Windbg, or etc.) may allow you to see API calls that are in process. The Sysinternals tools can show you system information to some degree of detail of usage.
Although I would recommend against this for the most part -- is it possible to contact the solution provider and get documentation? One reason for that is fragility -- if a vendor is not expecting users to utilize that aspect of the interface, they are more likely to change it without notice.
I have a C# module responsible for acquiring the list of network adapters that are "connected to the internet" on a windows Vista machine. The module uses the "Network List Manager API" (or NLM API) to iterate over all network connections and returns all those for which the IsConnectedToInternet value is true.
I received some suggestions for the implementation of this module in this SO question
To test this module I've decided to write a helper that returns the list of internet connected interfaces based on another logic, so it would be a sort of a "reality check" for the original module's logic. Note that for the test helper I am willing to use detection methods that might be considered bad practice for production code (e.g. relying on some internet resource like "Google" to be available - in case it shuts down, blocked by our internal firewall etc. it's relatively easy to fix the test as opposed to a deployed product base).
The alternative detection method I chose was to try to connect to "www.google.com:80" with a TcpClient. My problem: When I have more than one connected adapter (e.g. both wireless and LAN) the detection method fails for one of them with the error "A connect request was made on an already-connected socket".
My question is three fold:
How would you go about testing such a module in general? Do you support the idea of doing the same thing in a different way and comparing the results or is it an overkill and I should rely on the system's API? My main problem here, is that it's very hard to pre-configure the system so that I'll know what the expected results are in advance.
What alternative logic would you suggest? One thing that was suggested in the aforementioned question was looking at the routing table - what about considering each adapter that has a routing entry with a destination of 0.0.0.0 as "connected to the internet"? Other suggestions?
Do you understand why I get the "already-connected" error with the current test logic?
I can only answer your question about the unit test.
The code you're testing is, in your own words, "a C# module responsible for acquiring the list of network adapters that are 'connected to the internet' on a windows Vista machine. The module uses the 'Network List Manager API' (or NLM API) to iterate over all network connections and returns all those for which the IsConnectedToInternet value is true."
If I were writing this module, I would first use an interface for the NLM API, call it...NLMAPIService. Now, for the real code, create an Adapter that implements NLMAPIService and adapts the real NLM API.
For testing, create a class FakeNLMAPI that implements NLMAPIService and has all of its data in-memory somewhere, or in an XML file, or whatever. Your module calls methods only on the NLMAPIService, so you don't have to change any "real" code depending on whether you're testing or not.
Therefore, in your test setup method, you can instantiate FakeNLMAPI and pass it to your module, and in production, instantiate your NLM API Adapter.
I'm going to assume that you can instantiate and modify the object that represents a network connection. If not, you can follow the same pattern for faking the actual network connection object.
Dependency Injection is a very handy pattern to deal with issues like this. Instead of simply using the NLM API components directly in your code define an interface and a class that implements it and serves as a proxy to the NLM API. Pass an instance of this class to your module in the constructor and have your module use it. In your unit tests, instead of the real proxy object, use a mock object that returns known information -- it doesn't even have to reference the NLM API -- to use in testing the logic of your module. Granted, your proxy class will need some testing as well, but the logic in it is much simpler -- probably just some data marshaling. You might be able to convince yourself of its correctness or, if not, do some manual testing on it to make sure that it is working properly.
UnitTests shouldn't access to external resources. To UnitTest your method, I would stub out the Network List Manager API.
You still need an acceptance test layer. In that test environment you should replicate various configurations you expect to support in your environment, setup your own webhosts, routers, machine config. Acceptance testing should be done at the user experience level using a tool like Fitnesse.