I've build a distributed system consisting of several web-services and some web applications consuming them.
They are all hosted on Heroku.
Is there some way for request between these applications to be done "inside heroku" without going through the web.
Something analog to using localhost.
You are maybe in luck: such a feature has currently reached the experimental phase.
Let me take a moment to underscore that: this feature may disappear or change at any time. It's not supported, but bug reports are appreciated. Don't build a bank with it. Don't get yourself in a position to be incredibly sad if severe problems are found that render it unshippable and it's aborted.
However, it is still cool, and here it is: containerized-network
You can use, for example, the pub-sub interface of any of the hosted Redis solutions. Or any of the message brokers (IronMQ, RabbitMQ) to pass messages.
Related
I work with a Windows application that uses RabbitMQ, which the vendor provided. The only value at all in the advanced.config is a timeout value that the application vendor gave me when troubleshooting some issue with the connections timing out. I have been searching for performance tuning documentation as it pertains to Windows deployments of RabbitMQ, but all of the most useful stuff that I am finding is for Linux or containerized deployments. Unfortunately, the application that is using RabbitMQ is Windows dependent, so I can't just go a different/better route with the RabbitMQ deployment.
The problem we're running into is that we have close to 13,000 queues without any way to trim that down substantially, and even though the CPU and RAM resources don't ever seem taxed, RabbitMQ itself seems to be struggling. We're often seeing unacknowledged messages in the queues, and even the web interface itself sometimes stalls out. With all of the Linux information I am seeing, I feel like there has to be a list of Windows best practices too, or values that can be added and then tweaked in the advanced.config. Any advice great appreciated.
Currently, I deal with microservices on a daily basis at my 9-5. Most everything that I touch is written in PHP, and as only a software engineer, SysOps manages everything that has to do with apps running, etc. I have a little familiarity in how the infrastructure and build pipeline is setup, but I still am not a SysOps or DevOps guy.
With that said, I love Golang and for a side project, I am creating a fairly large web application with a lot of moving parts. Writing and designing the code is easy as I have learned a lot from my day job, but deploying and managing Golang web apps (as they are executables) is quite different than updating files for apache to serve.
I have researched a lot on how I would build and deploy my microservice apps, but I keep on thinking of more problems that will need to be solved along the way. I have tinkered with the idea of using Docker for all of this, but I would rather not have the added complexity of learning that and managing storage for all of the images as that could be large.
Is there a best practice or a good way to manage Golang applications after they have been deployed? I would need a way to keep track of all the microservice processes to be able to see if they are still up and to be able to stop them when a new build is going to be deployed.
As for the setup, just assume that all the microservices will be run on the server, not in a container or in a VM. They will all need to be managed, but also able to act upon independently. Jenkins will be used for building and deploying. I will be using Consul for service discovery and possibly configuration, and most likely health checks on the services. I'm thinking of having each microservice register itself to consul when started and deregister when stopping.
Again, I am looking for a solution that is hopefully not just "Docker". I also had thoughts into creating a deploy service that manages the services (add and remove), as well as registering them in Consul. So if I cannot find a better solution, I might go that path. Any help is appreciated.
** Sorry if my question was confusing, but since a couple people answered on the wrong topic at hand, I will try to clarify. I don't need any help making the microservices, or even know anything about them. I brought that point up as to why I need to ask my question. Basically what I need is just the ability to manage the running go processes of all my microservices so I can do deployments and be able to stop and start processes to update the code. It is easier when you have to worry about one app, but when you can have up to 10-15 difference microservices they become harder to keep track of. After my own research, it seems that Supervisord is what I am looking for, but I'm not sure. That is the direction I am going in with this question. Thanks.
Golang is great to use for microservices, but I would say there is not so big difference of managing golang or other languages microservices.
What I would say is golang specific:
you don't need to install anything on servers since golang is compiled to single library
you can take advantage of std lib golang rpc package and gob binary decoding, instead of usin 3rd party solution (gorpc, protocol buffer etc)
Other than that you need to use your own judgement. There is plenty of ways of doing one things in microservices world; one day you will implement solution A but when after 3 month you will see that its better to do B, do that.
In internet, there is so much reading about microservices. I will recommend you 2 good resurces: https://books.google.co.uk/books/about/Building_Microservices.html?id=RDl4BgAAQBAJ&source=kp_cover&redir_esc=y&hl=en
And article: http://highscalability.com/blog/2014/4/8/microservices-not-a-free-lunch.html
Remember, microservices are not a golden bullet, they often can help making application easier to maintain and grow, but from the other side require lot of additional work, consequence in specifying API contracts and strong devops culture.
I need to be able to enable access through a firewall to a server for an app that is built atop Heroku. Unfortunately the IP's coming from Heroku's AWS instances seem to vary quite a bit. Is there a "correct" way of determining what subnet to expect from Heroku's AWS platform for an app?
As unfortunate as this is -- there isn't a good way to continuously get this information. On the AWS forums, however, the EC2 engineers tend to occasionally post their IP ranges (here is a recent example: https://forums.aws.amazon.com/ann.jspa?annID=1701).
The downside to this, however, is that it requires a lot of manual work.
There is no reliable way to accept Heroku public IPs in firewalls. Even if there was, you would be compromising your application and opening up an attack vector via other apps on Heroku.
The solution is to have an adequate authentication layer in your exposed services.
This question was asked a few years ago back when services like Proximo didn’t exist -- or weren’t known within the Heroku community.
Today, if you want your outbound traffic to come through a static IP which you can whitelist in your firewall, you can use a proxy service like Proximo (Fixie is another example).
There are a few downsides for using these services:
1) Intrusive Setup
Although the setup of these addons is relatively simple, it’s important to understand how they affect the application.
In case of Proximo, for example, you’ll be required to wrap your processes in a special utility.
This utility will “automatically forward outbound TCP connections made by the wrapped process over your proxy.”
2) Latency
To make your outbound traffic come from a static IP, these services route the traffic through a proxy. This means you’ll add another hop to your outbound communication.
I know that applications that run on Heroku usually aren’t very sensitive to network latency, but it’s important to take this issue into a consideration.
3) Uptime
Although these services are relatively stable, it should be noted that routing the traffic through a specialized third-party proxy adds another point of failure and may affect the overall stability of your applications.
To summarize, these services will help you solve the problem. However, I would consider using them as a temporary workaround, not a complete solution.
Rest assured that these kinds of fixes can hold for a very long time, but if security becomes increasingly more important for the applications you’re running on Heroku, it can be a good idea to start planning a migration to AWS.
If you’re wondering when can be the best time for your team to make the transition to AWS, I’ve shared a few notes here: “Will Heroku always be perfect?”
Hope that helps.
I'm building an application that will provide a service to other applications (let's pretend like it solves differential equations). So my DifEq service will be running all the time and a client application can send it requests to solve DifEqs at any point.
This would be trivial using sockets or pipes.
The problem is some applications nefariously want to send linear equations instead of differential equations, so I want to register applications that I know are sending proper DifEqs to my application.
Traditional sockets break down here, as far as I know.
Ideally, I'd like to be able to look at some information about the application that is making a request of me and (either through some meta-data on that application, through communication with my web site, or through some other, unkown method) determine it is an acceptable DifEq app. Furthermore, this ideal method would not be spoofable without a root/admin-level compromise of the underlying OS. If the linear equation app is also a root kit, I'll concede to being broken. :)
I need to be able to do this on Windows, OS X, and Linux (and maybe Android); but I recognize that it may not be the same solution on all platforms. So, how would you accomplish this (specify the platform you are focusing on, if appropriate)? I've done a lot of server-side development, but it's been way too many years since I've done any client-side development outside the browser and the world is very different today than it was then.
I think your question is a little confusing when it comes to talking about DifEQ vs LinearEQ.
It sounds to me like you are just looking for a routine way to verify that clients are authorized to connect. There is a lot to read up on this subject. Common methods would be to use SSL certificates to verify the identity of clients. You can also tunnel over SSH, or use OAUTH, etc, etc.
You'll have to do some more digging around the web to see what kind of authentication fits your scenario. You mention 'not spoofable'. I think that people generally end up compiling-in a certificate of private key into their application. This will stop all but the very dedicated and experienced hackers.
Is it possible to run axis2 on the server and the clients sending the soap messages not use axis? This uses an in-out process. If so, have you seen any issues from doing this?
Yes. The whole idea behind web services is interoperability.
There can be issues. I'm not really sure what you're trying to do so it's hard to say what those issues would be.