What I want to achieve is:
Filter Error logs from Heroku logs
Send only the Error logs to a Slack Channel
I have checked deployhooks but these only work for Deployments and also Heroku ChatOps is also only for Deployments.
Is there a way to constantly tail the logs and redirect the error log to Slack?
One way I can achieve this is using:
heroku logs -t --app app-name | grep "error" > redirect to slack based on webhook
But this requires configuring Heroku cli on a machine and also allowing messages to be posted to slack from that machine.
Can this be directly configured to Heroku itself?
No. Sorry to say that there's nothing baked into the platform to support streaming exceptions/errors to Slack. That said, you should be able to accomplish this a number of ways. Some more work than others.
For instance, on the low-effort ends you can set up Rollbar for the application in question and it looks like Rollbar natively supports a Slack integration. It's more work but you could also build a custom application to more or less pipe things directly to Slack while listening to the log stream.
As #RangerRanger mentioned there is no way to do this on Heroku without using addons.
I found two add-ons which are easy to configure and allow filtration of logs in real-time and sends messages to slack.
You can use PaperTrail which is easier to configure or use Coralogix which gives you Kibana support.
Our application is running on Google Kubernetes Engine and pulling messages from a Google Cloud Pub/Sub Subscription. We have one pod running in idle state, and horizontal pod autoscaling is set up to 10 pods depending on cpu usage. The subscription is mostly empty, and when a batch job kicks in, it writes into to Pub/Sub topic. The autoscaling is working well. It immediately (within 30 seconds) scales up to 10 pods once there are outstanding messages in the Pub/Sub subscription. But the issue is only a few pods are pulling the messages from the subscription and rest of them are just sitting even though there are still messages in the subscription.
Pub/Sub Client settings are:
MaxExtension: 600
MaxOutstandingMessages: 100 (also tried with 25)
Synchronous: true (also tried with false)
Google Cloud Pub/Sub Subscription Settings:
Pull-based
Ack Deadline is 600 seconds
And once the batch job kicks in, it writes 20k messages into the Pub/Sub topic. And the application can process 2 messages/sec in average.
The application is written in golang and we're using cloud.google.com/go v0.44.1 package version.
Do you know why the pods are sitting and not pulling messages even though there's a backlog in the Cloud Pub/Sub subscription?
Can you try to set this:
sub.ReceiveSettings.NumGoroutines = 10 * runtime.NumCPU()
sub.ReceiveSettings.MaxOutstandingMessages = -1
sub.ReceiveSettings.MaxOutstandingBytes = -1
Maybe by removing some limit, it could be better?
let me know
We just developed a system that integrates azure queue with an azure cloud service to process batch items. One requirement we had was to have items be set in the future to process. So for example, we batch it now, but tell it not to start for 5 hours.
This is built right into azure queues AddMessage using initialVisibilityDelay, so we did not see this as being an issue. However, we just noticed when we add auto scale on our Cloud Service, it is going off the total items in queue. In our situation we added 100,000 queue items to be sent 5 days from now, however it is scaling assuming these 100,000 are ready to go right now.
So in our situation, we would basically have dozens of instances of our app running until these messages can even send, 5 days from now.
I feel like there is something simple we are missing here.
Any feedback would be very helpful.
Anthony
Have you considered using one queue for the waiting messages and another queue for the actual messages to be processed and scaling on that latter queue?
I have inherited from a colleague who left a system module that integrates with Exchange Online in Office 365. Essentially what this module does is interact with with the remote Exchange service via EWS Managed API; subscribe for push notifications on changes in a user's calendar.
Change events do get posted to our web service and that is fine. And based on the frequency parameter we defined, the status checks messages also get posted on expected intervals, as per description about the subscription keep-alive behaviour.
The problem is, in observation, the subscription does not perpetuate despite responding with SubscriptionStatusType.OK to keep it ongoing. We never send SubscriptionStatusType.Unsubscribe since there is no error condition found in notification in messages. It seems to only last 9 to 14 hours before the Exchange service stops sending any status checks or change notification messages. When we make subscriptions from two separate web servers (different notification callback URLs), their subscriptions seem to die out around the same time.
Haven't found any clues that would cause the Exchange service to cancel/expire our subscriptions. What other conditions might contribute to this premature unsubscribe?
Exchange will "lose" subscriptions on a fairly regular basis, especially in the O365 environment because mailboxes are continually being shuffled onto different servers in order to load balance the entire ecosystem. Even in an on-prem Exchange you can lose subscription if a CAS restarts. Unfortunately to build a reliable app, you have to check periodically that you heard from Exchange via some kind of notification or heartbeat.
I'm working on a web application that submits tasks to a master/worker system that farms out the tasks to any of a series of worker instances. The work queue master runs as a separate process (on a separate machine altogether) and tasks are submitted to the master via HTTP/REST requests. Once tasks are submitted to the work queue, client applications can submit another HTTP request to get status information about tasks.
For my web application, I'd like it to provide some sort of progress bar view that gives the user some indication of how far along task processing has come. The obvious way to implement this would be an AJAX progress meter widget that periodically polls the work queue for status on the tasks that have been submitted. My question is, is there a better way to accomplish this without the frequent polling?
I've considered having the client web application open up a server socket on which it could listen for notifications from the work master. Another similar thought I've had is to use XMPP or a similar protocol for the status notifications. (Of course, the master/worker system would need to be updated to provide notifications either way but I own the code for that so can make any necessary updates myself.)
Any thoughts on the best way to set up a notification system like this? Is the extra effort involved worth it, or is the simple polling solution the way to go?
Polling
The client keeps polling the server to get the status of the response.
Pros
Being really RESTful means cacheable and scaleable.
Cons
Not the best responsiveness if you do not want to poll your server too much.
Persistent connection
The server does not close its HTTP connection with the client until the response is complete. The server can send intermediate status through this connection using HTTP multiparts.
Comet is the most famous framework to implement this behaviour.
Pros
Best responsiveness, almost real-time notifications from the server.
Cons
Connection limit is limited on a web server, keeping a connection open for too long might, at best load your server, at worst open the server to Denial of Service attacks.
Client as a server
Make the server post status updates and the response to the client as if it were another RESTful application.
Pros
Best of every worlds, no resources are wasted waiting for the response, either on the server or on the client side.
Cons
You need a full HTTP server and web application stack on the client
Firewalls and routers with their default "no incoming connections at all" will get in the way.
Feel free to edit to add your thoughts or a new method!
I guess it depends on a few factors
How accurate the feedback can be (1 percent, 5 percent, 50 percent) Accurate feedback makes it worth pursuing some kind of progress bar and comet style push. If you can only say "Busy... hold on... almost there... done" then a simple ajax "are we there yet" poll is certainly easier to code.
How timely the Done message has to be seen by the client
How long each task takes (1 second, 10 seconds, 10 minutes)
1 second makes it a bit moot. 10 seconds makes it worth it. 10 minutes means you're better off suggesting the user goes for a coffee break :-)
How many concurrent requests there will be
Unless you've got a "special" server, live push style systems tend to eat connections and you'll be maxed out pretty quickly. Having to throw more webservers in for a fancy progress bar might hurt the budget.
I've got some sample code on 871184 that shows a hand rolled "forever frame" which seems to work out well. The project I developed that for isn't hammered all that hard though, the operations take a few seconds and we can give pretty accurate percent. The code uses asp.net and jquery, but the general techniques will work with any server and javascript framework.
edit As John points out, status reporting probably isn't the job of the RESTful service. But there's nothing that says you can't open an iframe on the client that hooks to a page on the server that polls the service. Theory says the server and the service will at least be closer to one another :-)
Look into Comet. You make a single request to the server and the server blocks and holds the connection open until an update in status occurs. Once that happens the response is sent and committed. The browser receives this response, handles it and immediately re-requests the same URL. The effect is that of events being pushed to the browser. There are pros and cons and it might not be appropriate for all use cases but would provide the most timely status updates.
My opinion is to stick with the polling solution, but you might be interested in this Wikipedia article on HTTP Push technologies.
REST depends on HTTP, which is a request/response protocol. I don't think you're going to get a pure HTTP server calling the client back with status.
Besides, status reporting isn't the job of the service. It's up to the client to decide when, or if, it wants status reported.
One approach I have used is:
When the job is posted to the server, the server responds back a pubnub-channel id (one could alternatively use Google's PUB-SUB kind of service).
The client on browser subscribes to that channel and starts listening for messages.
The worker/task server publishes status on that pubnub channel to update the progress.
On receiving messages on the subscribed pubnub-channel, the client updates the web UI.
You could also use self-refreshing iframe, but AJAX call is much better. I don't think there is any other way.
PS: If you would open a socket from client, that wouldn't change much - PHP browser would show the page as still "loading", which is not very user-friendly. (assuming you would push or flush buffer to have other things displayed before)