Heroku is imposing 300MB limit on slugsize. Normally, this should be way more than enough for most of the web apps. However, our company uses libraries that are frequently 50MB or more each, and there are a lot of those.
Is there anyway to increase the slug size limit on Heroku? Has anyone had any success with overcoming this limit?
Reading a bit from the documentation, it seems that it is not something Heroku does. The limitation is related to their expectations of storage consumed per application and circumventing it may result in malfunctioning services. In case they consider it a contract breach, they may take down your service. So I would not advise trying really hard. The documentation tells that smaller slugs will deploy faster and your question is directly against it.
Related
I implemented oplog on our server and that time our application response time improved but after some hour response time increased and application response was very slow.
Can you let me know the
Disadvantage of Oplog
Impact of Oplog on Meteor Application
what needs to take care while implementing oplog.
Please help me I go through several video and link but not find any satisfactory answer, thanks.
As mentioned in the comments, this is a very broad question and the "correct" answer completely depends on your specific situation (e.g. app needs, use case, etc.). Nevertheless, here is my attempted answer based upon having been thru the challenges of scaling Meteor apps.
If at all possible, you will want to enable oplog tailing in your production Meteor app. If you have done any Meteor development, then you are used to using the oplog because it is enabled by default.
The symptom you describe of the application response times increasing over time and becoming very slow is likely the result of something else going on with your app or hosting environment / infrastructure. I have run multiple production Meteor apps with 100+ simultaneous users and have never experienced this occurring.
There is one specific situation where you would NOT want to use the oplog and that is if you have a complex query where the bulk of the resulting data gets updated often. This can causing CPU spikes and/or thrashing and will kill your apps performance. I have one such application that falls into this category and after extensive testing, I found that it is much better to disable oplog on the query and increase the pollingThrottleMs accordingly. Again, this is an exception case and represents about the only time that you would want to stay away from using the oplog.
These are just some very surface level thoughts on the use of oplog based on my experience. I encourage you to experiment and see what works best for your app.
I understand that there are limitations with regards number of connections etc. but it doesn't have a tick next to "Production Ready"
What does this mean? I cannot use it for production?
There's really not much documentation on this. It's a play to get you paying by suggesting that general performance will be subpar until you do. Which is nonetheless probably very true. Based on performance I've had with Ignite in the past, I certainly wouldn't want to have more than a few people connecting at a given time. Plus, you'll almost certainly tap out of the allowed 5mb of storage very quickly if you're doing anything resource intensive anyway.
Is there a tool or a way to learn how many connections can manage my Heroku app simultaneously (with one dyno) before giving slow response times or time outs? I read of Blitz and New Relic but I am unsure of how to use them!
There's no quick and easy way to understand how your app scales. But the process usually goes along these lines:
Launch your target environment (a single dyno in your case)
Set up monitoring on all the possible metrics you care about. Usually this will include: CPU load, memory usage, I/O operations, database connections, etc. as well as any relevant applicative metrics. For Heroku, I recommend using Librato for a complete monitoring set.
Run load tests that resemble typical usages of your application, this means not just simple reads of static pages, but also dynamic operations such as user registrations, complex API calls, and anything else you think is relevant. The tools used here really depend on what your app does and how it is built.
See where you hit your limits, assume nothing, you might be bound by any of the resources you are using.
Resolve bottlenecks, rinse, repeat.
This will give your more or less a clue as to where your application will require further resources in order to scale.
We have a new project for a web app that will display banners ads on websites (as a network) and our estimate is for it to handle 20 to 40 billion impressions a month.
Our current language is in ASP...but are moving to PHP. Does PHP 5 has its limit with scaling web application? Or, should I have our team invest in picking up JSP?
Or, is it a matter of the app server and/or DB? We plan to use Oracle 10g as the database.
No offense, but I strongly suspect you're vastly overestimating how many impressions you'll serve.
That said:
PHP or other languages used in the application tier really have little to do with scalability. Since the application tier delegates it's state to the database or equivalent, it's straightforward to add as much capacity as you need behind appropriate load balancing. Choice of language does influence per server efficiency and hence costs, but that's different than scalability.
It's scaling the state/data storage that gets more complicated.
For your app, you have three basic jobs:
what ad do we show?
serving the add
logging the impression
Each of these will require thought and likely different tools.
The second, serving the add, is most simple: use a CDN. If you actually serve the volume you claim, you should be able to negotiate favorable rates.
Deciding which ad to show is going to be very specific to your network. It may be as simple as reading a few rows from a database that give ad placements for a given property for a given calendar period. Or it may be complex contextual advertising like google. Assuming it's more the former, and that the database of placements is small, then this is the simple task of scaling database reads. You can use replication trees or alternately a caching layer like memcached.
The last will ultimately be the most difficult: how to scale the writes. A common approach would be to still use databases, but to adopt a sharding scaling strategy. More exotic options might be to use a key/value store supporting counter instructions, such as Redis, or a scalable OLAP database such as Vertica.
All of the above assumes that you're able to secure data center space and network provisioning capable of serving this load, which is not trivial at the numbers you're talking.
You do realize that 40 billion per month is roughly 15,500 per second, right?
Scaling isn't going to be your problem - infrastructure period is going to be your problem. No matter what technology stack you choose, you are going to need an enormous amount of hardware - as others have said in the form of a farm or cloud.
This question (and the entire subject) is a bit subjective. You can write a dog slow program in any language, and host it on anything.
I think your best bet is to see how your current implementation works under load. Maybe just a few tweaks will make things work for you - but changing your underlying framework seems a bit much.
That being said - your infrastructure team will also have to be involved as it seems you have some serious load requirements.
Good luck!
I think that it is not matter of language, but it can be be a matter of database speed as CPU processing speed. Have you considered a web farm? In this way you can have more than one machine serving your application. There are some ways to implement this solution. You can start with two server and add more server as the app request more processing volume.
In other point, Oracle 10g is a very good database server, in my humble opinion you only need a stand alone Oracle server to commit the volume of request. Remember that a SQL server is faster as the people request more or less the same things each time and it happens in web application if you plan your database schema carefully.
You also have to check all the Ad Server application solutions and there are a very good ones, just try Google with "Open Source AD servers".
PHP will be capable of serving your needs. However, as others have said, your first limits will be your network infrastructure.
But your second limits will be writing scalable code. You will need good abstraction and isolation so that resources can easily be added at any level. Things like a fast data-object mapper, multiple data caching mechanisms, separate configuration files, and so on.
I have been doing some catching up lately by reading about cloud hosting.
For a client that has about the same characteristics as StackOverflow (Windows stack, same amount of visitors), I need to set up a hosting environment. Stackoverflow went from renting to buying.
The question is why didn't they choose cloud hosting?
Since Stackoverflow doesn't use any weird stuff that needs to run on a dedicated server and supposedly cloud hosting is 'the' solution, why not use it?
By getting answers to this question I hope to be able to make a weighted decision myself.
I honestly do not know why SO runs like it does, on privately owned servers.
However, I can assume why a website would prefer this:
Maintainability - when things DO go wrong, you want to be hands-on on the problem, and solve it as quickly as possible, without needing to count on some third-party. Of course the downside is that you need to be available 24/7 to handle these problems.
Scalability - Cloud hosting (or any external hosting, for that matter) is very convenient for a small to medium-sized site. And most of the hosting providers today do give you the option to start small (shared hosting for example) and grow to private servers/VPN/etc... But if you truly believe you will need that extra growth space, you might want to count only on your own infrastructure.
Full Control - with your own servers, you are never bound to any restrictions or limitations a hosting service might impose on you. Run whatever you want, hog your CPU or your RAM, whatever. It's your server. Many hosting providers do not give you this freedom (unless you pay up, of course :) )
Again, this is a cost-effectiveness issue, and each business will handle it differently.
I think this might be a big reason why:
Cloud databases are typically more
limited in functionality than their
local counterparts. App Engine returns
up to 1000 results. SimpleDB times out
within 5 seconds. Joining records from
two tables in a single query breaks
databases optimized for scale. App
Engine offers specialized storage and
query types such as geographical
coordinates.
The database layer of a cloud instance
can be abstracted as a separate
best-of-breed layer within a cloud
stack but developers are most likely
to use the local solution for both its
speed and simplicity.
From Niall Kennedy
Obviously I cannot say for StackOverflow, but I have a few clients that went the "cloud hosting" route. All of which are now frantically trying to get off of the cloud.
In a lot of cases, it just isn't 100% there yet. Limitations in user tracking (passing of requestor's IP address), fluctuating performance due to other load on the cloud, and unknown usage number are just a few of the issues that have came up.
From what I've seen (and this is just based on reading various blogged stories) most of the time the dollar-costs of cloud hosting just don't work out, especially given a little bit of planning or analysis. It's only really valuable for somebody who expects highly fluctuating traffic which defies prediction, or seasonal bursts. I guess in it's infancy it's just not quite competitive enough.
IIRC Jeff and Joel said (in one of the podcasts) that they did actually run the numbers and it didn't work out cloud-favouring.
I think Jeff said in one of the Podcasts that he wanted to learn a lot of things about hosting, and generally has fun doing it. Some headaches aside (see the SO blog), I think it's a great learning experience.
Cloud computing definitely has it's advantages as many of the other answers have noted, but sometimes you just want to be able to control every bit of your server.
I looked into it once for quite a small site. Running a small Amazon instance for a year would cost around £700 + bandwidth costs + S3 storage costs. VPS hosting with similar specs and a decent bandwidth allowance chucked in is around £500. So I think cost has a lot to do with it unless you are going to have fluctuating traffic and lots of it!
I'm sure someone from SO will answer it but "Isn't just more hassle"? Old school hosting is still cheap and unless you got big scalability problems why would you do cloud hosting?