I recently launched my app for iPhone/Android running with AppEngine backend. This is my first experience using AppEngine in Production.
As I get more traffic, I am starting to experience serious latency issues. Currently minimum idle instance is 1, max_pending_latency is 1s.
Yes, there are rooms for optimizations on my side, however I do not understand
Why the latency is not correlated with request/sec, traffic, memoryUsage, memcacheUsage, anything. I do not understand why there was no significant latency on Sep 21.
Why the call to memcached needs to be as slow as 500ms. (Usually it is 10 times faster). I am using NDB and 1GB dedicated memcached. Increasing to 5GB had no effect.
Is this simply how AppEngine works? I would like to get your insight.
Thanks
I forgot to update this.... I remember the issue was caused by me creating datastore keys by myself. Basically, not well distributed keys introduced "hot tablet" problem. Once I stopped creating my own keys and let the AppEngine create them, the issue seemed to be resolved.
We experienced a very long time in deserialization when we stored a lot of entities under the same memcache key. It can take a long time if you store a big array of some entities with a lot of structured properties.
You cannot store object size more than 1Mo in the same cache key. You can use Titan for App Engine to divide your cache key in several others cache key using the sharded memcache. It's transparent.
I hope it will help you.
Related
In my office, we have a server that has Ncache installed for storing and retrieving data and our applications are also hosted there.
There was an issue where application was getting timed out. In depth, i found that getting cache method from Ncache is taking 8-9 seconds, which was previously taking 0.5 seconds. The application isn't changed recently and it was working fine previously. All of a sudden this issue has occurred. Some one told me that there was an issue where all of a sudden all clustered cache were deleted from ncache manager and we resolved it by setting basic values from tutorial available online. But this issue seems to be never getting solved. Can anyone through some light on it that we can do to overcome this time out issue?
This seems like some application/environment related issue where a working application is now showing slow fetch time while it was working fine previously. Also, if your console app is getting results in less than a second then it again shows that issue is not from NCache server end but isolated to the application.
I will suggest to review what has been changed in the application to start off. You can also profile your application on which calls are taking more time now. NCache client side windows performance counters can also be reviewed to rule out if it is slow because of NCache or due to some application related issue.
Moreover, caching an object which is huge in size is generally not recommended. You should always break your bigger objects to smaller objects and then cache them. This will reduce network and storage overhead for your application. If you have to use bigger object then consider using compression.
NCache default settings are already tuned for optimum performance and should not slow things down. You should check firewall between client and NCache server to rule out any environmental issues.
I have no experience with Graph DB applications but I'm trying to write one. I intend to host on Heroku.
I can see there are 2 Graph DB service providers with free plans but I can't decide which one to use, they are both marketing themselves using different attributes, and I can't compare ! For example:
GrapheneDB mentions only the node and relationship count limit, and the query time limit. But nothing about the storage limit.
Graph Story mentions the RAM limit, `storage limit and data transfer limit.
Other properties are mentioned too but they aren't comparable between both providers.
Has anyone tried any of these services on Heroku and could share his experience please ?
EDIT: I found this page which give an idea about how much space does neo4j need.
I'll take a spin at answering this question by staying as much as objective as possible, as, I and some other frequent answerers here, have good relationships with both providers.
Both have their own pro's and con's, and I think looking only at the Heroku side is maybe not a good choice.
There is also one difference between both that you need to know, GraphStory provide Neo4j enterprise while GrapheneDB provide Neo4j Community, this is a fact. However I am personally thinking that if you run neo4j on heroku, then you don't need enterprise because "enterprise" users of Neo4j are using their own environment with clustering on servers with "real" RAM and SSD's, which in fact can be managed by both providers with a licence and support.
You speak about the storage limit. Well the storage depends about your amount of nodes, relationships and properties in the database, so if there is a limit of 1000 nodes you don't need to care about the storage limit I think.
I tried both on heroku, and except the nodes limit, there is not that much difference in matters of performance when you deploy free dynos.
If you are a startup, running Neo4j on heroku is great if you take the paid plan of course, both providers have cool support and both are rewarding their long term customers.
If you look only at the free dynos, then you don't need to care about the limitations, because it will just be LIMITED, in any way !
Outside of Heroku, here are some points I viewed :
GrapheneDB runs on all platforms including Azure which is a cool stuff
GraphStory runs enterprise so you can benefit from the high performance cache
GrapheneDB has an accessible API for creating neo4j servers on the fly and destroying it.
Depending of your location, you may want support from Europe or from US.
basic plans, on both, are suffering of some latency or boot time when not used for a long time
Both have support for spatial
Both are actors in the Neo4j community with cool stuff, you can meet them in real :)
Now, you can test them, both, for free !!!
I tried yesterday one CRUD application deployed in 2 Heroku app: the first with Graph Story and the other with GrapheneDB.
I had monitored with NewRelic and I detected Graph Story app have a medium latency variable from 1 to 2 seconds, instead the GrapheneDB service need only from 20 to 40 ms to perform the same operations.
Graph Story latency:
GrapheneDB latency:
I wanted to try a paid plan for some minutes in Graph Story, but for doing that you need to contact the assistance and waiting for an unknown time. Instead, GrapheneDB allow you to change plan autonomously without any issue.
I tried to export db in Graph Story, but the operation is not in real time: You need to wait for a link dispatched via email. I initiate the operation for 2 times, but the email after 10 hours not yet arrived.
Instead in GrapheneDB the export is immediate without waiting anxious emails
Graph Story offers the following features that differentiate it from other offerings:
Graph Story offers the Enterprise version of Neo4j
There are no limits on nodes or relationships on the free plan
Max query time is 30 seconds
You wouldn't want to use the free plan in production, of course, but it's excellent for proofs-of-concept, learning Neo4j, small hobby projects, etc.
(Full disclosure: I'm the CTO at Graph Story.)
I have recently encountered a question that since redis is not distributed and don't support parallelism(multi-core), isn't elastic search be a better choice to use instead of redis for caching purpose.
This is all in reference to a simple web, where we used redis to cache db query.
I have kind of got the idea here,
but still not sure whether it has any real benefits. Opening up this thread to discuss the advantages/disadvantages in doing so.
It's not really what you asked for but you might want to have a look at Aerospike.
Redis is an in-memory data structure store known for speed and often used as a cache. Both Redis and Aerospike are open source, however, when applications need persistence or when applications must scale but servers have maxed out of RAM, developers should consider Aerospike, a distributed key-value store that is just as fast or faster than Redis, but scales more simply and with the economics of flash/SSDs.
We are using AppFabric 1.1 for caching in a multi services application.
The base functionalities works just fine but now we're fine tuning and we stumble across some memory issues.
We saw that AppFabric is using more and more memory as time goes by, yet we're supposed to have a finite volume of stored objects.
I spent some time reading online and checking my code but I couldn't find why. So before engaging in war against "memory leak", I mean storage of un-referenced data or not referenced anymore, I'd like to know if this is a standard behavior in AppFabric ?
Could the massive use of GetLock-PutUnlock and massive readings and writings could cause the memory to go up to the limit allowed for the cache cluster ?
We used different size limits (1024 - 1536 - 2048) and different watermarks but it still goes up to either the maximum size limit or High watermark (we tried 75, 80 and 90).
If you need anymore information please tell me.
Thanks in advance.
EDIT
we decided that we had to abandon AppFabric as there seems to be no valid work-around for this issue. Instead we'll use a Windows Azure hosted Redis cache. Hopefully there will be everything we need in Redis.
If a fix or a solution is known I'd still like to know of it, if anyone has it.
Thanks.
I would like to know if any of you had, or may propose a solution to increase efficiency for online store based on Magento platform.
We currently use multifront architecture (front == each separate server) using load-balancing and two Memcache servers.
We're considering connection for each separate front an Memcache server, but at this point a problem arises with memcache synchronization, so that each store the same value.
Any advice appreciated :)
If you are running memcached on its own hardware, there is no benefit to giving each store its own memcached server.
Configure all front ends to use both memcached instances. This way, all front ends will go to the same memcached instance for a given key. Plus, you get automatic fail-over if one instance croaks, and you can scale up almost linearly as the demand for cache increases.
One of the biggest improvements I've seen for more advanced setups like these is to use a PHP Opcode cache like Xcode. Since Magento uses many PHP files, the opcode cache will end up saving a lot of compilation between runs. Also make sure that all your caches are turned on and leveraged as much as possible. Enable flat catalog and flat product tables as well.