By default, the Symfony2 profiler only saves a few* of the last requests (Don't know how many excactly - something between 25 and 40). I think the list removes the older ones after a cupple of minutes..
I'd like to do a long-time profiling and measuring of some data. Is it possible to configure the max profiling amount? Or maybe store all the data in my (MongoDB) Database? (I'm currently using Symfony 2.2)
Update: I found out, that clearing the cache, also clears the profiler data - so I guess the key issue here is, to save the profiler data somewhere else..
Check this out.
You can override the default profiler.storage service, which is defined here.
You could either define your own class (implementing the ProfilerStorageInterface) or use one of the provided implementations (check here for the whole list).
For example if you want to use the MongoDbProfilerStorage you can redefine the service as follows:
<service id="profiler.storage" class="Symfony\Component\HttpKernel\Profiler\MongoDbProfilerStorage" public="false">
<argument>mongodb://[USER:PWD#]HOST/DATABASE/COLLECTION</argument>
<argument></argument>
<argument></argument>
<argument>86400</argument>
</service>
This way you wouldn't delete profiler data whenever you clear the cache.
Related
I've tried diffrent ways of using memcached. I did not find complete explanation, solution, good practice on how to use it with Symfony 3.3.
Question 1: https://symfony.com/blog/new-in-symfony-3-3-memcached-cache-adapter.
Maybe I'm not seeing something here but how do I tell Symfony what I would like to cache(metadata, query, results - I am using doctrine)? Where I can find more configuration options? Is there something I can read about it?
Question 2: http://www.inanzzz.com/index.php/post/7rum/using-doctrine-memcached-caching-with-query-builder-in-symfony
Running stats commands showed some items in memory but there's no performance increase. Nothing at all. I followed steps very carefully. Does any one know do I have to do something more to see the actual results?
I don't think it matters but as example I am using simple endpoint which is getting list of posts from database.
Question 1: how to use cache?
Please see this example from the blog you referred to (New in Symfony 3.3: Memcached Cache Adapter):
$cacheProduct = $this->get('app.cache.products')->getItem($productId);
if (!$cacheProduct->isHit()) {
$product = ...
$cacheProduct->set($product);
$this->get('app.cache.products')->save($cacheProduct);
} else {
$product = $cacheProduct->get();
}
This cache adapter will allow you to store data in memory. This example uses memcache, but you can also use Redis for example. You have to decide which part of your code is too 'expensive' to perform at every request and can be cached. It's also up to you when the cache should be invalidated, which can be a real pain:
There are two hard things in computer science: cache invalidation, naming things, and off-by-one errors.
https://twitter.com/codinghorror/status/506010907021828096
The blog is about Symfony's Cache Component, but there is DoctrineCacheBundle too. The difference between them can cause some confusion, because you can use Doctrine Cache without Doctrine ORM and vice versa.
If you want Doctrine to use metadata_cache, result_cache or query_cache take a look at the configuration reference. Unfortunately I couldn't find any useful documentation about it's usage on Symfony.com.
Question 2: no performance increase?
I think you're not using any caching mechanisms right now, so try to do so.
You can open a separate question with more details if you still encounter problems after you implemented caching mechanisms.
I am very new to Redis. I've implemented caching in our application and it works nicely. I want to store two main data types: a directory listing and file content. It's not really relevant, but this will cache files served up via WebDAV.
I want the file structure to remain almost forever. The file content needs to be cached for a short time only. I have set up my expiry/TTL to reflect this.
When the server reaches memory capacity is it possible to priorities certain cached items over others? i.e. flush a key, flush a whole database or flush a whole instance of Redis.
I want to keep my directory listing and flush the file content when memory begins to be an issue.
EDIT: Reading this article seems to be what I need. I think I will need to use volatile-ttl. My file content will have a much shorter TTL set, so this should in theory clear that first. If anyone has any other helpful advice I would love to hear it, but for now I am going to implement this.
Reading this article describes what I needed. I have implemented volatile-ttl as my memory management type.
I'm reading this article:
http://37signals.com/svn/posts/3113-how-key-based-cache-expiration-works
I'm not using rails so I don't really understand their example.
It says in #3:
When the key changes, you simply write the new content to this new
key. So if you update the todo, the key changes from
todos/5-20110218104500 to todos/5-20110218105545, and thus the new
content is written based on the updated object.
How does the view know to read from the new todos/5-20110218105545 instead of the old one?
I was confused about that too at first -- how does this save a trip to the database if you have to read from the database anyway to see if the cache is valid? However, see Jesse's comments (1, 2) from Feb 12th:
How do you know what the cache key is? You would have to fetch it from the database to know the mtime right? If you’re pulling the record from the database already, I would expect that to be the greatest hit, no?
Am I missing something?
and then
Please remove my brain-dead comment. I just realized why this doesn’t matter: the caching is cascaded, so yes a full depth regeneration incurs a DB hit. The next cache hit will incur one DB query for the top-level object—all the descendant objects are not queried because the cache for the parent object includes cached versions for the children (thus, no query necessary).
And Paul Leader's comment 2 below that:
Bingo. That’s why is works soooo well. If you do it right it doesn’t just eliminate the need to generate the HTML but any need to hit the db. With this caching system in place, our data-vis app is almost instantaneous, it’s actually useable and the code is much nicer.
So given the models that DHH lists in step 5 of the article and the views he lists in step 6, and given that you've properly setup your relationships to touch the parent objects on update, and given that your partials access your child data as parent.children, or even child.children in nested partials, then this caching system should have a net gain because as long as the parent's cache-key is still valid then the parent.children lookup will never happen and will also be pulled from cache, etc.
However, this method may be pointless if your partials reference lots of instance variables from the controller since those queries will already have been performed by the time Rails sees the calls to cache in the view templates. In that case you would probably be better off using other caching patterns.
Or at least this is my understanding of how it works. HTH
Im trying to find a good way to handle memcache keys for storing, retrieving and updating data to/from the cache layer in a more civilized way.
Found this pattern, which looks great, but how do I turn it into a functional part of a PHP application?
The Identity Map pattern: http://martinfowler.com/eaaCatalog/identityMap.html
Thanks!
Update: I have been told about the modified memcache (memcache-tag) that apparently does do a lot of this, but I can't install linux software on my windows development box...
Well, memcache use IS an identity map pattern. You check your cache, then you hit your database (or whatever else you're using). You can go about finding information about the source by storing objects instead of just values, but you'll take a performance hit for that.
You effectively cannot ask the cache what it contains as a list. To mass invalidate, you'll have to keep a list of what you put in and iterate it, or you'll have to iterate every possible key that could fit the pattern of concern. The resource you point out, memcache-tag can simplify this, but it doesn't appear to be maintained inline with the memcache project.
So your options now are iterative deletes, or totally flushing everything that is cached. Thus, I propose a design consideration is the question that you should be asking. In order to get a useful answer for you, I query thus: why do you want to do this?
Does SQLAlchemy support some kind of caching so if I repeatedly run the same query it returns the response from cache instead of querying the database? Is this cache automatically cleared when the DB is updated?
Or what's the best way to implement this on a CherryPy + SQLAlchemy setup?
We have a pretty comprehensive caching solution, as an example in conjunction with embedded hooks, in 0.6. It's a recipe to subclass Query, make it aware of Beaker, and allow control of query caching for explicit queries as well as lazy loaders via query options.
I'm running it in production now. The example itself is in the dist and the intro documentation is at http://www.sqlalchemy.org/docs/orm/examples.html#beaker-caching .
UPDATE: Beaker has now been replaced with dogpile caching: http://docs.sqlalchemy.org/en/latest/orm/examples.html#module-examples.dogpile_caching
Not an answer to your second question, but from the comments in this link indicates that SQLAlchemy does not support cacheing : http://spyced.blogspot.com/2007/01/why-sqlalchemy-impresses-me.html
Raven said...
Does SQLAlchemy do any kind of internal caching?
For example, if you ask for the same data twice (or an obvious subset
of the initially requested data) will the database be hit once or twice?
I recently wrote a caching database abstraction layer for an
application and (while fun) it was a fair bit of work to get it to a
minimally functional state. If SQLAlchemy did that I would seriously
consider jumping on the bandwagon.
I've found things in the docs that imply something like this might be
going on, but nothing explicit.
4:36 PM
Jonathan Ellis said...
No; the author of SA [rightly, IMO] considers caching a separate concern.
What you saw in the docs is probably the SA identity map, which makes it so
if you load an instance in two different places, they will refer
to the same object. But the database will still be queried twice, so it is
not a cache in the sense you mean.
SQLAlchemy supports two types of caches:
Caching the result set so that repeatedly running the same query hits the cache instead of the database. It uses dogpile which supports many different backends, including memcached, redis, and basic flat files.
Docs are here: http://docs.sqlalchemy.org/en/latest/orm/examples.html#module-examples.dogpile_caching
Caching the query object so that Python interpreter doesn't have to manually re-assemble the query string every time. These queries are called baked queries and the cache is called baked. Basically it caches all the actions sqlalchemy takes BEFORE hitting the database--it does not cut down on database calls. Initial benchmarks show speedups of up to 40% in query generation time at the tradeoff of a slight increase in code verbosity.
Docs are here: http://docs.sqlalchemy.org/en/latest/orm/extensions/baked.html
Or use an application-level cache via weak references dicts (weakref.WeakValueDictionary), see an example here : http://www.sqlalchemy.org/trac/wiki/UsageRecipes/UniqueObject