I've tried diffrent ways of using memcached. I did not find complete explanation, solution, good practice on how to use it with Symfony 3.3.
Question 1: https://symfony.com/blog/new-in-symfony-3-3-memcached-cache-adapter.
Maybe I'm not seeing something here but how do I tell Symfony what I would like to cache(metadata, query, results - I am using doctrine)? Where I can find more configuration options? Is there something I can read about it?
Question 2: http://www.inanzzz.com/index.php/post/7rum/using-doctrine-memcached-caching-with-query-builder-in-symfony
Running stats commands showed some items in memory but there's no performance increase. Nothing at all. I followed steps very carefully. Does any one know do I have to do something more to see the actual results?
I don't think it matters but as example I am using simple endpoint which is getting list of posts from database.
Question 1: how to use cache?
Please see this example from the blog you referred to (New in Symfony 3.3: Memcached Cache Adapter):
$cacheProduct = $this->get('app.cache.products')->getItem($productId);
if (!$cacheProduct->isHit()) {
$product = ...
$cacheProduct->set($product);
$this->get('app.cache.products')->save($cacheProduct);
} else {
$product = $cacheProduct->get();
}
This cache adapter will allow you to store data in memory. This example uses memcache, but you can also use Redis for example. You have to decide which part of your code is too 'expensive' to perform at every request and can be cached. It's also up to you when the cache should be invalidated, which can be a real pain:
There are two hard things in computer science: cache invalidation, naming things, and off-by-one errors.
https://twitter.com/codinghorror/status/506010907021828096
The blog is about Symfony's Cache Component, but there is DoctrineCacheBundle too. The difference between them can cause some confusion, because you can use Doctrine Cache without Doctrine ORM and vice versa.
If you want Doctrine to use metadata_cache, result_cache or query_cache take a look at the configuration reference. Unfortunately I couldn't find any useful documentation about it's usage on Symfony.com.
Question 2: no performance increase?
I think you're not using any caching mechanisms right now, so try to do so.
You can open a separate question with more details if you still encounter problems after you implemented caching mechanisms.
Related
I am new to Sphinx and want to make it index a 2 million row table (~1.5GB in size). I will use plain indexes.
At the moment, I don't know how much memory should I put in the mem_limit config. My idea is that I could simply keep the default, and then I could see how many results are being swapped (stay in disk) or expired (how frequently used results in memory go to disk).
I'm not sure yet exactly how Sphinx works, anyway, but this is my understanding for now. However, how can I see stats like these, just like we can see the STATS for Memcached?
Having some kind of stats would definitely help me know how to better tune Sphinx for my application.
In case it's relevant, I use MariaDB and PHP on CentOS.
In case its not clear mem_limit is ONLY used by the indexer program. For use during creating the index.
... frankly the setting isnt all that critical. Just set it as high as you have available memory.
It's not applicable to searchd which actually answers queries.
There is 'SHOW STATUS' command, but it doesnt really have anything about menory
http://sphinxsearch.com/docs/current.html#sphinxql-show-status
... memory usage (and there are no variables to control it!) can be got from general OS commands. On linux for example, possibly something like memstat
For some reporting issues I have to use codes as
{{ Model1.Model2.Model3.name }}
in for loops. I know it is not the best way (or may be the worst) to use but things happened and now I have to figure out some way to make this load faster. Because although there are 300 rows it takes nearly 10 secs to load.
My question is, how can I cache some of these results which are not actually queries on the backend? Or would you suggest an other way to make page loading faster?
Have you tried using a different strategy to get the model meta data ? I mean, if you didn't setup anything on your config file, everytime you make a query, Phalcon must query the database first to know the "metadata" of the table(Columns, column type, nullable, etc).
You can change your strategy to annotation, or at least cache the table metadata.
Please check Phalcon documentation
Im trying to find a good way to handle memcache keys for storing, retrieving and updating data to/from the cache layer in a more civilized way.
Found this pattern, which looks great, but how do I turn it into a functional part of a PHP application?
The Identity Map pattern: http://martinfowler.com/eaaCatalog/identityMap.html
Thanks!
Update: I have been told about the modified memcache (memcache-tag) that apparently does do a lot of this, but I can't install linux software on my windows development box...
Well, memcache use IS an identity map pattern. You check your cache, then you hit your database (or whatever else you're using). You can go about finding information about the source by storing objects instead of just values, but you'll take a performance hit for that.
You effectively cannot ask the cache what it contains as a list. To mass invalidate, you'll have to keep a list of what you put in and iterate it, or you'll have to iterate every possible key that could fit the pattern of concern. The resource you point out, memcache-tag can simplify this, but it doesn't appear to be maintained inline with the memcache project.
So your options now are iterative deletes, or totally flushing everything that is cached. Thus, I propose a design consideration is the question that you should be asking. In order to get a useful answer for you, I query thus: why do you want to do this?
I am fetching some questions from the server (database) and showing it to client (user) in the browser. The client will answer the question and based on his/her answer the next set of questions will be fetched from the database. Now, I want to pre-fetch the next set of questions while the user read the present question so that the waiting time for user to see the next question will be shorter.
My questions is, how to store the pre-fetched questions i.e. which data structure should I use to store the pre-fetched questions in the memory so that I can get better performance? I want a "cache" type of thing. Also once the user hit any question from the cache the question won't be there any more.
PS: Each question has unique Id.
Thanks
Naveen
There are multiple options to go about it. One that makes a big difference, one that makes little.
Little difference would be to fetch questions and store it in user's session. It's basically depends on where your session is stored, could also be database, or a file. This only makes sense if your db tables are very denormalized and it requires lots of joins to get the answer. I doubt that's the case so this won't make much difference for the user no matter which data structure used.
Big difference would make prefetching them with AJAX using javascript straight into the browser. In this case a simple array would suffice. JS gives you flexibility to build any objects with any properties, anything would be good enough. So write a poller in JS which fetches the questions from server while user is looking at the question, return them using JSON for example. JSON will become a simple object. Since each user stores only a couple of questions prefetched in their browser particular data structure choice won't make a difference here either.
Try using LinkedHashMap as You will have LRU algorithm implemented quickly with good performance.
Read this link as well :
LinkedHashMap as cache
First a few questions to adapt to your context :
assuming you use Java ?
using Hibernate also ?
If you want to prefetch in the server, many caching solutions exists.
Taking into account your unique id (see PS), if this ID is database related and you are using Hibernate, the easiest solution would be to configure the Hibernate second-level cache for that entity. Then, your only code would be to run the query in advance....
If theses requisites do not fit, I used EhCache as the caching solutions.
Somehow easy to start using, and it has plenty of features available when you later need them.
Does SQLAlchemy support some kind of caching so if I repeatedly run the same query it returns the response from cache instead of querying the database? Is this cache automatically cleared when the DB is updated?
Or what's the best way to implement this on a CherryPy + SQLAlchemy setup?
We have a pretty comprehensive caching solution, as an example in conjunction with embedded hooks, in 0.6. It's a recipe to subclass Query, make it aware of Beaker, and allow control of query caching for explicit queries as well as lazy loaders via query options.
I'm running it in production now. The example itself is in the dist and the intro documentation is at http://www.sqlalchemy.org/docs/orm/examples.html#beaker-caching .
UPDATE: Beaker has now been replaced with dogpile caching: http://docs.sqlalchemy.org/en/latest/orm/examples.html#module-examples.dogpile_caching
Not an answer to your second question, but from the comments in this link indicates that SQLAlchemy does not support cacheing : http://spyced.blogspot.com/2007/01/why-sqlalchemy-impresses-me.html
Raven said...
Does SQLAlchemy do any kind of internal caching?
For example, if you ask for the same data twice (or an obvious subset
of the initially requested data) will the database be hit once or twice?
I recently wrote a caching database abstraction layer for an
application and (while fun) it was a fair bit of work to get it to a
minimally functional state. If SQLAlchemy did that I would seriously
consider jumping on the bandwagon.
I've found things in the docs that imply something like this might be
going on, but nothing explicit.
4:36 PM
Jonathan Ellis said...
No; the author of SA [rightly, IMO] considers caching a separate concern.
What you saw in the docs is probably the SA identity map, which makes it so
if you load an instance in two different places, they will refer
to the same object. But the database will still be queried twice, so it is
not a cache in the sense you mean.
SQLAlchemy supports two types of caches:
Caching the result set so that repeatedly running the same query hits the cache instead of the database. It uses dogpile which supports many different backends, including memcached, redis, and basic flat files.
Docs are here: http://docs.sqlalchemy.org/en/latest/orm/examples.html#module-examples.dogpile_caching
Caching the query object so that Python interpreter doesn't have to manually re-assemble the query string every time. These queries are called baked queries and the cache is called baked. Basically it caches all the actions sqlalchemy takes BEFORE hitting the database--it does not cut down on database calls. Initial benchmarks show speedups of up to 40% in query generation time at the tradeoff of a slight increase in code verbosity.
Docs are here: http://docs.sqlalchemy.org/en/latest/orm/extensions/baked.html
Or use an application-level cache via weak references dicts (weakref.WeakValueDictionary), see an example here : http://www.sqlalchemy.org/trac/wiki/UsageRecipes/UniqueObject