I am developig an import module which update product data. To speed up the process, I put index to manual mode.
$processes = Mage::getSingleton('index/indexer')->getProcessesCollection();
$processes->walk('setMode', array(Mage_Index_Model_Process::MODE_MANUAL));
$processes->walk('save');
and after the import is finished, I reindex data and put index mode back to auto
$processes = Mage::getSingleton('index/indexer')->getProcessesCollection();
$processes->walk('reindexAll');
$processes->walk('setMode', array(Mage_Index_Model_Process::MODE_REAL_TIME));
$processes->walk('save');
But I am not sure if I also need to clear cache. So my question is how index and cache are related. For example if I clear cache, does it also reindex all data? And on the other site, if I reindex all data, does it clear cache? Or do I need to trigger everytime both processes if I have index mode set to manual? I am not quite sure about this, I hope anybody could confirm it for sure.
Thank you
Magentos System -> Cache Managment and System -> Index Managment are both stand-alone features. If you rebuild such index, no matter whether thru backend or directly using reindexAll(), Magento will not automatically refresh any cache and vice versa.
The answer to Do I need both? (caches and indexes) is: it depends.
If you're running Magento with caches COLLECTION_DATA and/or EAV enabled, you should refresh those caches after importing and reindexing product data.
The refreshing is necessary because your importer has updated/inserted product data which the caches are not aware of, but not, because you've reindexed.
If you're running Magento with all caches disabled, you don't need both. Technically, there is no need to refresh disabled cache. Magento would be slower of course, but still be fully functional.
Related
when the new products get uploaded they are not visible on the front end. They told me we need to re-index every time. However re-index gets stuck.Please look into this issue for me and let me know if we can have it reindex automatically.
Reindex mode is Update on save and currently showing processing in index management
If it shows as processing then it's probably running. It may take a long time depending on how many products you have. The core url rewrite index is the most intensive. The reason you need to index is most likely because you have enabled the Use Catalog Flat Tables options under system configuration which is good for performance and speed of the site. Downside is you need to re-index these flat tables when you add new items. You can have the indexes run automatically by configuring cron to call magentos index process.
e.g.
* * * * * /usr/bin/php /var/www/magentosite/cron.php
5 0 * * * /usr/bin/php /var/www/magentosite/shell/indexer.php --reindex all
If it is stuck and you are sure it's not running, then you can change the status of the index process back to pending manually in the database by editing the table index_process. Re-run it manually from the admin and see if it fails. The index most likely causing your products not to show up is Flat Catalog Product Indexes so run this one first.
Yes, When you insert new product so re-indexing is require and also disable you magento cache.
and also flush your magento cache.
I have never used memcached before and I am confused on the following basic question.
Memcached is a cache right? And I assume we cache data from a DB for faster access. So when the DB is updated who is responsible to update the cache? Our code is does memcached "understand" when the DB has been updated?
Memcached is a cache right? And I assume we cache data from a DB for
faster access
Yes it is a cache, but you have to understand that a cache speed up the access when you are often accessing same data. If you access thousand times data/objects which are always different each other a cache doesn't help.
To answer your question:
So when the DB is updated who is responsible to update the cache?
Always you but you don't have to worry about if you are doing the right thing.
Our code is does memcached "understand" when the DB has been updated?
memcached doesn't know about your database. (actually the client doesn't know even about servers..) So when you use an object of your database you should check if is present in cache, if not you put in cache otherwise you are fine.. that is all. When the moment comes memcache will free the memory used by old data, or you can tell memcached to free data after a time you choose(read the API for details).
You are responsible to update the cache (or some plugin).
What happens is that the query is compressed to some key features and these are hashed. This is tested against the cache. If the value is in the cache, the data is returned directly from cache. Otherwise the query is performed, stored in cache and returned to the user.
In pseudo code:
key = query_key(your_sql_query)
if key in cache:
return cache.get(key)
else:
results = execute(your_sql_query)
cache.set(key, results, time_to_live)
return results.
The cache is cleared once in a while, you can give a time to live to a key, then your cached results are refreshed.
This is the most simple model, but can cause some inconsistencies.
One strategy is that if your code is also the only app that updates data, then your code can also refresh memcached as a second step after it has updated the database. Or at least evict the stale data from memcached, so the next time an app wants to read it, it will be forced to re-query the current data from the database and restore that latest data to memcached.
Another strategy is to store data in memcached with an expiration time, so memcached automatically purges that data element after a certain time. You pick the expiration time, based on your knowledge of how frequently the data might be updated, and how tolerant your app is of reading stale data.
So ultimately, you are the one responsible for putting data into memcached. Only you know what data is worth storing in the cache, what format you want to store it in, how frequently you expect to query it, and when to refresh it. You make this judgment on a case-by-case basis, because you know better than any automatic system the likely behavior of your data and your app.
I have a Magento site in which the cross-selling products do not seem to be appearing.
After looking on Stack and Google it seems that 'reindexing the data' has solved this issue for a lot of individuals.
My question is, are there any risks associated with performing this task? Or is it a relatively straight forward procedure?
Indexing is a fundamental part of Magento and will not effect your site in a negative way.
Magento uses a complex EAV (entity-attribute-value) database structure that can sometimes require heavy database queries to retrieve simple results. Because of this, the Magento developers have implemeted Index tables that query all of this data, and store it into a single table structure. This allows Magento to quickly query the single Index table, rather than making complex joins across multiple tables.
With that being said, Reindexing does not alter your existing data. It simply queries your existing data and copies it to it's own tables.
To reindex your site, you can simple go to System > Index Management, check off all the indexes that you wish to reindex, then submit.
If you have a large set of products, I recommend reindexing your site from the shell command line.
Login to your site using an SSH program (such as Putty)
Once logged in, cd to your magento/shell/ (where magento is your Magento root directory)
Run the following command to reindex your site: php indexer.php reindexall
Wait for the index processes to complete.
Lastly, ensure that your Catalog is using the Flat index tables. To do this:
Go to System > Configuration > Catalog > Frontend (section)
Set Use Flat Catalog Category to Yes
Set Use Flat Catalog Product to Yes
Click Save Config
No, you're safe to reindex whenever you see the notice appear.
If you know you're going to make a lot of changes, you can wait until you're done, saving yourself some time but only running it once at the end.
The only exception where this is not safe is if you have tens of thousands of products and/or lots of store views. It may end up running for hours and hours, slowing down your site leading to an undesirable experience for the customer.
I have found on sites with a large number of products, running the price reindex can cause a database lock, which can cause certain actions to be unavailable and for orders to be duplicated during that time. It also can affect performance and eat resources. I recommend performing this late at night only if possible.
I am importing about 10.000 products and updating their data with import custom script on regular basic. I use Magento object to save product data. The problem is that for each product save the process is slower. On 1000 products save it becomes really slow. When I clear cache, everything is ok again.
I have now couple of questions to understand the thing:
Does anybody have any idea why is that?
Should I disable "Collections Data" cache, or maybe any other type of cache as well?
Or is there any way to tell Magento not to cache collection data on product save?
If not, will disabling Collections Data cache slow the page a lot?
Thank you
The reason for the slowness is your indexes are getting larger. Unless specified Magento will reindex for each new product, you can speed this up during your imports by disabling it, however you will need to reindex at some point to be able to present the newly imported products to the frontend.
A solution to consider:
Magento API: Rebuild Indexes after adding new products
I'm developing a Magento extension that introduces a new table to the DB. Whenever I release a new version of the extension that makes modifications to the table schema, I find that users are forced to manually click the "Flush Cache Storage" button under System > Cache Management.
I'd like to have my extension automatically clear the cache upon being installed. I know how to do the same thing as the button programmatically but I'd prefer not to, because this deletes the entire Magento cache folder and impacts negatively on performance.
Might anyone know how to write code that will clear the caching of my table's schema and do so as specifically as possible - leaving unrelated cached data unharmed?
Update: I've found the file containing my table's schema cache here: /var/cache/mage-f/mage---d07_DB_PDO_MYSQL_DDL_<table_name>_1d . Now how do I target it in code? :)
This is what I've been able to come up with:
$app = Mage::app();
if ($app != null)
{
$cache = $app->getCache();
if ($cache != null)
{
$cache->clean('matchingTag', array('DB_PDO_MYSQL_DDL'));
}
}
This will delete only the cache entries and metadata files that hold information about DB schemas.
Note that it will delete these entries for all tables. There's no simple way to clear a specific table's cached schema and leave the rest untouched.