I'm currently stuck at my magento shop and I can't find a solution. I want to import almost 600.000 products. The process of it is really slow. The more product are in the system, the slower the import of new products works. I also can't open some category pages. Its simply not loading.
Any idea what could be the problem?
If your indexes are set to auto-update you should switch them to manual mode before importing. this is a common reason for decreasing performance (and db locks)
$collection = Mage::getSingleton('index/indexer')->getProcessesCollection();
foreach ($collection as $process) {
$process->setMode(Mage_Index_Model_Process::MODE_MANUAL)->save();
}
after import you could switch back to
$process->setMode(Mage_Index_Model_Process::MODE_REAL_TIME)->save();
You should look into https://github.com/avstudnitz/AvS_FastSimpleImport
This one is based on Mage_ImportExport_Model_Import_Entity_Product and is therefore much faster, than instantiating new product model on each iteration
For Faster import, you can use Magmi : http://wiki.magmi.org/index.php?title=Main_Page
once you upload products through, reindex through system > Index Management....
Related
I have created a extension that filters product collection based on a attribute.
Below is the controllers, block and view template code.
Controller
$url = Mage::getUrl('no-route');
if(Mage::app()->getRequest()->getParam('ajax')){
echo $this->getLayout()->createBlock('catalogextensions/bestsellers_home_list')
->setTemplate('catalog/landings/bestseller.phtml')
->toHtml();
}
else{
$this->loadLayout();
$this->getLayout()->getBlock('head')->setTitle('Besesellers');
$this->renderLayout();
}
Block product collection function
$storeId = Mage::app()->getStore()->getId();
$products = Mage::getResourceModel('catalog/product_collection')
->addAttributeToSelect('*')
->addAttributeToFilter(array( array( 'attribute'=>'top_seller', 'eq' => '1' )));
Mage::getSingleton('catalog/product_status')->addVisibleFilterToCollection($products);
Mage::getSingleton('catalog/product_visibility')->addVisibleInCatalogFilterToCollection($products);
Mage::getSingleton('cataloginventory/stock')->addInStockFilterToCollection($products);
$products->getSelect()->limit(4,$this->get_cur_page());
return $products ;
Product collection is iterated on view.phtml template.
Now, it is taking 35 seconds to get the output, below are the statistics of execution that i was able to get by making use of PHP's microtime() function
For
Block(product collection ) it taking around 0.01 second
Template Rendering its taking around 0.12 second
But for controller function it is taking around 35 second
I am not able to find what to check, because controller function just create a Block at runtime.
*Note:I am making use of a Paid full page cache Extension "Mirasvit FPC".
how can i find it what is taking time
Server configuration is
30GB Ram with 4 vCPU.
Application server : Nginx + php5-fpm.
Version: Magento CE 1.8.0.1
Thanks.
You use a bit deprecated way to call the filters. I don't know if that can really influate the time of loading but using addFieldToFilter() would limit the calls to database and can be usefull. You can use :
Magento: Filter products by Status
filtering product collection on "is_salable"
The block is maybe not cached correctly. I mean the products are load each time the page is called.
Make sure your flat product is set to on and all attributes needed for your filters are set as filter.
If a listing comes from EAv tables it might consume lots of resources thus slowing down.
Additionally apply optimization tricks.
First, are you sure you need to add all attributes to select?
You can replace the
addAttributeToSelect('*')
with
addAttributeToSelect(array('attribute_code', 'attribute_code'))
Next, what exactly is that
$this->get_cur_page()
doing? Rest of code shouldn't make disastrous performance issues.
PS
This should have been a comment, but I'm unable to comment yet.
I am updating the Stock on Hand Inventory Qty on a Magento store through an extension that I have developed, using the following code:
Mage::getModel("cataloginventory/stock_item")
->loadByProduct($pid)
->setQty($qty)
->save();
Now, from my testing, this works fine, however I am a little concerned if this is having any negative affects on the different types of product that can be created in Magento (such as simple and complex products).
Is the above the correct way to update the SOH, and do I need to handle Complex products any differently? My gut feeling is that I don't need to do anything differently with complex products as they all end up deriving from a Simple product which has its own Stock on Hand?
Any advice appreciated
As long as you only update simples you'll be fine like this. Indeed all other non-virtual product types derive their stock from the simples.
You might even want to add
$stockItem = Mage::getModel("cataloginventory/stock_item")
->loadByProduct($pid)
->setQty($qty);
if ($stockItem->getCanBackInStock() && $stockItem->getQty() > $stockItem->getMinQty()) {
$stockItem->setIsInStock(true)
->setStockStatusChangedAutomaticallyFlag(true);
}
$stockItem->save();
See Mage_CatalogInventory_Model_Stock::backItemQty() to see how Magento adds stock.
using magento 1.7.0.2 here, having already tested lot of recommended solutions (reindexing, clearing cache, assuring no url rewrite is done, etc) I give up and ask the question.
We are importing a bunch of products (around 150) into the demo store, together with 2 new categories and lot of new attributes (around 200). After some issues we finally managed to get the products shown both on backend and frontend, great.
But when clicking on any newly imported product we get the 404 error page from Magento with the "Whoops, our bad...". After 2 days searching and trying different approaches still haven't found a solution. Reindexing, clearing cache, checking the url rewrite option, changing 'localhost' for '127.0.0.1' on the core_config_data table, and some other solutions that worked for others, no use for us.
Any clue?
Other than the solutions found we have checked that the products are enabled, visibility is set for both catalog and search, there is a website assigned and quantity is bigger than zero. Also we manually created a product with all the correct attributes values and its description page is visible. Comparing (both using backend and database tables/fields) with the imported products averything seems to be exactly the same.
I'm self answering. Thanks to #sulman that pointed in the correct direction.
Yes, visibility was ok for all the products and it was correctly settled as for both Catalog and Search. But the problem was that in our import script we didn't check for the existence and correct assignment of the attribute to the imported product. Therefore the attribute existed but it was not linked to the already existing visbility attribute.
Here a sample code in case is needed, see the opening if (isset($attributeId)) that checks if the attribute exists or not.
if (isset($attributeId))
{
$entityAttributeModel = Mage::getModel('catalog/product_attribute_api');
$attributeList = $entityAttributeModel->items($attributeSetId);
$attributeLink = $this->search($attributeList, 'attribute_id', $attributeId);
if (sizeof($attributeLink) == 0)
{
$setup = new Mage_Eav_Model_Entity_Setup('core_setup');
$setup->addAttributeToGroup($this->getEntityTypeId(), $attributeSetId, $groupId, $attributeId);
}
$model = Mage::getModel('catalog/resource_eav_attribute');
$attribute = $model->load($attributeId);
}
today I'm fighting with Magento again :) and I found a difference between
$product = Mage::getModel('catalog/product')->loadByAttribute('sku', $product_sku);
and
$product = Mage::getModel('catalog/product')->load($product_id);
Can anyone exaplain me a difference between these two approaches? I found that when I'm loading a product by sku then when I 'try to re-save it with changed data then I get error exception 'Varien_Exception' with message 'Invalid method Varien_Object::save in app\code\core\Mage\CatalogInventory\Model\Observer.php(153): Varien_Object->__call('save', Array) that's true because once you try to load by sku then another observer sets product's stock item as Varien_Object, that looks like pitfall or I just dont understand it enough, but
I do daily Magento development from its beginnig so I know a lot about system and this is new for me. Thanks in advance, Jaro.
Interesting. While both methods will net you a single product model instance with fully loaded EAV data (provided the third parameter of loadByAttribute() is not passed or is *), the observers which add stock-related data are different for products vs. product collections, yet both stock information objects are added to the product data key "stock_item". It's debatable, but this feels like a bug. I would think that Mage_CatalogInventory_Model_Observer::saveInventoryData() or Mage_CatalogInventory_Model_Observer::_prepareItemForSave() would handle this.
You could resolve this issue by setting the product stock item fully on your product instance using the stock_item object.
loadByAttribute is a serious misnomer in my opinion because it doesn't actually trigger a load(); rather it uses getResourceCollection():
public function loadByAttribute($attribute, $value, $additionalAttributes = '*')
{
$collection = $this->getResourceCollection()
->addAttributeToSelect($additionalAttributes)
->addAttributeToFilter($attribute, $value)
->setPage(1,1);
Because it doesn't trigger the observer events associated with load() it means the resulting product object doesn't include the full set of product data you might want. In my case I needed the "description" attribute and it wasn't included.
There are several ways to resolve this:
Use a different method to load by SKU:
$product = Mage::getModel("catalog/product");
$product->load($product->getIdBySku("whatever"));
Force the desired attribute data to be included in the default product resource data by visiting Magento Admin > Catalog > Attributes > Edit attribute > "Used in Product Listing" = "Yes" and then reindexing. You should then be able to use the attribute data (in the frontend, at least) using loadByAttribute().
See also https://magento.stackexchange.com/a/197286/18855
I'm using a script to generate a csv file for another site.
I am getting all products and on configurable ones I add in description some attributes.
Here is the code inside the function called on each configurable:
$children_ids = $product->getTypeInstance()->getUsedProductIds();
foreach($children_ids as $child_id){
$child = Mage::getModel('catalog/product')->load($child_id);
$attributeValue = $child->getAttributeText($attributeLabel);
$child_qty = (int)$child->getStockItem()->getQty();
$mage_size[$attributeValue] = $child_qty;
$simple_qty += $child_qty;
}
I noticed that on large amount of products somehow the output suddenly stop without getting all. I tested with and without this code. Without I get as expected.
I have set_time_limit(0);
But no clue why my script terminate before getting information from all products.
Any ideea whay may couse this behaviour?
It's not a proper solution but works on my script.
I added
ini_set('memory_limit', '512M');
and works fine. I know 512 is way too much but just to be sure and to prevent that behaviour in the future when will be more products.
You would both have to increase the memory limit and PHP's execution time - magento is freakin' resource hungry.
I also generate product csv's with a custom script. I would recommend to run these kind of tasks from the command line.