Advice with Magento large productsfile csv import - magento

Question

I need to import 40.000 products.
These products together are generating 600.000 attribute values
There is a csv file with product info (Sku, Name, Description and Ean) and a seperate sheet with only the attributes(600K)
On each row i have:
Sku - Attribute - Value
e.g.
123 Color Green
123 Length 120
123 Size XL
456 Color White
456 Length 260
etc..
I have filtered all duplicates out which resulted in 2200 unique Attributes.
What should i Do with these attributes?
Put them all in one attributeset? Will it hurt the performance of the webshop?
And what about the attributesheet?
How should i convert the structure of the presented data so it will be usefull for for import in magent? Cause magento needs all attribute names as columnheaders?
I have tried to collect teh attribute values with VLOOKUP but run into memory problems on my MACbook Pro. Fill down one column with a formula doesn't work wit this amount of records.
Maybe there is a solution programmticly.
Thanks!

Answer 1

I would suggest to create product pragmatically.
Import product into a mysql table (tables) and create using magento code directly.
You can create configurable attributes and use its id when creating products.

Answer 2

You should think about using uRapidFlow.
While I am not a huge fan of the fact that it uses direct sql to import products, it has been around for some time and is known to be very reliable and very fast.

Answer 3

//add new products
$product = Mage::getModel('catalog/product');
$product->setSku($sku);
$product->setStatus(1);
$product->setName($name);
$product->setDescription($details);
$product->setShortDescription($description);
$product->setPrice(0);
$product->setTypeId('simple');
$product->setAttributeSetId(4); // enter the catalog attribute set id here
$product->setCategoryIds($categoryIds); // id of categories
$product->setWeight(1.0);
$product->setTaxClassId($taxClassId);
$product->setVisibility($visibility);
$product->setStatus($productStatus);
$product->setColor('color',$color_id); // u need to get attribute code for dropdown items
$product->setData('material', $avid);
$product->setBrand($brand);
/*$product->setStockData(
array(
'manage_stock' => 1,
'is_in_stock' => 1,
'qty' => $qty
)
);*/
// assign product to the default website
$product->setWebsiteIds(array(1,2,3,4));
try{
$product->save();
$i++;
;
}catch (Exception $e) {
echo Mage::logException($e->getMessage());
}
$id = Mage::getModel('catalog/product')->getIdBySku(trim($sku)); //get product id after create
//add dropdown items for attribute u created already, u need to make this as function to get option id before add a product
$attribute_model = Mage::getModel('eav/entity_attribute');
$attribute_options_model= Mage::getModel('eav/entity_attribute_source_table') ;
$attribute_code = $attribute_model->getIdByCode('catalog_product', $attribute_code);
$attribute = $attribute_model->load($attribute_code);
$attribute_table = $attribute_options_model->setAttribute($attribute);
$options = $attribute_options_model->getAllOptions(false);
foreach($options as $option)
{
if (strtolower($option['label']) == strtolower($label))
{
$optionId=$option['value'];
break;
}
}
//u need to run thru a loop to add products