I'm looking for the best method to export some datas from elasticsearch.
My index structure is very simple : ID + 4/5 simple fields
The ID is the ID of French companies,
The other fields are, for example, the activity code, the legal form code, employee size, zip code, ...
The total of records is about 30M.
The count is very fast, but if I want to extract only the IDs, it take a long time.
I need this list "online" for other treatments in a webpage.
How can I do this ?
My query for search is :
{"query":{"bool":{"must":[{"bool":{"should":[{"match":
{"etatadministratifunitelegale":"A"}}]}},{"bool":{"should":[{"match":
{"etatadministratifetablissement":"A"}}]}},{"bool":{"should":[{"match":
{"activiteprincipaleunitelegale":"68.20A"}}]}}], "must_not":[], "should":[] } } }
It's $params in my php code.
This query gives more than 650 000 IDs. To extract them :
$hosts = [
'http://xxxxxxxxx:9200'
];
$client = Elasticsearch\ClientBuilder::create()
->setHosts($hosts)
->build();
$xparams['index'] = 'vecteur';
$xparams['type'] = '_doc';
$xparams['stored_fields'] = [];
$xparams['scroll'] = '1s';
$xparams['size'] = '50000';
$xparams['body'] = $params;
$res = $client->search($xparams);
$scroll_id = $res['_scroll_id'];
$zzz = $res['hits']['hits'];
$xstr = "";
foreach ($zzz as $elt) {
$xstr .= $elt['_id']."\n";
}
fwrite($myfile, $xstr);
while (\true) {
$params2['scroll'] = '1s';
$params2['scroll_id'] = $scroll_id;
$response = $client->scroll($params2);
$zzz = $response['hits']['hits'];
$xstr = "";
foreach ($zzz as $elt) {
$xstr .= $elt['_id']."\n";
}
fwrite($myfile, $xstr);
if (count($response['hits']['hits']) > 0) {
$scroll_id = $response['_scroll_id'];
} else {
break;
}
}
fclose($myfile);
$myfile is a CSV file. Is it clear ?
Related
so i got imported my large excel file with huge informations with 1 sec and i did get all the information i need to display within 1 sec to , but i got a big probleme when displaying the array in the view coz i m testing if the cells are correct or not i m using 5 ifs and 3 foreach and it takes mote than 2 mins , i need help to display all the info in a short time this is my array , and thanks
and there is my code of the view which takes to much time to display infos
and thanks
//here we get our final result of true and false fields
$finale_array = [];
// here we get all our
$current_table2 = [];
$results_applied = [];
$current_result = [];
$columns = SCHEMA::getColumnListing('imports');
// here we get all our conditions
$conditions = DB::table('conditions')->select('number', 'field', 'value')->get();
// here we get all our data
$imports = DB::table('imports')->get();
$results = DB::table('results')->get();
$x = 0;
$default_value = 0;
foreach ($imports as $key => $imported) {
$res = get_object_vars($imported);
foreach ($conditions as $value) {
$array = get_object_vars($value);
$result = $this->test($columns, $array['field']); // the result of our test function
if ($result == "ok") {
if ($res[$array['field']] == $array['value']) {
foreach ($results as $value_result) {
$res_resultat = get_object_vars($value_result);
$test_field = $this->test($columns, $res_resultat['field']);
// testing if the condtion numder match with the result number
if (($res_resultat['condition_number'] == $array['number'])) {
if (($test_field == 'ok')) {
if ($res['id'] != $default_value) {
// here test if the difference between the id and the default value is different from the current id to insert
if (($res['id'] - $default_value) != $res['id']) {
array_push($current_table2, $results_applied);
array_push($finale_array, $current_table2);
$current_table2 = [];
$results_applied = [];
$default_value = $res['id'];
}
$current_table2 = [$res, $res['id']];
}
$current_result = [$res_resultat['field'], $res[$res_resultat['field']]];
if ($res_resultat['value'] == $res[$res_resultat['field']]) {
$current_result[2] = 'true';
$current_result[3] = $res_resultat['value'];
} else {
$current_result[2] = 'false';
$current_result[3] = $res_resultat['value'];
}
$current_result[4] = $array['number'];
array_push($results_applied, $current_result);
}
}
}
}
}
}
$default_value = $res['id'];
}
array_push($current_table2, $results_applied);
array_push($finale_array, $current_table2);
dd($finale_array);
return view('Appliedconditions', ['imports' => $finale_array, 'columns' => $columns]);
}
From what I can see, you are recovering all the data without paging it.
You must paginate the data using the paginate(#elements_per_page) directive in your query, where #elements_per_page is the number of elements you want to display per page.
For example:
$elements = Elements::select('*')->paginate(10);
and in your blade view you can retreive pagination links after the closing table tag in this way: {{ $elements->links() }}
I am building a pharmacy management system. I have a condition where I need to build a section
As shown in the picture, I need to store the data where the upper three should be same for all of the rows inputted below.
The form data is submitted as in the below picture.
But the data when looped and saved is not being saved as desired. Only the last row of the data is being inserted and I am also confused to store supplier, purchase date and note since these data should be repeated as many as the number of rows are added.
PurchaseController.php
public function storePurchase(PurchaseStoreRequest $request)
{
$purchase = new Purchase();
$count = count($request->name);
// return response()->json($request->all());
for ($i = 0; $i < $count; $i++) {
$purchase->supplier = $request->supplier;
$purchase->purchase_date = $request->purchase_date;
$purchase->description = $request->description;
$purchase->category = $request->category[$i];
$purchase->name = $request->name[$i];
$purchase->generic_name = $request->generic_name[$i];
$purchase->batch_number = $request->batch_number[$i];
$purchase->company = $request->company[$i];
$purchase->strength = $request->strength[$i];
$purchase->expiry_date = $request->expiry_date[$i];
$purchase->quantity = $request->quantity[$i];
$purchase->selling_price = $request->selling_price[$i];
$purchase->purchase_price = $request->purchase_price[$i];
$purchase->save();
}
return response()->json(['message' => 'Purchase Saved Successfully']);
}
Can someone help me to store these three fields in the database repeating the number of rows submitted ?
Currently only the last row is being inserted into the database.
This might be an another way to accomplish what you're looking for.
$sharedKeys = ['supplier', 'purchase_date', 'description'];
$sharedData = $request->only($sharedKeys);
$multiKeys = ['category', 'name', 'generic_name', 'batch_number', 'company', 'strength', 'expiry_date', 'quantity', 'selling_price', 'purchase_price'];
$multiData = $request->only($multiKeys);
for ($i = 0; $i < count($request->name); $i++) {
$individualData = array_combine($multiKeys, array_column($multiData, $i));
Purchase::create($sharedData + $individualData);
}
For every loop, you need to create a new instance, then It will create a new record on each loop :
public function storePurchase(PurchaseStoreRequest $request)
{
// $purchase = new Purchase();
$count = count($request->name);
// return response()->json($request->all());
for ($i = 0; $i < $count; $i++) {
$purchase = new Purchase(); // create new instance end with (); on each loop
$purchase->supplier = $request->supplier;
$purchase->purchase_date = $request->purchase_date;
$purchase->description = $request->description;
$purchase->category = $request->category[$i];
$purchase->name = $request->name[$i];
$purchase->generic_name = $request->generic_name[$i];
$purchase->batch_number = $request->batch_number[$i];
$purchase->company = $request->company[$i];
$purchase->strength = $request->strength[$i];
$purchase->expiry_date = $request->expiry_date[$i];
$purchase->quantity = $request->quantity[$i];
$purchase->selling_price = $request->selling_price[$i];
$purchase->purchase_price = $request->purchase_price[$i];
$purchase->save();
}
return response()->json(['message' => 'Purchase Saved Successfully']);
}
I want to make a doctrine query in which each parameters can have multiple values (coming from a select multiple).
I have a table with a 'type' parameter that can have value of 1, 2, 3 or 4 and an 'online' parameter that can be 0 or 1.
My query so far is the following :
$query = $this->createQueryBuilder('properties');
if (array_key_exists('type', $searchValues)) {
$types = $searchValues['type'];
$iterator = 0;
foreach ($types as $type) {
if ($iterator == 0) {
$query->andWhere('properties.idPropertyType = ' . $type);
} else {
$query->orWhere('properties.onlineProperties = ' . $type);
}
$iterator++;
}
}
if (array_key_exists('status', $searchValues)) {
$status = $searchValues['status'];
$iterator = 0;
foreach ($status as $statu) {
if ($iterator == 0) {
$query->andwhere('properties.onlineProperties = ' . $statu);
} else {
$query->andWhere('properties.onlineProperties = ' . $statu);
}
$iterator++;
}
}
$properties = $query->getQuery()->getResult();
In the case of a search with parameter type = 1 and online = 0 and 1, I have results where type is another value than 1. I understand the reason why but I cannot figure out a proper way to make my query.
You don't need to build your query by hand manually, just use the (in) function from the QueryBuilder class. Try this:
$query = $this->createQueryBuilder('properties');
if(array_key_exists('type', $searchValues)){
$types = $searchValues['type'];
$query->andWhere($query->expr()->in('properties.idPropertyType', $types));
}
if(array_key_exists('status', $searchValues)){
$status = $searchValues['status'];
$query->andwhere($query->expr()->in('properties.onlineProperties', $status));
}
$properties = $query->getQuery()->getResult();
I have used more time with Laravel 5.0 on database transaction but when I change to Laravel 5.3.* it not work as I want even I have disabled commit() method all data still continue insert into database.
if ($request->isMethod('post')) {
DB::beginTransaction();
$cats = new Cat();
$cats->parent_id = $this->request->input('category_id');
$cats->status = ($this->request->input('status')) ? $this->request->input('status') : 0;
if ($res['cat'] = $cats->save()) {
$catD = new CategoryDescriptions();
$catD->category_id = $cats->id;
$catD->language_id = 1;
$catD->name = $this->request->input('en_name');
if ($res['cat2'] = $catD->save()) {
$catD2 = new CategoryDescriptions();
$catD2->category_id = $cats->id;
$catD2->language_id = 2;
$catD2->name = $this->request->input('kh_name');
$res['all'] = $catD2->save();
}
}
if ($res) {
//DB::commit();
return $res;
}
return [false];
}
Check this line ($res['cat'] is a boolean):
if($res['cat'] = $cats->save()) {
And this ($res is an array. You compare an array as boolean!):
if($res){
The right condition should be:
$res = array(); // the first line of your method
// .... your http method check, start transaction, etc.
if (!in_array(false, $res, true)) { // check if array doesn't contain 'false values'
DB::commit();
}
i am tyring to add categories programmatically for german and english languages.
website-ID: 1
website store-ID: 1
view-ID english, code "en" => 1
view-ID german, code "de" => 2
$category['general']['path'] = $v['cat_name_de'];
$category['general']['name'] = $v['cat_name_de'];
$category['general']['meta_title'] = "";
$category['general']['meta_description'] = "";
$category['general']['is_active'] = 1;
$category['general']['url_key'] = urlencode($v['cat_name_de']);
$category['general']['display_mode'] = "PRODUCTS";
$category['general']['is_anchor'] = 0;
$category['category']['parent'] = $v['magento_cat_id'];
$storeId = 0; // default
$magento_subcat_id = createCategory($category, $storeId);
...
function createCategory($data, $storeId) {
echo "Erzeuge '{$data['general']['name']}' - Elternkategorie:
[{$data['category']['parent']}] ...";
$category = Mage::getModel('catalog/category');
$category->setStoreId($storeId);
if (is_array($data)) {
$category->addData($data['general']);
if (!$category->getId()) {
$parentId = $data['category']['parent'];
if (!$parentId) {
if ($storeId) {
$parentId = Mage::app()->getStore($storeId)->getRootCategoryId();
} else {
$parentId = Mage_Catalog_Model_Category::TREE_ROOT_ID;
}
}
$parentCategory = Mage::getModel('catalog/category')->load($parentId);
$category->setPath($parentCategory->getPath());
}
if ($useDefaults = $data['use_default']) {
foreach ($useDefaults as $attributeCode) {
$category->setData($attributeCode, null);
}
}
$category->setAttributeSetId($category->getDefaultAttributeSetId());
if (isset($data['category_products']) &&
!$category->getProductsReadonly()) {
$products = array();
parse_str($data['category_products'], $products);
$category->setPostedProducts($products);
}
try {
$category->save();
return $category->getId();
} catch (Exception $e) {
return FALSE;
}
}
}
What i am puzzling with at the moment is setting view-ID individually for passing the $category['general']['name'] for each language.
I tried to set each language string with a foreach setting english/german texts. But this did not work out, because i somehow mixed up setting the view correctly.
To make it short:
Creating categories and subcategories like this does work very good, but i do not get it to work to pass english/german categoty titles.
Thanks guys
For all of you who are struggeling with this situation i made a workaound like this:
I added all those values to the affected database tables directly. This is for sure a quick and dirty way, but it does the job.
For all in need of some ideas here is the (undocumented) snipped belonging to my initial post.
// Get values for next language insert
$entity_type_id = $category
->getResource()
->getTypeId();
$entity_id = $category->getEntityId();
$read_entity = Mage::getSingleton('core/resource')
->getConnection('core_write');
$sql_entity = "SELECT * FROM eav_attribute WHERE
`attribute_code` = 'name' AND
`entity_type_id` = " . $entity_type_id . ";";
$row_entity = $read_entity
->fetchRow($sql_entity);
$attribute_id = $row_entity['attribute_id'];
// prepare value for name field in different langauge
$val = Mage::getSingleton('core/resource')
->getConnection('default_write')
->quote($v['en']);
$sql_insert = "INSERT INTO `catalog_category_entity_varchar`
(`entity_type_id`,`attribute_id`,`store_id`,`entity_id`,`value`)
VALUES ($entity_type_id,$attribute_id,$storeId,$entity_id,$val);";
$read_entity->query($sql_insert);
I hope this helps someone out who is in need of this.