I want to do some mass DB population from Excel file.
The most time economic way is to use INSERT INTO statement with lots of values to be stored in one transaction:
INSERT INTO `assortment`(`id`, `sku`, `agroup`, `subgroup`, `title`, `measure_unit`, `price`, `discount`, `imageUrl`, `fileUrl`)
VALUES ([value-1],[value-2],[value-3],[value-4],[value-5],[value-6],[value-7],[value-8],[value-9],[value-10]),
([value-1],[value-2],[value-3],[value-4],[value-5],[value-6],[value-7],[value-8],[value-9],[value-10]),
([value-1],[value-2],[value-3],[value-4],[value-5],[value-6],[value-7],[value-8],[value-9],[value-10]),
([value-1],[value-2],[value-3],[value-4],[value-5],[value-6],[value-7],[value-8],[value-9],[value-10]),
...
Yet, to avoid SQL injection i wish to bind params, the yii providing functionality for that. Yet, it seems impossible for me to do it for hundredes/thousands of values. Isn't it?
To keep SQL hygene i did the simple insert thru Active Record attributes (Yii AR functionality sanitizes input data by default):
$auxarr = array();
for ($i = 0; $sheetData[$i]; $i++)
{
$model = new Assortment();
$j = 0;
foreach ($labels as $label)
{
$auxarr[$label] = $sheetData[$i][$j++];
}
$model->attributes = $auxarr;
if (!$model->save())
throw new CHttpException(400, 'Error db storing');
}
This approach obviously being time non-efficient.
Is there any way that would feature both security and time efficiency in the bulk SQL inserting?
Yii is using conventional PDO in CDbCommand.
So, you can create a string consists of series values like this
(?,?,?),(?,?,?),(?,?,?),(?,?,?),(?,?,?),(?,?,?)
then create an array with values for all these placeholders
and finally execute all the stuff
My approach is
$sql = "INSERT INTO `assortment`(`id`, `sku`, `agroup`, `subgroup`, `title`, `measure_unit`, `price`, `discount`, `imageUrl`, `fileUrl`) VALUES "
$params = array();
$cntRows = count($sheetData);
for ($i = 0; $i < $cntRows; $i++)
{
$j = 0;
$rowParams = array();
foreach ($labels as $label)
{
$rowParams[":{$label}_{$i}_{j}"] = $sheetData[$i][$j++];
}
$params = array_merge($params, $rowParams);
$sql . = "(" . implode(",", array_keys($rowParams) ) .")"
}
/*
Sql now is : INSERT INTO assortment (....) VALUES ( :id_1_1 , sku_1_1 , ... ) (:id_2_1 , sku_2_1 , ...)
AND $params is { :id_1_1 => [value] ........ }
*/
$cmd = Yii::app()->db->createCommand($sql);
$cmd->execute($params);
We excute insert sql in one transaction, no multiple transactions or use ActiveRecord (waste memory and many functions are executed) and avoid SQL injections. If your data is large you can split it to multiple transactions.
Related
everyone.
In the codeigniter there is update_batch function by using it we are able to bulk update multiple rows.
$this->db->update_batch('table_name', $update_data_array, 'where_condition_field_name');
I want similar functionality in core PHP in one function or in one file. Is there any workaround?
Is there any way to extract update_batch function from Codeigniter in one file/function?
I tried to extract this function but it is very lengthy process and there will be many files / functions should be extracted.
Please help me in this regard
Thanks in advance
You can also insert multiple rows into a database table with a single insert query in core php.
(1)One Way
<?php
$mysqli = new mysqli("localhost", "root", "", "newdb");
if ($mysqli == = false) {
die("ERROR: Could not connect. ".$mysqli->connect_error);
}
$sql = "INSERT INTO mytable (first_name, last_name, age)
VALUES('raj', 'sharma', '15'),
('kapil', 'verma', '42'),
('monty', 'singh', '29'),
('arjun', 'patel', '32') ";
if ($mysqli->query($sql) == = true)
{
echo "Records inserted successfully.";
}
else
{
echo "ERROR: Could not able to execute $sql. "
.$mysqli->error;
}
$mysqli->close();
? >
(2)Second Way
Let's assume $column1 and $column2 are arrays with same size posted by html form.
You can create your sql query like this:-
<?php
$query = 'INSERT INTO TABLE (`column1`, `column2`) VALUES ';
$query_parts = array();
for($x=0; $x<count($column1); $x++){
$query_parts[] = "('" . $column1[$x] . "', '" . $column2[$x] . "')";
}
echo $query .= implode(',', $query_parts);
?>
You can easily construct similar type of query using PHP.
Lets use array containing key value pair and implode statement to generate query.
Here’s the snippet.
<?php
$coupons = array(
1 => 'val1',
2 => 'va2',
3 => 'val3',
);
$data = array();
foreach ($coupons AS $key => $value) {
$data[] = "($key, '$value')";
}
$query = "INSERT INTO `tbl_update` (id, val) VALUES " . implode(', ', $data) . " ON DUPLICATE KEY UPDATE val = VALUES(val)";
$this->db->query($query);
?>
Alternatively, you can use CASE construct in UPDATE statement. Then the query would be something
like:
UPDATE tbl_coupons
SET code = (CASE id WHEN 1 THEN 'ABCDE'
WHEN 2 THEN 'GHIJK'
WHEN 3 THEN 'EFGHI'
END)
WHERE id IN(1, 2 ,3);
I need to do a bulk insert of thousands of records (5k up to 20k).
The scenario is User<->n:m<->Group. The list of users is obtained by a complex query with many joins.
I have access to the QueryBuilder that generates the list.
The simpliest approach to add the users to the group is
$users = $this->repository->findRecipientsByCriteria($group->getCriteria());
foreach ($users as $user){
$group->addUser($user);
}
But for the number of users involved i don't think it's a good idea (in term of performances).
I can't even iterate results because of the fetch join relations.
I would like to inject the QueryBuilder Dql (or Sql) to the INSERT statement?
I mean something like:
$qb = $this->repository->getRecipientsByCriteriaQueryBuilder($group->getCriteria());
$qb->select("'".$group->getId()."' AS gruppo_id, U.id AS utente_id");
$d = $qb->getQuery()->getSQL();
$q = $this->entityManager->createNativeQuery('INSERT INTO `msg_gruppo_utente` (`gruppo_id`, `utente_id`) '.$d, new ResultSetMapping());
$q->execute();
But this results in
INSERT INTO `msg_gruppo_utente` (`gruppo_id`, `utente_id`) SELECT '64f105a3-a6ab-460a-8378-84b0c3258601' AS sclr_0, s0_.id AS id_1 FROM security_utente s0_ INNER JOIN security_utente_cliente s1_ ON s0_.id = s1_.utente_id INNER JOIN api_cliente a2_ ON s1_.cliente_id = a2_.id INNER JOIN api_indirizzo_cliente a3_ ON a2_.id = a3_.cliente_id INNER JOIN api_contratto a4_ ON a2_.id = a4_.cliente_id WHERE s1_.verificato = ? AND a3_.city = ?
Where parameters are not set, while i thought thatthe parameters should have been be set in getRecipientsByCriteriaQueryBuilder
Due to doctrine native SQL resctrictions
If you want to execute DELETE, UPDATE or INSERT statements the Native
SQL API cannot be used and will probably throw errors. Use
EntityManager#getConnection() to access the native database connection
and call the executeUpdate() method for these queries.
I've used this solution (executeUpdate is deprecated in favor of executeStatement)
$usersQueryBuilder = $this->repository->getRecipientsByCriteriaQueryBuilder($group->getCriteria());
$usersQueryBuilder->select("'".$group->getId()."' AS gruppo_id, U.id AS utente_id");
$parameters = $usersQueryBuilder->getParameters();
$p = [];
/** #var Parameter $parameter */
foreach ($parameters as $parameter){
$p[] = $parameter->getValue();
}
$conn = $this->entityManager->getConnection();
$conn->executeStatement('INSERT INTO `msg_gruppo_utente` (`gruppo_id`, `utente_id`) '.
$usersQueryBuilder->getQuery()->getSQL(), $p);
I'm a newbie in laravel. I have to generate multiple random characters at once in my laravel project and then save them to database. Any example or advise that quite easy to understand? thank you
A for loop may be appropriate for you in this case. If you don't have an object for the codes, you can do something like this:
$total_wanted = 100;
for ( $i = 0; $i < $total_wanted; $i++ ) {
DB::table('codes')->insert(['code' => uniqid("prefix")]);
}
If you do have a Code object, you can do it like this instead:
$total_wanted = 100;
for ( $i = 0; $i < $total_wanted; $i++ ) {
$code = new Code;
$code->code = uniqid("prefix");
$code->save();
}
echo uniqid('code_', true);
You generate a random, unique string based on current timestamp, so it can not be a duplicate.
Then simple store it in the database as a varchar
More about this function here
now, i get list Units from Database:
var sections = scheduler.serverList("type");
scheduler.createUnitsView({
name:"unit",
property:"type",
list:sections
});
$list = new OptionsConnector($res, $dbtype);
$list->render_table("doctors","id","id(value),first_name(first_name),last_name(last_name),middle_name(middle_name),spec(spec),image(image)");
$scheduler = new schedulerConnector($res, $dbtype);
$scheduler->set_options("type", $list);
But i want some query from Database for this. No render all values from Databases, just result from "Select *******"
Is it possible? Render_sql? Thank you advance
you can use render_sql method for this:
$list = new OptionsConnector($res, $dbtype);
$list->render_sql(
"SELECT * FROM doctors WHERE someColumn > " .$list->sql->escape("someValue") . " AND anotherColumn < ". $list->sql->escape("anotherValue"),
"id",
"id(value),first_name(first_name),last_name(last_name),middle_name(middle_name),spec(spec),image(image)"
);
$scheduler = new schedulerConnector($res, $dbtype);
$scheduler->set_options("type", $list);
The first parameter takes the sql query, and the rest parameters are the same as in render_table.
Also note that connectors don't support parameterized queries, but you can use $connector->sql->escape method if you need to insert request values into the query
I have a query in laravel:
...
$query = $model::group_by($model->table().'.'.$model::$key);
$selects = array(DB::raw($model->table().'.'.$model::$key));
...
$rows = $query->distinct()->get($selects);
this works fine and gives me the fields keys' that I need but the problem is that I need to get all the columns and not just the Key.
using this:
$selects = array(DB::raw($model->table().'.'.$model::$key), DB::raw($model->table().'.*'));
is not an option, cuz it's not working with PostgreSQL, so i used $rows to get the rest of columns:
for ($i = 0; $i<count($rows); $i++)
{
$rows[$i] = $model::find($rows[$i]->key);
}
but as you see this is it's so inefficient, so what can i do to make it faster and more efficient?
you can find the whole code here: https://gist.github.com/neo13/5390091
ps. I whould use join but I don't know how?
Just don't pass anything in to get() and it will return all the columns. Also the key is presumably unique in the table so I don't exactly understand why you need to do the group by.
$models = $model::group_by( $model->table() . '.'. $model::$key )->get();