I'm needing to chunk my query as it's making PHP run out of memory, however the below code just dumps null:
$chunked = $query->chunk(25, function ($events) {
//dd($events);
});
dd($chunked);
However, when I do the below, it dumps the first chunk of 25:
$chunked = $query->chunk(25, function ($events) {
dd($events);
});
//dd($chunked);
No combination of changing dd($events); to the likes of return $events, return true, iterating over each item in each chunk and returning that - works.
Am I stupid/doing something wrong or is this not working like it should?
chunk() is a helper method which you can use on an instance of Query Builder or Eloquent Builder. In both cases, the method returns void:
Query Builder
Eloquent Builder
This means that your $chunked variable will be always empty.
The syntax you need to employ is the following:
Query Builder
DB::table('users')->chunk(100, function($users) {
foreach ($users as $user) {
//
}
});
Eloquent
Flight::chunk(200, function ($flights) {
foreach ($flights as $flight) {
//
}
});
Basically, all you need to do is to use a loop within the callback function of chunk() to modify your results.
Related
I would like to use the query function, because I have a slow performance in laravel datatable. I have this select:
$audits = \OwenIt\Auditing\Models\Audit::select(
'audits.id',
'audits.user_type',
'audits.user_id',
'audits.event',
'audits.auditable_id',
'audits.auditable_type',
'audits.old_values',
'audits.new_values',
'audits.created_at',
'users.login'
)
->join('users','users.id','audits.user_id');
and I tried to use the query function this way:
$audits = \OwenIt\Auditing\Models\Audit::query()->select(
'audits.id',
'audits.user_type',
'audits.user_id',
'audits.event',
'audits.auditable_id',
'audits.auditable_type',
'audits.old_values',
'audits.new_values',
'audits.created_at',
'users.login'
)
->join('users','users.id','audits.user_id');
But is not working the code above. I thought to use Laravel Pagination, however in the project that I work has datatables in so many tables and I want to keep the same standard. Someone can help me?
You have a misunderstanding of what query() does. Simply put it does nothing differently and it will not help you with any optimization for your queries.
Model::select(args)
When you do Model::select('col1', 'col2'), this is what happens:
There is no Model::select public static function, so __callStatic(method: 'select', parameters: ['col1', 'col2']) gets called instead.
All __callStatic does in Model is try to call a public function method passing in parameters.
# Illuminate\Database\Eloquent\Model
public static function __callStatic($method, $parameters)
{
return (new static)->$method(...$parameters);
}
There is no Model::select public function, so __call(method: 'select', parameters: ['col1', 'col2']) gets called instead.
All __call does is forward the call to the object returned from the public function Model::newQuery. So, (new Model)->newQuery()->select(['col1', 'col2'])
# Illuminate\Database\Eloquent\Model
public function __call($method, $parameters)
{
if (in_array($method, ['increment', 'decrement'])) {
return $this->$method(...$parameters);
}
if ($resolver = ($this->relationResolver(static::class, $method))) {
return $resolver($this);
}
return $this->forwardCallTo($this->newQuery(), $method, $parameters);
}
To summarize, when you do Model::select(args), it's the same as doing (new Model)->newQuery()->select(args).
Model::query()->select(args)
This is the source code for the Model::query public static function.
public static function query()
{
return (new static)->newQuery();
}
As you can see, it's the same.
The only "optimization" that comes out of using query() is making the call stack shorter.
Model::select()
calls __callStatic
calls __call
calls forwardCallTo
calls newQuery
Model::query()->select()
calls newQuery
There are still a few extra steps that happen between newQuery and select, but they are the same.
As for possible optimizations:
Creating an index on audits 's user_id column
Pagination
Lazily loading the results (cursor, lazy)
https://laravel.com/docs/9.x/eloquent#cursors
https://laravel.com/docs/9.x/eloquent#chunking-using-lazy-collections
Loading the data through ajax
https://datatables.net/reference/option/ajax
Within a map function i would like to use dd(); dump but i only get the first object.
When i use print_r(); within a map function i get all objects
$valeurCategorieCible = $CategoriesItineraires->map(function ($item) {
return $item->ciblesParCategorie->map(function ($ciblesParCategorie) {
return $ciblesParCategorie->cibles->map(function ($items) {
print_r($items); // Return all objects
dd($items); // Return only the first object then stop !
});
});
});
According to the documentation for dd() it "dumps the given variables and ends execution of the script" - the important part is "ends execution of the script"!
If you don't want to end execution use dump() instead of dd()
$jobs = Job::with(['topics', 'topic'])->get();
// This works as intented, does not make extra queries.
foreach ($jobs as $job) {
$job->topics;
}
// This code produces extra queries even tho "topic" is loaded above.
foreach ($jobs as $job) {
$job->getLogo();
}
// Function in Job Model
function getLogo() {
return $this->topic->getLogo();
}
// Function in Topic Model
function getLogo() {
return $this->logo;
}
Even though $job has loaded topic it still makes extra queries. How do I resolve this?
It looks like your post is mostly code; please add some more details.
I have ~5-6k $items that I need to update in the database. Each item needs a HTTP request to get the data from the page. In the HTTP GET request I get arrays that are massive (~500-2500) and I need to insert only those lines that are not in the database. It seems to take a lot of time with my current script (1 item every 2-4 minutes) on my vagrant scotch box.
Simplified example:
<?php
namespace App\Http\Controllers;
use Illuminate\Http\Request;
use App\Http\Requests;
use GuzzleHttp\Client;
use App\Item;
use App\ItemHistory;
use Carbon\Carbon;
use DB;
class UpdateController extends Controller
{
public function getStart() {
// Don't cancel the script
ignore_user_abort(true);
set_time_limit(0);
$client = new Client();
$items = Item::where('updated_at', '<=', Carbon::now()->subDay())->get();
foreach($items as $item) {
$response = $client->request('GET', 'API_URL');
// get the body
$body = $response->getBody()->getContents();
$hugeArray = $body['history']; // can be from 100 to 5 000 lines and I use regex to get the "history" array from the body
$arrayCollection = collect($hugeArray);
foreach($arrayCollection->take(-100) as $row) { // I take the last 100 since each row = 1 hour, so I get items in the last 100 hours
$date = new \DateTime($row['created_at']);
if( ! ItemHistory::whereItemId($item->id)->whereSoldAt($date)->count()) { // Checking if it already exists
// I insert the new rows..
$history = new ItemHistory;
// ....
$history->save();
}
}
}
}
}
I actually crawl the data and use regex to find the arrays in the body response.
Am I doing something wrong? It takes quite a while until it moves onto the next $item.
I can provide a simplified answer - synchronous execution, object hydration, and bulk database querys.
Consider the following example:
$requests = function () use ($items) {
foreach ($items as $item) {
yield new GuzzleHttp\Psr7\Request($method, $uri);
}
};
$client = new GuzzleHttp\Client();
foreach ($requests() as $request) {
$client->sendAsync($request)
->then(
function(Psr7\Http\Message\ResponseInterface) {
// process the response into array;
return $arrayFromResponse;
})
->then(
function ($unfilteredArray) {
// filter the array as necessary
return $filteredArray;
})
->then(
function($filteredArray) {
// create the array for bulk insert / update
return $sqlArray;
})
->then(
function($sqlArray) {
// perform bulk db operations.
}
);
}
Synchronous Http queries - The above example highlight's some of Guzzle's asynchronous capabilities, while breaking out the processing steps. The code you linked above is synchronous. Perform a request, wait for a response, process response, rince & repeat. Asynchronous Http requests will ensure that data is being downloaded while other information is being processed. I should note that your results will vary, and depending on your particular use case, may see increased resource usage.
Object Hydration - aka what your ORM is doing when you perform a query and it returns an object instance (rather than an array), is time consuming and memory intensive. #orcamius (one of Doctrine's developers) wrote a fairly technical article on the subject. While this is not Eloquent specific, it does provide insight into operations that go on behind the scenes for all ORM's. The code snippet performs many of these (reference $itemHistory, $history, Item::where).
Bulk Database Operations - a widely known fact is that database operations are slow. This time is further increased when coupled with object hydration. It is much better to perform a single insert with 1000x records vs 1000x inserts. To do this, code will have to be modified from using the ORM to using the DB tables directly. Bulk inserts can be performed by DB::table('itemHistory')->insert($arrayOfValues) as seen in the docs
Update: Although not shown then() has a method signature of then(callable $fulfilled, callable $onError). If something goes awry with the request you could do something like
// promise returned from a request
$p->then(
function (Psr\Http\Message\ResponseInterface $response) use ($p)
if ($response->getResponseCode() >= 400) {
$p->cancel();
}
//perform processing
return $someArray;
},
function (RequestException $e) {
echo $e->getMessage() . "\n";
echo $e->getRequest()->getMethod();
})
->then(
function($someArray) use ($p) {
// filter or other processing
});
Additional information on Guzzle's Promises can be found within the Github Repo
I'm adding some models to a project, and was wondering if there is a "best practice" kind of approach to creating models:
Does it make sense to create a function for each specific query?
I was starting to do this, then had the idea of creating a generic function that I could pass parameters to. e.g:
Instead of
function getClients(){
return $this->db->query('SELECT client_id,last FROM Names ORDER BY id DESC');
}
function getClientNames($clid){
return $this->db->query('SELECT * FROM Names WHERE client_id = '.$clid);
}
function getClientName($nameID){
return $this->db->query('SELECT * FROM Names WHERE id ='.$nameID);
}
}
Something like
function getNameData($args,$cond){
if($cond==''){
$q=$this->db->query('SELECT '.$args.' FROM Names');
return $q;
}else{
$q=$this->db->query('SELECT '.$args.' FROM Names WHERE '.$cond);
return $q;
}
}
where I can pass the fields and conditions (if applicable) to the model. Is there a reason the latter example would be a bad idea?
Thanks!
I think it would actually be a better idea to use CI's Active Record to compile the queries.
An example:
function all_clients($select)
{
$this->db->select($select);
return $this->_get_client_data();
}
function single_client($select, $id = "")
{
// validate $id
$this->db->select($select);
$this->db->where("id", $id);
$this->db->limit(1);
return $this->_get_client_data();
}
// Only called by a method above once the query parameters have been set.
private function _get_client_data()
{
$q = $this->db->get("clients");
if($q->num_rows() > 0)
{
return $q->result_array();
}
return FALSE;
}
CI's Active Record makes all the stuff you were wanting to much easier. You can imagine setting up your public functions to conditionally set a number of options before actually calling $this->db->get().
I guess you would call _get_client_data a catch-all (?) and running all your data retrieval through a single method makes stuff like error handling much easier to maintain.
NOTE: Always remember to validate data like this. I know you do, but I'm just repeating it.