I am querying a large data sets from the table and then iterating through a loop for creating a json file.
$user = App\User::all();
foreach($user as $val){
// logic goes here for creating the json file
}
Now the problem i am facing is that when iterating through the loop it is consuming memory and i am getting error 'Allowed memory size exhausted'.And also the cpu usage of the server becomng so high.
My question how i should use the laravel lazy collections to get rid of this issue.I have gone through the offcial docs but couldnt find the way.
Just replace the all method with the cursor one.
$user = App\User::cursor();
foreach($user as $val){
// logic goes here for creating the json file
}
For more informations about the methods you can chain, refer to the official documentation
Related
What would be the fastest way using as less as possible memory to seed 1M records in Laravel?
for( $i=0; $i<1000; $i++){
User::factory()->count(1000)->make();
}
Or using chunk method or everytime the loop has 1000 records make the records and empty the variable? Or are there any other ways this could be done faster?
Eloquent models give a significant overhead. Laravel factories are a nice pattern to get some data in your database, but you will notice a slow-down for 1000s of models, since each of the models first get constructed from the PHP class (and if you would persist the factory models using ->create() instead of ->make(), it sends out a bunch of creating/created/saving/saved events as well). So the most performant way forward is to not use the factory code but instead use simple DB statements like so:
$data = [['username' => 'foo', ...], ['username' => 'bar', ...]];
Illuminate\Support\Facades\DB::table('users')->insert($data);
At a certain $data size you could run into an SQL prepared statement placeholder limit, which basically means the query is becoming too big. In that case, you can also just chunk the code above to:
// Use `collect()` to create a Collection object (glorified array)
// which can chunk/paginate the data:
collect($data)->chunk(100)->each(function($chunk) {
DB::table('users')->insert($chunk);
});
That being said, the Laravel factories are not bad to use and they are very handy in testing.
I am loading data from excel. In foreach I am checking for each record if it does exist in database:
$recordExists = $this->checkIfExists($record);
function checkIfExists($record) {
$foundRecord = $this->repository->newQuery()
->where(..., $record[...])
->where(..., $record[...])
...
->get();
}
When the excel contains up to 1000 values which is relatively small piece of data - the code runs around 2 minutes. I am guessing this is very inefficient way to do it.
I was thinking of passing the array of loaded data to the method checkIfExists but then I could not query on the data.
What would be a way to proceed?
You can use laravel queue if you want to do a lot of work within a very short time. Your code will run on backend. Client can not recognize the process. just show a message to client that this process is under queue. Thats it
You can check the Official Documentation From Below Url
https://laravel.com/docs/5.8/queues
If you passes all the data from the database to the function (so no more queries to the database), you can use laravel collections functions to filter.
On of them is where => https://laravel.com/docs/5.8/collections#method-where
function checkIfExists($record, Collection $fetchedDataFromDatabase) {
// laravel collectons 'where' function
$foundRecord = $fetchedDataFromDatabase
->where(..., $record[...])
->where(..., $record[...]);
}
other helpful functions.
filter
contains
I was thinking about using cache/storage to store source for api routes which doesnt update that much. I mean i would like to update stored data every one minute (example) and my routes will use that data as source instead of database as source. It will make less database queries. Im I thinking in good way? How to achieve that in Laravel?
You could setup an instance of memcached which is supported by Laravel. The Laravel caching documentation is a good place to start learning how to setup a cache driver for memcached. Once setup, your logic could look something like this:
if (Cache::has('key')) {
//your key exists in the cache. get it.
$value = Cache::get('key');
//and use it
useMyValue($value);
}
else
{
//the cache does not contain the key you are looking for
//you can get it from the DB and cache it.
//the next time your function runs it will get the value from cache instead
//of reading from the db
$value = Cache::get('key', function () {
return DB::table(...)->get();
});
//and use your value now like normal
useMyValue()
}
I'm trying to get data from an Eloquent query as follows:
$submission->find($id)->first()->with('user', 'clientform')->get();
I'm sending the submission to the view and attempting to access properties from the user model like so:
{{ $submission->clientform->name }}
However, Laravel is throwing the following error:
Undefined property: Illuminate\Database\Eloquent\Collection::$clientform
What am I doing wrong with the way my query is formatted?
You're overdoing it!
Let's break it down:
$submission->find($id);
Will return an instance of the model which corresponds to the entry having the primary key of $id.
$submission->find($id)->first();
This is an unncessary repetition; the find method already gives you a single entry, no need to call first on it.
$submission->find($id)->first()->with('user', 'clientform');
This is where you start going the wrong way; when calling with on a model, Laravel will transform that model into a query builder again, even though it was already resolved by calling find.
$submission->find($id)->first()->with('user', 'clientform')->get();
Finally, the get method resolves the builder into a Illuminate\Support\Collection, regardless of the number of entries found (i.e. it could be a Collection of only one item). It's worth noting, though, that your query will most likely have been fully reset by calling with. It would be the same as instantiating a fresh Submission model and building your query with it. That means you're probably using a collection of all your submission entries, not just the one with $id.
Long story short, this is what you want:
$submission->with('user', 'clientform')->find($id);
It will fetch the matching $id, with the user and clientform relations eager-loaded.
Although I can tentatively see a solution to this, I was wondering if there may be a glaringly obvious simpler approach.
My aim is to use the first segment of a given URI to query the DB as to which controller should be run.
I assume I would have to reform the URI with the resultant controller name in segment 1, then allow the system to continue processing as normal (hence a pre_system hook).
Although not essential I would also like to hold a couple of other variables from the same DB request to be used later in the call stack, and assume this would have to be done using global variables?
Any better suggestions would be gladly received.
Thanks.
Should it be of use to anyone else, here is the code to acheive the desired result. This does however not take into account passing additional variables because I can live without them.
function set_controller()
{
include_once APPPATH.'config/database.php'; //Gather the DB connection settings
$link = mysql_connect($db[$active_group]['hostname'], $db[$active_group]['username'], $db[$active_group]['password']) or die('Could not connect to server.' ); //Connect to the DB server
mysql_select_db($db[$active_group]['database'], $link) or die('Could not select database.'); //Select the DB
$URI = explode('/',key($_GET)); //Break apart the URL variable
$query = 'SELECT * FROM theDomainTable WHERE domainName = "'.$URI[1].'"'; //Query the DB with the URI segment
if($results = mysql_fetch_array(mysql_query($query))){ //Only deal with controller requests that exist in the database
$URI[1] = $results['controllerName']; //Replace the controller segment
$_GET = array(implode('/',$URI)=>NULL); //Reconstruct and replace the GET variable
}
mysql_close($link); //Close the DB link
}
I wouldn't use global variables, Id prefer to store it in a library for retrieval later if possible. Global variables are kind of messy in the context of CI.
Although at pre_system Only the benchmark and hooks class have been loaded at this point. This means you're pretty-much stuck with global variables unless you can find a way to select the controller on pre_controller as all the base-classes are loaded and you can put the data somewhere more logical.