Incomplete json response in a large eloquent-collection's ajax request - laravel

Question

I'm having an error in my Laravel 8 app which uses yajra/laravel-datatables-oracle with server-side rendering, and I can't figure it out so far. What makes it frustating is the fact that this have been working just fine for the last couple of months, and suddenly I got a report about this error just this morning.
So I have a page which shows a table. The table itself is pretty large, with more than 25 columns shown, thousands of total records, and with some nested relationships as well. For this, I use the yajra/laravel-datatables-oracle package with ajax/server-side rendering as I mentioned above. The page has pagination option enabled, and is set to show 50 records by default. The page is functioning correctly when opened in its default condition.
However, an error will happen when user changes the table's page-length to "All". After checking the the route that handles the table's ajax rendering, turns out the error happened because the returned json is not a valid json. So instead of getting a valid json like these:
"data": [
{"id": 1, "name": "Foo"},
// ....
{"id": 999, "name": "Bar"}
]
I get:
"data": [
{"id": 1, "name": "Foo"},
// ....
{"id": 500,
]
// or...
"data": [
{"id": 1, "name": "Foo"},
// ....
{"id": 500, "name": "Ja
]
The point is that the returned response is incomplete, which makes it invalid as a json, which makes the datatables throws a json bad parsing error.
These are the things for you to know:
Remember, this thing have been working fine for the last couple of months.
Just like the example above, the json response (which is incomplete) is always changing everytime I refresh the url. Meaning that it's likely not related to "abnormal" record.
The error occurs only if the selected page-lenght pass a certain amount of number:
It's working when just being selected (up to) 180 items per page (each item has a total of >100 fields).
More than 180 and the error will occur. However, the incompleteness could be anywhere, not in the 181'th item. It could be in the 70th-ish, 90th-ish, etc. In fact, It never returns more than 100 records when the error occur.
I had thought that this might be caused by PHP's memory exhaustion. I tried to set the memory_limit of the PHP to -1 or 999999M, but nothing changes.
This is actually not just happening in one page, but many.
However, there are another pages, that's working totally fine without the error above. The size of the data that it has is roughly the same.
I really have no idea what's the cause of this. Is this because of the server's PHP config, the database (mysql), the ajax request, or what?
Please help. You're much appreciated.
Additional Info
This is one of the method's controller where the error appears.
use App\Models\Instansi;
use App\Models\InstrukturKelasPeriodeTahapan;
use App\Models\Tahapan;
use App\Models\Tahun;
use Illuminate\Http\Request;
use Yajra\DataTables\Facades\DataTables;
public function datatable(Request $request) {
// validate request etc...
$jadwals = InstrukturKelasPeriodeTahapan::query()
->withInstruktur()->withKelasPeriodeTahapan()
->withTarifInTahunAndPeriodeOf($tahun, $periode)
->withPphInTahunAndPeriodeOf($tahun, $periode)
->withCount([
'pamongDalamKelasPeriodeTahapan as jumlah_pamong_yg_mengajar'
])
->whereHasTahunOf($tahun)
->whereHasPeriodeOf($periode)
->whereHasTahapanOf(Tahapan::PPP);
if ($request->instansi) {
if ($request->instansi == Instansi::NON_FOO) {
$jadwals->whereHasInstansiOfNonFoo();
} else {
$jadwals->whereHasInstansiOf($request->instansi);
}
}
return DataTables::of($jadwals)
->addIndexColumn()
->make(true);
}
This should get a total of 1134 records and used to be working just fine. Now, when the error occurs, I get less than 100 records, and the last returned record is incomplete.
These different return syntaxes still giving me incomplete result.
return response()->json(
DataTables::of($jadwals)
->addIndexColumn()
->make(true)
);
return DataTables::of($jadwals)
->addIndexColumn()
->toJson();
return $jadwals->get();
return response()->json($jadwals->get());
However, these sintaxes is returning the complete 1134 records. But I can't use it as it's not using datatable's response format.
return dd($jadwals->get());
return $jadwals->get()->toJson();
And this is a data example of how an item/record should look like when returned completely, with 120 lines in total:
{
"id": 9999,
"kelas_periode_tahapan_id": 999,
"instruktur_id": 999,
"jumlah_pamong_yg_mengajar": 9,
"instruktur": {
"id": 9999,
"id_ref": 9999,
"nama": "Dr. Xxxxx",
"email": "xxxxxxxx#gmail.com",
"tgl_lahir": "95-09-9977",
"nip": "999999999999999999",
"nik": "99999999999999999",
"npwp": "99999999999999999",
"golongan_id": 9,
"jab_fungsional_id": 9,
"instansi_id": 9,
"no_rek": "99999999999",
"nama_bank": "BANK XXXXXXXX",
"bank_cabang": "XXX",
"nama_rek": "Xxxxxxxx Xxxxx Xxxxxxxxx",
"is_active": 9,
"golongan": {
"id": 9,
"description": "XXXXX",
"periodes": [
{
"id": 9,
"tahun_id": 9,
"periode": 9,
"pivot": {
"golongan_id": 9,
"periode_id": 9,
"pph_npwp": 9,
"pph_non_npwp": 9
}
}
]
},
"jab_fungsional": {
"id": 9,
"description": "Xxxxxxxx",
"periodes": [
{
"id": 9,
"tahun_id": 9,
"periode": 9,
"pivot": {
"jab_fungsional_id": 9,
"periode_id": 9,
"tarif": 99999999
}
}
]
},
"instansi": {
"id": 9,
"description": "Xxxxxxxxxxx Xxxxxxxxxxxxx Xxxxxxxxxxxx"
}
},
"kelas_periode_tahapan": {
"id": 999,
"kelas_id": 99,
"periode_tahapan_id": 99,
"is_verified": 9,
"is_locked": 9,
"kelas": {
"id": 99,
"periode_prodi_id": 9,
"description": "Xxxxxxxxxxx Xxxxxxxxxxxxx Xxxxxxxxxxxx",
"nama_grup_wa": "Xxxxxxxx Xxxxxxxxxxx",
"link_grup_wa": "https://xxxxxxxxxxxxxxxxxxxxxxxxx",
"periode_prodi": {
"id": 9,
"prodi_id": 9,
"periode_id": 9,
"prodi": {
"id": 9,
"nama": "Xxxxxxxxxxxxx Xxxxxxxxxxxxx",
"kode": "999",
"is_active": 9
}
}
},
"periode_tahapan": {
"id": 99,
"periode_id": 9,
"tahapan_id": 9,
"ket_hari_ke": 99,
"jadwal": "09-08-9099",
"materi": "Xxxxxxxxxxx Xxxxxxxx Xxxxxxxxxxxxxxxxxxxxx",
"jp_full": 9,
"jp_pamong": 9,
"tahapan": {
"id": 9,
"description": "Xxxxxxxxxxx Xxxxxxxxxxxxx Xxxxxxxxxxxx",
"akronim": "XXX",
"satuan": "XXX"
},
"periode": {
"id": 9,
"tahun_id": 9,
"periode": 9,
"tahun": {
"id": 9,
"tahun": 9099
},
"persentase_honor": {
"id": 9,
"periode_id": 9,
"full_dosen": 900,
"full_gpamong": 900,
"persentase_dosen": 99,
"persentase_gpamong": 99
}
}
}
},
"DT_RowIndex": 1
}