403 Forbidden error is received on file upload - laravel

Question

I'm trying to upload a file in my Laravel 9 application. On the localhost, it works just fine, however, the same action fails when I try it from my application running on an Amazon Linux EC2.
What I know:
The action does not throw any exceptions, it just receives an HTTP 403 error. (I've enabled debug in my .env)
I'm only receiving this error in places where I'm trying to upload an image. Other controllers work fine.
What I suspect the issue is:
I believe the issue is with Apache/server configuration. I've added a dd() call in the first line of the update() function and it doesn't even get there. The 403 error is thrown before that. see below.
What I've done to troubleshoot:
Made sure the S3 bucket policy is ok (the same controller works fine on localhost)
Made sure the file permissions are ok (see screenshot below)
Made sure the authorize() function inside the request class returns true
looked at the error logs and don't see anything relevant in there either.
Made sure file_uploads = On, upload_max_filesize = 4M and max_file_uploads = 20 inside php.ini
Here is what my controller looks like (the authorize() function inside UpdateContactRequest returns true):
//https://myapp.com/contacts/1
//App\Http\Controllers\ContactController
public function update(UpdateContactRequest $request, Contact $contact)
{
dd($request); //This does not get executed. The 403 error happens before reaching this.
$contact->update($request->all());
$this->uploadAvatar($request, $contact);
Flash::success('Contact updated successfully.');
return redirect(route('dealers.contacts.index', $contact->dealer->id));
}
Here is the uploadAvatar function I use to do the upload:
private function uploadAvatar(Request $request, Contact $contact)
{
if ($request->hasFile('avatar')) {
try {
$contact
->addMediaFromRequest('avatar')
->sanitizingFileName(function ($fileName) {
return strtolower(str_replace(['#', '/', '\\', ' '], '-', $fileName));
})
->toMediaCollection('avatars');
} catch (\Spatie\MediaLibrary\MediaCollections\Exceptions\FileUnacceptableForCollection $e) {
Flash::error($e->getMessage());
}
}
}
How the routes are defined:
Route::get('/', function () {
return view('welcome');
});
Route::get('/test', App\Http\Controllers\TestController::class);
Route::get('/embed-iframe/{uuid}', [App\Http\Controllers\EmbedController::class, 'iframe']);
Route::get('/embed-js/{uuid}', [App\Http\Controllers\EmbedController::class, 'js']);
Auth::routes();
Route::middleware('admin')->group(function () {
Route::get('/home', [App\Http\Controllers\HomeController::class, 'index'])->name('home');
Route::resource('dealers', App\Http\Controllers\DealerController::class);
Route::post('refreshDealerCRMData', [App\Http\Controllers\DealerController::class, 'refreshCRMData']);
Route::post('loadCRMView', [App\Http\Controllers\DealerController::class, 'loadCRMView']);
Route::resource('cms', App\Http\Controllers\CmsController::class);
Route::resource('crms', App\Http\Controllers\CrmController::class);
Route::resource('leads', App\Http\Controllers\LeadController::class);
Route::resource('contacts', App\Http\Controllers\ContactController::class);
Route::resource('attachment-categories', App\Http\Controllers\AttachmentCategoryController::class);
Route::resource('CRMAttachments', App\Http\Controllers\CRMAttachmentController::class);
Route::resource('dealers.leads', App\Http\Controllers\DealerLeadController::class)->scoped([
'dealers' => 'dealer.id',
'leads' => 'lead.id',
]);
Route::resource('dealers.contacts', App\Http\Controllers\DealerContactController::class)->scoped([
'dealers' => 'dealer.id',
'contacts' => 'contact.id',
]);
});
Here is a screenshot of my root directory (/var/www/html):
Here are my virtual hosts defined in /etc/httpd/conf/httpd.conf
<VirtualHost *:443>
ServerAdmin user#domain.com
ServerName myapp.com
ServerAlias www.myapp.com
DocumentRoot /var/www/html/public
<Directory /var/www/html>
Options Indexes FollowSymLinks MultiViews
AllowOverride All
Require all granted
</Directory>
</VirtualHost>
<VirtualHost *:80>
ServerAdmin user#domain.com
ServerName myapp.com
ServerAlias www.myapp.com
DocumentRoot /var/www/html/public
<Directory /var/www/html>
Options Indexes FollowSymLinks MultiViews
AllowOverride All
Require all granted
</Directory>
</VirtualHost>
Here is my IAM policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::mybucket",
"arn:aws:s3:::mybucket/*"
]
}
]
}
Here is a screenshot of the exact error being thrown:
How can I further troubleshoot this when no actual exception is thrown?

Answer 1

For other users who run into mysterious 403 Errors using AWS, here is an approach to debugging this and problems like it.
First, verify if your request is even hitting your application. An excellent way to do this is to try to dd() in the AppServiceProvider. In this case, it would not have returned anything, which is your clue that it is outside your application. If it would have been inside, continue debugging using dd() through your application or logging into the Laravel log.
The next step is ensuring you are hitting the server—two approaches for this when using AWS.
Using Sampled Requests View
Aws has a thorough walk-through for viewing a sample of web requests - this will allow you to view any incoming requests that AWS WAF has inspected and either allowed or blocked.
AWS Waf Logs
Additionally, when setting up new projects, I recommend always enabling AWS Waf Logs, allowing you to see which rule is the terminating rule blocking the request.
Both of these methods should be sufficient in debugging this type of error.

Answer 2

The issue was with an AWS WAF rule preventing HTTP post size from exceeding a certain number. Removed the rule, and it's now working again.

Answer 3

I'm guessing this 403 issue you were experiencing was having place in a shared hosting. When you create the symlink with php artisan storage:link it contains the full absolute path to the storage/app/public folder. This is what probably causes the 403 response by the server.
The solution is create a symlink with a relative path from the project root like
ln -s ../storage/app/public public/storage