folder structure for scraping in Laravel, using Goutte - laravel

Question

I am a bit confused about my folder structure for the scraping code. Using console/commands, not the controller. So, in the handle function I am writing the whole scraping code. But should I suppose to do that? Or... what is the best approach for this?
UPDATED
If I understand correctly the answer below. It should look like this right now.
calling services
class siteControl extends Command
{
protected $signature = 'bot:scrape {website_id}';
protected $description = 'target a portal site and scrape';
public function __construct()
{
parent::__construct();
}
public function handle()
{
$website_id = $this->argument("website_id");
if ($website_id == 1) {
$portal = "App\Services\Site1";
}
$crawler = new $portal;
$crawler->run();
}
}
in handle method
class Site1 extends Utility
{
public function __construct()
{
parent::__construct();
}
public function run()
{
echo "method runs";
}
}
abstract:
use Goutte\Client;
abstract class Utility implements SiteInterfaces
{
protected $client;
public function __construct()
{
$this->client = new Client();
}
}
interfaces:
namespace App\Services;
interface SiteInterfaces
{
public function run();
}
and finally, I should write the whole scraping code inside the run() method? Please correct me If wrong about this... I am searching the best solution.

Answer 1

A best practice would be to call a separate service from your command handle() method. That way you could reuse that same service in a controller for instance.
The technical version:
Your application is given a specific thing to do (a command if you will). This command comes from outside of your application, which can be a anything from a web controller, to an API controller or a CLI application. In terms of hexagonal architecture this is called a port.
Once the application receives such a command it should not care which port it came from. By handling all similar commands in a single spot (a command handler) it does not have to worry about the origins of the command.
So to give you a short overview:
[Web request] [CLI command] <-- these are ports
\ /
\ /
\ /
[Command] <--- this is a method call to your service
|
|
|
[Command handler] <--- this is the service doing the actual work
Updated my answer
Based on the code you provided I implemented what I mentioned above like so:
app/Console/Command/BotScrapeCommand.php
This is the CLI command I mentioned above. All this class has to do is:
1. Gather input arguments; (website_id) in this case
2. Wrap those arguments in a command
3. Fire off the command using the command handler
namespace App\Console\Commands;
use App\Command\ScrapePortalSiteCommand;
use CommandHandler\ScrapePortalSiteCommandHandler;
class BotScrapeCommand extends Command
{
protected $signature = 'bot:scrape {website_id}';
protected $description = 'target a portal site and scrape';
public function handle(ScrapePortalSiteCommandHandler $handler)
{
$portalSiteId = $this->argument("website_id");
$command = new ScrapePortalSiteCommand($portalSiteId);
$handler->handle($command);
}
}
app/Command/ScapePortalSiteCommand.php
This is the Command I mentioned above. Its job is to wrap all input arguments in a class, which can be used by a command handler.
namespace App\Command;
class ScrapePortalSiteCommand
{
/**
* #var int
*/
private $portalSiteId;
public function __construct(int $portalSiteId)
{
$this->portalSiteId = $portalSiteId;
}
public function getPortalSiteId(): int
{
return $this->portalSiteId;
}
}
app/CommandHandler/ScrapePortalSiteCommandHandler.php
The command handler should implement logic based on its command. In this case that's figuring out which crawler to pick, then fire that one off.
namespace App\CommandHandler;
use App\Command\ScrapePortalSiteCommand;
use App\Crawler\PortalSite1Crawler;
use App\Crawler\PortalSiteCrawlerInterface;
use InvalidArgumentException;
class ScrapePortalSiteCommandHandler
{
public function handle(ScrapePortalSiteCommand $command): void
{
$crawler = $this->getCrawlerForPortalSite($command->getPortalSiteId());
$crawler->crawl();
}
private function getCrawlerForPortalSite(int $portalSiteId): PortalSiteCrawlerInterface {
switch ($portalSiteId) {
case 1:
return new PortalSite1Crawler();
default:
throw new InvalidArgumentException(
sprintf('No crawler configured for portal site with id "%s"', $portalSiteId)
);
}
}
}
app/Crawler/PortalSiteCrawlerInterface.php
This interface is there to make sure all crawlers can be called in similar fashion. Additionally it makes for nice type hinting.
namespace App\Crawler;
interface PortalSiteCrawlerInterface
{
public function crawl(): void;
}
app/Crawler/PortalSite1Crawler.php
This is where the implementation of the actual scraping goes.
namespace App\Crawler;
class PortalSite1Crawler implements PortalSiteCrawlerInterface
{
public function crawl(): void
{
// Crawl your site here
}
}
Another update
As you had some additional questions I've updated my answer once more.
:void
The use of : void in a method declaration means the method will not return anything. In a same way public function getPortalSiteId(): int means this method will always return an integer. The use of return typehints was added to PHP 7 and is not specific to Laravel. More information on return typehints can be found in the PHP documentation.
Commands and handlers
The use of commands and command handlers is a best practice which is part of the command bus pattern. This pattern describes an universal way of dealing with user input (a command). This post offers a nice explanation on commands and handlers. Additionally, this blog post describes in more details what a command bus is, how it's used and what the advantages are. Please note that in the code I've provided the bus implementation itself is skipped. In my opinion you do not need it per se, but in some cases it does add value.