Thu thập dữ liệu bằng Laravel, proxy và HTML Dom đơn giản

Thứ Sáu, 22/11/2019

Tram Ho

Hôm nay mình sẽ giới thiệu các bạn làm sao có thể lấy được dữ liệu từ bất kỳ một trang web nào sử dụng laravel, proxy và html dom
Trong bài viết này mình sẽ lấy ví dụ là crawl product của amazon

Cài đặt

Đầu tiên các bạn vào site này download file simple_html_dom.php để vào thư mục Helpers của laravel chẳng hạn (thư mục mình tự tạo ra, bạn có thể bỏ vào bất cứ thu mục nào bạn muốn).
sau đó mở file composer.json ra và thêm đường dẫn file vừa tạo vào phần autoload

<span class="token property">"autoload"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
        <span class="token property">"files"</span><span class="token operator">:</span> <span class="token punctuation">[</span>
            <span class="token string">"app/Helpers/simple_html_dom.php"</span> # thêm vào đây
        <span class="token punctuation">]</span><span class="token punctuation">,</span>
        <span class="token property">"psr-4"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
            <span class="token property">"App\"</span><span class="token operator">:</span> <span class="token string">"app/"</span>
        <span class="token punctuation">}</span><span class="token punctuation">,</span>
        <span class="token property">"classmap"</span><span class="token operator">:</span> <span class="token punctuation">[</span>
            <span class="token string">"database/seeds"</span><span class="token punctuation">,</span>
            <span class="token string">"database/factories"</span>
        <span class="token punctuation">]</span>
    <span class="token punctuation">}</span><span class="token punctuation">,</span>

"autoload": {

"files": [

"app/Helpers/simple_html_dom.php" # thêm vào đây

],

"psr-4": {

"App\": "app/"

},

"classmap": [

"database/seeds",

"database/factories"

]

},

rồi chạy composer dumpautoload để file này được load vào thư viện của laravel.

Code

Để crawl dữ liệu mình sẽ tạo ra file command sau đó từ command gọi sang phần jobs của laravel. nếu dùng thế này mình có thể đẩy toàn bộ tác vụ crawl chạy tự động cũng như đẩy phần chạy vào queue rồi chúng ra có thể dùng supervisor để start 1 lúc nhiều process lên chạy cùng 1 lúc. nhưng mình khuyên là nên để tối đa 5 process chạy 1 lúc thôi, amazone sẽ block IP nào có nhiều request đến trong 1 khoảng thời gian ngắn (cái này có thể dùng public proxy hoặc private proxy để vượt qua được).

Đầu tiên bạn cần tạo 1 file AwsProductCrawler.php trong thưc mục app/Console/Commands có nội dung như sau:

<span class="token php language-php"><span class="token delimiter important">&lt;?php</span>

<span class="token keyword">namespace</span> <span class="token package">AppConsoleCommands</span><span class="token punctuation">;</span>

<span class="token keyword">use</span> <span class="token package">AppJobsAwsCrawlerLink</span><span class="token punctuation">;</span>
<span class="token keyword">use</span> <span class="token package">DB</span><span class="token punctuation">;</span>
<span class="token keyword">use</span> <span class="token package">IlluminateConsoleCommand</span><span class="token punctuation">;</span>

<span class="token keyword">class</span> <span class="token class-name">AwsProductCrawler</span> <span class="token keyword">extends</span> <span class="token class-name">Command</span>
<span class="token punctuation">{</span>
    <span class="token comment">/**
     * The name and signature of the console command.
     *
     * @var string
     */</span>
    <span class="token keyword">protected</span> <span class="token variable">$signature</span> <span class="token operator">=</span> <span class="token single-quoted-string string">'aws:product'</span><span class="token punctuation">;</span>

    <span class="token comment">/**
     * The console command description.
     *
     * @var string
     */</span>
    <span class="token keyword">protected</span> <span class="token variable">$description</span> <span class="token operator">=</span> <span class="token single-quoted-string string">'aws product crawler, run one time a week'</span><span class="token punctuation">;</span>

    <span class="token comment">/**
     * Create a new command instance.
     *
     * @return void
     */</span>
    <span class="token keyword">public</span> <span class="token keyword">function</span> <span class="token function">__construct</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
    <span class="token punctuation">{</span>
        <span class="token keyword">parent</span><span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">__construct</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
    <span class="token punctuation">}</span>

    <span class="token keyword">const</span> <span class="token constant">LIMIT</span> <span class="token operator">=</span> <span class="token number">25</span><span class="token punctuation">;</span>

    <span class="token comment">/**
     * Execute the console command.
     *
     * @return mixed
     */</span>
    <span class="token keyword">public</span> <span class="token keyword">function</span> <span class="token function">handle</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
    <span class="token punctuation">{</span>
        <span class="token constant">DB</span><span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">table</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'merchants'</span><span class="token punctuation">)</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">orderBy</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'id'</span><span class="token punctuation">)</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">chunk</span><span class="token punctuation">(</span>self<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token constant">LIMIT</span><span class="token punctuation">,</span> <span class="token keyword">function</span> <span class="token punctuation">(</span><span class="token variable">$merchants</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
            <span class="token keyword">foreach</span> <span class="token punctuation">(</span><span class="token variable">$merchants</span> <span class="token keyword">as</span> <span class="token variable">$merchant</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
                AwsCrawlerLink<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">dispatch</span><span class="token punctuation">(</span><span class="token variable">$merchant</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
            <span class="token punctuation">}</span>
        <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span>

        <span class="token keyword">return</span><span class="token punctuation">;</span>
    <span class="token punctuation">}</span>
<span class="token punctuation">}</span>
</span>

<?php

namespace AppConsoleCommands;

use AppJobsAwsCrawlerLink;

use DB;

use IlluminateConsoleCommand;

class AwsProductCrawler extends Command

{

/**

* The name and signature of the console command.

* @var string

*/

protected $signature = 'aws:product';

/**

* The console command description.

* @var string

*/

protected $description = 'aws product crawler, run one time a week';

/**

* Create a new command instance.

* @return void

*/

{

parent::__construct();

}

const LIMIT = 25;

/**

* Execute the console command.

* @return mixed

*/

public function handle()

{

DB::table('merchants')->orderBy('id')->chunk(self::LIMIT, function ($merchants) {

foreach ($merchants as $merchant) {

AwsCrawlerLink::dispatch($merchant);

}

});

return;

}

file này có nội dung khá đơn giản, nó chỉ là lấy những merchants nào cần crawl có trong database thôi, trong bản merchant các bạn cần có merchant_id để có thể vào list của merchant này get toàn bộ product về.

Tiếp theo bạn cần tạo 1 file AwsCrawlerLink.php trong thư mục Jobs của laravel có nội dung sau:

<span class="token php language-php"><span class="token delimiter important">&lt;?php</span>

<span class="token keyword">namespace</span> <span class="token package">AppJobs</span><span class="token punctuation">;</span>

<span class="token keyword">use</span> <span class="token package">AppHelpersAwsClient</span><span class="token punctuation">;</span>
<span class="token keyword">use</span> <span class="token package">IlluminateBusQueueable</span><span class="token punctuation">;</span>
<span class="token keyword">use</span> <span class="token package">IlluminateContractsQueueShouldQueue</span><span class="token punctuation">;</span>
<span class="token keyword">use</span> <span class="token package">IlluminateFoundationBusDispatchable</span><span class="token punctuation">;</span>
<span class="token keyword">use</span> <span class="token package">IlluminateQueueInteractsWithQueue</span><span class="token punctuation">;</span>
<span class="token keyword">use</span> <span class="token package">IlluminateQueueSerializesModels</span><span class="token punctuation">;</span>
<span class="token keyword">use</span> <span class="token package">IlluminateSupportFacadesLog</span><span class="token punctuation">;</span>
<span class="token keyword">use</span> <span class="token package">MockeryException</span><span class="token punctuation">;</span>

<span class="token keyword">class</span> <span class="token class-name">AwsCrawlerLink</span> <span class="token keyword">implements</span> <span class="token class-name">ShouldQueue</span>
<span class="token punctuation">{</span>
    <span class="token keyword">use</span> <span class="token package">Dispatchable</span><span class="token punctuation">,</span> InteractsWithQueue<span class="token punctuation">,</span> Queueable<span class="token punctuation">,</span> SerializesModels<span class="token punctuation">;</span>

    <span class="token keyword">const</span> <span class="token constant">CLASS_DETAIL_PRODUCT</span> <span class="token operator">=</span> <span class="token single-quoted-string string">'.a-text-normal'</span><span class="token punctuation">;</span>

    <span class="token keyword">const</span> <span class="token constant">TIME_OUT</span> <span class="token operator">=</span> <span class="token number">300</span><span class="token punctuation">;</span>

    <span class="token keyword">protected</span> <span class="token variable">$seller</span><span class="token punctuation">;</span>

    <span class="token comment">/**
     * Create a new job instance.
     * @param  $seller
     * @return void
     */</span>
    <span class="token keyword">public</span> <span class="token keyword">function</span> <span class="token function">__construct</span><span class="token punctuation">(</span><span class="token variable">$seller</span><span class="token punctuation">)</span>
    <span class="token punctuation">{</span>
        <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">seller</span> <span class="token operator">=</span> <span class="token variable">$seller</span><span class="token punctuation">;</span>
    <span class="token punctuation">}</span>

    <span class="token comment">/**
     * Execute the job.
     *
     * @return void
     */</span>
    <span class="token keyword">public</span> <span class="token keyword">function</span> <span class="token function">handle</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
    <span class="token punctuation">{</span>
        <span class="token keyword">try</span> <span class="token punctuation">{</span>
            <span class="token function">set_time_limit</span><span class="token punctuation">(</span>self<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token constant">TIME_OUT</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
            Log<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">debug</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'Start crawl product link, seller = '</span> <span class="token punctuation">.</span> <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">seller</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">merchant_id</span><span class="token punctuation">)</span><span class="token punctuation">;</span>

            <span class="token variable">$awsBaseUrl</span> <span class="token operator">=</span> <span class="token function">env</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'BASE_AWS_URL'</span><span class="token punctuation">,</span> <span class="token function">config</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'common.default_aws_url'</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
            <span class="token variable">$sellerBaseUrl</span> <span class="token operator">=</span> <span class="token variable">$awsBaseUrl</span> <span class="token punctuation">.</span> <span class="token single-quoted-string string">'/s?me='</span> <span class="token punctuation">.</span> <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">seller</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">merchant_id</span><span class="token punctuation">;</span>
            <span class="token variable">$endPage</span> <span class="token operator">=</span> <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">countPage</span><span class="token punctuation">(</span><span class="token variable">$sellerBaseUrl</span><span class="token punctuation">)</span><span class="token punctuation">;</span>

            <span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token variable">$i</span> <span class="token operator">=</span> <span class="token number">1</span><span class="token punctuation">;</span> <span class="token variable">$i</span> <span class="token operator">&lt;=</span> <span class="token variable">$endPage</span><span class="token punctuation">;</span> <span class="token variable">$i</span><span class="token operator">++</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
                <span class="token variable">$urlWithPage</span> <span class="token operator">=</span> <span class="token variable">$sellerBaseUrl</span> <span class="token punctuation">.</span> <span class="token single-quoted-string string">'&amp;page='</span> <span class="token punctuation">.</span> <span class="token variable">$i</span><span class="token punctuation">;</span>

                Log<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">debug</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'start get list products $urlAwsSeller ='</span> <span class="token punctuation">.</span> <span class="token variable">$urlWithPage</span><span class="token punctuation">)</span><span class="token punctuation">;</span>

                <span class="token variable">$html</span> <span class="token operator">=</span> AwsClient<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">getContent</span><span class="token punctuation">(</span><span class="token variable">$urlWithPage</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
                <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token function">is_array</span><span class="token punctuation">(</span><span class="token variable">$html</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
                    <span class="token keyword">continue</span><span class="token punctuation">;</span>
                <span class="token punctuation">}</span>

                <span class="token variable">$html</span> <span class="token operator">=</span> <span class="token function">str_get_html</span><span class="token punctuation">(</span><span class="token variable">$html</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
                <span class="token keyword">foreach</span> <span class="token punctuation">(</span><span class="token variable">$html</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">find</span><span class="token punctuation">(</span>self<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token constant">CLASS_DETAIL_PRODUCT</span><span class="token punctuation">)</span> <span class="token keyword">as</span> <span class="token variable">$productDetailUrl</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
                    <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span><span class="token function">empty</span><span class="token punctuation">(</span><span class="token variable">$urlDetail</span> <span class="token operator">=</span> <span class="token variable">$productDetailUrl</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">href</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
                        <span class="token variable">$urlDetail</span> <span class="token operator">=</span> <span class="token function">env</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'BASE_AWS_URL'</span><span class="token punctuation">,</span> <span class="token single-quoted-string string">'https://www.amazon.co.jp'</span><span class="token punctuation">)</span> <span class="token punctuation">.</span> <span class="token variable">$urlDetail</span><span class="token punctuation">;</span>
                        AwsCrawlerDetail<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">dispatch</span><span class="token punctuation">(</span><span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">seller</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">id</span><span class="token punctuation">,</span> <span class="token variable">$urlDetail</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
                    <span class="token punctuation">}</span>
                <span class="token punctuation">}</span>
            <span class="token punctuation">}</span>
        <span class="token punctuation">}</span> <span class="token keyword">catch</span> <span class="token punctuation">(</span><span class="token class-name">Exception</span> <span class="token variable">$exception</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
            <span class="token function">report</span><span class="token punctuation">(</span><span class="token variable">$exception</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token punctuation">}</span>

        Log<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">debug</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'End crawl product, seller = '</span> <span class="token punctuation">.</span> <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">seller</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">merchant_id</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token keyword">return</span><span class="token punctuation">;</span>
    <span class="token punctuation">}</span>

    <span class="token keyword">private</span> <span class="token keyword">function</span> <span class="token function">countPage</span><span class="token punctuation">(</span><span class="token variable">$urlAwsSeller</span><span class="token punctuation">)</span>
    <span class="token punctuation">{</span>
        Log<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">debug</span><span class="token punctuation">(</span><span class="token double-quoted-string string">"Start get count page, url= <span class="token interpolation"><span class="token punctuation">{</span><span class="token variable">$urlAwsSeller</span><span class="token punctuation">}</span></span>"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token variable">$html</span> <span class="token operator">=</span> AwsClient<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">getContent</span><span class="token punctuation">(</span><span class="token variable">$urlAwsSeller</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token function">is_array</span><span class="token punctuation">(</span><span class="token variable">$html</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
            <span class="token keyword">return</span> <span class="token number">0</span><span class="token punctuation">;</span>
        <span class="token punctuation">}</span>
        <span class="token variable">$html</span> <span class="token operator">=</span> <span class="token function">str_get_html</span><span class="token punctuation">(</span><span class="token variable">$html</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token comment">// find end page more than 9 page</span>
        <span class="token variable">$page</span> <span class="token operator">=</span> <span class="token variable">$html</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">find</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'.a-disabled'</span><span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token variable">$page</span> <span class="token operator">&amp;&amp;</span> <span class="token function">isset</span><span class="token punctuation">(</span><span class="token variable">$page</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">plaintext</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
            Log<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">debug</span><span class="token punctuation">(</span><span class="token double-quoted-string string">"page count is <span class="token interpolation"><span class="token punctuation">{</span><span class="token variable">$page</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">plaintext</span><span class="token punctuation">}</span></span>"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
            <span class="token variable">$pageCount</span> <span class="token operator">=</span> <span class="token variable">$page</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">plaintext</span><span class="token punctuation">;</span>
        <span class="token punctuation">}</span>

        <span class="token comment">// find end page not more than 9 page</span>
        <span class="token variable">$page</span> <span class="token operator">=</span> <span class="token variable">$html</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">find</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'.a-normal'</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token variable">$page</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
            <span class="token variable">$page</span> <span class="token operator">=</span> <span class="token function">end</span><span class="token punctuation">(</span><span class="token variable">$page</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
            <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token function">isset</span><span class="token punctuation">(</span><span class="token variable">$page</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">plaintext</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
                Log<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">debug</span><span class="token punctuation">(</span><span class="token double-quoted-string string">"page count is <span class="token interpolation"><span class="token punctuation">{</span><span class="token variable">$page</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">plaintext</span><span class="token punctuation">}</span></span>"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
                <span class="token variable">$pageCount</span> <span class="token operator">=</span> <span class="token variable">$page</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">plaintext</span><span class="token punctuation">;</span>
            <span class="token punctuation">}</span>
        <span class="token punctuation">}</span>

        AwsClient<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">cleanHtml</span><span class="token punctuation">(</span><span class="token variable">$html</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token function">isset</span><span class="token punctuation">(</span><span class="token variable">$pageCount</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
            <span class="token keyword">return</span> <span class="token variable">$pageCount</span><span class="token punctuation">;</span>
        <span class="token punctuation">}</span>

        Log<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">error</span><span class="token punctuation">(</span><span class="token double-quoted-string string">"======= Cannot get countPage urlAwsSeller = <span class="token interpolation"><span class="token punctuation">{</span><span class="token variable">$urlAwsSeller</span><span class="token punctuation">}</span></span> or maybe count Page = 1"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>

        <span class="token keyword">return</span> <span class="token number">1</span><span class="token punctuation">;</span>
    <span class="token punctuation">}</span>
<span class="token punctuation">}</span>
</span>

100

101

102

103

104

105

106

107

108

109

110

<?php

namespace AppJobs;

use AppHelpersAwsClient;

use IlluminateBusQueueable;

use IlluminateContractsQueueShouldQueue;

use IlluminateFoundationBusDispatchable;

use IlluminateQueueInteractsWithQueue;

use IlluminateQueueSerializesModels;

use IlluminateSupportFacadesLog;

use MockeryException;

class AwsCrawlerLink implements ShouldQueue

{

use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

const CLASS_DETAIL_PRODUCT = '.a-text-normal';

const TIME_OUT = 300;

protected $seller;

/**

* Create a new job instance.

* @param $seller

* @return void

*/

{

$this->seller = $seller;

}

/**

* Execute the job.

* @return void

*/

{

try {

set_time_limit(self::TIME_OUT);

Log::debug('Start crawl product link, seller = ' . $this->seller->merchant_id);

$awsBaseUrl = env('BASE_AWS_URL', config('common.default_aws_url'));

$sellerBaseUrl = $awsBaseUrl . '/s?me=' . $this->seller->merchant_id;

$endPage = $this->countPage($sellerBaseUrl);

for ($i = 1; $i <= $endPage; $i++) {

$urlWithPage = $sellerBaseUrl . '&page=' . $i;

$html = AwsClient::getContent($urlWithPage);

if (is_array($html)) {

continue;

}

$html = str_get_html($html);

foreach ($html->find(self::CLASS_DETAIL_PRODUCT) as $productDetailUrl) {

if (!empty($urlDetail = $productDetailUrl->href)) {

$urlDetail = env('BASE_AWS_URL', 'https://www.amazon.co.jp') . $urlDetail;

AwsCrawlerDetail::dispatch($this->seller->id, $urlDetail);

}

} catch (Exception $exception) {

report($exception);

}

Log::debug('End crawl product, seller = ' . $this->seller->merchant_id);

return;

}

private function countPage($urlAwsSeller)

{

Log::debug("Start get count page, url= {$urlAwsSeller}");

$html = AwsClient::getContent($urlAwsSeller);

return 0;

}

// find end page more than 9 page

$page = $html->find('.a-disabled', 1);

if ($page && isset($page->plaintext)) {

Log::debug("page count is {$page->plaintext}");

$pageCount = $page->plaintext;

}

// find end page not more than 9 page

$page = $html->find('.a-normal');

if ($page) {

$page = end($page);

if (isset($page->plaintext)) {

Log::debug("page count is {$page->plaintext}");

}

AwsClient::cleanHtml($html);

if (isset($pageCount)) {

return $pageCount;

}

Log::error("======= Cannot get countPage urlAwsSeller = {$urlAwsSeller} or maybe count Page = 1");

return 1;

}

File này có nhiệm vụ lấy toàn bộ url của product về, xong nó tiếp tục đẩy sang AwsCrawlerDetail để lấy chi tiết thông tin của products

function getContent có nội dung như sau:

 <span class="token keyword">public</span> <span class="token keyword">static</span> <span class="token keyword">function</span> <span class="token function">getData</span><span class="token punctuation">(</span><span class="token variable">$url</span><span class="token punctuation">,</span> <span class="token variable">$proxy</span> <span class="token operator">=</span> <span class="token boolean">false</span><span class="token punctuation">)</span>
    <span class="token punctuation">{</span>
        <span class="token variable">$client</span> <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">Client</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>

        <span class="token keyword">try</span> <span class="token punctuation">{</span>
            <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span><span class="token variable">$proxy</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
                <span class="token variable">$content</span> <span class="token operator">=</span> <span class="token variable">$client</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">get</span><span class="token punctuation">(</span><span class="token variable">$url</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
                <span class="token keyword">return</span> <span class="token variable">$content</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">getBody</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">getContents</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
            <span class="token punctuation">}</span>

            <span class="token variable">$content</span> <span class="token operator">=</span> <span class="token variable">$client</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">get</span><span class="token punctuation">(</span><span class="token variable">$url</span><span class="token punctuation">,</span> <span class="token punctuation">[</span>
                <span class="token single-quoted-string string">'proxy'</span> <span class="token operator">=</span><span class="token operator">&gt;</span> <span class="token variable">$proxy</span><span class="token punctuation">,</span>
                <span class="token single-quoted-string string">'connect_timeout'</span> <span class="token operator">=</span><span class="token operator">&gt;</span> <span class="token number">20</span><span class="token punctuation">,</span> 
                <span class="token single-quoted-string string">'timeout'</span> <span class="token operator">=</span><span class="token operator">&gt;</span> <span class="token number">60</span><span class="token punctuation">,</span>
                <span class="token single-quoted-string string">'allow_redirects'</span> <span class="token operator">=</span><span class="token operator">&gt;</span> <span class="token boolean">false</span><span class="token punctuation">,</span>
                <span class="token single-quoted-string string">'headers'</span> <span class="token operator">=</span><span class="token operator">&gt;</span> <span class="token punctuation">[</span>
                    <span class="token single-quoted-string string">'User-Agent'</span> <span class="token operator">=</span><span class="token operator">&gt;</span> <span class="token single-quoted-string string">'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13'</span><span class="token punctuation">,</span>
                <span class="token punctuation">]</span>
            <span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
            <span class="token keyword">return</span> <span class="token variable">$content</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">getBody</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">getContents</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token punctuation">}</span> <span class="token keyword">catch</span> <span class="token punctuation">(</span><span class="token class-name">Exception</span> <span class="token variable">$exception</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
            Log<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">error</span><span class="token punctuation">(</span><span class="token double-quoted-string string">"(getData) Exception messages  = <span class="token interpolation"><span class="token punctuation">{</span><span class="token variable">$exception</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">getMessage</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">}</span></span>"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
            Log<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">error</span><span class="token punctuation">(</span><span class="token double-quoted-string string">"(getData) status code = <span class="token interpolation"><span class="token punctuation">{</span><span class="token variable">$exception</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">getCode</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">}</span></span>"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>

            <span class="token keyword">return</span> <span class="token punctuation">[</span>
                <span class="token single-quoted-string string">'error'</span> <span class="token operator">=</span><span class="token operator">&gt;</span> <span class="token boolean">true</span><span class="token punctuation">,</span>
                <span class="token single-quoted-string string">'code'</span> <span class="token operator">=</span><span class="token operator">&gt;</span> <span class="token variable">$exception</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">getCode</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
            <span class="token punctuation">]</span><span class="token punctuation">;</span>
        <span class="token punctuation">}</span>
    <span class="token punctuation">}</span>

public static function getData($url, $proxy = false)

{

$client = new Client();

try {

if (!$proxy) {

$content = $client->get($url);

return $content->getBody()->getContents();

}

'proxy' => $proxy,

'connect_timeout' => 20,

'timeout' => 60,

'allow_redirects' => false,

'headers' => [

'User-Agent' => 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13',

]

]);

Log::error("(getData) Exception messages = {$exception->getMessage()}");

Log::error("(getData) status code = {$exception->getCode()}");

return [

'error' => true,

'code' => $exception->getCode()

];

}

Cuối cùng là phần lấy thông tin chi tiết của product, cũng là phần dài và khó nhất có nội dung như sau AwsCrawlerDetail.php:

<span class="token php language-php"><span class="token delimiter important">&lt;?php</span>

<span class="token keyword">namespace</span> <span class="token package">AppJobs</span><span class="token punctuation">;</span>

<span class="token keyword">use</span> <span class="token package">AppHelpersAwsClient</span><span class="token punctuation">;</span>
<span class="token keyword">use</span> <span class="token package">AppModelsCategory</span><span class="token punctuation">;</span>
<span class="token keyword">use</span> <span class="token package">AppModelsProduct</span><span class="token punctuation">;</span>
<span class="token keyword">use</span> <span class="token package">AppModelsProductDetail</span><span class="token punctuation">;</span>
<span class="token keyword">use</span> <span class="token package">AppModelsProductStar</span><span class="token punctuation">;</span>
<span class="token keyword">use</span> <span class="token package">CarbonCarbon</span><span class="token punctuation">;</span>
<span class="token keyword">use</span> <span class="token package">IlluminateBusQueueable</span><span class="token punctuation">;</span>
<span class="token keyword">use</span> <span class="token package">IlluminateContractsQueueShouldQueue</span><span class="token punctuation">;</span>
<span class="token keyword">use</span> <span class="token package">IlluminateFoundationBusDispatchable</span><span class="token punctuation">;</span>
<span class="token keyword">use</span> <span class="token package">IlluminateQueueInteractsWithQueue</span><span class="token punctuation">;</span>
<span class="token keyword">use</span> <span class="token package">IlluminateQueueSerializesModels</span><span class="token punctuation">;</span>
<span class="token keyword">use</span> <span class="token package">IlluminateSupportFacadesLog</span><span class="token punctuation">;</span>

<span class="token keyword">class</span> <span class="token class-name">AwsCrawlerDetail</span> <span class="token keyword">implements</span> <span class="token class-name">ShouldQueue</span>
<span class="token punctuation">{</span>
    <span class="token keyword">use</span> <span class="token package">Dispatchable</span><span class="token punctuation">,</span> InteractsWithQueue<span class="token punctuation">,</span> Queueable<span class="token punctuation">,</span> SerializesModels<span class="token punctuation">;</span>

    <span class="token keyword">protected</span> <span class="token variable">$seller_id</span><span class="token punctuation">;</span>

    <span class="token keyword">const</span> <span class="token constant">TIME_OUT</span> <span class="token operator">=</span> <span class="token number">300</span><span class="token punctuation">;</span> <span class="token comment">// second</span>

    <span class="token keyword">protected</span> <span class="token variable">$detailProductUrl</span><span class="token punctuation">;</span>
    <span class="token keyword">protected</span> <span class="token variable">$asin</span><span class="token punctuation">;</span>

    <span class="token comment">/**
     * Create a new job instance.
     * @param $sellerId
     * @param $detailProductUrl
     * @param $asin
     * @return void
     */</span>
    <span class="token keyword">public</span> <span class="token keyword">function</span> <span class="token function">__construct</span><span class="token punctuation">(</span><span class="token variable">$sellerId</span><span class="token punctuation">,</span> <span class="token variable">$detailProductUrl</span><span class="token punctuation">,</span> <span class="token variable">$asin</span> <span class="token operator">=</span> <span class="token keyword">null</span><span class="token punctuation">)</span>
    <span class="token punctuation">{</span>
        <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">seller_id</span> <span class="token operator">=</span> <span class="token variable">$sellerId</span><span class="token punctuation">;</span>
        <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">detailProductUrl</span> <span class="token operator">=</span> <span class="token variable">$detailProductUrl</span><span class="token punctuation">;</span>
        <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">asin</span> <span class="token operator">=</span> <span class="token variable">$asin</span><span class="token punctuation">;</span>
    <span class="token punctuation">}</span>

    <span class="token comment">/**
     * Execute the job.
     *
     * @return void
     */</span>
    <span class="token keyword">public</span> <span class="token keyword">function</span> <span class="token function">handle</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
    <span class="token punctuation">{</span>
        <span class="token function">set_time_limit</span><span class="token punctuation">(</span>self<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token constant">TIME_OUT</span><span class="token punctuation">)</span><span class="token punctuation">;</span>

        Log<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">debug</span><span class="token punctuation">(</span><span class="token double-quoted-string string">"Start crawl product detail, url = "</span> <span class="token punctuation">.</span> <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">detailProductUrl</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">checkAlreadyCrawl</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
            Log<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">debug</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'product already crawl, end this product!'</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
            <span class="token keyword">return</span><span class="token punctuation">;</span>
        <span class="token punctuation">}</span>

        <span class="token keyword">try</span> <span class="token punctuation">{</span>
            <span class="token variable">$html</span> <span class="token operator">=</span> AwsClient<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">getContent</span><span class="token punctuation">(</span><span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">detailProductUrl</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
            <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token function">is_array</span><span class="token punctuation">(</span><span class="token variable">$html</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
                Log<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">debug</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'ignore this product content = '</span><span class="token punctuation">,</span> <span class="token variable">$html</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
                <span class="token keyword">return</span><span class="token punctuation">;</span>
            <span class="token punctuation">}</span>

            <span class="token variable">$html</span> <span class="token operator">=</span> <span class="token function">str_get_html</span><span class="token punctuation">(</span><span class="token variable">$html</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
            <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span><span class="token variable">$html</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
                Log<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">debug</span><span class="token punctuation">(</span><span class="token double-quoted-string string">"content null"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
                AwsClient<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">cleanHtml</span><span class="token punctuation">(</span><span class="token variable">$html</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
                <span class="token keyword">return</span><span class="token punctuation">;</span>
            <span class="token punctuation">}</span>

            <span class="token variable">$arrProduct</span> <span class="token operator">=</span> <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">getProductData</span><span class="token punctuation">(</span><span class="token variable">$html</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
            <span class="token variable">$productDetail</span> <span class="token operator">=</span> <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">getProductDetail</span><span class="token punctuation">(</span><span class="token variable">$html</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
            <span class="token variable">$productStar</span> <span class="token operator">=</span> <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">getProductReviewStartDetail</span><span class="token punctuation">(</span><span class="token variable">$html</span><span class="token punctuation">,</span> <span class="token variable">$productDetail</span><span class="token punctuation">[</span><span class="token single-quoted-string string">'review_count'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">;</span>

            <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span><span class="token function">empty</span><span class="token punctuation">(</span><span class="token variable">$arrProduct</span><span class="token punctuation">[</span><span class="token single-quoted-string string">'asin'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
                Log<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">debug</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'save product: '</span><span class="token punctuation">,</span> <span class="token variable">$arrProduct</span><span class="token punctuation">)</span><span class="token punctuation">;</span>

                <span class="token variable">$product</span> <span class="token operator">=</span> Product<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">saveProduct</span><span class="token punctuation">(</span><span class="token variable">$arrProduct</span><span class="token punctuation">,</span> <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">asin</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
                ProductDetail<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">saveProductDetail</span><span class="token punctuation">(</span><span class="token variable">$product</span><span class="token punctuation">,</span> <span class="token variable">$productDetail</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
                ProductStar<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">saveProductStar</span><span class="token punctuation">(</span><span class="token variable">$product</span><span class="token punctuation">,</span> <span class="token variable">$productStar</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
                Category<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">saveCategory</span><span class="token punctuation">(</span><span class="token variable">$product</span><span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token single-quoted-string string">'name'</span> <span class="token operator">=</span><span class="token operator">&gt;</span> <span class="token variable">$productDetail</span><span class="token punctuation">[</span><span class="token single-quoted-string string">'category'</span><span class="token punctuation">]</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
            <span class="token punctuation">}</span>
            AwsClient<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">cleanHtml</span><span class="token punctuation">(</span><span class="token variable">$html</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token punctuation">}</span> <span class="token keyword">catch</span> <span class="token punctuation">(</span><span class="token class-name">Exception</span> <span class="token variable">$exception</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
            Log<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">error</span><span class="token punctuation">(</span><span class="token variable">$exception</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">getMessage</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token punctuation">}</span>
        <span class="token keyword">return</span><span class="token punctuation">;</span>
    <span class="token punctuation">}</span>

    <span class="token keyword">public</span> <span class="token keyword">function</span> <span class="token function">checkAlreadyCrawl</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
    <span class="token punctuation">{</span>
        <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">asin</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
            <span class="token variable">$product</span> <span class="token operator">=</span> Product<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">where</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'asin'</span><span class="token punctuation">,</span> <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">asin</span><span class="token punctuation">)</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">first</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span>
            <span class="token variable">$product</span> <span class="token operator">=</span> Product<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">where</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'detail_aws_url'</span><span class="token punctuation">,</span> <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">detailProductUrl</span><span class="token punctuation">)</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">first</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token punctuation">}</span>

        <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span><span class="token variable">$product</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
            <span class="token keyword">return</span> <span class="token boolean">false</span><span class="token punctuation">;</span>
        <span class="token punctuation">}</span>

        <span class="token keyword">return</span> ProductDetail<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">where</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'product_id'</span><span class="token punctuation">,</span> <span class="token variable">$product</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">id</span><span class="token punctuation">)</span>
            <span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">whereDate</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'created_at'</span><span class="token punctuation">,</span> Carbon<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">now</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">format</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'Y-m-d'</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
            <span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">first</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
    <span class="token punctuation">}</span>

    <span class="token comment">/**
     * get product data from html dom
     * @param $html
     * @return array
     */</span>
    <span class="token keyword">public</span> <span class="token keyword">function</span> <span class="token function">getProductData</span><span class="token punctuation">(</span><span class="token variable">$html</span><span class="token punctuation">)</span>
    <span class="token punctuation">{</span>
        <span class="token variable">$asin</span> <span class="token operator">=</span> <span class="token variable">$html</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">find</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'#cerberus-data-metrics'</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token variable">$asin</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
            <span class="token variable">$asin</span> <span class="token operator">=</span> <span class="token variable">$asin</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">getAllAttributes</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token punctuation">}</span>

        <span class="token variable">$img</span> <span class="token operator">=</span> <span class="token variable">$html</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">find</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'#imgTagWrapperId img'</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token variable">$img</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
            <span class="token variable">$img</span> <span class="token operator">=</span> <span class="token variable">$img</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">getAttribute</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'data-old-hires'</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token punctuation">}</span>

        <span class="token variable">$sellAt</span> <span class="token operator">=</span> <span class="token variable">$html</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">find</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'.date-first-available .value'</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">plaintext</span> <span class="token operator">?</span><span class="token operator">?</span> <span class="token keyword">null</span><span class="token punctuation">;</span>

        <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span><span class="token variable">$sellAt</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
            <span class="token variable">$sellAt</span> <span class="token operator">=</span> <span class="token variable">$html</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">find</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'#productDetailsTable ul li'</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">)</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">plaintext</span> <span class="token operator">?</span><span class="token operator">?</span> <span class="token keyword">null</span><span class="token punctuation">;</span>
            <span class="token variable">$sellAt</span> <span class="token operator">=</span> <span class="token function">str_replace</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'Amazon.co.jp での取り扱い開始日:'</span><span class="token punctuation">,</span> <span class="token single-quoted-string string">''</span><span class="token punctuation">,</span> <span class="token variable">$sellAt</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token punctuation">}</span>

        <span class="token keyword">return</span> <span class="token punctuation">[</span>
            <span class="token single-quoted-string string">'name'</span> <span class="token operator">=</span><span class="token operator">&gt;</span> <span class="token variable">$html</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">find</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'#productTitle'</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">plaintext</span> <span class="token operator">?</span><span class="token operator">?</span> <span class="token keyword">null</span><span class="token punctuation">,</span>
            <span class="token single-quoted-string string">'url_img'</span> <span class="token operator">=</span><span class="token operator">&gt;</span> <span class="token variable">$img</span><span class="token punctuation">,</span>
            <span class="token single-quoted-string string">'asin'</span> <span class="token operator">=</span><span class="token operator">&gt;</span> <span class="token variable">$asin</span><span class="token punctuation">[</span><span class="token single-quoted-string string">'data-asin'</span><span class="token punctuation">]</span> <span class="token operator">?</span><span class="token operator">?</span> <span class="token keyword">null</span><span class="token punctuation">,</span>
            <span class="token single-quoted-string string">'seller_id'</span> <span class="token operator">=</span><span class="token operator">&gt;</span> <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">seller_id</span><span class="token punctuation">,</span>
            <span class="token single-quoted-string string">'sell_at'</span> <span class="token operator">=</span><span class="token operator">&gt;</span> <span class="token function">trim</span><span class="token punctuation">(</span><span class="token variable">$sellAt</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
            <span class="token single-quoted-string string">'detail_aws_url'</span> <span class="token operator">=</span><span class="token operator">&gt;</span> <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">detailProductUrl</span><span class="token punctuation">,</span>
        <span class="token punctuation">]</span><span class="token punctuation">;</span>
    <span class="token punctuation">}</span>

    <span class="token comment">/**
     * get product detail from html dom
     * @param $html
     * @return array
     */</span>
    <span class="token keyword">public</span> <span class="token keyword">function</span> <span class="token function">getProductDetail</span><span class="token punctuation">(</span><span class="token variable">$html</span><span class="token punctuation">)</span>
    <span class="token punctuation">{</span>
        <span class="token variable">$asin</span> <span class="token operator">=</span> <span class="token variable">$html</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">find</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'#cerberus-data-metrics'</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token variable">$asin</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
            <span class="token variable">$asin</span> <span class="token operator">=</span> <span class="token variable">$asin</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">getAllAttributes</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token punctuation">}</span>

        <span class="token variable">$avgReview</span> <span class="token operator">=</span> <span class="token variable">$html</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">find</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'#acrPopover'</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">title</span> <span class="token operator">?</span><span class="token operator">?</span> <span class="token number">0</span><span class="token punctuation">;</span>
        <span class="token variable">$acrCustomerReviewText</span> <span class="token operator">=</span> <span class="token variable">$html</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">find</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'#acrCustomerReviewText'</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">plaintext</span> <span class="token operator">?</span><span class="token operator">?</span> <span class="token number">0</span><span class="token punctuation">;</span>

        <span class="token variable">$ranking</span> <span class="token operator">=</span> <span class="token variable">$html</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">find</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'#SalesRank .value'</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">innertext</span> <span class="token operator">?</span><span class="token operator">?</span> <span class="token number">0</span><span class="token punctuation">;</span>

        <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span><span class="token variable">$ranking</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
            <span class="token variable">$ranking</span> <span class="token operator">=</span> <span class="token variable">$html</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">find</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'#SalesRank'</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">innertext</span> <span class="token operator">?</span><span class="token operator">?</span> <span class="token number">0</span><span class="token punctuation">;</span>
        <span class="token punctuation">}</span>

        <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span><span class="token variable">$ranking</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
            <span class="token variable">$ranking</span> <span class="token operator">=</span> <span class="token variable">$html</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">find</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'.pdTab'</span><span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">)</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token property">innertext</span> <span class="token operator">?</span><span class="token operator">?</span> <span class="token number">0</span><span class="token punctuation">;</span>
        <span class="token punctuation">}</span>

        <span class="token variable">$ranking</span> <span class="token operator">=</span> <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">getCatAndRank</span><span class="token punctuation">(</span><span class="token variable">$ranking</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token keyword">return</span> <span class="token punctuation">[</span>
            <span class="token single-quoted-string string">'price'</span> <span class="token operator">=</span><span class="token operator">&gt;</span> <span class="token function">str_replace</span><span class="token punctuation">(</span><span class="token single-quoted-string string">','</span><span class="token punctuation">,</span> <span class="token single-quoted-string string">''</span><span class="token punctuation">,</span> <span class="token variable">$asin</span><span class="token punctuation">[</span><span class="token single-quoted-string string">'data-asin-price'</span><span class="token punctuation">]</span><span class="token punctuation">)</span> <span class="token operator">?</span><span class="token operator">?</span> <span class="token number">0</span><span class="token punctuation">,</span>
            <span class="token single-quoted-string string">'currency_code'</span> <span class="token operator">=</span><span class="token operator">&gt;</span> <span class="token variable">$asin</span><span class="token punctuation">[</span><span class="token single-quoted-string string">'data-asin-currency-code'</span><span class="token punctuation">]</span> <span class="token operator">?</span><span class="token operator">?</span> <span class="token single-quoted-string string">'JPY'</span><span class="token punctuation">,</span>
            <span class="token single-quoted-string string">'avg_review'</span> <span class="token operator">=</span><span class="token operator">&gt;</span> <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">getAvgReviewFromString</span><span class="token punctuation">(</span><span class="token variable">$avgReview</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
            <span class="token single-quoted-string string">'review_count'</span> <span class="token operator">=</span><span class="token operator">&gt;</span> <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">getNumberFromString</span><span class="token punctuation">(</span><span class="token variable">$acrCustomerReviewText</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
            <span class="token single-quoted-string string">'ranking'</span> <span class="token operator">=</span><span class="token operator">&gt;</span> <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">getRankingFromString</span><span class="token punctuation">(</span><span class="token variable">$ranking</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
            <span class="token single-quoted-string string">'category'</span> <span class="token operator">=</span><span class="token operator">&gt;</span> <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">getCatFromString</span><span class="token punctuation">(</span><span class="token variable">$ranking</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
        <span class="token punctuation">]</span><span class="token punctuation">;</span>
    <span class="token punctuation">}</span>

    <span class="token comment">/**
     * get start count for product review count
     * @param $html
     * @param $total
     * @return mixed
     */</span>
    <span class="token keyword">public</span> <span class="token keyword">function</span> <span class="token function">getProductReviewStartDetail</span><span class="token punctuation">(</span><span class="token variable">$html</span><span class="token punctuation">,</span> <span class="token variable">$total</span><span class="token punctuation">)</span>
    <span class="token punctuation">{</span>
        <span class="token variable">$arr</span><span class="token punctuation">[</span><span class="token single-quoted-string string">'total_star'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token variable">$total</span><span class="token punctuation">;</span>
        <span class="token keyword">for</span> <span class="token punctuation">(</span><span class="token variable">$i</span> <span class="token operator">=</span> <span class="token number">1</span><span class="token punctuation">;</span> <span class="token variable">$i</span> <span class="token operator">&lt;=</span> <span class="token number">5</span><span class="token punctuation">;</span> <span class="token variable">$i</span><span class="token operator">++</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
            <span class="token variable">$star</span> <span class="token operator">=</span> <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">getAStart</span><span class="token punctuation">(</span><span class="token variable">$html</span><span class="token punctuation">,</span> <span class="token variable">$i</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">// percent</span>
            <span class="token variable">$star</span> <span class="token operator">=</span> <span class="token punctuation">(</span><span class="token variable">$star</span> <span class="token operator">*</span> <span class="token variable">$total</span><span class="token punctuation">)</span> <span class="token operator">/</span> <span class="token number">100</span><span class="token punctuation">;</span>
            <span class="token variable">$arr</span><span class="token punctuation">[</span><span class="token double-quoted-string string">"star_<span class="token interpolation"><span class="token variable">$i</span></span>"</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token punctuation">(</span>int<span class="token punctuation">)</span><span class="token function">round</span><span class="token punctuation">(</span><span class="token variable">$star</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token punctuation">}</span>

        <span class="token keyword">return</span> <span class="token variable">$arr</span><span class="token punctuation">;</span>
    <span class="token punctuation">}</span>

    <span class="token comment">/**
     * get data a star
     * @param $html
     * @param $int
     * @return int
     */</span>
    <span class="token keyword">public</span> <span class="token keyword">function</span> <span class="token function">getAStart</span><span class="token punctuation">(</span><span class="token variable">$html</span><span class="token punctuation">,</span> <span class="token variable">$int</span><span class="token punctuation">)</span>
    <span class="token punctuation">{</span>
        <span class="token variable">$star</span> <span class="token operator">=</span> <span class="token variable">$html</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">find</span><span class="token punctuation">(</span><span class="token double-quoted-string string">"#histogramTable .<span class="token interpolation"><span class="token punctuation">{</span><span class="token variable">$int</span><span class="token punctuation">}</span></span>star"</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span><span class="token variable">$star</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
            <span class="token keyword">return</span> <span class="token number">0</span><span class="token punctuation">;</span>
        <span class="token punctuation">}</span>

        <span class="token variable">$star</span> <span class="token operator">=</span> <span class="token variable">$star</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">getAttribute</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'aria-label'</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token variable">$star</span> <span class="token operator">=</span> <span class="token variable">$this</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token function">getNumberFromString</span><span class="token punctuation">(</span><span class="token variable">$star</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token variable">$star</span> <span class="token operator">=</span> <span class="token punctuation">(</span>int<span class="token punctuation">)</span><span class="token function">preg_replace</span><span class="token punctuation">(</span><span class="token double-quoted-string string">"/<span class="token interpolation"><span class="token variable">$int</span></span>/"</span><span class="token punctuation">,</span> <span class="token single-quoted-string string">''</span><span class="token punctuation">,</span> <span class="token variable">$star</span><span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token keyword">return</span> <span class="token variable">$star</span><span class="token punctuation">;</span>
    <span class="token punctuation">}</span>

    <span class="token comment">/**
     * convert rate string to number
     * @param $str
     * @return mixed
     */</span>
    <span class="token keyword">public</span> <span class="token keyword">static</span> <span class="token keyword">function</span> <span class="token function">getAvgReviewFromString</span><span class="token punctuation">(</span><span class="token variable">$str</span><span class="token punctuation">)</span>
    <span class="token punctuation">{</span>
        <span class="token keyword">try</span> <span class="token punctuation">{</span>
            <span class="token variable">$str</span> <span class="token operator">=</span> <span class="token function">explode</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'うち'</span><span class="token punctuation">,</span> <span class="token variable">$str</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
            <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span><span class="token variable">$str</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
                Log<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">error</span><span class="token punctuation">(</span><span class="token double-quoted-string string">"cannot get rate 1"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
                <span class="token keyword">return</span> <span class="token number">0</span><span class="token punctuation">;</span>
            <span class="token punctuation">}</span>

            <span class="token variable">$matches</span> <span class="token operator">=</span> <span class="token function">array_map</span><span class="token punctuation">(</span><span class="token single-quoted-string string">'floatval'</span><span class="token punctuation">,</span> <span class="token variable">$str</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
            <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token function">empty</span><span class="token punctuation">(</span><span class="token variable">$matches</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
                <span class="token keyword">return</span> <span class="token number">0</span><span class="token punctuation">;</span>
            <span class="token punctuation">}</span>

            <span class="token variable">$matches</span> <span class="token operator">=</span> <span class="token function">array_filter</span><span class="token punctuation">(</span><span class="token variable">$matches</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
            <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token function">empty</span><span class="token punctuation">(</span><span class="token variable">$matches</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
                <span class="token keyword">return</span> <span class="token number">0</span><span class="token punctuation">;</span>
            <span class="token punctuation">}</span>

            <span class="token keyword">return</span> <span class="token function">min</span><span class="token punctuation">(</span><span class="token variable">$matches</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token punctuation">}</span> <span class="token keyword">catch</span> <span class="token punctuation">(</span><span class="token class-name">Exception</span> <span class="token variable">$exception</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
            <span class="token function">report</span><span class="token punctuation">(</span><span class="token variable">$exception</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token punctuation">}</span>

        Log<span class="token punctuation">:</span><span class="token punctuation">:</span><span class="token function">error</span><span class="token punctuation">(</span><span class="token double-quoted-string string">"cannot get rate 2"</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
        <span class="token keyword">return</span> <span class="token number">0</span><span class="token punctuation">;</span>
    <span class="token punctuation">}</span>

    <span class="token comment">/**
     * get category and product ranking
     * @param $str
     * @return string
     */</span>
    <span class="token keyword">public</span> <span class="token keyword">function</span> <span class="token function">getCatAndRank</span><span class="token punctuation">(</span><span class="token variable">$str</span><span class="token punctuation">)</span>
    <span class="token punctuation">{</span>
        <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token operator">!</span><span class="token variable">$str</span><span class="token punctuation">)</span> <span class="token punctuation">{</span>
            <span class="token keyword">return</span> <span class="token variable">$str</span><span class="token punctuation">;</span>
        <span class="token punctuation">}</span>

        <span class="token variable">$str</span> <span class="token operator">=</span> <span class="token function">preg_replace</span><span class="token punctuation">(</span>'<span class="token shell-comment comment">#(&lt;a.*?&gt;</span></span>).*?(<span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>a</span><span class="token punctuation">&gt;</span></span>)#m', '$1$2', $str);
        $str = preg_replace('#(<span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>ul.*?</span><span class="token punctuation">&gt;</span></span>).*?(<span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>ul</span><span class="token punctuation">&gt;</span></span>)#m', '$1$2', $str);
        $str = preg_replace('#(<span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>b.*?</span><span class="token punctuation">&gt;</span></span>).*?(<span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>b</span><span class="token punctuation">&gt;</span></span>)#m', '$1$2', $str);
        $str = preg_replace('#(<span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>tr.*?</span><span class="token punctuation">&gt;</span></span>).*?(<span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>tr</span><span class="token punctuation">&gt;</span></span>)#m', '$1$2', $str);
        $str = preg_replace('#(<span class="token tag"><span class="token tag"><span class="token punctuation">&lt;</span>style.*?</span><span class="token punctuation">&gt;</span></span><span class="token style language-css"><span class="token punctuation">)</span>.*?<span class="token punctuation">(</span></span><span class="token tag"><span class="token tag"><span class="token punctuation">&lt;/</span>style</span><span class="token punctuation">&gt;</span></span>)#m', '$1$2', $str);
        $str = trim(strip_tags($str));
        $str = str_replace('()', '', $str);

        return trim($str);
    }

    /**
     * get product ranking from string
     * @param $str
     * @return int
     */
    public function getRankingFromString($str)
    {
        if (!$str) {
            return $str;
        }

        $ranking = explode('-', $str);

        if (!isset($ranking[1])) {
            return 0;
        }

        return $this-&gt;getNumberFromString($ranking[1]);
    }

    /**
     * get category from string
     * @param $str
     * @return int|string
     */
    public function getCatFromString($str)
    {
        if (!$str) {
            return '未定';
        }

        $ranking = explode('-', $str);

        if (!isset($ranking[0])) {
            return '未定';
        }

        return trim($ranking[0]);
    }

    /**
     * get number from string
     * @param $str
     * @return int
     */
    public function getNumberFromString($str)
    {
        if (!$str) {
            return 0;
        }

        return (int)filter_var($str, FILTER_SANITIZE_NUMBER_INT);
    }
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

<?php

namespace AppJobs;

use AppHelpersAwsClient;

use AppModelsCategory;

use AppModelsProduct;

use AppModelsProductDetail;

use AppModelsProductStar;

use CarbonCarbon;

use IlluminateBusQueueable;

use IlluminateContractsQueueShouldQueue;

use IlluminateFoundationBusDispatchable;

use IlluminateQueueInteractsWithQueue;

use IlluminateQueueSerializesModels;

use IlluminateSupportFacadesLog;

class AwsCrawlerDetail implements ShouldQueue

{

protected $seller_id;

protected $detailProductUrl;

protected $asin;

/**

* Create a new job instance.

* @param $sellerId

* @param $detailProductUrl

* @param $asin

* @return void

*/

public function __construct($sellerId, $detailProductUrl, $asin = null)

{

$this->seller_id = $sellerId;

$this->detailProductUrl = $detailProductUrl;

$this->asin = $asin;

}

/**

* Execute the job.

* @return void

*/

{

Log::debug("Start crawl product detail, url = " . $this->detailProductUrl);

if ($this->checkAlreadyCrawl()) {

return;

}

try {

$html = AwsClient::getContent($this->detailProductUrl);

return;

}

if (!$html) {

return;

}

$arrProduct = $this->getProductData($html);

$productDetail = $this->getProductDetail($html);

$productStar = $this->getProductReviewStartDetail($html, $productDetail['review_count']);

if (!empty($arrProduct['asin'])) {

$product = Product::saveProduct($arrProduct, $this->asin);

ProductDetail::saveProductDetail($product, $productDetail);

ProductStar::saveProductStar($product, $productStar);

Category::saveCategory($product, ['name' => $productDetail['category']]);

}

Log::error($exception->getMessage());

}

return;

}

public function checkAlreadyCrawl()

{

if ($this->asin) {

$product = Product::where('asin', $this->asin)->first();

} else {

$product = Product::where('detail_aws_url', $this->detailProductUrl)->first();

}

if (!$product) {

return false;

}

return ProductDetail::where('product_id', $product->id)

->whereDate('created_at', Carbon::now()->format('Y-m-d'))

->first();

}

/**

* get product data from html dom

* @param $html

* @return array

*/

public function getProductData($html)

{

$asin = $html->find('#cerberus-data-metrics', 0);

if ($asin) {

$asin = $asin->getAllAttributes();

}

$img = $html->find('#imgTagWrapperId img', 0);

if ($img) {

$img = $img->getAttribute('data-old-hires');

}

$sellAt = $html->find('.date-first-available .value', 0)->plaintext ?? null;

if (!$sellAt) {

$sellAt = $html->find('#productDetailsTable ul li', 4)->plaintext ?? null;

$sellAt = str_replace('Amazon.co.jp での取り扱い開始日:', '', $sellAt);

}

return [

'name' => $html->find('#productTitle', 0)->plaintext ?? null,

'url_img' => $img,

'asin' => $asin['data-asin'] ?? null,

'seller_id' => $this->seller_id,

'sell_at' => trim($sellAt),

'detail_aws_url' => $this->detailProductUrl,

];

}

/**

* get product detail from html dom

* @param $html

* @return array

*/

public function getProductDetail($html)

{

}

$avgReview = $html->find('#acrPopover', 0)->title ?? 0;

$acrCustomerReviewText = $html->find('#acrCustomerReviewText', 0)->plaintext ?? 0;

$ranking = $html->find('#SalesRank .value', 0)->innertext ?? 0;

if (!$ranking) {

$ranking = $html->find('#SalesRank', 0)->innertext ?? 0;

}

$ranking = $html->find('.pdTab', 1)->innertext ?? 0;

}

$ranking = $this->getCatAndRank($ranking);

return [

'price' => str_replace(',', '', $asin['data-asin-price']) ?? 0,

'currency_code' => $asin['data-asin-currency-code'] ?? 'JPY',

'avg_review' => $this->getAvgReviewFromString($avgReview),

'review_count' => $this->getNumberFromString($acrCustomerReviewText),

'ranking' => $this->getRankingFromString($ranking),

'category' => $this->getCatFromString($ranking),

];

}

/**

* get start count for product review count

* @param $html

* @param $total

* @return mixed

*/

public function getProductReviewStartDetail($html, $total)

{

$arr['total_star'] = $total;

for ($i = 1; $i <= 5; $i++) {

$star = $this->getAStart($html, $i); // percent

$star = ($star * $total) / 100;

$arr["star_$i"] = (int)round($star);

}

return $arr;

}

/**

* get data a star

* @param $html

* @param $int

* @return int

*/

public function getAStart($html, $int)

{

$star = $html->find("#histogramTable .{$int}star", 0);

if (!$star) {

return 0;

}

$star = $star->getAttribute('aria-label');

$star = $this->getNumberFromString($star);

$star = (int)preg_replace("/$int/", '', $star, 1);

return $star;

}

/**

* convert rate string to number

* @param $str

* @return mixed

*/

public static function getAvgReviewFromString($str)

{

try {

$str = explode('うち', $str);

if (!$str) {

return 0;

}

$matches = array_map('floatval', $str);

if (empty($matches)) {

return 0;

}

$matches = array_filter($matches);

return 0;

}

return min($matches);

}

return 0;

}

/**

* get category and product ranking

* @param $str

* @return string

*/

public function getCatAndRank($str)

{

return $str;

}

$str = preg_replace('#(<a.*?>).*?(</a>)#m', '$1$2', $str);

$str = preg_replace('#(<ul.*?>).*?(</ul>)#m', '$1$2', $str);

$str = preg_replace('#(<b.*?>).*?(</b>)#m', '$1$2', $str);

$str = preg_replace('#(<tr.*?>).*?(</tr>)#m', '$1$2', $str);

$str = preg_replace('#(<style.*?>).*?(</style>)#m', '$1$2', $str);

$str = trim(strip_tags($str));

$str = str_replace('()', '', $str);

return trim($str);

}

/**

* get product ranking from string

* @param $str

* @return int

public function getRankingFromString($str)

{

if (!$str) {

return $str;

}

$ranking = explode('-', $str);

if (!isset($ranking[1])) {

return 0;

}

return $this->getNumberFromString($ranking[1]);

}

/**

* get category from string

* @param $str

* @return int|string

public function getCatFromString($str)

{

if (!$str) {

return '未定';

}

$ranking = explode('-', $str);

if (!isset($ranking[0])) {

return '未定';

}

return trim($ranking[0]);

}

/**

* get number from string

* @param $str

* @return int

public function getNumberFromString($str)

{

if (!$str) {

return 0;

}

return (int)filter_var($str, FILTER_SANITIZE_NUMBER_INT);

}

Để chạy crawl này bạn chạy lệnh sau:

php artisan aws:product

1 2	php artisan aws:product

Proxy

Bạn để ý trong function getContent mình có để 1 tham số nữa là proxy, bạn có thể truyền proxy vào theo dạng http:192.162.1.15:8080 để ẩn ip server hiện tại của bạn đi.
proxy này kiếm ở đây nhé: https://hidemy.name/en/, mình đã mua code của trang này để xử dụng, các bạn có thể mua hoặc kiếm proxy từ một nguồn khác cũng được.

Chia sẻ bài viết ngay

Nguồn bài viết : Viblo

Thu thập dữ liệu bằng Laravel, proxy và HTML Dom đơn giản

Cài đặt

Code

Proxy

TikTok trở thành nền tảng mảng xã hội lớn thứ hai ở Nam Phi

Mất giá nhanh nhất sau 9 tháng ra mắt, iPhone 14 Pro Max tiếp tục phá đáy tại Việt Nam

Hướng dẫn sử dụng ngôn ngữ R cho người mới bắt đầu

10 plugin cần thiết của SublimeText dành cho các lập trình viên JavaScript