Create crawling application bing with Golang, MySQL driver

Tram Ho

Hi everybody,

We meet each other once in a while, to continue the series to share knowledge about tech, today I will learn and share about a hot language in the IT community that is Golang .

As everyone knows Golang is a language designed and developed by Google. It is expected to help the software industry exploit the multi-core platform of the processor and multitask better. Therefore, it has extremely fast processing speed and is preferred in large projects that require high processing speed in the 4.0 technology trend.

Today I have made a small example of bing’s clone data to see how Golang works and how difficult it is, invite everyone to read and practice.

I. Setup golang

1.1 Download golang

Open up a terminal and download golang with the following command:

This command will download the golang installation file as tar.gz and install it.

1.2. Add PATH

Adding the PATH variable for golang is so that when we type the go command, golang will automatically run golang Open .bashrc file with the command:

and add to the bottom of the PATH export command file:

Then run the command below to reload bashrc

Thus my PATH variable was successfully added.

1.3 Inspection

To check if the installation is successful or not? And what version is golang using? You run the following command:

After running the test, you should see your version

So, the first step is to install golang , then I will initialize the project and install the package

You can visit golang ‘s golang to see details on how to install:

II. Install package

Create your project directory anywhere and run the following commands to install the package:

In the scope of this project, I install 2 packages: gocolly/colly , go-sql-driver/mysql

  • colly : you can easily extract structured data from web pages, which can be used for a variety of applications, like data mining, data processing or storage. Link github
  • mysql : The mysql driver for convenient and easy DB manipulation. This github 2 package link is quite hot and is used a lot in golang because it meets the needs of the user and is easy to use.

After the installation is complete, in your project directory will create a file go.mod . This file is similar to PHP’s composer or nodejs package.json. It is used to store package names, version used in the project.

Ok, finished setting, install package, I will start crawling data from bing.

III. Starting craw

3.1 Objectives

Scan data from Bing url: and save title , link , description in the database.

3.2 Craw bing data

Because golang has a working mechanism that will start from main() function in a main run file, I will create main.go file to be the main run file for project and main() function.

In the above code, I used package colly to crawl data, and already explained in each line, it’s really not too difficult, if you want to see more details about data extraction please refer to the doc at the page. master colly:

Run the command to execute:

And here are the crawl results:

Run craw bing data

IV. Initialize DB

After getting data from bing, I will initialize the database to manipulate the DB.

Here I have modified the import, added 3 packages and 1 function to initialize database connection. So, the initialization of the database connection, I see quite similar to nodeJs, just a little different in error handling, variable declaration.

V. Insert database

Next, I will insert the crawled data into the database.

Execute the main.go file main.go to check the results:

Craw and insert into DB

Check in the database:

Database result


BECAUSE. Conclude

So I have completed the crawling and inserting process into the database and then, with my limited knowledge and what I have learned, this is my first share about Golang .

Hopefully this article will help you understand and know a little more about craw data and insert data with golang .

Personally, I think golang executes really fast, writing style is not too fussy, and creating a project is also very fastttt. Let’s get started and try the code!

Thank you!


Share the news now

Source : Viblo