Implement centralized log storage with Graylog

Nowadays, when the application of Microservice architecture becomes more and more popular, we not only manage logs of one application on a server but now will be dozens, hundreds of different containers, logs lying garbage. on multiple servers. Debugging and monitoring the application becomes more difficult, so the build applications aim to focus the application’s logs on one place to become effective tools.

In this article, I will share how to build a Graylog application suite and how to configure containers to send logs to Graylog?

About Graylog

Graylog is considered a centralized log management application. Graylog can store every terabytes of logs every day, process and store it in memory. Graylog’s web interface allows you to search for records, quickly retrieve information, organize and display them in the form of maps, tables, …

Collecting Data: Graylog’s architecture allows input to be any type of data structure, including log messages and network traffic from:

• • Syslog (TCP, UDP, AMQP, Kafka)
• • GELF (TCP, UDP, AMQP, Kafka, HTTP)
• • AWS – AWS Logs, FlowLogs, CloudTrail
• • Beats / Logstash
• • CEF (TCP, UDP, AMQP, Kafka)
• • JSON Path from HTTP API
• • Netflow (UDP)
• • Plain / Raw Text (TCP, UDP, AMQP, Kafka) Organizing Data: Raw logs, raw messages are analyzed and enriched with information, making them useful for managers and downstream systems.

Implement Graylog with Docker

I will use Docker Compose to deploy the application. In production environment, we are not recommended to use Docker Compose , instead of Docker Swarm , it is also very easy to migrate the syntax of Docker Compose via Docker Swarm.

I have used 3 more folders used for the purpose of volumes of services, so I need to grant read-write permission to those folders first.

Then just run docker-compose up and wait for the application to pop up, you just need to visit the address: http: // localhost: 9000 to use the web interface. You log in with your account / password as admin / admin .

So I turned on the application, until the next step only.

Configure to send logs to Graylog.

Configure Input for Graylog

You access here to create Input configuration for Graylog http: // localhost: 9000 / system / inputs , I will send logs from Syslog via TCP so I configure the following:

Click Save and wait for it to switch to the RUNNING state.

Configure on the application to send them to Graylog

Here I use Rsyslog to post logs to Graylog. Depending on your operating system, Rsyslog needs to be installed.

Configure the output of Rsyslog in /etc/rsyslog.conf :

Then you restart service rsyslog to get the new configuration

I will test by logging in / logout of the computer =)) and BOM

The log has been successfully submitted to Graylog.

Conclusion

Above are the basic configuration steps to build Graylog in order to collect logs from different sources to a place. A little more advanced, you can configure in Graylog filters to filter logs by each application, each log type, each different log level.