ELK stack
bao gồm Elasticsearch
, Logstash
và Kibana
, trong đó Logstash
thu thập logs trong các ứng dụng của bạn đưa về lưu trữ trong Elasticsearch
và Kibana
sẽ trình diễn chúng trên một giao diện thân thiện với bạn hơn. Ngoài ra để thu thập logs cho Kuberbernettes
thì mình sử dụng thêm Fluentd
.
I. Thu thập logs từ Docker
Điều kiện cần
Để thuận tiện cho Logstash có thể thu thập logs từ các Docker Container, đầu tiên hãy sửa logging driver của Docker nằm ở /etc/docker/daemon.json
về sử dụng syslog
:
1 2 3 4 5 6 7 | <span class="token punctuation">{</span> <span class="token property">"log-driver"</span><span class="token operator">:</span> <span class="token string">"syslog"</span><span class="token punctuation">,</span> <span class="token property">"log-opts"</span><span class="token operator">:</span> <span class="token punctuation">{</span> <span class="token property">"syslog-address"</span><span class="token operator">:</span> <span class="token string">"tcp://127.0.0.1:9600"</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span> |
Trong đó tcp://127.0.0.1:9600 tương ứng là giao thức TCP để truyền tin đến Logstash được đặt ở IP 127.0.0.1 và port là 9600.
Tìm hiểu thêm ở: Configure logging drivers
Triển khai ELK stack bằng Docker Stack
Docker Stack file và config của ELK mình sử dụng như bên dưới:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 | <span class="token key atrule">version</span><span class="token punctuation">:</span> <span class="token string">'3.3'</span> <span class="token key atrule">services</span><span class="token punctuation">:</span> <span class="token key atrule">elasticsearch</span><span class="token punctuation">:</span> image<span class="token punctuation">:</span> docker.elastic.co/elasticsearch/elasticsearch<span class="token punctuation">:</span>7.4.1 ports<span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token string">"9200:9200"</span> <span class="token punctuation">-</span> <span class="token string">"9300:9300"</span> configs<span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">source</span><span class="token punctuation">:</span> elastic_config target<span class="token punctuation">:</span> /usr/share/elasticsearch/config/elasticsearch.yml volumes<span class="token punctuation">:</span> <span class="token punctuation">-</span> /u01/elasticsearch/data<span class="token punctuation">:</span>/usr/share/elasticsearch/data environment<span class="token punctuation">:</span> ES_JAVA_OPTS<span class="token punctuation">:</span> <span class="token string">"-Xms512m -Xmx512m"</span> ELASTIC_PASSWORD<span class="token punctuation">:</span> changeme ES_HEAP_SIZE<span class="token punctuation">:</span> 1g MAY_LOCKED_MEMORY<span class="token punctuation">:</span> unlimited node.max_local_storage_nodes<span class="token punctuation">:</span> <span class="token number">20</span> networks<span class="token punctuation">:</span> <span class="token punctuation">-</span> elk deploy<span class="token punctuation">:</span> mode<span class="token punctuation">:</span> replicated replicas<span class="token punctuation">:</span> <span class="token number">1</span> <span class="token key atrule">logstash</span><span class="token punctuation">:</span> <span class="token key atrule">image</span><span class="token punctuation">:</span> docker.elastic.co/logstash/logstash<span class="token punctuation">:</span>7.4.1 <span class="token key atrule">ports</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token string">"5000:5000"</span> <span class="token punctuation">-</span> <span class="token string">"9600:9600"</span> <span class="token key atrule">configs</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">source</span><span class="token punctuation">:</span> logstash_config <span class="token key atrule">target</span><span class="token punctuation">:</span> /usr/share/logstash/config/logstash.yml <span class="token punctuation">-</span> <span class="token key atrule">source</span><span class="token punctuation">:</span> logstash_pipeline <span class="token key atrule">target</span><span class="token punctuation">:</span> /usr/share/logstash/pipeline/logstash.conf <span class="token key atrule">environment</span><span class="token punctuation">:</span> <span class="token key atrule">LS_JAVA_OPTS</span><span class="token punctuation">:</span> <span class="token string">"-Xmx256m -Xms256m"</span> <span class="token key atrule">networks</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> elk <span class="token key atrule">deploy</span><span class="token punctuation">:</span> <span class="token key atrule">mode</span><span class="token punctuation">:</span> replicated <span class="token key atrule">replicas</span><span class="token punctuation">:</span> <span class="token number">1</span> <span class="token key atrule">depends_on</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> elasticsearch <span class="token key atrule">kibana</span><span class="token punctuation">:</span> <span class="token key atrule">image</span><span class="token punctuation">:</span> docker.elastic.co/kibana/kibana<span class="token punctuation">:</span>7.4.1 <span class="token key atrule">ports</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token string">"8601:5601"</span> <span class="token key atrule">configs</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">source</span><span class="token punctuation">:</span> kibana_config <span class="token key atrule">target</span><span class="token punctuation">:</span> /usr/share/kibana/config/kibana.yml <span class="token key atrule">networks</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> elk <span class="token key atrule">deploy</span><span class="token punctuation">:</span> <span class="token key atrule">mode</span><span class="token punctuation">:</span> replicated <span class="token key atrule">replicas</span><span class="token punctuation">:</span> <span class="token number">1</span> <span class="token key atrule">depends_on</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> elasticsearch <span class="token key atrule">configs</span><span class="token punctuation">:</span> <span class="token key atrule">elastic_config</span><span class="token punctuation">:</span> <span class="token key atrule">file</span><span class="token punctuation">:</span> ./elasticsearch/config/elasticsearch.yml <span class="token key atrule">logstash_config</span><span class="token punctuation">:</span> <span class="token key atrule">file</span><span class="token punctuation">:</span> ./logstash/config/logstash.yml <span class="token key atrule">logstash_pipeline</span><span class="token punctuation">:</span> <span class="token key atrule">file</span><span class="token punctuation">:</span> ./logstash/pipeline/logstash.conf <span class="token key atrule">kibana_config</span><span class="token punctuation">:</span> <span class="token key atrule">file</span><span class="token punctuation">:</span> ./kibana/config/kibana.yml <span class="token key atrule">networks</span><span class="token punctuation">:</span> <span class="token key atrule">elk</span><span class="token punctuation">:</span> <span class="token key atrule">driver</span><span class="token punctuation">:</span> overlay |
Chúng ta sẽ deploy thông qua Docker Stack bằng command.
1 2 3 4 | $ docker stack deploy --compose-file<span class="token operator">=</span>docker-stack.yaml elk // --compose-file chính là đường dẫn đến stack <span class="token function">file</span> // elk là tên của serivce. |
Lưu ý đối với containter Elasticsearch
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | <span class="token key atrule">version</span><span class="token punctuation">:</span> <span class="token string">'3.3'</span> <span class="token key atrule">services</span><span class="token punctuation">:</span> elasticsearch<span class="token punctuation">:</span> image<span class="token punctuation">:</span> docker.elastic.co/elasticsearch/elasticsearch<span class="token punctuation">:</span>7.4.1 ports<span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token string">"9200:9200"</span> <span class="token punctuation">-</span> <span class="token string">"9300:9300"</span> configs<span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">source</span><span class="token punctuation">:</span> elastic_config target<span class="token punctuation">:</span> /usr/share/elasticsearch/config/elasticsearch.yml volumes<span class="token punctuation">:</span> <span class="token punctuation">-</span> /u01/elasticsearch/data<span class="token punctuation">:</span>/usr/share/elasticsearch/data environment<span class="token punctuation">:</span> ES_JAVA_OPTS<span class="token punctuation">:</span> <span class="token string">"-Xms512m -Xmx512m"</span> ELASTIC_PASSWORD<span class="token punctuation">:</span> changeme ES_HEAP_SIZE<span class="token punctuation">:</span> 1g MAY_LOCKED_MEMORY<span class="token punctuation">:</span> unlimited node.max_local_storage_nodes<span class="token punctuation">:</span> <span class="token number">20</span> networks<span class="token punctuation">:</span> <span class="token punctuation">-</span> elk deploy<span class="token punctuation">:</span> mode<span class="token punctuation">:</span> replicated replicas<span class="token punctuation">:</span> <span class="token number">1</span> |
- Container của Elasticsearch sẽ mount thư mục
/u01/elasticsearch/data
từ ngoài máy host vào trong container, thư mục này cần được cấp quyền cho elasticsearch sử dụng được nên cần phải sửa chown cho nó
1 2 | $ <span class="token function">sudo</span> <span class="token function">chown</span> -R 1000:1000 /u01/elasticsearch/data |
Elasticsearch
expose ra 2 cổng9200, 9300
và sẽ join vào Docker networkelk
để có thể giao tiếp với các container khác trong cùng network
Lưu ý với container Logstash
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | <span class="token punctuation">...</span> logstash<span class="token punctuation">:</span> image<span class="token punctuation">:</span> docker.elastic.co/logstash/logstash<span class="token punctuation">:</span>7.4.1 ports<span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token string">"5000:5000"</span> <span class="token punctuation">-</span> <span class="token string">"9600:9600"</span> configs<span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">source</span><span class="token punctuation">:</span> logstash_config target<span class="token punctuation">:</span> /usr/share/logstash/config/logstash.yml <span class="token punctuation">-</span> <span class="token key atrule">source</span><span class="token punctuation">:</span> logstash_pipeline target<span class="token punctuation">:</span> /usr/share/logstash/pipeline/logstash.conf environment<span class="token punctuation">:</span> LS_JAVA_OPTS<span class="token punctuation">:</span> <span class="token string">"-Xmx256m -Xms256m"</span> networks<span class="token punctuation">:</span> <span class="token punctuation">-</span> elk deploy<span class="token punctuation">:</span> mode<span class="token punctuation">:</span> replicated replicas<span class="token punctuation">:</span> <span class="token number">1</span> depends_on<span class="token punctuation">:</span> <span class="token punctuation">-</span> elasticsearch |
File config cho Logstash nằm trong thư mục logstash/
gồm config để giao tiếp với Elasticsearch
và pipeline
để cấu hình luồng logs input và output của Logstash.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | input <span class="token punctuation">{</span> syslog <span class="token punctuation">{</span> type =<span class="token punctuation">></span> syslog port =<span class="token punctuation">></span> 9600 <span class="token punctuation">}</span> <span class="token punctuation">}</span> Add your filters / logstash plugins configuration here filter <span class="token punctuation">{</span> grok <span class="token punctuation">{</span> match =<span class="token punctuation">></span> <span class="token punctuation">{</span> "message" =<span class="token punctuation">></span> "<span class="token punctuation">...</span>" // Thêm regex pattern match với log của bạn vào đây <span class="token punctuation">}</span> add_field =<span class="token punctuation">></span> <span class="token punctuation">{</span> "type" =<span class="token punctuation">></span> "message_full" <span class="token punctuation">}</span> <span class="token punctuation">}</span> if "_grokparsefailure_sysloginput" in <span class="token punctuation">[</span>tags<span class="token punctuation">]</span> <span class="token punctuation">{</span> drop <span class="token punctuation">{</span><span class="token punctuation">}</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span> output <span class="token punctuation">{</span> elasticsearch <span class="token punctuation">{</span> hosts =<span class="token punctuation">></span> "elasticsearch<span class="token punctuation">:</span>9200" user =<span class="token punctuation">></span> "elastic" password =<span class="token punctuation">></span> "changeme" <span class="token punctuation">}</span> <span class="token punctuation">}</span> |
Trong phần config pipeline có phần filter sử dụng grok
để lọc log gửi về từ Docker container
. Grok
sẽ tìm field tên là message
và thực hiện extract thông tin trong message
đó thành 1 object có giá trị, dễ hiểu để sử dụng ở trên Kibana
. Trong ví dụ, nếu bạn sử dụng Grok pattern
này
1 2 3 | <span class="token operator">/</span><span class="token operator">/</span> <span class="token variable">Pattern</span> <span class="token punctuation">(</span>?<span class="token operator"><</span><span class="token atom">time</span><span class="token operator">></span><span class="token punctuation">(</span><span class="token operator"><</span><span class="token comment">%{WORD}>%{MONTH} %{MONTHDAY} %{TIME} %{WORD}[%{WORD}])): %{TIMESTAMP_ISO8601:timestamp} | %{LOGLEVEL:log_level}.*| (?<detail>(%{WORD} | %{WORD} | %{WORD})) | (?<container>(.+?)) | (?<project_name>(@%{WORD}.%{WORD}@)) |(?<correlation_id>(.+?))|(?<client_ip>(.+?))| (?<msg>([.*].*))</span> |
1 2 3 | // Message log chạy qua Logstash <30>Nov 19 11:35:32 c3e4e76fc8bb[31175]: 2019-11-19T04:35:32.224Z | INFO | VDS | AppLog | Java | 3324.container-0-C-1 | @<a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="90e0e2fffaf5f3e4befef1fdf5d0">[email protected]</a> | 111112vwsdf | 129.0.0.1 | [Consumer clientId=consumer-2, groupId=anonymous.2ff2fs2d-965a-4b1e-a873-ea27d6ce9fc9] Group coordinator 10.0.244.255:9000 (id: 1111111 rack: null) is unavailable or invalid, will attempt rediscovery |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | // Output nhận được sẽ là <span class="token punctuation">{</span> <span class="token property">"time"</span><span class="token operator">:</span> <span class="token punctuation">[</span> <span class="token string">"<30>Nov 19 11:35:32 c3e4e76fc8bb[31175]"</span> <span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token property">"timestamp"</span><span class="token operator">:</span> <span class="token punctuation">[</span> <span class="token string">"2019-11-19T04:35:32.224Z"</span> <span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token property">"log_level"</span><span class="token operator">:</span> <span class="token punctuation">[</span> <span class="token string">"INFO"</span> <span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token property">"detail"</span><span class="token operator">:</span> <span class="token punctuation">[</span> <span class="token string">"VDS | AppLog | Java"</span> <span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token property">"container"</span><span class="token operator">:</span> <span class="token punctuation">[</span> <span class="token string">"3324.container-0-C-1"</span> <span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token property">"project_name"</span><span class="token operator">:</span> <span class="token punctuation">[</span> <span class="token string">"@<a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="2e5e5c41444b4d5a00404f434b6e">[email protected]</a>"</span> <span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token property">"correlation_id"</span><span class="token operator">:</span> <span class="token punctuation">[</span> <span class="token string">" 111112vwsdf "</span> <span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token property">"client_ip"</span><span class="token operator">:</span> <span class="token punctuation">[</span> <span class="token string">" 129.0.0.1 "</span> <span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token property">"msg"</span><span class="token operator">:</span> <span class="token punctuation">[</span> <span class="token string">"[Consumer clientId=consumer-2, groupId=anonymous.c75c47ea-965a-4b1e-a873-ea27d6ce9fc9] Group coordinator 10.0.244.255:9000 (id: 1111111 rack: null) is unavailable or invalid, will attempt rediscovery "</span> <span class="token punctuation">]</span> <span class="token punctuation">}</span> |
Tham khảo thêm Grok pattern debugger tại đây .
Kibana
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | <span class="token punctuation">...</span> <span class="token key atrule">kibana</span><span class="token punctuation">:</span> image<span class="token punctuation">:</span> docker.elastic.co/kibana/kibana<span class="token punctuation">:</span>7.4.1 ports<span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token string">"8601:5601"</span> configs<span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">source</span><span class="token punctuation">:</span> kibana_config target<span class="token punctuation">:</span> /usr/share/kibana/config/kibana.yml networks<span class="token punctuation">:</span> <span class="token punctuation">-</span> elk deploy<span class="token punctuation">:</span> mode<span class="token punctuation">:</span> replicated replicas<span class="token punctuation">:</span> <span class="token number">1</span> depends_on<span class="token punctuation">:</span> <span class="token punctuation">-</span> elasticsearch |
Kibana
sẽ expose cổng 5601
ra cổng 8601
trên máy host để có thể truy cập tới được.
Sử dụng Kibana
B1: User đăng nhập mặc định là elastic, mật khẩu là changeme . tại địa chỉ http://127.0.0.1:8601/
B2: Tạo
index pattern
để hiển thị log từLogstash
Điềnlogstash-*
vào ô Index pattern -> Next stepB3: Chọn @timestamp -> Create index pattern
B4: Xem logs được đổ về ở http://127.0.0.1:8601/app/kibana#/discover?_g=()
Ở đây bạn chọn vào nút +Add filter sau đó điền các giá trị như trong hình bên dưới để lọc logs theo tên project. Giá trị Value này là 1 Regex pattern match với các message có chứa thông tin @project.name@.
II. Thu thập logs từ trong Kubernetes.
Sử dụng Fluentd.
Fluentd
là một ứng dụng hoàn toàn miễn phí và open-source giúp bạn có thể lấy được log ở web server và xử lý log data đó. Dữ liệu log lấy đươc rất có thể từ nginx, apache hoặc đơn giản là từ các gói tin tcp hay các message của http. Để thu thập logs từ Kubernettes mình sẽ triển khai Fluentd
bằng object là DaemonSet
.
DaemonSet
thường được sử dụng để triển khai 1 service trên tất cả các Pod
Tạo namespace kube-logging
1 2 3 4 5 | <span class="token key atrule">kind</span><span class="token punctuation">:</span> Namespace <span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> v1 <span class="token key atrule">metadata</span><span class="token punctuation">:</span> name<span class="token punctuation">:</span> kube<span class="token punctuation">-</span>logging |
tạo namespace bằng lệnh
1 2 | $ kubectl apply -f kube-logging.yaml |
Tạo Daemonset Fluentd
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 | <span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> v1 <span class="token key atrule">kind</span><span class="token punctuation">:</span> ServiceAccount <span class="token key atrule">metadata</span><span class="token punctuation">:</span> name<span class="token punctuation">:</span> fluentd namespace<span class="token punctuation">:</span> kube<span class="token punctuation">-</span>logging labels<span class="token punctuation">:</span> app<span class="token punctuation">:</span> fluentd <span class="token punctuation">---</span> <span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> rbac.authorization.k8s.io/v1 <span class="token key atrule">kind</span><span class="token punctuation">:</span> ClusterRole <span class="token key atrule">metadata</span><span class="token punctuation">:</span> name<span class="token punctuation">:</span> fluentd labels<span class="token punctuation">:</span> app<span class="token punctuation">:</span> fluentd <span class="token key atrule">rules</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">apiGroups</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token string">""</span> resources<span class="token punctuation">:</span> <span class="token punctuation">-</span> pods <span class="token punctuation">-</span> namespaces verbs<span class="token punctuation">:</span> <span class="token punctuation">-</span> get <span class="token punctuation">-</span> list <span class="token punctuation">-</span> watch <span class="token punctuation">---</span> <span class="token key atrule">kind</span><span class="token punctuation">:</span> ClusterRoleBinding <span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> rbac.authorization.k8s.io/v1 <span class="token key atrule">metadata</span><span class="token punctuation">:</span> name<span class="token punctuation">:</span> fluentd <span class="token key atrule">roleRef</span><span class="token punctuation">:</span> kind<span class="token punctuation">:</span> ClusterRole name<span class="token punctuation">:</span> fluentd apiGroup<span class="token punctuation">:</span> rbac.authorization.k8s.io <span class="token key atrule">subjects</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">kind</span><span class="token punctuation">:</span> ServiceAccount name<span class="token punctuation">:</span> fluentd namespace<span class="token punctuation">:</span> kube<span class="token punctuation">-</span>logging <span class="token punctuation">---</span> <span class="token key atrule">apiVersion</span><span class="token punctuation">:</span> apps/v1 <span class="token key atrule">kind</span><span class="token punctuation">:</span> DaemonSet <span class="token key atrule">metadata</span><span class="token punctuation">:</span> name<span class="token punctuation">:</span> fluentd namespace<span class="token punctuation">:</span> kube<span class="token punctuation">-</span>logging labels<span class="token punctuation">:</span> app<span class="token punctuation">:</span> fluentd <span class="token key atrule">spec</span><span class="token punctuation">:</span> selector<span class="token punctuation">:</span> matchLabels<span class="token punctuation">:</span> app<span class="token punctuation">:</span> fluentd template<span class="token punctuation">:</span> metadata<span class="token punctuation">:</span> labels<span class="token punctuation">:</span> app<span class="token punctuation">:</span> fluentd spec<span class="token punctuation">:</span> serviceAccount<span class="token punctuation">:</span> fluentd serviceAccountName<span class="token punctuation">:</span> fluentd tolerations<span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">key</span><span class="token punctuation">:</span> node<span class="token punctuation">-</span>role.kubernetes.io/master effect<span class="token punctuation">:</span> NoSchedule containers<span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> fluentd image<span class="token punctuation">:</span> fluent/fluentd<span class="token punctuation">-</span>kubernetes<span class="token punctuation">-</span>daemonset<span class="token punctuation">:</span>v1.4.2<span class="token punctuation">-</span>debian<span class="token punctuation">-</span>elasticsearch<span class="token punctuation">-</span><span class="token number">1.1</span> env<span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> FLUENT_ELASTICSEARCH_HOST value<span class="token punctuation">:</span> "IP<span class="token punctuation">...</span>" //Public IP của Elasticsearch đã cài ở trên <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> FLUENT_ELASTICSEARCH_PORT value<span class="token punctuation">:</span> <span class="token string">"9200"</span> <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> FLUENT_ELASTICSEARCH_USER value<span class="token punctuation">:</span> <span class="token string">"elastic"</span> <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> FLUENT_ELASTICSEARCH_PASSWORD value<span class="token punctuation">:</span> <span class="token string">"changeme"</span> <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> FLUENT_ELASTICSEARCH_SCHEME value<span class="token punctuation">:</span> <span class="token string">"http"</span> <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> FLUENTD_SYSTEMD_CONF value<span class="token punctuation">:</span> disable resources<span class="token punctuation">:</span> limits<span class="token punctuation">:</span> memory<span class="token punctuation">:</span> 512Mi requests<span class="token punctuation">:</span> cpu<span class="token punctuation">:</span> 100m memory<span class="token punctuation">:</span> 200Mi volumeMounts<span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> varlog mountPath<span class="token punctuation">:</span> /var/log <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> varlibdockercontainers mountPath<span class="token punctuation">:</span> /var/lib/docker/containers readOnly<span class="token punctuation">:</span> <span class="token boolean important">true</span> terminationGracePeriodSeconds<span class="token punctuation">:</span> <span class="token number">30</span> volumes<span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> varlog hostPath<span class="token punctuation">:</span> path<span class="token punctuation">:</span> /var/log <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> varlibdockercontainers hostPath<span class="token punctuation">:</span> path<span class="token punctuation">:</span> /var/lib/docker/containers |
Thực hiện tạo DaemonSet bằng command
1 2 | $ kubectl apply -f fluentd.yaml |
Trong file cấu hình của fluentd có đoạn cấu hình:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | <span class="token key atrule">containers</span><span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> fluentd image<span class="token punctuation">:</span> fluent/fluentd<span class="token punctuation">-</span>kubernetes<span class="token punctuation">-</span>daemonset<span class="token punctuation">:</span>v1.4.2<span class="token punctuation">-</span>debian<span class="token punctuation">-</span>elasticsearch<span class="token punctuation">-</span><span class="token number">1.1</span> env<span class="token punctuation">:</span> <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> FLUENT_ELASTICSEARCH_HOST value<span class="token punctuation">:</span> "IP<span class="token punctuation">...</span>" //Public IP của Elasticsearch đã cài ở trên <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> FLUENT_ELASTICSEARCH_PORT value<span class="token punctuation">:</span> <span class="token string">"9200"</span> <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> FLUENT_ELASTICSEARCH_USER value<span class="token punctuation">:</span> <span class="token string">"elastic"</span> <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> FLUENT_ELASTICSEARCH_PASSWORD value<span class="token punctuation">:</span> <span class="token string">"changeme"</span> <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> FLUENT_ELASTICSEARCH_SCHEME value<span class="token punctuation">:</span> <span class="token string">"http"</span> <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> FLUENTD_SYSTEMD_CONF value<span class="token punctuation">:</span> disable |
Fluentd sẽ gửi logs về cho Elasticsearch trên server IP thông qua port 9200 được config ở Docker stack bên trên.
Logs sẽ được hiển thị bên trong http://127.0.0.1:8601/
Tạm kết
Trên đây mình vừa đưa ra cấu hình cho ELK và Fluentd để thu thập logs từ Docker và Kubernettes. Tuy nhiên bộ config trên chưa thực sự tối ưu trên môi trường production khi số lượng container, pod đồng thời gửi logs về hệ thống Logs tập trung này nhiều lên, trong bài viết tới, mình sẽ chỉ ra những điểm cần thay đổi, cấu hình cho khả năng scaling của hệ thống.