Skip to main content

Logging

Introduction​

Logging Infrastructure​

Grafana Loki is a log aggregation system, Prometheus for logs. Loki differs from Prometheus by focusing on logs instead of metrics, and collecting logs via push, instead of a pull mechanism.

Promtail is an agent used to ship the contents of local logs to a Grafana Loki instance. It is deployed to every machine that runs applications which needs to ship logs to Loki.

alt text

How to send Logs to infra-loki Datasource​

The infra-loki Datasource is the self-hosted Loki instance managed by the infrastructure team. infra-loki has a 30 days retention period.
As earlier stated, you need Promtail to forward logs to Loki. The Promtail ansible-role can help reduce the complexity of setting up Promtail.

The Promtail ansible-role runs Promtail as a Docker container.

Create an Ansible playbook:​

Create a ./playbooks/run-promtail.yml Ansible playbook.

- name: Start Promtail
  hosts: all
  gather_facts: false
  become: true
  roles:
    - role: chainsafe.general.promtail

Set Ansible variables:​

Promtail ansible-role requires variables to run, get the Loki credentials from the infra team.

###### Promtail vars #####
loki_push_url: "https://loki.chainsafe.dev/loki/api/v1/push"
loki_basic_auth_user: "{{ vault_metrics_auth_user }}"
loki_basic_auth_password: "{{ vault_metrics_auth_password }}"

Set Docker labels:​

The Promtail config automatically supports log collection from every container with the promtail_logging: "true" Docker label. To make your log filtering experience seamless, please set the following labels, (more is good).

docker_labels:
  promtail_logging: "true"
  instance: "{{inventory_hostname}}"

Run Ansible playbook:​

Run the run-promtail.yml playbook:

$ ansible-playbook  ./playbooks/run-promtail.yml -l <your_hostname_or_group>

View Logs:​

With Promtail sending logs to Loki, logs will be visible at Grafana cloud. alt text