Gathering OpenShift master logs in 3.10 and later

OpenShift 3.10 introduced the move from Master API and Controller systemd services to containers. This move to containers changed the way in which logging is stored and gathered. A script (/usr/local/bin/master-logs) was introduced to aide in gather the logs. Below is an Ansible playbook to build on that script and help with quickly gathering logs from all OpenShift Masters at once.

Usage

Download the playbook

wget https://access.redhat.com/sites/default/files/attachments/master-loglevel.yml

Increase the log

ansible-playbook -i inventory master-loglevel.yml -t loglevel -e debug_loglevel=6

Do your test case

Wait the api and controller to be up and running again, before proceed.

Collect the logs

ansible-playbook -i inventory master-loglevel.yml -t collect

Logs are saved under /tmp/master-logs/ of your ansible host, for each master <hostname>.tar.gz.
It contains OpenShift Master API, Controllers and ETCD logs.

Restore log level

ansible-playbook -i inventory master-loglevel.yml -t loglevel -e debug_loglevel=2      

Source code

- hosts: masters
  serial: 1
  vars:
  - debug_loglevel: 2
  tasks:
  - name: Ensure log level to {{ debug_loglevel }}
    lineinfile:
      path: /etc/origin/master/master.env
      regexp: '^DEBUG_LOGLEVEL='
      line: "DEBUG_LOGLEVEL={{ debug_loglevel}}"
      state: present
    tags: loglevel

  - name: Restart masters
    shell: /usr/local/bin/master-restart {{ item }}
    failed_when: false
    with_items:
    - api
    - controllers
    tags: loglevel

  - block:
    - name: Clean up old logs
      file:
        path: /tmp/{{ item }}
        state: absent
      with_items:
      - api.log
      - controllers.log
      - etcd.log
      - '{{ ansible_hostname }}.tar.gz'

    - name: Clean up localhost
      file: 
        path: /tmp/master-logs/{{ ansible_hostname }}.tar.gz
        state: absent
      delegate_to: localhost
      run_once: true

    - name: Save logs
      shell: |
        /usr/local/bin/master-logs {{ item }} {{ item }} 2> /tmp/{{ item }}.log
      with_items:
      - api
      - controllers
      - etcd

    - name: Create a gz archive of multiple log, rooted at /tmp/{{ ansible_hostname }}.tar.gz
      archive:
        path:
        - /tmp/api.log
        - /tmp/controllers.log
        - /tmp/etcd.log
        dest: /tmp/{{ ansible_hostname }}.tar.gz
        format: gz

    - name: Fetch logs
      fetch:
        src: /tmp/{{ ansible_hostname }}.tar.gz
        dest: /tmp/master-logs/
        flat: true
    tags: collect