To send our logs to CloudWatch, AWS provides a daemon called awslogs. We are going to install and configure it through Ansible.

Go into your ansible roles directory:

$ cd ansible/roles  

Create a new role called awslogs:

$ ansible-galaxy init awslogs
- awslogs was created successfully  

We will first edit the task file awslogs/tasks/main.yml. Our first operation will be to install the package. For that, we will use the yum module:

--- 
# tasks file for awslogs  

- name: install awslogs 
  yum: 
    name: awslogs 
    state: present 

We will want to configure the service dynamically with Ansible. For that, we will want to create a handler to restart awslogs when the configuration changes.

Edit the file awslogs/handlers/main.yml and add the following:

--- 
# handlers file for awslogs 

- name: restart awslogs 
  service: 
    name: awslogs 
    state: restarted 

We can now configure the service. You can refer to http://amzn.to/2qMhaEt for the full documentation of the configuration of the service. In our case, we will keep it very simple. The service is configured through a set of INI files. The first one goes into /etc/awslogs/awslogs.conf. We will create the file using the file module from Ansible.

Create a new file in awslogs/files/, call it awslogs.conf, and put the following in it:

[general] 
state_file = /var/lib/awslogs/agent-state 

Now that the file is created, we are going to copy it to its target destination, /etc/awslogs/awslogs.conf. For that, we will use the copy module. Back in the awslogs/tasks/main.yml task file, we will add the following:

- name: copy global configuration 
  copy: 
    src: awslogs.conf 
    dest: /etc/awslogs 
  notify: restart awslogs 

Thanks to the notify handler we created, if we were to change our configuration file, awslogs would automatically restart and, through that, load the new configuration.

We now want to use Ansible to configure which file needs to be collected. We will do that by creating new INI files inside the /etc/awslogs/config directory. To make it easy to operate, we will create one INI file per log file we want to collect. We will take advantage of the INI file module that Ansible provides to implement that. At the bottom of the task file, create a new command as follows:

- name: configure awslogs to collect {{ file }} 
  ini_file: 

We want our role to be generic and able to collect a variety of logs. As such, we will take advantage of the variable system that Ansible provides. The idea is that whenever we call this role, we will provide the information needed, such as the file to collect and its name.

The INI module requires us to provide the path of the INI file we want to configure. We will do that as follows:

    path: "/etc/awslogs/config/{{ name }}.conf" 

Here as well, we are calling the variable name that will be provided when we instantiate the module. The section of the configuration file will be the filename we want to collect:

    section: "{{ file }}" 

Now that the section is created, we are going to configure the different options. We will want to configure six different options. To keep the code dry, we will take advantage of the with_items keyword to iterate over the list of options and values:

    option: "{{ item.option }}" 
    value: "{{ item.value }}" 

Finally, we can list the different options and values as follows:

  with_items: 
    - { option: file, value: "{{ file }}" } 
    - { option: log_group_name, value: "{{ file }}" } 
    - { option: log_stream_name, value: "{instance_id}" } 
    - { option: initial_position, value: "start_of_file" } 
    - { option: buffer_duration, value: "5000" } 
    - { option: datetime_format, value: "{{ datetime_format | default('%b %d %H:%M:%S') }}" } 

Here, too, we rely heavily on the fact that most values will be provided later. Note how we are making the datetime_formatfield optional. If it's not provided, we will try to read our logs as if they were formatted by the syslog. You can refer to the Python datetime.strptime() documentation for the full list of format variables.

We will conclude this call with a notify to restart awslogs if the configuration changed:

  notify: restart awslogs 

The last call of our task file will be to start the service and enable it so that awslogs starts right away, and upon reboot:

- name: start awslogs and enable it 
  service: 
    name: awslogs 
    state: started 
    enabled: yes 

Our role is ready, and we will use it inside our nodeserver.yml file. We will collect /var/log/messages and /var/log/helloworld/helloworld.log. Each entry will have a unique name (messages and helloworld) and their full path. In addition, we need to specify the logging format that our helloworld application is using:

--- 
- hosts: "{{ target | default('localhost') }}" 
  become: yes 
  roles: 
    - nodejs 
    - codedeploy 
    - { role: awslogs, name: messages, file: /var/log/messages }    
    - {        
        role: awslogs,        
        name: helloworld,        
        file: /var/log/helloworld/helloworld.log,
        datetime_format: "%Y-%m-%dT%H:%M:%S.%f"
      }

Your code should be similar to http://bit.ly/2v3bp9G.

Before we commit those changes, we are going to update our CloudFormation template to add proper permissions.