Our logging library, Winston, has a system to extend some of its functionalities, including its transport system. We will install a new transport system that can talk to Kinesis Firehose.

In your terminal, go to the root directory of your helloworld application and run the following command:

$ npm install [email protected] --save --save-exact  

This will install specifically version 1.0.6 of the winston-firehose package and update the package.json accordingly. We need to enforce this version, because our EC2 instance is running an old version of Node.js.

Then, open the helloworld.js file with your code editor.

After the declaration of our winston variable, we will define a new variable as follows:

var WFirehose = require('winston-firehose') 

When we used the awslogs agent or the ECS logging driver, the service was able to specify the hostname or the container ID that the logs were coming from. In addition to this, each log file was in its own log group, which made it easy to identify what service and what instance of this service emitted a given log. The new architecture we are migrating to doesn't offer this. We will make some changes to our code to expose the service name and the host that produced the logs.

After the creation of the version variable, add the following:

var hostname = process.env.HOSTNAME 

This will get the hostname of the server from our environment.

Later on, we will edit the rewriter to include this extra information as follows:

logger.rewriters.push(function(level, msg, meta) { 
  meta.version = version 
  meta.hostname = hostname 
  meta.appname = "helloworld" 
  return meta 
}) 

Lastly, we will replace the logger variable definition. Find the following code:

var logger = new winston.Logger({ 
  transports: [new winston.transports.Console({ 
    timestamp: function() { 
       var d = new Date(); 
       return d.toISOString() 
    }, 
  })] 
}) 

Replace this code with the reference to our Firehose endpoint:

var logger = new (winston.Logger)({ 
  transports: [new WFirehose({ 
    'streamName': 'FirehoseLogs', 
    'firehoseOptions': { 
      'region': 'us-east-1' 
    } 
  })] 
}) 

Once those changes are in place, we can add the new module to git, commit, and push the changes:

$ git add helloworld.js package.json node_modules
$ git commit -m "Sending logs to Firehose directly"
$ git push  

Within a few minutes, CloudWatch logs should stop receiving logs, while your Firehose delivery service should start seeing traffic. You can open https://console.aws.amazon.com/firehose/home?region=us-east-1#/details/FirehoseLogs?edit=false to verify that the last changes are working:

We can now look at our logs in Kibana.