Automate.Insights Part 3; Practical deep dive (clustered)

​This is part 3 in a 3 part series. To visit the prior examples, click below

 

In part two of this series, I created a standalone elkstack server using the Automate.Insight tool and knife topo. While it was the simplest to setup, a standalone monolith is not a true mimicking of production. Sometimes you will have separate logstash and elasticsearch nodes. Sometimes you will have multiple elasticsearch masters to make it even more complex. For the sake of simplicity in this example, let's have Logstash and Elasticsearch on separate servers, and have a third server exporting it's syslogs to the Logstash server. 

Clustered Setup

After starting with a fresh environment by running

vagrant destroy -f 
vagrant up

You will have three fresh Ubuntu machines to work with. For the sake of this tutorial, you will want to bootstrap specific machines with specific configurations

  • ai-elkstack-1(10.0.1.2): Logstash and logstash forwarder application. Required because of Ipaddress specific SSL cert
  • ai-elkstack-2(10.0.1.3): Elasticsearch and kibana
  • ai-elkstack-3(10.0.1.4) mimicks "application server" Meaning that you only use logstash-forwarder to push it's logs to the actual logstash server (10.0.1.2)

Next, if you don't have your Chef-server set up from the part 2 of the blog series, you will need to upload your cookbooks to the chef server. Start by creating your chef-zero server locally.

 chef-zero -d -H10.0.1.1 

This will run a chef-zero server as a background process on your workstation. If you need to kill that chef-zero server later, you will need to run 

 ps aux | grep chef-zero 

and kill the listed process id for the chef-zero server

Next, you will need to pull your cookbook dependencies in and upload them to the chef-server. I have included a Berksfile in the repo that will allow you to pull in dependencies. To pull and upload, run 

berks vendor
knife cookbook upload --all --cookbook-path berks-cookbooks/

This will:

  • pull the respective dependencies down from the remote git repos
  • upload them to the Chef-Zero server

(Note: Trying to run knife cookbook upload without specifying the path will throw false errors about cookbooks missing. You will not get these if you follow the procedure above ) 

This should give you all the cookbooks that you need initially.

Next, run these separately in order to configure each of the servers in your topology. 

knife bootstrap 10.0.1.3 -x vagrant -P vagrant --sudo -N df_box_elasticsearch --bootstrap-version 12.0.3 -r "recipe[df_java],recipe[df_elasticsearch],recipe[df_kibana],recipe[df_kibana::kibana_nginx]"

knife bootstrap 10.0.1.2 -x vagrant -P vagrant --sudo -N df_box_logstash --bootstrap-version 12.0.3 -r "recipe[df_java],recipe[df_logstash],recipe[df_logstash::logstash_forwarder]"

knife bootstrap 10.0.1.4 -x vagrant -P vagrant --sudo -N df_box_application --bootstrap-version 12.0.3 -r "recipe[df_java],recipe[df_logstash::logstash_ssl],recipe[df_logstash::logstash_forwarder]"

(Note: There have been some timeout issues for the logstash package installing, so you might get some chef-run errors. If so, just re run the bootstrap and it should go through)

This will 

  • set up elasticsearch and kibana on a separate server (node name df_box_elasticsearch)
  • set up logstash and logstash forwarder on it's own server (node name df_box_logstash: MUST KEEP ON 10.0.1.2)
  • create a third application server that will forward logs to logstash. (node name df_box_application)

We can now hit the kibana UI at 10.0.1.3, and see if everything is working properly 

As we can see, something didn't work quite right... We are not getting the logs sent to the Elasticsearch server. We can verify this from the Logstash server (run "vagrant ssh ai-elkstack-1" in the root of the repo first) by trying to hit the elasticsearch node. 

You can run the command yourself by running 

 curl 'http://10.0.1.3:9200/_search?pretty' 

As you can see, there is a connection refused going on with the server. This is because we have our current cookbooks to run everything through localhost. This doesn't work on a distributed setup. The best solution is to adjust the attributes to point to the specified hosts.

Rather than fix it in the cookbooks and reupload, we will use the Automate.Insights tool to change those attributes. But first, we need to get it into the Automate.Insights system! Export your cluster topology by running 

 knife topo export df_box_elasticsearch df_box_logstash df_box_application --topo elkstackcluster > elkstack2.json 

The next steps are very much the same as importing a standalone cluster.

First, you need to set up a prime blueprint from Chef. Navigate to the business system df_elkblog like in part 2.

Then you will prime the topology blueprint like before 

Make sure to load the topology.json file that is for the cluster. For us it was elkstack2.json

make sure all your recipes are named as you would want them to be.

What is really cool is that you can see that there are three nodes listed instead of one like last time. You can edit their node types to be meaningful names.

We can click next in the import process until we hit the attributes section. This is where you will want to select your list of attributes to edit. Make sure you include all attributes that start with df_, as these are the ones we will need later. 

Once you have finished, you can navigate to the elkstackcluster blueprint

Here, you will see that there are several nodes with attributes that you can edit. To make the necessary changes, choose the node type, and edit the attributes that are specific to that. 

For df_box_elasticsearch you will need to edit. 

  • df_elasticsearch.network_host - change from 'localhost' to '10.0.1.3'
  • df_kibana.elasticsearch_host - change from 'localhost' to '10.0.1.3'

For the Logstash host

  • df_logstash.elasticsearch_host - change '127.0.0.1' to '10.0.1.3'

 

for the application host you will not have to adjust any attribute values. 

Once you have made the attribute changes in the UI, set each node that you plan to provision in your topology (keep logstash on 10.0.1.2 for ssl cert verification) 

For Elasticsearch host

For Logstash host

For the appserver

once you have done this, export the components to Chef. Make sure they are via a topology cookbook like the example in part 2

Save your topology layout from the Automate.Insights tool to an elkstackcluster.json file in your projects directory. Once this is done, you will want to import it to your chef node itself

knife topo import elkstackcluster.json

Note: I have included an elkstackcluster.json file in the topologies/ folder of the repo if you would like to use that instead. 

Now you have a cluster setup defining your entire topology. Let's start with some fresh Ubuntu Vm's for this.

vagrant destroy -f 
vagrant up

Then, you can run a simple command to build the entire topology of nodes.

knife topo create elkstackcluster --bootstrap --sudo -xvagrant -Pvagrant

(Note: There are sometimes timeout issues on the logstash package installation, so you might have to re run this. Press y when it prompts you to update the topologies that already exist)

This is: 

  • Uploading the topology information to the Chef server
  • bootstrapping the nodes
  • configuring the specific hosts with the software defined in our topology.json file.

To verify, simply hit the 10.0.1.3 in your browser, and verify that it is receiving logs. You can also re run the command from earlier, which this time should output JSON formatted information 

 curl 'http://10.0.1.3:9200/_search?pretty' 

 

So there you have it, a working topology of nodes, adjusted to work with the automate.inights tool, provisioned with Chef, and put together with Knife topo. This small example is a simple proof of concept to spinning up a collection of machines. Obviously what you choose to build with it is your shops choice, but the potential is quite powerful