ELK Stack for Kubernetes Metrics

Part of the work of the Platform Services team at Alfresco is around deployment, specifically Kubernetes.

Monitoring a Kubernetes (K8s) cluster is critical in production and there are many great tools out there like Prometheus and Grafana, but I wanted to ‘play’ with getting K8s metrics into the Elasticsearch, Logstash, and Kibana (ELK) stack, all deployed in my local Docker Desktop Kubernetes environment.

I found several resources on various parts of the solution, but didn’t find one that tied all of it together, so hopefully this post will be of use to someone.

Prerequisites

1. Deploy the ELK Stack

Helm provides excellent package management of K8s and is always my first choice for deployment so I started there for the ELK stack, and sure enough, there’s a Helm chart for that (elastic-stack).

You can easily override/merge the default values of a chart with your own values file so I created one that enabled filebeat and pointed it to logstash and the elasticsearch client and pointed logstash to the elasticsearch client:

my-elastic-stack.yaml:

logstash:
  enabled: true
  elasticsearch:
    host: elastic-stack-elasticsearch-client

filebeat:
  enabled: true
  config:
    output.file.enabled: false
    output.logstash:
      hosts: ["elastic-stack-logstash:5044"]
  indexTemplateLoad:
    - elastic-stack-elasticsearch-client:9200

Then deployed with those values:

helm install --name elastic-stack stable/elastic-stack -f ~/Desktop/my-elastic-stack.yaml

You’ll see some output detailing what you’ve deployed and you can check the status of the pods with kubectl:

kubectl get pods -l "release=elastic-stack"

2. Port Forward Kibana and Setup Index

Once the pods are all showing proper READY status (which may take a minute or two) you can set up port forwarding to the Kibana pod:

export POD_NAME=$(kubectl get pods --namespace default -l "app=kibana,release=elastic-stack" -o jsonpath="{.items[0].metadata.name}");
kubectl port-forward --namespace default $POD_NAME 5601:5601

Then open http://127.0.0.1:5601 in your browser to access the Kibana UI.

You can then of course create an index pattern for filebeat and browse those entries:

3. Deploy kube-state-metrics

The elastic metricbeat module (which we’ll deploy in a moment) relies on kube-state-metrics. Sure enough, there’s a chart for that (kube-state-metrics):

helm install --name kube-state-metrics stable/kube-state-metrics

4. Deploy Metricbeat

Metricbeat collects (easy guess) metrics and, through its Kubenetes modules, can get data from K8s nodes and the API. Sure enough, (you know where this is going…) there’s a chart for that (metricbeat).

In this case we have to override values to:

  • Use the open source version of the metricbeat Docker image (metricbeat-oss)
  • Point the daemonset Kubernetes module to the proper K8s SSL metrics host (and disable SSL verification)
  • Point the deployment to the elasticsearch client and kibana in the cluster

my-elastic-metricbeat.yaml

image:
  repository: docker.elastic.co/beats/metricbeat-oss
daemonset:
  config:
    output.file:
    output.elasticsearch:
      hosts: ["elastic-stack-elasticsearch-client:9200"]
  modules:
    kubernetes:
      config:
        - module: kubernetes
          metricsets:
            - node
            - system
            - pod
            - container
            - volume
          period: 10s
          host: ${NODE_NAME}
          hosts: ["https://${HOSTNAME}:10250"]
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          ssl.verification_mode: "none"
deployment:
  config:
    output.file:
    output.elasticsearch:
      hosts: ["elastic-stack-elasticsearch-client:9200"]
    setup.kibana:
      host: "elastic-stack-kibana:443"
    setup.dashboards.enabled: true

Then deploy with those values:

helm install --name elastic-metricbeat stable/metricbeat -f ~/Desktop/my-elastic-metricbeat.yaml

5. Setup Metricbeat Index and K8s Dashboard

Metricbeat will start feeding the K8s metrics to Elasticsearch and should setup another index pattern and load several dashboards into Kibana.

You can then open the [Metricbeat Kubernetes] Overview dashboard:

Of course, there are lots ways this could be done, and this could all be easily tied together in a parent Helm chart for easier release management.

Tools like Prometheus and Grafana may be targeted more specifically for this sort of task out of the box, but it’s great to see the Elastic stack as another viable option.

Are any of you using the ELK stack for K8s monitoring in production?

Adding Document Library Views to Alfresco Share

Overview

In this post we’ll go through how to develop and deploy additional view types beyond the default simple and detailed views in the document library in Share.

The techniques here require Alfresco 4.0.2 or later, or a recent checkout of community HEAD.

The Document Library Component

We won’t go into all the details on how the document library is rendered but it’s a fairly standard webscript-based component with ‘server-side’ resources like FreeMarker files for HTML fragments, Javascript for manipulating the data model, and localization properties files, and ‘client-side’ resources sent to the browser like CSS and Javascript.

While adding a document library view will involve changes to both client-side and server-side tiers, most of our notable changes will be to client-side Javascript files sent to the browser. The server-side webscripts changes are primarily to load and instantiate the client-side objects.

The component we’re interested in is at:
alfresco/site-webscripts/org/alfresco/components/documentlibrary/documentlist.*

and note that within that directory resources from include/documentlist.* are included by some of the primary files.

DocumentList and DocumentListViewRenderer

The DocumentList Javascript object makes up the bulk of the document library view and is defined in the client-side documentlist.js file along with event handler methods and other display setup. The DocumentList object sets up a YUI DataTable which handles most of the work of AJAX calls, pagination, etc.

Each column in the DataTable is assigned a method which dictates how to render that column given the data row.  For example, the DataTable calls DocumentList.fnRenderCellThumbnail for the ‘thumbnail’ column of each row it renders.

The DocumentList also contains a set of registered DocumentListViewRenderers, ‘simple‘ and ‘detailed‘ by default, which do the work of generating the HTML for their view type. In other words, DocumentList.fnRenderCellThumbnail actually determines the current view renderer based on the current viewRendererName and hands off its work to DocumentListViewRenderer.renderCellThumbnail.

This allows developers to create their own objects which extend and override the DocumentListViewRenderer with methods for their specific needs.

For this example our goal will be to add a view which is very similar to the existing detail view but with larger thumbnails.

Steps to Add a View

Define the View

For a new view we need to define a new DocumentListViewRenderer object.  That customization could be just a new instance of the base DocumentListViewRenderer with a different name so that alternate CSS is applied and metadata rendered.

The name of the view renderer specified in the constructor is used for CSS class names among other things so a simple example of adding a second detailed view with a larger thumbnail might look like:

new Alfresco.DocumentListViewRenderer("large");

In more complex scenarios you probably want to extend DocumentListViewRenderer in a client-side documentlist-custom.js file, which is what we’ll do in this example.

First we want to define a constructor:

Alfresco.DocumentListLargeViewRenderer = function(name)
{
   Alfresco.DocumentListLargeViewRenderer.superclass.constructor.call(this, name);
   // Defaults to large but we'll copy the metadata from detailed view
   this.metadataBannerViewName = "detailed";
   this.metadataLineViewName = "detailed";
   this.thumbnailColumnWidth = 200;
   return this;
};

then make it extend from DocumentListViewRenderer:

YAHOO.extend(Alfresco.DocumentListLargeViewRenderer, Alfresco.DocumentListViewRenderer);

and we’ll override the renderCellThumbnail method with almost the same code in the detail view, but adding a documents-large CSS class to the thumbnail container and changing the call to generateThumbnailUrl to use the imgpreview thumbnail definition:

Alfresco.DocumentList.generateThumbnailUrl(record, 'imgpreview')

(Full code)

Register the View

Once the view renderer has been defined it needs to be registered with the DocumentList object via its registerViewRenderer method.  This needs to be done after the DocumentList object has been created and setup, so a good place to perform the registration is after the YUI Bubbling Event postSetupViewRenderers has fired.  To register the same view above:

YAHOO.Bubbling.subscribe("postSetupViewRenderers", function(layer, args) {
   var scope = args[1].scope;
   var largeViewRenderer = new Alfresco.DocumentListLargeViewRenderer("large");
   scope.registerViewRenderer(largeViewRenderer);
});
Accessing the View

The user has to be able to navigate to this new view and even though we’ve registered it, the DocumentList needs to know which view renderers are enabled and in what order, both of which are defined in the DocumentList’s options under viewRendererNames.  We can add our custom view above to the server-side FreeMarker model:

model.viewRendererNames.push("large");

Note that the name here must match the name of the view renderer constructor.

Not only does this define that the ‘large‘ view is enabled, the navigation buttons are also rendered from this viewRendererNames list, so we’ve taken care of the interface as well.

CSS

Our overridden renderCellThumbnail method above adds the documents-large CSS class so we’ll need to define that, as well as the display of the view selection button. Here’s a portion of the documentlist-custom.css file:

.doclist .thumbnail.documents-large {
	height: 200px !important;
	width: 200px !important;
}
.doclist .thumbnail.documents-large img {
    max-width: 280px;
}

Packaging

There are of course a number of ways this code could be packaged and deployed but this type of customization lends itself particularly well to a share extensibility module which Erik Winlöf and David Draper have blogged about in detail.  We’ll assume a standard Maven or Gradle Java project layout for the code, and to keep things simple the example deploys as a Share jar rather than an AMP.

Server-side Resources

We need to define the customization itself:

src/main/resources/alfresco/site-data/extensions/custom-view-renderer.xml

<extension>
    <modules>

        <module>
            <id>Example :: Document List Custom View</id>
            <auto-deploy>true</auto-deploy>
            <customizations>
                <customization>
                    <targetPackageRoot>org.alfresco</targetPackageRoot>
                    <sourcePackageRoot>com.example.alfresco-share-doclib-views-example</sourcePackageRoot>
                </customization>
            </customizations>
        </module>

    </modules>
</extension>

We could place the javascript which creates the new view renderer a number of places, even purely client-side, but as we’ll see in another post it can be handy to have access to the FreeMarker data model so we’ll place it in:

src/main/resources/alfresco/site-webscripts/com/example/alfresco-share-doclib-views-example/components/documentlibrary/documentlist.get.html.ftl

Note that the the wrapping <@markup> directive there indicates that we want to inject the Javascript after documentListContainer.

We’ll place the server-side webscript javascript which enables the view at:

src/main/resources/alfresco/site-webscripts/com/example/alfresco-share-doclib-views-example/components/documentlibrary/documentlist.get.js

which will get its label from the localization file at:
src/main/resources/alfresco/site-webscripts/com/example/alfresco-share-doclib-views-example/components/documentlibrary/documentlist.get.properties

Client-side Resources

For including client-side CSS, Javascript, and image resources in a jar we’ll place them in the META-INF dir at:
src/main/resources/META-INF/alfresco-share-doclib-views-example/components/documentlibrary/*

So at this point src/main/resources should look like:

Package that as jar or AMP and deploy to your Share instance.

Results

The next time you go to a site’s document library you should see the additional
button for your view and the larger thumbnails:

Since we implemented this as a standard extensibility module you can completely disable it by going to your Share deployment’s module management page at something like http://localhost:8080/share/page/modules/deploy and disable the module.

What’s Next?

I’ll go into more advanced customizations in future posts, but in the meantime, what kind of doclib view are you going to build?

My Mullet Business Card

I recently had to redesign my business card and decided to take a step back and think about the different functions it serves at different times. What resulted was my new mullet cards: business in the front, party in the back.

A business card can be much more than just a means to contact you at a later time.

It’s a way to connect with people on a personal level

At networking events I notice that people often jump to the address when handed a business card in order to strike up a conversation about people or places they know near there.  There’s a good chance we have a lot more in common than that so I threw several work and personal interests in a sort of thin word cloud.  Maybe they’re an avid cyclist, maybe they’ve just started coworking, or maybe they want to blow off this networking event and go get a good beer.

Who the hell was this and why do I have their card?

If I don’t process (I use CardMunch) the cards I’ve collected at an event with 24 hrs (which I usually do) I often can’t place a person by their name or company alone if I didn’t have more than a brief conversation with them.  My interests might help someone recall who handed them the card, but a face will almost certainly do the trick.  Realtors usually have a headshot on their cards, why shouldn’t I?

Use QR codes wisely

QR codes are meant to ease entry and access for mobile devices.  Often people put all of their contact info in one QR code in something like a vCard format, which can work, but when you try to put all of your information in a single code it can greatly affect the ability of QR readers recognize it, especially in bad lighting.  Even if the vCard does get captured it most often gets neatly tucked away in their address book, never to be viewed again.

I chose instead to go with two simple codes; one that is the URL for my About Me page, and one that is my Google Voice number.  If I’m telling someone how cool Untappd is or discussing cycling routes they can quickly snap the About Me QR code I just handed them and get right to my Untappd or RunKeeper profiles from there.  If weeks later they find my card and decide they want to chat they can snap the Call Me code which is only the phone number so most readers will just ask for a quick confirmation then dial directly, no hunting through address books.

Don’t buck the trend too much

The front is pretty much your traditional card with basic contact info.  Obviously not everyone knows what a QR code is, let alone how to read them, or is going to want to look at my ugly mug. Business in the front.

A shout out to the printers

I used MOO.  I was very impressed with the ease of submitting and verifying, speed of printing and shipping, and quality of the final product.  They did a fantastic job and I highly recommend them.

This was a good first pass, I’ll probably modify them a bit for the next round.  Do you have any suggestions?

Inconsistent CMIS Query Results? It’s not You, It’s your Locale.

A simple CMIS query like:

SELECT D.* FROM cmis:document AS D WHERE D.cmis:name LIKE '%Flower%' OR D.cmis:contentStreamFileName LIKE '%Flower%'

was giving me a fit lately, not working at all in RightsPro‘s CMIS plugin which uses OpenCMIS and yielding inconsistent results in CMIS Workbench where re-posting the same query ten times would give the expected result maybe half the time.

Thanks to Florian Müller and this post, it seems as though Alfresco doesn’t always behave as expected when the locale is set in the CMIS session.

Removing the locale session parameters got things working in RightsPro, but I didn’t immediately see an easy way to change the locale in a Workbench session (the log shows that it’s using a default of en_US), and I still don’t know what’s up with the inconsistent results there, perhaps a coincidental caching issue.

This was all using OpenCMIS 0.3 against Alfresco Enterprise 3.3.1 over SSL.

Searching Custom Aspects via CMIS

I didn’t immediately see any simple, working examples of how to search custom aspect properties via CMIS so I’ll post a brief one here.

The Aspect

In this example we’re using a CMIS query to search for the IPTC caption property provided by the IPTC/EXIF project. After installing that module the properties are applied using an aspect which is defined in $ALFRESCO_HOME/WEB-INF/classes/alfresco/module/iptcexif/model/iptcModel.xml.

Within that file you’ll see:

    <aspects>
      <aspect name="iptc:iptc">

The Query

In our case the target Alfresco repository is running enterprise 3.3.1 and aspects are implemented via JOIN syntax as very briefly stated in the wiki.

So the resulting query for this particular search of the IPTC caption field ends up looking like:

SELECT D.*, IPTC.* FROM cmis:document AS D JOIN iptc:iptc AS IPTC ON D.cmis:objectId = IPTC.cmis:objectId WHERE IPTC.iptc:caption LIKE '%searchTerm%'

The Changes

There were big changes in Alfresco 3.4 in terms of metadata extraction and I haven’t yet had a chance to update the forge projects (or determine if they’re still even needed) and there are proposals for aspects in the next CMIS spec so I’ll be back to update this post.

MyFaces Tomahawk and the HTML BASE Tag

This one incited a bit of facepalm.

When trying to changing part of RightsPro‘s header implementation to use the HTML BASE tag the element seemed to be ignored completely and resource references were still relative to the page URL.  The tag was present, correct, and closed, but having no affect.

It turns out the BASE tag must appear ‘before any element that refers to an external source’ but MyFaces Tomahawk adds any script references needed immediately after the opening HEAD tag, rendering the BASE tag useless.

I don’t have time to modify the Tomahawk source at the moment for a proper fix but did submit a bug.

As a workaround you can put the BASE tag above the head element. It’s not valid HTML but most browsers will still obey it and render the page properly.

A short post, but it may save a few precious hairs from being pulled out. I’ll update if and when the bug is resolved.