Four Elasticsearch enthusiasts from LiteBreeze attended the inaugural Elastic user group meetup at Kochi on 25th July 2018. We attended the meeting with the purpose to connect with other like-minded Elasticsearch users, to gain a better understanding of Elastic stack and to discuss challenges as well as the different methods of using Elasticsearch in their applications.

There were nearly 100 attendees for the meeting, which is impressive for an inaugural meetup. They were a mixture of professionals including freshers, professionals and seasoned users of Elasticsearch.

This meetup included fairly basic sessions to make the attendees aware of the Elastic stack and the features it offers, and the ways they could use it in their applications.

The sessions also stressed that most of the Elastic stack is available free and the source code is open for anyone to use and contribute to.

Session details

There were three sessions in this meetup:

1. A welcome speech and a quick intro to the applications in the Elastic stack by Alfred Santa Molison from IBS.
2. A detailed introduction to the Elastic stack and its inner working, with a specific focus on Elasticsearch, taken by Aravind Putrevu, developer advocate from Elasticsearch.
3. A session on illustrating the rich integration between Apache Spark and Elasticsearch, with real-time data, taken by Eldhose K Shibu, CEO of RightVows Technologies.

The second session had proved to be the most informative session for us and helped us get a better understanding of the Elastic stack.

The session started with the introduction of the Elastic stack components and the licensing used in each of them. It was noted that most of the components in the Stack are open source and available to use freely. Even for the licensed components, the source code has been made openly available. X-Pack is one such component.

The massive adoption of Elastic stack by startups was discussed. The latest entrant in the Elastic stack, Beats, was explained in detail, along with the official beats offering available, and the much larger set of community beats available for ingesting data from other sources.

Apache Lucene was introduced as the core on which Elasticsearch is built upon. Basic terminologies related to Elasticsearch like index, mapping, nodes, clusters, etc. were discussed in detail.

The concept of an inverted index was explained in detail with an example for easy understanding. The importance of tokenizers and analyzers in Elasticsearch was discussed at length.

The latest developments and offerings on the Elastic stack were also discussed. Unsupervised machine learning jobs which currently work only with time-series data can help detect anomalies and make predictions on real-time data.

Aravind also did a quick demo of the Elastic cloud, their SaaS offering. It’s easy to deploy, operate and scale Elastic products on the cloud in a seamless manner.

They provide a 14-day trial offering which can be used by new users to check out Elasticsearch without messing with the manual installation. Also, the users who need professional support from Elasticsearch team can make use of the hosted Elastic cloud offering.

At the end of three sessions, there was a fun quiz hosted by Aravind to review the information shared during the session. The icing on the cake was that two of us from LiteBreeze won the 1st and 2nd prizes in it, along with Eldhose, who hosted the 3rd session.

The attendees were also provided fun takeaways such as quality stickers of the main Elasticstack tools.

Main takeaways

The main takeaway for us was a better understanding about the inner workings of Elasticsearch like the use of analyzers and tokenizers and introduction to the newest additions to the Elastic-Stack namely Beats, which is a platform for lightweight shippers that send data from edge machines to Logstash and Elasticsearch, and the exciting new experimental support for SQL to fetch data from Elasticsearch.

Also, the open forum QA session helped us clear some queries we had on Elasticsearch and the use of Logstash.

Experience with Elasticsearch

We have used Elasticsearch in multiple projects, and have production indices handling more than 500GB of data deployed in Elastic cloud. We mainly use Elasticsearch for indexing, retrieving and aggregation of data, along with providing advanced search features in the applications.

We also use it for processing web server logs and monitoring key application parameters. For the development environments, we use Elastic instances deployed in our AWS cloud.

Future plans with Elastic stack

We plan to make effective use of Kibana to get a better visualization of our application’s data like analytics and logs. We also plan to replace the use of Logstash in one of our productions apps, with File Beat to process and send log-data to the relevant Elasticsearch index.

The current Logstash based solution has been identified to be a major memory bottleneck in the current production stack. We expect Beats – designed to be lightweight shippers and written in GO – to be more efficient with its memory use.

Expectations for the next meetup

I applaud IBS for hosting this first-ever Elastic meetup at Kochi in a very neat and organized manner.  I look forward to future next meetups and hope that it will be deeper and include more discussions about use-cases and challenges with Elastic stack, as well as the solutions that different teams have come up with. We at LiteBreeze believe that it is useful to share knowledge and continually learn new things in technology.