Implementing the Speed Layer of Lambda Architecture

In this post, we will implement the third part of our system, which is the Speed Layer of Lambda architecture. We will use Spark Structured Streaming to read data from Kafka’s “TwitterStreaming” topic and analyze this data in real time.

Real-time Twitter Analysis

In this tutorial, we will use TwitterStreaming API to get Twitter’s tweets in real time and publish it to a Kafka topic. We then use Spark as a Consumer to read Twitter data from Kafka and analyse that data. To

Integrating Kafka with Spark using Structured Streaming

In the previous tutorial (Integrating Kafka with Spark using DStream), we learned how to integrate Kafka with Spark using an old API of Spark – Spark Streaming (DStream) . In this tutorial, we will use a newer API of Spark,

Integrating Kafka with Spark using DStream

In this tutorial, we will learn how to integrate Kafka with Spark by writing a Spark application that get data from a Kafka topic and then perform some analysis on the received data. We normally have two ways to integrate