In the previous posts, we implemented four components of our system: Data collector and storage (Data collection and storage), Batch layer (Implementing the Batch Layer of Lambda Architecture), Speed Layer (Implementing the Speed Layer of Lambda Architecture) and Serving layer (Implementing the Serving Layer of Lambda Architecture). In this post, we will finalize our project by writing code for Server-side and Client-side. The Server-side is implemented as follows:

The use of EnvRunner.start(), CassandraDB.runDB(), KafkaTwitterStreaming.run() and AkkaServer.start() are already shown in Data collection and storage and Implementing the Serving Layer of Lambda Architecture. Here we implement the runProcessing() function responsible for running both Batch layer and Speed layer to ensure that each time we start running Batch Processing, we also restart the Real-time Processing (by deleting results of the previous Streaming Query and restart the Streaming Query as explained in the tutorial Implementing the Speed Layer of Lambda Architecture). We also need to run the realtimeActor in BatchProcessingActor and thus the code of BatchProcessingActor is slightly changed compare to the one in Implementing the Batch Layer of Lambda Architecture (full code of the project can be downloaded on our Github at here).

On the Client-side, we will create a simple UI to help users retrieve data from Server and display it on the screen. This UI is implemented using GUI Form of IntelliJ with 2 files: ClientUI.java and ClientUI.form (the code of ClientUI.java can be download at here). Run ClientUI.java and click “Top HashTag” button, we get the following result:

Wait for a few seconds and click “Top HashTag” button again, we can see that the number of hashtag and the count value of some hastags are increased since the processing result are constantly updated (as explained in Implementing the Serving Layer of Lambda Architecture)

So, we have completed building our project which aim to develop a big data processing system based on Lambda Architecture using SMACK stack. The full code of this project can be downloaded on the Gihub of ITechSeeker at here.

March 18, 2019
ITechSeeker