Step by step guide for writing a custom read data source in Apache Spark 3.0.x with location-aware and multi-partition support — Apache Spark is a very powerful distributed execution engine. As we read through its documentation and examples, even with all its complex functionalities, we find it relatively easy to use. As we delve deep into it and try to solve real-life use cases, despite its features packed functionalities we need…

A practical design guide for the complex systems — Have you ever struggled with the Software Design or Do you find it difficult? If your answer is yes, you are NOT ALONE. Many of us face the same problem with the design. Most of the time, the complexity of the system and the enormity of the task in hand…

Actionable guidelines to make Apache Ignite production-ready. — This article was originally published at https://www.bugdbug.com/post/how-to-make-apache-ignite-production-ready In this article, we will discuss the guidelines that should be followed while deploying Apache Ignite in the production environment. It will help you to get the most out of it and will make sure it performs better without any problem. I assume…

This article is originally published at https://www.bugdbug.com/post/lombok-remove-boilerplate-code-from-java-source-files Project Lombok is the boilerplate code generator library for Java. It autogenerates Java bytecode in .class files for different annotations used, during the build process. It is a very popular framework among open source community and is a part of many open source…

Apache Ignite is primarily an in-memory distributed caching and data processing framework. In the last article, we discussed how we can handle network segmentation in Apache Ignite. In this article, we will discuss how to secure Apache Ignite cluster using Ignite security plugin. This article was originally published at https://www.bugdbug.com/post/how-to-secure-apache-ignite-cluster …

Apache Ignite is primarily an in-memory distributed caching and data processing framework. It also provides different capabilities like service-grid, messaging etc. Like any other distributed system, it needs to handle network partitioning. This article was originally published at https://www.bugdbug.com/post/how-to-handle-network-segmentation-in-apache-ignite Apache Ignite, like any other distributed system, follows the CAP theorem…

In the last article, we learned how to write a Apache Spark custom data source which could be used for connecting to any legacy data store. In this article, we will learn how to write a custom streaming data source. With the rise in popularity of IoT devices, there are…

In today’s big data computing world, Apache Spark is the most popular distributed execution engine. Fairly justifying its popularity, Apache Spark can connect to multiple data sources natively. Different data sources that Spark supports are Parquet, CSV, Text, JDBC, AVRO, ORC, HIVE, Kafka, Azure Cosmos, Amazon S3, Redshift, etc. Parquet…

IGFS is Apache Ignite’s in-memory distributed file system, which is the Hadoop File System compatible. — Apache Ignite is a distributed in-memory caching and data processing framework. One of the unique capability of Apache Ignite is IGFS, it’s distributed in-memory file system. IGFS implements Hadoop FileSystem API because of which it can easily be plugged-in with the Hadoop and Spark deployments.