Open in app

Sign In

Write

Sign In

Amar Gajbhiye
Amar Gajbhiye

71 Followers

Home

About

Published in

Level Up Coding

·Nov 2, 2020

Easy Guide to Create a Custom Write Data Source in Apache Spark 3

Step by step guide for creating a custom transactional write data source in Apache Spark 3.0.x — This is a second article in the series to write a custom data source in Apache Spark 3.0.x. In the first article, we learned about data source APIs in Apache Spark 3.0.x, their significance and overview of the read APIs. First, we learned to create a simple custom read data…

Apache Spark

7 min read

Easy Guide to Create a Write Data Source in Apache Spark 3
Easy Guide to Create a Write Data Source in Apache Spark 3
Apache Spark

7 min read


Published in

Level Up Coding

·Oct 19, 2020

Easy Guide to Create a Custom Read Data Source in Apache Spark 3

Step by step guide for writing a custom read data source in Apache Spark 3.0.x with location-aware and multi-partition support — Apache Spark is a very powerful distributed execution engine. As we read through its documentation and examples, even with all its complex functionalities, we find it relatively easy to use. As we delve deep into it and try to solve real-life use cases, despite its features packed functionalities we need…

Apache Spark

6 min read

Easy Guide to Create a Custom Read Data Source in Apache Spark 3
Easy Guide to Create a Custom Read Data Source in Apache Spark 3
Apache Spark

6 min read


May 10, 2020

Domain-Driven Approach Towards Software Design

A practical design guide for the complex systems — Have you ever struggled with the Software Design or Do you find it difficult? If your answer is yes, you are NOT ALONE. Many of us face the same problem with the design. Most of the time, the complexity of the system and the enormity of the task in hand…

Software Development

6 min read

Domain-Driven Approach Towards Software Design
Domain-Driven Approach Towards Software Design
Software Development

6 min read


Feb 14, 2020

How to Make Apache Ignite Production Ready

Actionable guidelines to make Apache Ignite production-ready. — This article was originally published at https://www.bugdbug.com/post/how-to-make-apache-ignite-production-ready In this article, we will discuss the guidelines that should be followed while deploying Apache Ignite in the production environment. It will help you to get the most out of it and will make sure it performs better without any problem. I assume…

Apache Ignite

4 min read

How to Make Apache Ignite Production Ready
How to Make Apache Ignite Production Ready
Apache Ignite

4 min read


Feb 10, 2020

Lombok: Remove boilerplate code from Java source files

This article is originally published at https://www.bugdbug.com/post/lombok-remove-boilerplate-code-from-java-source-files Project Lombok is the boilerplate code generator library for Java. It autogenerates Java bytecode in .class files for different annotations used, during the build process. It is a very popular framework among open source community and is a part of many open source…

Java

4 min read

Lombok: Remove boilerplate code from Java source files
Lombok: Remove boilerplate code from Java source files
Java

4 min read


Sep 14, 2019

How to Secure Apache Ignite cluster?

Apache Ignite is primarily an in-memory distributed caching and data processing framework. In the last article, we discussed how we can handle network segmentation in Apache Ignite. In this article, we will discuss how to secure Apache Ignite cluster using Ignite security plugin. This article was originally published at https://www.bugdbug.com/post/how-to-secure-apache-ignite-cluster …

Distributed Systems

3 min read

How to Secure Apache Ignite cluster?
How to Secure Apache Ignite cluster?
Distributed Systems

3 min read


Sep 14, 2019

How to handle Network Segmentation in Apache Ignite?

Apache Ignite is primarily an in-memory distributed caching and data processing framework. It also provides different capabilities like service-grid, messaging etc. Like any other distributed system, it needs to handle network partitioning. This article was originally published at https://www.bugdbug.com/post/how-to-handle-network-segmentation-in-apache-ignite Apache Ignite, like any other distributed system, follows the CAP theorem…

Big Data

3 min read

How to handle Network Segmentation in Apache Ignite?
How to handle Network Segmentation in Apache Ignite?
Big Data

3 min read


Aug 19, 2019

Apache Spark: Custom streaming data source

In the last article, we learned how to write a Apache Spark custom data source which could be used for connecting to any legacy data store. In this article, we will learn how to write a custom streaming data source. With the rise in popularity of IoT devices, there are…

Big Data

4 min read

Apache Spark: Custom streaming data source
Apache Spark: Custom streaming data source
Big Data

4 min read


Jul 29, 2019

Apache Spark Custom Data Source

In today’s big data computing world, Apache Spark is the most popular distributed execution engine. Fairly justifying its popularity, Apache Spark can connect to multiple data sources natively. Different data sources that Spark supports are Parquet, CSV, Text, JDBC, AVRO, ORC, HIVE, Kafka, Azure Cosmos, Amazon S3, Redshift, etc. Parquet…

Apache Spark

5 min read

Apache Spark Custom Data Source
Apache Spark Custom Data Source
Apache Spark

5 min read


Jul 5, 2019

Apache Spark and In-memory Hadoop File System (IGFS)

IGFS is Apache Ignite’s in-memory distributed file system, which is the Hadoop File System compatible. — Apache Ignite is a distributed in-memory caching and data processing framework. One of the unique capability of Apache Ignite is IGFS, it’s distributed in-memory file system. IGFS implements Hadoop FileSystem API because of which it can easily be plugged-in with the Hadoop and Spark deployments.

Hadoop

3 min read

Apache Spark and in-memory Hadoop File System (IGFS)
Apache Spark and in-memory Hadoop File System (IGFS)
Hadoop

3 min read

Amar Gajbhiye

Amar Gajbhiye

71 Followers

Technology Enthusiast | Big Data Developer | Amateur Cricketer | Technical Lead Engineer @ eQ Technologic | https://www.bugdbug.com

Following
  • Sean Kernan

    Sean Kernan

  • M.G. Siegler

    M.G. Siegler

  • Ethan Siegel

    Ethan Siegel

  • Darius Foroux

    Darius Foroux

  • Gregg Williams, MFT

    Gregg Williams, MFT

See all (70)

Help

Status

Writers

Blog

Careers

Privacy

Terms

About

Text to speech

Teams