How to handle Network Segmentation in Apache Ignite?
--
Apache Ignite is primarily an in-memory distributed caching and data processing framework. It also provides different capabilities like service-grid, messaging etc. Like any other distributed system, it needs to handle network partitioning.
This article was originally published at https://www.bugdbug.com/post/how-to-handle-network-segmentation-in-apache-ignite
Apache Ignite, like any other distributed system, follows the CAP theorem. So, in the event of Network partitioning, it can either have Availability or Consistency. Since Apache Ignite is a system with the ACID guarantees, it needs to make sure that data consistency is maintained in the event of partitioning.
Segmentation causes
Network partitioning could occur because of a network glitch, node failure or long GC pauses. Because of network partitioning, the cluster could be divided into one or more segments with one or more nodes. These segments could act as independent clusters and may cause data inconsistency if kept working and writing data.
Now, to identify true network segmentation cases and rule out the false ones, we need to have some sort of mechanism in place. Also, we need to identify nodes in the valid segment and keep them running and take appropriate action about the ones in invalid segments.
Fortunately, Apache Ignite has provided this mechanism called Network Segment validation. Each node checks if it is a part of a valid segment using SegmentationResolver. Gridgain has provided three implementations of it. Details regarding these can be found here. Though, this solution is not a part of its free distribution(i.e. Apache Ignite). But, we can define our implementation by writing a custom plugin for it by following these guidelines.
Custom segmentation plugin
Now, let’s see how we can define a custom network segmentation plugin for Apache Ignite.
- Every plugin needs to implement PluginConfiguration and register it using IgniteConfiguration.setPluginConfigurations.
2. PluginProvider implementation creates a plugin instance on the node startup. As per the documentation, name() and plugin() methods need to be implemented, which return name and plugin instance respectively, other methods can be null.
3. createComponent() creates an instance of GridSegmentationProcessor which handles segmentation using SegmentationResolver. GridSegmentationProcessor could use segmentation settings such as getSegmentationResolveAttempts(), getSegmentationResolvers(), isAllSegmentationResolversPassRequired(). These can be set using IgniteConfiguration.
4. Ignite plugins are loaded using ServiceLoader. We need to create a file with org.apache.ignite.plugin.PluginProvider name in the META-INF/services folder with the name of the implementation in it.
5. SegmentationPolicy defines what happens when segmentation occurs. The default value is STOP the JVM it is running in. One needs to change that according to the requirement.
Embedded Ignite
When Apache Ignite is running in an embedded mode, i.e. inside an already running process, segmentation needs to be handled differently. The default segmentation policy of stopping the process would not be appropriate. We can change that policy to SegmentationPolicy.NOOP and register a listener for the Segmentation event and can provide appropriate custom handling. One of such handling could be to restart the Ignite service which could wait at startup till segmentation is resolved. This could be configured using IgniteConfiguration.setWaitForSegmentOnStart().
Source code for the custom IgniteSegmentation Plugin can be found here.
If you like this article, check out similar articles here https://www.bugdbug.com/
Feel free to share your thoughts, comments.
If you find this article helpful, share it with a friend!
This article was originally published at https://www.bugdbug.com/post/how-to-handle-network-segmentation-in-apache-ignite
