Apache Spark and In-memory Hadoop File System (IGFS)

IGFS is Apache Ignite’s in-memory distributed file system, which is the Hadoop File System compatible.

IGFS

1. Start the Ignite Node with IGFS

<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
. . . . . . . . .
<property name="fileSystemConfiguration">
<list>
<bean class="org.apache.ignite.configuration.FileSystemConfiguration">
<!-- Distinguished file system name. -->
<property name="name" value="myFileSystem" />
<property name="ipcEndpointConfiguration">
<bean class="org.apache.ignite.igfs.IgfsIpcEndpointConfiguration">
<property name="type" value="TCP"/>
<property name="host" value="192.168.0.106"/>
<!--<property name="port" value="12345"/>-->
</bean>
</property>
</bean>
</list>
</property>
. . . . . . . . .
</bean>

2. Start Apache Spark node with IGFS connector

<dependency>
<groupId>org.apache.ignite</groupId>
<artifactId>ignite-spring</artifactId>
<version>2.7.5</version>
</dependency>
sparkConf.set("fs.igfs.impl", "org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem");
sparkSession.read().text("igfs://myFileSystem@192.168.0.106/file/file.txt")anddataFrame.write().text("igfs://myFileSystem@192.168.0.106/file/file.txt")

--

--

Technology Enthusiast | Big Data Developer | Amateur Cricketer | Technical Lead Engineer @ eQ Technologic | https://www.bugdbug.com

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Amar Gajbhiye

Technology Enthusiast | Big Data Developer | Amateur Cricketer | Technical Lead Engineer @ eQ Technologic | https://www.bugdbug.com