Though the local filesystem is not a good fit to store big data due to disk size limitations and lack of distributed nature, technically you can load data in distributed systems using the local filesystem. But then the file/directory you are accessing has to be available on each node.
Please note that if you are planning to use this feature to load side data, it is not a good idea. To load side data, Spark has a broadcast variable feature, which will be discussed in upcoming chapters.
Enriching data with side data is a very common use-case and we will cover how to do it in the subsequent chapters.
Enriching data with side data is a very common use-case and we will cover how to do it in the subsequent chapters.
In this recipe, we will look at how to load data in Spark from the local filesystem.