IBM® Spectrum Scale is flexible and scalable software-defined file storage for analytics workloads. Enterprises around the globe deployed IBM Spectrum™ Scale to form large data lakes and content repositories to perform High Performance Computing (HPC) and analytics workloads. It is known to scale performance and capacity without bottlenecks.

Hortonworks Data Platform (HDP) is a leader in Hadoop and Spark distributions. HDP addresses the needs of data-at-rest, powers real-time customer applications, and delivers robust analytics that accelerate decision making and innovation.

IBM Spectrum Scale™ solves the challenge of explosive growth of unstructured data against a flat IT budget. IBM Spectrum Scale provides unified file and object software-defined storage for high-performance, large-scale workloads that can be deployed on-premises or in the cloud.

IBM Spectrum Scale includes NFS, SMB, and Object services and meets the performance that is required by many industry workloads, such as technical computing, big data, analytics, and content management. IBM Spectrum Scale provides world-class, web-based storage management with extreme scalability, flash accelerated performance, and automatic policy-based storage tiering from flash through disk to the cloud, which reduces storage costs up to 90% and improves security and management efficiency in cloud, big data, and analytics environments.

IBM Elastic Storage™ Server is an optimized disk storage solution that is bundled with IBM hardware and innovative IBM Spectrum Scale RAID technology (based on erasure coding). It performs fast background disk rebuilds in minutes without affecting application performance. Traditional Hadoop analytics systems feature a dedicated cluster to run Hadoop services, as shown in Figure 1 on page 2.

Data scientists waste days just copying data into Hadoop Distributed File System (HDFS) for analytics and copying data out of HDFS for traditional applications.

In this kind of analytics system, users and system administrators face the following key challenges:

IBM Spectrum Scale is POSIX compatible, which supports various applications and workloads. By using IBM Spectrum Scale HDFS Transparency Hadoop connector, you can analyze file and object data in-place with no data transfer or data movement. Traditional systems and the analytics systems are using and sharing data that is hosted on IBM Spectrum Scale file system, as shown in Figure 2 on page 3.

Hadoop services can use a storage system to save IT cost because no special purpose storage is required to perform the analytics. IBM Spectrum Scale features a rich set of enterprise-level data management and protection features, such as snapshots, information lifecycle management (ILM), compression, and encryption, which provide more value than traditional analytic systems.

Spark is a fast and general engine for large-scale data processing. It runs programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk.

IBM Spectrum Scale supports for in-place analytics by using Spark, which can be configured with POSIX interface through IBM Spectrum Scale native client to mount the file system directly. It also supports the use of the HDFS interface through IBM Spectrum Scale HDFS Transparency.

IBM Spectrum Scale also supports various Spark distributions, such as Spark service in Hortonworks Data Platform and IBM Spectrum Conductor™ with Spark.

 

The following benefits can be realized by using IBM Spectrum Scale with HDP:

IBM Spectrum Scale is a parallel architecture. With a parallel architecture, no single metadata node can become a bottleneck. Every node in the cluster can serve data and metadata, which allows a single IBM Spectrum Scale file system to store billions of files. This ability enables clients to grow their HDP environments seamlessly as the amount of data grows.

By using IBM Spectrum Scale global namespace, clients can create active, remote data copies and enable real-time, global collaboration. This feature allows global organizations to form data lakes across the globe, and host their distributed data under one namespace. IBM Spectrum Scale also enables multiple Hadoop clusters to access a single file system while still providing all the required data isolation semantics.

The IBM Spectrum Scale Transparent Cloud Tiering feature can archive data into S3/SWIFT-compatible cloud Object Storage system, such as IBM Cloud Object Storage or Amazon S3 by using the powerful IBM Spectrum Scale ILM policies.

IBM Spectrum Scale features the most comprehensive support for data access protocols. It supports data access that uses NFS, SMB, Object, POSIX, and the HDFS API. This feature eliminates the need to maintain separate copies of the same data for traditional applications and for purposes of analytics.

You can deploy IBM Spectrum Scale as software directly on commodity storage-rich servers that are running the HDP stack. It also can be deployed as part of a pre-integrated system that uses the IBM Elastic Storage Server. Clients can use software-only options to start small and still use enterprise storage benefits. By using the IBM Elastic Storage Server, clients can control cluster sprawl and grow storage independently of the compute infrastructure.

IBM Elastic Storage Server uses erasure coding, which eliminates the need for the three-way replication for data protection that is required with other solutions. IBM Elastic Storage Server requires only 30% extra capacity to offer similar data protection benefits.

Along with IBM Elastic Storage Server, IBM Power Systems™ offers the most optimized hardware stack for running analytics workloads. Clients can enjoy up to 3x reduction of storage and compute infrastructure by moving to Power Systems and IBM Elastic Storage Server compared to commodity scale-out x86 systems. IBM Elastic Storage Server also is available in all flash models, which allows for accelerated analytics that are required for certain data sets.

To support security and regulatory compliance requirements of organizations, IBM Spectrum Scale offers Federal Information Processing Standard (FIPS) compliant data encryption for secure data at rest, policy-based tiering and ILM, cold data compression, disaster recovery, snapshots, backup, and secure erase. The HDP Atlas and Ranger components provide more data governance capabilities and the ability to define and manage security policies.

In addition to comparable or better performance, IBM Spectrum Scale provides more enterprise-level storage services and data management capabilities, as listed in Table 1.

IBM Spectrum Scale for big data and analytics solution often is used for the following reasons:

IBM Spectrum Scale is a preferred platform for running big data and analytics workloads. IBM Spectrum Scale in-place analytics for file and object data solves traditional analytics solution challenges.

HDFS Remote Procedure Call (RPC)-based IBM Spectrum Scale HDFS transparency Hadoop connector provides enhanced high availability capability, performance, and security for big data and analytics workloads. The IBM Spectrum Scale big data and analytics solution is deployed in Storage Rich Server architecture, SAN shared storage, and an integrated system Elastic Storage Server.

POSIX compatibility with various enterprise class features provides flexible data management and protection for big data and analytics workloads.

For more information, see the following resources:

 

Here’s an opportunity to spotlight your skills, grow your career, and become a published author—all at the same time! Join an ITSO residency project and help write a book in your area of expertise, while honing your experience using leading-edge technologies. Your efforts will help to increase product acceptance and customer satisfaction, as you expand your network of technical contacts and relationships. Residencies run from two to six weeks in length, and you can participate either in person or as a remote resident working from your home base.

Find out more about the residency program, browse the residency index, and apply online at: