IBM Cloud Object Storage File Access (COS FA) is a software defined offering that provides SMB and NFS protocol interfaces to applications to store, archive and retrieve infrequently accessed files on IBM Cloud Object Storage.

The IBM COS FA Solution includes the following components:

The IBM COS FA Portal is the management component of the offering, which enables the creation, delivery and management of the services mentioned below. The IBM COS FA Portal interacts with the IBM COS FA Gateways and efficiently handles file data exchange between these applications and users and the on-premise or public IBM Cloud Object Storage instance side. A centralized management console makes it possible to effectively manage a very large number of connected IBM COS FA Gateways.

The IBM COS FA Gateway is the component that the application and other data sources are connected to, and allows LAN speed writes via SMB and NFS protocols, and is in charge of onboarding the data to IBM Cloud Object Storage instantly and seamlessly.

The IBM COS FA Gateway works in caching mode, which means that it has a dedicated local disk space to allow local LAN speed ingestion. The main storage is on the IBM COS FA Portal in the cloud with stubs saved on the IBM COS FA Gateway. A stub is a file with a tiny footprint that contains the metadata about the file, such as the file name, size, and modification date. Only the folder and file metadata and not the actual file content is saved locally. This results in the cost of storage being significantly lower.

Also, systems with many file changes, where only some of the files are required locally, don’t over use bandwidth between the cloud and IBM COS FA Gateway. Only the required files are passed across the wire. When a user accesses a file stub, the file is opened without delay, where possible by streaming the file content from the cloud. After the download has completed, the file is unstubbed. Any changes to the file are synced back to the IBM COS FA Portal. Folders that are always required can be pinned, in which case the files in the folders, and not the stubs, are stored on the IBM COS FA Gateway.

IBM COS FA Gateways are virtual appliances, which can be installed on any customer provided ESXi, Hyper-V, or KVM/OpenStack environment.

The COS FA deployment configuration should be able to handle the load generated by a specific user environment. This section describes the COS FA Archiving use case.

The focus of this use case is the ability to actively archive data from the edge/datacenter/any location that is sourcing data to be archived, into cloud storage to lower cost of archival data.

The following examples can demonstrate the capabilities of the use case:

The Archiving use case utilizes the direct to storage write capabilities to fully utilize the available network capacity to the archive tier storage target and gain fast and reliable ingestion of data to be archived.

A COS FA installation comprises a cluster of one or more servers (VMs). The relevant roles are divided into IBM COS FA Portal VMs and IBM COS FA Gateway VMs.

The application server (VM) is in charge of all the communication between the moving parts in the system. It receives all requests from IBM COS FA Gateways and administrators and communicates them to the database (DB) node. The application VM is stateless and is in active-active mode with at least one more application VM as part of a production implementation. This means that every production implementation will have at least two application VMs.

A main database server (VM) holds administrative information related to the various cloud storage services, and metadata for files stored on the IBM COS storage infrastructure. Only one VM can host the main database node. The VM that hosts the main database node is called the master server and is fully replicated to a replica DB for full HA and DR capabilities. This means that every production implementation will have two DB VMs.

The IBM COS FA Gateway is the component that makes the SMB and NFS shares that the applications and users write to, available. This means it has to be always on and includes another DR IBM COS FA Gateway in a full production environment. This means that each ingestion site will have at least two FA Gateways.

While the application is busy navigating the relevant requests to the other parts of the system, and the DB is in charge of the application database and metadata of all the files that are archived, the IBM COS FA Gateway uses the Direct to Storage feature, allowing direct writes to the IBM COS bucket. This feature dramatically increases data ingestion based on the available network between the IBM COS FA Gateway and the storage, with speeds of around 350MB/s write to COS for archiving workflows.

Figure 7-1 shows a high level architecture of this deployment use case.

The IBM COS FA Portal was designed to scale from tens to hundreds and thousands of connected IBM COS FA Gateways and to support an easy to scale file system with PBs of data and more.

The IBM COS FA Portal is capable of supporting both scale-up and scale-out deployment schemes: administrators may deploy the IBM COS FA Portal software on increasingly more powerful compute platforms, thus scaling the deployment up. Alternatively, they can distribute the IBM COS FA Portal processes on multiple concurrent compute platforms, thus scaling the deployment out.

In addition, the file system is fully scalable by enlarging the database to accommodate data capacity growth.