Unraveling the Storage Landscape: A Guide to Optimizing Backend Data Management

The backbone of any backend system is the storage which holds the data. The storage landscape has grown far beyond simple hard drives and SSDs. In this post, we aim to gain an understanding on which solutions work best to optimize our backend data management.

One size does not fit all

Before we dive in deeper, it’s important to understand that there is no such thing as a “one-size-fits-all” storage solution. Storage solutions are designed for varying use cases. Such as: Long-term storage, high-speed access, high availability, caching, max capacity, and so on. Though you can use some solutions for varying use cases, you will find that there are certainly better solutions out there for optimal performance.

At a high, and basic level, let’s compare hard disk drives (HDDs) and solid-state drives (SSDs). HDDs are great for long-term storage where you are not frequently accessing the data on these drives as they are typically slower, but have a higher capacity. SSDs, however, are great for high-speed access and are typically used for caching and high-speed data access. Cost is usually another factor to consider between the two, but as of lately, due to the rise of AI and machine learning_[1] the costs of hard drives, generally, have been increasing. However, we can take away from this that even at a high level, there are different storage solutions for different use cases.

Volatile vs Non-Volatile

A concept that is important to understand when it comes to storage is the difference between volatile and non-volatile storage. Let’s take a quick look at what these terms mean.

Volatile Storage

Volatile storage is storage which is temporary and is lost when a machine is powered off. This is typically seen in cases, such as Random Access Memory (RAM).

Non-Volatile Storage

Unlike volatile storage, non-volatile storage will not be lost when a machine is powered off. This is what most is most commonly though of when we think of storage solutions, such as hard drives and SSDs.

Storage mediums

There are many different storage mediums available to us today. For the purpose of this post, we will only focus on the ones which directly impact the backend data management of our systems.

Hard Disk Drives (HDDs)

Hard Disk Drives are an older technology, but still has its place in the storage landscape. You may hear HDDs referred to as “mechanical drives” or “spinning drives.” This is because HDDs has platters that rotate at a high speed (measured in RPMs) and a magnetic head which is used to read and write data to these platters. There are quite a few different interfaces which hard drive manufactures provide. Let’s look at two common interfaces seen on HDDs.

Serial ATA (SATA)

SATA is more commonly seen in consumer-grade systems. It is a slower interface as it uses half-duplex communication.

Serial Attached SCSI (SAS)

Serial Attached SCSI is more commonly seen in enterprise-grade systems. It is a faster interface as it uses full-duplex communication.

Fibre Channel (FC)

Fibre Channel in storage systems is different than the previous interfaces mentioned. The previous interfaces use what we call “Direct Attached Storage” (DAS) where the storage is directly connected to the system. Fibre Channel uses a network to connect the storage to the system. This is known as “Storage Area Network” (SAN). Fibre Channel is excellent for environments which require large amounts of storage and/or storage which needs to be accessed by multiple systems.

Solid-State Drives (SSDs)

Solid-State Drives are much faster than HDDs as SSDs have no moving parts and are much more reliable than HDDs. They also provide better IOPS (Input/Output Operations Per Second) than HDDs as well as lower latency. SSDs are on their way to becoming a standard in most systems_[2]. You will find that SSDs share the same interfaces as HDDs, such as SATA and SAS. However, there are other interfaces which SSDs use as well.

Non-Volatile Memory Express (NVMe)

NVMe is a newer interface which is designed specifically for SSDs. It is much faster than SATA and SAS as it uses the PCIe interface and prioritizes low latency. SSDs also have the ability to compute in parallel as they do not have the drawbacks of having to wait for a mechanical head to read and write data. This plays another big role in NVMe’s speed. NVMe is typically seen in high-performance systems such as caching and high-speed data access.

Hybrid Drives

Hybrid drives are a combination of both HDDs and SSDs. They are not typically seen in enterprise-grade systems, but are more commonly seen in consumer-grade systems. Hybrid drives are great for systems which require a balance between speed and capacity. They can, however, be great for:

• Smaller projects or budget-friendly systems.
• Applications that regularly swap between hot and cold data and can benefit from the SSD cache.
• Backup and archival systems that require quick retrieval of data.

That being said, mission-critical applications or high-performance systems should opt for SSDs or HDDs, depending on their specific needs as you still have to make a compromise with hybrid drives.

Tape Drives

Tape drives are something which, most people have never heard of. Tape drives are used for long-term storage exclusively. They are not used for high-speed access, but are great for storing data for long periods of time. Tape drives are typically used in enterprise-grade systems where data needs to be stored for long periods of time and is not frequently accessed, such as, archival storage. You will most often see these used in tape libraries where multiple tapes are stored and can be accessed by a robotic arm.

Storage Solutions

Now that we have an idea of the mediums that data is stored on, let’s take a look at some storage solutions which are available to us.

File Storage

File storage is the most common and basic storage solution. This is simply exactly what it sounds like. You are storing files on a storage medium. The best use case for this approach is when you need your data to be stored in a hierarchical structure, you are storing files in its raw format, or want a simplified solution for sharing files across systems.

Distributed File System

Distributed file systems are a more advanced version of file storage. Distributed file systems are file systems that span across multiple storage nodes and provide a unified view of the files stored on these nodes. This is great for when you need to store large amounts of data and need to access this data across multiple systems._[4]

Block Storage

Block storage stores data in fixed-size blocks. This is not far off from how data is stored on a hard drive. For this reason, block storage is great for when you have situations where the data needs to be addressed by an individual block. You typically see this in workloads which require high-speed access, such as databases and virtual machines. _[5]

Object Storage

Object storage has become wildly popular in recent years due to the rise of cloud storage. Object storage stores data in objects which are stored in a flat structure without any hierarchy. This provides a great platform for several use cases, such as, backups, content delivery networks (CDNs)_[3], serverless applications, and big data analytics.

Serverless Storage

You can think of serverless storage as “un-managed storage.” This is storage which is managed by a cloud provider and you typically don’t have to worry about where the data is stored. This is great for when you need to store data, but don’t want to worry about managing the storage system.

Attached Storage

There are a few classifications of storage that fall in the “attached storage” category.

Direct Attached Storage (DAS)

As mentioned earlier, DAS is storage which is directly connected to the system. This is great for when you need high-speed access to the data and the data is not shared across multiple systems.

Network Attached Storage (NAS)

NAS is storage which is connected to a network and can be accessed by multiple systems. This is great for when you need to share data across multiple systems and need a centralized location for the data.

Storage Area Network (SAN)

SAN is like NAS, but far more advanced. SAN is a network completely dedicated to storage. This is great for when you need high-speed access to the data and need to share data across multiple systems.

Final Thoughts

The storage landscape may seem vast and complex, but by understanding your specific use case and requirements, you can make an informed decision on the right storage solution for your backend systems. Remember, there is no one-size-fits-all approach - the key is to carefully evaluate factors like data volume, access patterns, performance needs, budget, and disaster recovery.

By asking yourself the right questions upfront, you can start to narrow down the options and identify the storage technology that will best support your applications and deliver the optimal performance, scalability, and reliability your business demands. And if you’re ever unsure, don’t hesitate to consult with a storage expert who can provide valuable guidance and help you make the most informed choice.

The world of data storage is constantly evolving, so stay curious, keep learning, and don’t be afraid to adapt your strategy as your needs change. With the right storage foundation in place, you’ll be well-equipped to tackle the challenges of the modern digital landscape and drive your business forward.

What has your experience been with selecting and implementing storage solutions for your backend systems? I’d love to hear your insights and lessons learned in the comments below.

References

1. Techradar posted an article which covers the topic of hard drives becoming more expensive due to AI.
2. According to a Statista graph on Shipments of hard and solid state disk (HDD/SSD) drives worldwide from 2015 to 2021.
3. Digital Ocean wrote an article on how to Maximize efficiencies with object storage + CDN.
4. GlusterFS has quite a few use cases for distributed file systems available on their website.
5. Akamai has detailed docs on use cases for block storage.