Introduction: Beyond the Digital Filing Cabinet
For most of us, cloud storage is a simple and invisible utility. We save photos, share work documents, and post hilarious dog memes, all without a second thought for the massive digital infrastructure humming away in the ether. We treat it like an infinite digital filing cabinet—a straightforward place to put our data.
But, for a traditional block and file guy like me, the technology that powers the cloud, particularly object storage, operates in ways that are often surprising and deeply counter-intuitive. The totality of cloud storage isn’t just a bigger version of the hard drive on my personal computer; it's a fundamentally different architecture, designed from the ground up for the scale of the internet.
Peeling back the layers reveals a system of clever illusions and powerful design choices. So, let’s look "under the hood" to uncover six surprising (to me, anyway) truths about how the cloud really stores our data.
--------------------------------------------------------------------------------
1. It's an Abstraction, Not a Destination
It's natural to think of object, file, and block storage as distinct types of physical hardware, but that's often not the case. At the most fundamental level, all data has been historically stored on physical media like HDDs or SSDs, which operate by reading and writing fixed-size chunks of data—this is block storage. Just to mention that Everpure systems are built on distributed flash modules (DFMs) which are revolutionizing the industry!
File systems are an abstraction built on top of block storage to organize those blocks into the familiar hierarchy of files and folders. Object storage is another layer of abstraction, but its implementation varies. While some systems run on top of a file system, hyperscale object storage services like Amazon S3 often bypass traditional file systems to manage blocks more directly, avoiding performance and scaling bottlenecks.
This separation of concerns is the key to cloud scale. By decoupling the interface (the object API) from the physical media management (how blocks are placed on disk), cloud providers can build massively distributed, resilient, and cost-effective backend systems without exposing that complexity to the user. You aren't choosing a different kind of disk; you're choosing the right interface for the job and the operational model it enables.
They are just interfaces.
--------------------------------------------------------------------------------
2. The "Folders" Are an Illusion
When you see a path in an object store like photos/vacation/beach.jpg, your brain instinctively sees a folder named "photos" containing a subfolder named "vacation." This is a useful illusion, but it's not what's actually happening. Object storage doesn't have a hierarchical directory structure; it uses a completely flat address space, often called a "storage pool." This architecture is the foundation for modern "data lakes."
In this architecture, there are no folders. The entire string photos/vacation/beach.jpg is the object's single, unique name, also known as its "key." The slashes are just characters in the name that user interfaces can interpret to mimic the appearance of folders for our convenience.
This flat structure is a critical design choice. By eliminating the need to navigate complex, nested directory trees, object storage systems avoid the performance bottlenecks that plague traditional file systems at extreme scales. This is precisely what allows object storage to scale almost infinitely, managing trillions of objects without breaking a sweat.
--------------------------------------------------------------------------------
3. Metadata Is Its Superpower
Perhaps the most significant advantage of object storage lies in its handling of metadata—the data about your data. A traditional file system offers a minimal, fixed set of metadata: file name, creation date, size, and permissions. Object storage, however, allows you to attach rich, extensive, and fully customizable metadata to every single object.
This means an X-ray image can be tagged with a patient ID, a doctor's name, and the type of scan. A video file can carry metadata for the project name, director, and scene number. A legal document can be tagged with a case number, compliance retention policy, and confidentiality level.
This capability is a game-changer. It transforms a simple storage repository into a powerful, searchable database. Using S3 tables, you can query your storage directly to find all objects related to a specific project or patient. Even better, you can create automated data management policies—such as automatically moving files to a cheaper archive tier or deleting them after a certain date—based entirely on these custom metadata tags.
--------------------------------------------------------------------------------
4. The Proprietary API That Became the Standard
Many of the foundational standards of technology have been hammered out by large industry working groups over many years. The S3 API, the universal language of modern object storage, is NOT one of them. When Amazon launched its Simple Storage Service (S3) in 2006, it introduced its own proprietary, RESTful interface for developers to interact with the service.
The impact was quite profound. As other companies—from cloud providers to on-premises hardware vendors—rushed to offer their own object storage solutions, they found that the market had already chosen a winner. To be competitive, they had no choice but to make their products "S3-compatible." This single-company API, born out of a specific product launch, became the de facto industry standard by sheer force of market adoption.
The fact that Amazon controlled the S3 API (also referred to as the S3 protocol) enabled it to continually modify the API to meet fluid market demands by releasing service after service.
--------------------------------------------------------------------------------
5. You Can't Just "Edit" an Object
On your computer, if you change a single word in a large document, the file system updates only the tiny portion of the file that changed. In object storage, this is impossible. Objects are immutable, meaning that once an object is written, it cannot be modified.
To make a change—even a small one—you must upload the entire object again. The system then replaces the old object with the new one, often treating it as a new version. This might seem inefficient, but it's an intentional design choice that is a cornerstone of data integrity at scale. Immutability eliminates entire classes of problems related to partial updates, data corruption, and complex file-locking mechanisms that plague traditional systems. This simplicity is essential for reliably managing trillions of objects.
While the logical model is an atomic replacement, systems make this practical for large files with features like multipart uploads, which allow a file to be uploaded in parallel chunks for more resilient and efficient replacement.
--------------------------------------------------------------------------------
6. The Price Is Right for Storing, Not Always for Leaving
Cloud object storage is famously inexpensive for storing massive amounts of data. The per-gigabyte cost is often fractions of a cent, making it an incredibly attractive option for building vast data lakes and archives. However, there's a crucial, and often overlooked, part of the pricing model: data egress fees.
Egress fees are what cloud providers charge you to access, download, or move your data out of their storage platform. While storing a petabyte of data might be affordable, reading that data back frequently or moving it to another provider can result in surprisingly high costs.
These fees create a form of cost-based vendor lock-in and must be treated as a critical architectural consideration. An application that reads data frequently can turn an "inexpensive" storage service into one of the most significant cloud expenses. It is essential to model your application's data access patterns before choosing a provider to avoid major, unexpected costs down the road.
--------------------------------------------------------------------------------
Conclusion: An Object Foundation, Built for the Future
From illusory folders to immutable data, cloud object storage is a far more nuanced and powerful architecture than the simple "digital filing cabinet" it appears to be. Its design principles—a flat namespace for infinite scale, rich metadata for intelligence, and a standard API for interoperability—have made it the foundational storage layer for the modern internet.
Today, it is the silent workhorse powering everything from cloud-native applications and AI data lakes to global content delivery networks and mission-critical disaster recovery plans. Understanding its unique characteristics is no longer just an academic exercise; it's essential for anyone building for the future. As the world's data continues to explode, how will these powerful but specific architectural choices shape the next generation of applications?
The future of data storage is object and Everpure’s enterprise data cloud strategy addresses data-driven outcomes, whether the data lives on block, file, or object.
References:
Intro to Object Storage on Everpure FlashBlade
An exploration of Files and Objects for Data Storage
Unified Data Management with Fusion
Contributors: Russell Pope, Kevin Long, Chris Frank
