Today’s Database Landscape at a 50,000 Foot Level

DB_Part1We all know the drill—data is exploding in size, but it’s not just the volume of data that is wreaking havoc on organizations. It’s how quickly it’s growing, how many different forms it can take, and how it’s constantly changing. And that’s just scratching the surface. How can the potential of data truly be harnessed? The database technologies for organizing the data that we generate and manipulate continue to morph and multiply. The hugely successful relational database management systems (RDBMSs) continue to soldier on using principles now over 40 years old, while newer database technologies have come along and been widely adopted to address specific needs in the data storage, management, and analytics space. Over the next few weeks, I’ll be going through the evolution of database technology at a 50,000 foot level to highlight the old and the new, how they’re used today, and what vendors to keep an eye on.

The RDBMS and How It Affects Your Everyday Life

The first RDBMS came out in the 1970s. The idea was to shield programmers from the underlying systems complexity. With a few simple database commands, it became possible to create, read, update, and delete (CRUD) records of data. The relational database allowed users to easily explore relationships between data. It became robust and reliable—good enough to run large financial and ticket reservation systems. And they’re still in use today, keeping your bank account in order, your medical records accessible, and your flight seat reserved.

ACID and Strict Data Discipline

If data transactions like depositing money into your account or booking a trip to Australia are to be guaranteed, a database should respect four key principles:

  • Atomicity. Also known as the all-or-nothing rule. A transaction, like moving money between bank accounts, is either completely executed or not at all.
  • Consistency. Makes sure only valid data is written in the database. Stops you from trying to deposit "ABC" dollars into your account, instead of "123," for instance.
  • Isolation. Processes transactions securely and independently, finishing one before the next.
  • Durability. Once a transaction has been completed, it remains so, even if the database server crashes right after. When the server comes back up, so is the completed transaction.

Together, these make up ACID, a set of properties that’s great for keeping law and order among data. This of course comes at a price, specifically around making it more difficult for relational databases to grow to very large sizes.

NoSQL to Satisfy the Need for Speed

The development of web-based applications has paved the way for the arrival of big data, which not only creates challenges for RDBMS, but also requires higher levels of performance that cannot be satisfied by traditional RDBMS. They have given rise to a new type of database: NoSQL. NoSQL databases get their name from the fact that they operate beyond the strict confines of the RDBMS and their "structured query language" or SQL. NoSQL databases offer a new way to store data, speed up database operations, and easily spread huge loads of data processing across distributed server environments. On the other hand, ACID properties must often be sacrificed.

Next-generation SQL for the Best of Both Worlds

Ideally, a database should have the reliability and trustworthiness of ACID, together with the ability to efficiently handle huge quantities of data. Those are the design goals of next-generation SQL databases, the latest arrivals on the database scene. Is such a combination really possible? In the next series of blog posts, I’ll be diving into these three database technologies to answer that question and see which database choice makes sense as requirements differ.

Look out for the next blog in this series, Why Won't the RDBMS Go Away?

big data analysis

Topics: Data Platforms, Analytics, & AI ESG Validation Services