ESG Blogs

Data warehouse modernization has become an essential move to meet the demands of the modern business. And it is easy to get lost in the hype when it comes to modernizing with a “cloud-first” or “cloud-only” approach. Organizations are drawn to the promise of ultra-simplicity, unlimited scale, improved agility, and ubiquitous accessibility. But for some organizations that are on this path, they are starting to see the tradeoffs they have made. One of those tradeoffs comes with price/performance. Not that modern cloud data warehouses do not perform well, but if you want the low-latency performance to truly support real-time, you will either have to pay for it (especially at scale as more end-users want access to the data) or minimize your ability to truly achieve real-time responsiveness. And it is forcing organizations to rethink their cloud data warehouse strategies. Maybe an on-premises data warehouse does have a place? And so does a cloud data warehouse?

The day has finally arrived. Today, Snowflake will IPO under the ticker SNOW. Many knew this was coming, even before Frank Slootman was brought in as CEO. But I don’t think it was expected to have as much fanfare as we’re seeing. Buffet (Berkshire Hathaway) is on board. Benioff (Salesforce) is on board. It’s being called the hottest tech IPO of the year. So, what’s the draw? Is it the differentiable technology? Not really. The number of customers? Nope. Customer growth? Impressive, but no. What about the total market size? Getting there. What about the value propositions it offers customers? Well yes, but I would argue there are several competitors that offer similar value propositions. So, what is it? The opportunity. The opportunity is simply massive.

A couple weeks back, Google Cloud’s multi-week virtual event Next 20: OnAir started. There were a number of announcements, but the biggest was BigQuery Omni. By combining BigQuery and Anthos, BigQuery Omni enables organizations to embrace multi-cloud analytics by cost-effectively bringing Google Cloud's data warehouse to where the data resides across public cloud environments.

First came the traditional enterprise data warehouse (EDW). Structured data is integrated into an EDW from external data sources using ETLs (check out my recent blog post on this). The data can then be queried by end-users for BI and reporting. EDWs were purpose built for BI and reporting. But with the growing desire to incorporate more data, of different types, from different sources, of different change rates, the traditional EDW has fallen short. It does not support unstructured data (i.e., video, audio, unstructured text, etc.), streaming is for the most part out of the question, there is no data science or machine learning that can be done directly on the data, and because of their closed/proprietary nature, costs quickly skyrocket as organizations scale their deployments. Modern, cloud-based EDWs have looked to address several of these challenges and done a good job of it, but some challenges still remain, with the obvious being lack of unstructured data support.

Data integration is hard. Over the years, of all the technologies and processes that are part of an organization’s analytics stack/lifecycle, data integration continuously has been cited as a challenge. In fact, according to recent ESG research, more than 1 in 3 (36%) organizations say data integration is one of their top challenges with data analytics processes and technologies. The data silo problem is very real, but it’s about so much more than having data in a bunch of locations and needing to consolidate. It’s becoming more about the need to merge data of different types and change rates; the need to leverage metadata to understand where the data came from, who owns it, and how it’s relevant to the business; the need to properly govern data as more folks ask for access; and the need to ensure trust in data because if there isn’t trust in the data, how can you trust the outcomes derived from it?

The recent announcement from IBM to withdraw from all research, development, and offerings of facial recognition will not stop facial recognition from being used by law enforcement or government entities. There. I said it. Facial recognition will continue on its gray area trajectory with or without IBM. But what IBM has done, specifically Arvind Krishna, is bringing attention to a growing concern that needs far more national and global attention. The use of facial recognition needs to be scrutinized for bias and privacy concerns. It needs oversight. It needs guardrails. Usage, especially from law enforcement and governing entities, needs to be transparent. And frankly, the technology needs to be better for it to work the way people envision.

Over the last several months, automation has seen a jump in interest. Operational efficiency has been a top priority for years, but as of late, it’s an even greater priority. For businesses, tasks or processes that used to be viewed as manageable but inefficient are now being scrutinized. The inefficiency aspect is being amplified and organizations don’t have a choice but to act. And one of those actions is to look into a trendy buzzword that is proving to be so much more: robotic process automation (RPA).

I’ve talked through a couple ways that data is helping fight COVID-19, from detecting and tracking an outbreak, to detecting within people. In this blog, I’ll be focused on research and elimination. While the world sets its sights on a hopeful slowdown similar to what is experienced with a seasonal flu, a true vaccination or cure is the ultimate goal. With more and more researchers throwing their collective hats in the ring, data sharing and collaboration is becoming key. What new information is available on the virus makeup? How is it evolving/mutating? Where are we making progress on vaccination development? What approaches have a higher likelihood of success? How can progress be shared across the globe to spur new ideas or rapid insight? All of these questions tie back to data, and using data science and AI to help answer them is rapidly becoming a go-to approach.

My first blog in this series focused on using data, AI, and visualization to help detect and track the continued outbreak of COVID-19. The next topic in this blog series is focused on a different form of detection: detecting the virus in people. And it starts with understanding symptoms and ends with techniques to identify infected individuals at scale, which eventually helps shape individual treatment plans and resource allocation based on established virus hotspots.

While the world continues its fight against COVID-19, data is becoming one of the most prominent weapons for humans. Whether tracking an outbreak, detecting the virus on a case by case basis, preventing further spread, or eventually eliminating the virus altogether, data is fueling decision making from governing bodies to personal households. As we evolve to deal with the difficult norms of what today brings, the technology community is rising to answer a desperate call to arms. The value of trusted data has never been higher as actions are being taken based purely on statistical models and predictions. The need for collaboration on a global scale is paramount. Being able to rapidly respond to new information is now a matter of life or death. Over the next few weeks I’ll be revisiting and highlighting some of the ways data is being used in the ongoing fight against this historical and scary virus.