ESG Blogs

First came the traditional enterprise data warehouse (EDW). Structured data is integrated into an EDW from external data sources using ETLs (check out my recent blog post on this). The data can then be queried by end-users for BI and reporting. EDWs were purpose built for BI and reporting. But with the growing desire to incorporate more data, of different types, from different sources, of different change rates, the traditional EDW has fallen short. It does not support unstructured data (i.e., video, audio, unstructured text, etc.), streaming is for the most part out of the question, there is no data science or machine learning that can be done directly on the data, and because of their closed/proprietary nature, costs quickly skyrocket as organizations scale their deployments. Modern, cloud-based EDWs have looked to address several of these challenges and done a good job of it, but some challenges still remain, with the obvious being lack of unstructured data support.

Data integration is hard. Over the years, of all the technologies and processes that are part of an organization’s analytics stack/lifecycle, data integration continuously has been cited as a challenge. In fact, according to recent ESG research, more than 1 in 3 (36%) organizations say data integration is one of their top challenges with data analytics processes and technologies. The data silo problem is very real, but it’s about so much more than having data in a bunch of locations and needing to consolidate. It’s becoming more about the need to merge data of different types and change rates; the need to leverage metadata to understand where the data came from, who owns it, and how it’s relevant to the business; the need to properly govern data as more folks ask for access; and the need to ensure trust in data because if there isn’t trust in the data, how can you trust the outcomes derived from it?

The recent announcement from IBM to withdraw from all research, development, and offerings of facial recognition will not stop facial recognition from being used by law enforcement or government entities. There. I said it. Facial recognition will continue on its gray area trajectory with or without IBM. But what IBM has done, specifically Arvind Krishna, is bringing attention to a growing concern that needs far more national and global attention. The use of facial recognition needs to be scrutinized for bias and privacy concerns. It needs oversight. It needs guardrails. Usage, especially from law enforcement and governing entities, needs to be transparent. And frankly, the technology needs to be better for it to work the way people envision.

Over the last several months, automation has seen a jump in interest. Operational efficiency has been a top priority for years, but as of late, it’s an even greater priority. For businesses, tasks or processes that used to be viewed as manageable but inefficient are now being scrutinized. The inefficiency aspect is being amplified and organizations don’t have a choice but to act. And one of those actions is to look into a trendy buzzword that is proving to be so much more: robotic process automation (RPA).

I’ve talked through a couple ways that data is helping fight COVID-19, from detecting and tracking an outbreak, to detecting within people. In this blog, I’ll be focused on research and elimination. While the world sets its sights on a hopeful slowdown similar to what is experienced with a seasonal flu, a true vaccination or cure is the ultimate goal. With more and more researchers throwing their collective hats in the ring, data sharing and collaboration is becoming key. What new information is available on the virus makeup? How is it evolving/mutating? Where are we making progress on vaccination development? What approaches have a higher likelihood of success? How can progress be shared across the globe to spur new ideas or rapid insight? All of these questions tie back to data, and using data science and AI to help answer them is rapidly becoming a go-to approach.

My first blog in this series focused on using data, AI, and visualization to help detect and track the continued outbreak of COVID-19. The next topic in this blog series is focused on a different form of detection: detecting the virus in people. And it starts with understanding symptoms and ends with techniques to identify infected individuals at scale, which eventually helps shape individual treatment plans and resource allocation based on established virus hotspots.

While the world continues its fight against COVID-19, data is becoming one of the most prominent weapons for humans. Whether tracking an outbreak, detecting the virus on a case by case basis, preventing further spread, or eventually eliminating the virus altogether, data is fueling decision making from governing bodies to personal households. As we evolve to deal with the difficult norms of what today brings, the technology community is rising to answer a desperate call to arms. The value of trusted data has never been higher as actions are being taken based purely on statistical models and predictions. The need for collaboration on a global scale is paramount. Being able to rapidly respond to new information is now a matter of life or death. Over the next few weeks I’ll be revisiting and highlighting some of the ways data is being used in the ongoing fight against this historical and scary virus.

As of January 1^st, the California Consumer Privacy Act is now in effect. The CCPA lets anyone in California request all the information a company has on them as a consumer, including what data has been sold to /accessed by other companies. And when it comes to penalties, if a company is notified of being out of compliance (i.e., unable to provide all the data of their consumers), they have 30 days to comply or they will get fined per record. And that “per record” component is important because it highlights how quickly a fine could balloon into billions of dollars in fines. The interesting component of this is that if a company doesn’t comply, it opens companies to face class action lawsuits from consumers.

For the last few years, the processing space has been red hot. Between startups and mainstay chip vendors, it’s an ongoing arms race to address the specialized needs of modern workloads and applications in core data centers, at the edge, and in the public cloud. In fact, when ESG asked respondents to cite the aspects across the entire data pipeline that are most frequently responsible for causing delays, the top response was data processing. Organizations want speed, reliability, and cost effectiveness. And to get there, it’s forcing organizations to rethink their approach to computing, especially with the rapid adoption of AI technologies fueling the booming compute market.

AWS re:Invent 2019 has come and gone. The event was full of announcements, people (the entire Las Vegas strip was taken over by 65,000), and fun. Many announcements were shared prior to the event, but in a 3-hour marathon keynote, AWS CEO Andy Jassy shared more…and more….and more. And he did it all without saying “multi-cloud” or “hybrid cloud,” the latter being most impressive since, well, Outposts. Focusing on analytics, databases, and AI, here are some of my key takeaways.