Customer data needs to be firewalled and if protected properly can still be used for valuable analytics

In 2006, British mathematician Clive Humby declared that data is the new oil—and so could be the fuel source for a new, data-driven Industrial Revolution.

Given that he and his wife helped Tesco make £90m from its first attempt at a Clubcard, he should know. And it looks like the "derricks" out there are actually pumping that informational black gold up to the surface: the global big data analytics market is predicted to be more than $745bn by 2030 —and while it may not be the most dependable metric, Big Tech is throwing billions at AI at a rate described as "some of the largest infusions of cash in a specific technology in Silicon Valley history".

That's what we're being told is happening. But is it really? Are organisations genuinely and consistently monetising all the data they have—and so using it for competitive advantage and innovation?

More worryingly, are the safeguards we have put in place, like GDPR and other global data privacy restrictions, being deployed in ways that make them needlessly restrictive to the full flood of AI-driven innovation?

We were keen to understand where organisations really are with that exploitation, and if data monetisation and a defined, repeatable workflow really is in place.

Specifically, we wanted to check if organisations really are leveraging their data's full potential and are operating with the appropriate levels of protection for that data, and if that data is accessible in a time frame useful for the business.

Therefore, we commissioned research and spoke to 600 CIOs, CTOs, CISOs, heads of data and data managers in organisations in everything from the airline sector to retail and telecommunications.

And it was interesting to hear firsthand what they told us about the true state of data handling out there.

Innovation through data, yes. But enough of it? No

What these practitioners told us is that while they absolutely see the potential for innovation and profit opportunity in their data, they are struggling to capitalise on this cost-effectively.

For example, 56% of all respondents said they have already achieved significant improvements in both CX and EX (customer and employee experience); 44% confirmed data access has directly led to a profit advantage of 6 to 10%—a very impressive 20% put that figure at more like 11 to 20%. In 2023, over half (57%) of the organisations we spoke to had introduced new products and services based on insights they'd got directly from mining their data.

But 32% admitted it can take 3-6 months to access the data they need to power apps to do that. 37% said that process could take between one to two full months, and only a tiny percentage, a mere 2%, said that they could access data in "less than a week" or instantly.

To clarify what we mean by "data'': we're not talking transactional data such as EPOS data—we mean the potentially good data, about customers and their behaviours and attitudes.

When we say that, CISOs and Data Protection Officers will immediately slap the label "sensitive and classified data" over this—which leads to this data being put into purdah and many doors locked around it.

Which is as it should be—I don't mind others knowing what kind of sandwich I bought at the petrol station for example but I do mind if you connect that purchase to my NI number.

If the data protection wall is being set at too high a level, does that mean a miss of millions in potential profit through data exploitation (and potentially billions for UK plc) in lost opportunity?

Is putting all your effort into cyber potentially a big mistake?

Yes and no. We do need to protect our data —these laws exist because of shockingly bad behaviour by some technology vendors.

But there's a troubling confusion about what we are really protecting here. IT has somehow become obsessed with cyber security but hasn't yet really got its head around data security.

Thus, we have $180 billion annually spent on cybersecurity, while data security strategies are so much less robustly understood and resourced. At the end of the day all that cybersecurity exists to protect your networks, your infrastructure and endpoint... but 99 times out of 100, the bad agents don't care about any of that—they are after one thing, your data.

What is happening, then, is that CIOs are putting all their attention into cybersecurity and GDPR and other data privacy laws, thinking that solves the data problem. But if you lock down your data in ways that make it essentially 'dark' to the kind of AI analysis you want to run over it, then you're sitting on your oil, not pipelining it. The good news is there is a solution that will satisfy all the stakeholders in the custody and use of customer data.

Use the right technology to protect all that sensitive information

The way out of this is to understand that there are tried and trusted ways to let your data scientists and machine learning platforms get into the data in ways that 100% preserve my NI number from that visit to the petrol station, but still allow you to do something with the transaction (or sell it to third parties).

Step forward Privacy-Enhancing Technologies (PETs) like encryption and anonymisation, which can be used to protect sensitive information in ways that seal off anything that could worry the CIO, as that "sensitive and classified" personal data is replaced with a value that prevents individual consumer data from being directly identified.

What CIOs in this survey tell us is that most organisations are heavily invested in PET through encryption, with pseudonymisation and tokenisation less utilised or understood. 

Personally, we think that's a mistake—as pseudonymisation in particular is a highly efficient way of making your customer and partner data safe in privacy terms but open to rapid analysis.

Nonetheless, we were very encouraged by how our analysis showed the need to look again at data security, with almost all (96%) of all respondents saying they plan to invest a portion of their IT budget on the issue this year—with 49% thinking that needs to be between 11-15% of their entire IT budget.

Summing up, these numbers show that business owners know data is their biggest asset—but they are also crying out for a safe way to use it so they can analyse it all for advantage in ways that don't expose anybody in the company or their supply chain. 

Not long after that famous comparison between data and oil, Australian economist Michael Palmer pointed out that while oil in its raw state is valuable, "if unrefined it cannot really be used". Could thinking more about data than infrastructure security be the refining step we've been missing?

The full results of the research discussed in this article—The State Of Data Security Optimisation And Monetisation—can be found here