What is a Data Platform?

Is There a Difference Between a Data Platform and a Big Data Platform?

A “big data platform” is no different than a “data platform” — both are intended to handle data at scale.

The concept of “big data” was popularized in the 1990s, when the volume of data generated by humanity started on a path of exponential growth. But at this point, all data is big data. Individual consumers have access to hardware and cloud systems with petabytes of storage. Professional organizations — businesses and public sector alike — are generating staggering amounts of data. IDC estimates that by 2025, there will be 163 zettabytes of data in the digital universe.

There are three core characteristics that define “big data” (which we can call “data” going forward):

• Volume — The quantity of generated and stored data.
• Variety —The type and nature of the data.
• Velocity — The speed at which the data is generated and processed.

Because of the extraordinary growth along these three vectors, any data platform that can keep up with current organizational demands can be considered a big data platform.

The advantages of an enterprise data platform revolve around the combination of end-to-end features that replace point solutions previously used to provide data services. Many organizations make do with operational data stores (ODS), data warehouses (DW) or data marts (DMs) that may not work together effectively and limit the ability to scale. An EDP integrates the capabilities of those solutions and brings all the data into one place, where it can be secured, shared and used most effectively.

An EDP offers other significant benefits to large organizations, including:

• Centralizing and standardizing data functions in one cloud-based platform
• Greater resulting ease in adding new users (even acquired companies)
• Central management of the technology, rather than spread across departments
• Better, simplified reporting
• Better, faster, more comprehensive data analysis

In essence, an effective enterprise data platform will let you work with any and every data set, regardless of what it is, where it is (structured database or vast data lake), or how much of it there is. And at a speed, and with a degree of trust, that gives you actionable, real-time insights.

What is data architecture?

A data architecture is essentially a framework for an organization’s data environment. A data architecture is not a data platform. A data architecture is the plan for ingesting, storing and delivering the data, while the data platform is the machine that accesses, moves, analyzes, correlates and validates data for end users.

That’s the importance of a solid data architecture — it’s the backbone of a data-driven organization, the robust infrastructure that supports its existing data requirements and scales to match data and infrastructure growth. With technologies like edge computing and the Internet of Things (IoT), solid architectural principles are increasingly important.

A modern data architecture is built with these three characteristics in mind:

• Versatility: Data architectures are intended to manage the flow of data through an organization so that every business unit that needs to can easily access the data relevant to their goals with ease. Business needs and data sources (from new back-end apps to new social media platforms) frequently change, and the data architecture should scale and adapt to those changes swiftly and without incident.
• Intelligence: A data architecture should require minimal maintenance and upkeep — it should automate data ingestion and distribution as much as possible, reliably cataloging and delivering data to its destination. In addition to automation, a data architecture should leverage machine learning and artificial intelligence (AI) to respond to changes, repair erroneous data and continuously improve its ability to predict user needs.
• Security: Any worthwhile data architecture must balance all of the above qualities with security. Maintaining strict security and complying with privacy regulations is essential — and can be accomplished with robust data encryption practices and data lifecycle management. A strategy for securing data as the architecture scales is vital for protecting any organization, its customers and their combined futures.

What is a data strategy?

A data strategy is a plan for how an organization will gather, store, secure, manage, analyze and share data and use it to meet organizational goals. It is a central, integrated concept that articulates how data will enable and inspire business strategy. A company’s data strategy sets the foundation for everything it does related to data.

Every organization will have a different data strategy, but a typical data strategy will accomplish the following:

• Define how data will help the organization meet its business objectives and establish the path to that success
• Drive change in the organization to maximize the value of its data, and plan for how the company will make those changes
• Provide financial justification for the suggested changes and how the company will benefit from them using insights to increase profits and monetize the data

A data strategy combines data science with business objectives, and should be specific and include tactics for implementation, but it should also be flexible enough to adjust to fast-paced changes in the market.

How do you choose a data platform?

Choosing the right data platform comes down to six core considerations: on-premises vs. cloud, scalability, flexibility, usability/breadth, security/compliance and intelligence/automation. Driving all of them is the essential consideration that you want to be able to work with any data in your organization; regardless of source, format or time scale, you want to be able to ask any question and get actionable insight.

• On-premises, cloud or hybrid. Multiple factors determine whether you manage your data on site, through a cloud provider, or a combination of both. Those factors include: security and compliance requirements; costs of different software licensing models; which skills/functions you want to maintain in your in-house IT team, and which you’d acquire through your vendors.
• Scalability. A data platform must be able to perform at today’s scale and be adaptable to the inevitable growth of your data stores. (The need for scalability is one of the main forces behind the increased adoption of cloud-based data platforms.)
• Flexibility. Flexibility is essential. Can the platform currently serve multiple groups and use cases? Is it relatively straightforward to add new functions and use cases to the platform? Is there a robust ecosystem of applications and add-ons that can support new functions?
• Usability and breadth. Is the platform you’re considering simple to deploy and configure for users of varying skill levels? What’s the learning curve? Applying data to every decision requires that anyone in your organization — from IT wizards to less-technical employees — be able to work with that data.
• Security and compliance. Organizations need to ensure that their data is protected to prevent the sorts of data breaches that dominate headlines and put companies, customers and even nations at risk. That means ensuring that your data platform has robust security features built in, or tools that integrate with your existing security solutions. The same is true for compliance — a data management platform that adheres to the frameworks and guidelines established by a country or region’s regulatory bodies is essential if your organization does business in that country or region.
• Intelligence and automation. Vast quantities of data — for which a data platform is a requirement — exceed the capabilities of even the most dedicated analysts. Innovations in technology, particularly around machine learning (ML) and artificial intelligence (AI), have created new opportunities for organizations of every size to benefit from data-driven insights.

What is the future of data platforms?

In the future, data platforms will need to handle data sets of greater velocity, variety and volume, while allowing a range of users — from data scientists to business managers — to bring real-time data to every question, decision and action. A data platform must allow users to investigate, monitor and analyze data — and take effective action based on the insights revealed.

Splunk believes that as new technologies bring more data, in more formats, data platforms will have to evolve as well. To meet the challenges of the future, data platforms will need to integrate “smart” technologies — machine learning and artificial intelligence — to proactively assist organizations with their data-related goals. The Splunk approach allows you to complement the expertise of your organization and data with AI and machine learning for enhanced effectiveness and productivity, across industries, use cases and skill sets. Learn more about our approach with the Splunk platform.

Focus on your needs, not labels

With so many options available, choosing a data platform can seem like an overwhelming prospect. Set aside the enormous selection and the various labels for products, services and solutions, and approach the search by starting with your needs:

• Know your goals: Knowing what you want to achieve is vital when choosing a big data application. You can’t address your needs effectively if you don’t know what you hope to accomplish.
• Start small: Focus on a small-scale project to start — proving the efficacy of working with a data management platform is the best way to encourage wider adoption in your organization.
• Think big: Data is extraordinarily powerful and can be useful to every part of a business. Make sure that the platform you choose can be used wherever data can be useful — throughout your organization.
• Build a data culture: Making data analysis accessible to your organization is half of the equation — you have to create a culture that leads with insights from data. Have data and analytics teams collaborate across the business to help empower users to seek their own insights.