The data lake market is projected to grow by a compounded annual growth rate of 24.8 percent in the forecast period starting in 2021 and ending in 2027. There is a growing demand for centralized repositories that enable the storage of natural or raw data at different scales, in both their structured and unstructured formats.
This growing use of data lakes may mean advantages for organizations, but it can also pose difficulties. Maintaining and securing data lakes, in particular, can be challenging especially for companies that are new to the concept. For one, it can cause “data drowning” or the inability to cope with the overwhelming amounts of information and data management requirements.
More importantly, securing data in a data lake demonstrates the very situation forewarned by the cautionary idiom: don’t put all your eggs in one basket. As such, it is crucial to take measures to ensure not only simple data lake security but efficient security and maintenance.
Weigh options carefully with high-performance analytics
It is not impossible to achieve high performance with data lakes. Organizations just need to find the right database management system. Ample research and analysis should be done before picking a database management solution. Whether it’s a time series database like InfluxDB or a hybrid timeseries solution Druid and Clickhouse, it is important to conduct a meticulous evaluation of options. Doing Pinot vs Clickhouse or Clickhouse vs Druid comparisons, for example, will provide a clear grasp of the right choice to make.
It is worth noting, though, that hybrid time series databases are usually designed to achieve high levels of performance for specific “ad hoc” analytics like those that make use of group and data filters.
There’s the option to use Elastic analytics, which is excellent for search operations. However, it can be very costly to do analytics with it. Elastic is not created on a columnar store, so it necessitates the generation of indexes to enable queries. It is uncommon for organizations to build numerous indexes on Elastic, so the costs can easily rack up and loading time can also suffer.
Data warehouse expert Robert Meyer advises that choosing the right analytics engine is bound to become easier in the future as data warehouses are already becoming faster and better, capable of supporting more high-performance analytics. For now, though, organizations can make good choices by doing a thorough internal examination and coming up with an analytics roadmap that spans one to three years.
“You may find that the broader analytics capabilities, flexibility, and multi-workload support you get with a high-performance data warehouse better suits your broader needs than choosing yet another specialized analytics database one project at a time,” Meyer says.
Ensure proper data governance
As mentioned, data lakes are intended for all kinds of data. This sounds convenient, but it entails serious security consequences. Organizations need to establish a system to carefully vet data and sources and ensure proper data management, processing, and consumption. Good data governance allows organizations to rapidly identify important details such as data ownership, the security protocol to follow when handling sensitive data, as well as data and data source history, among others.
It is also important to have thoughtfully planned policies on role-based access, user authorization, access authentication, and at-rest and in-motion data encryption. These should be implemented in specific instances where they can deliver the best benefits for data security and privacy. Additionally, it is advisable to stick to the principle of least privilege. Requests for access should only be granted to a very limited extent, just enough to provide what is necessary to undertake a specific task.
Legal requirements on data protection should also be taken into account. There are national and international regulations on data access organizations have to bear in mind. One of the most effective approaches to achieve an efficient operation in view of such regulation is the use of zones within the storage layer and the configuration of access to these zones in such a way that it is highly limited but can eventually be adjusted in response to legitimate requests.
Use partitions and hierarchy and implement data life cycle management
Michael Chen of Oracle Big Data subscribes to the idea that data lakes need multiple standard zones to store data in accordance with its trustworthiness and readiness for use. These standard zones are as follows:
- Temporal – A zone where copies and streaming spools, as well as other ephemeral data, are stored before they are deleted
- Raw – This is the zone where raw data resides before processing. Sensitive data in this zone are usually encrypted.
- Trusted – As the name implies, this is the zone where validated data is stored, ready for the use of data scientists, analysts, and other end users.
- Refined – This is where trusted data is processed further through manipulation or enrichments. Examples of these are the final data output of data management tools.
Having a hierarchy like this benefits organizations by minimizing the chances of allowing the wrong people from accessing or tampering with data they should not be permitted to view or modify. Data access hierarchy and partitions also make role-based access management systems more effective.
Also, in his blog post about improving data lake security, Chen notes the importance of data life cycle management. “In a data lake environment, older stale data can be moved to a specific tier designed for efficient storage, ensuring that it is still available should it ever be needed but not taking up needed resources,” Chen explains. This enables efficiency, which makes data lakes function like a “well-oiled machine” that is unlikely to falter after getting overwhelmed by their own contents.
Employ machine learning
Machine learning is nothing new in the field of data management. It is particularly useful when it comes to data lakes as it expedites the processing and categorization of raw data, thereby minimizing the opportunities for cybercriminals to find opportunities they can exploit. Also, machine learning can be used to automate the identification of issues in raw data. It can red-flag certain parts of the raw data stored in a data lake for security investigation. There are machine learning techniques that effectively optimize the management of data in data lakes while improving data quality at the same time.
Additionally, machine learning helps address the problems associated with data silos. As explained in a conference paper presented at the International Conference on Data Mining and Big Data in 2017, data silos can turn into massive and overlapping data “monsters” in data lakes. “Machine Learning can distribute the architecture of data models and integrate the data silo with other organizations’ data to optimize the operational business processes within an organization in order to improve data quality and efficiency,” the paper writes.
Encrypt data
This is already a given in securing data in any type of data management situation, but it is still worth emphasizing the need to have data, particularly the sensitive ones, encrypted. Also, it is important to highlight the need for efficient encryption-decryption execution. Organizations cannot use any kind of encryption technology and processes just for the sake of doing encryption.
There has to be a sound data encryption strategy that is suitable for the processes and existing infrastructure of an organization. This strategy must provide the right protection in both the at-rest and in-motion states of data, especially when the data involve secrets, confidential information, and other sensitive content.
Towards efficient operation and security
Working with gargantuan amounts of data is unlikely to be an easy task. Securing such huge amounts of data is even more challenging. With the right tools and strategies, though, it is possible to achieve efficiency not only in the handling of data but also when it comes to securing it. This is true even when it comes to the highly complex nature of data lakes.
It is not impossible to have an efficient way to operate, maintain, and secure data lakes. The technologies to achieve high-performance analytics are no longer out of reach. Existing tools for data security and management as well as effective data governance strategies can be applied to data lakes. Organizations just need to find the right solutions for their specific requirements.
You may be interested in: 4 Things to Consider When Hiring Remote Workers