A Guide to Modernizing Legacy Data Analytics Technology

Introduction

Embracing change is part of life—and also, for successful companies. Over the years, companies have continued to use many data analytics tools as a strategy to gain a competitive advantage in the marketplace. Recently, technologies such as Hadoop, NOSQL, Amazon Redshift, and Azure Synapse (as well as many others) have been released for businesses to use to improve their overall data analytics strategy outlook. Although many companies are ready to migrate to these analytics tools, there are still many issues that plague businesses from adopting these modern offerings. This guide explores common issues companies face, as well as how to address them to be ready to migrate to a new modern data analytics platform.

Reasons to Modernize

There are a couple of main drivers in modernizing Data Analytics tools. One are changes in increased data volume and variety and the other is to desire to cut overhead costs associated with technology infrastructure. In years past, a traditional Data Warehouse (built on a Relational Database Engine) was used to drive data analytics strategy and decision making. However, due to today’s data needs, the rigid nature of a Relational Database Engine cannot scale large enough nor be one to handle disparate data types to utilize current data requirement. Tools such as Hadoop and NOSQL databases can be leveraged by companies for high volume and variety; however, they are not a one-size-fits-all solution for a deep analytics strategy.

Other Modern Data platforms are located in the cloud. The benefits of companies adopting a cloud solution strategy are well documented, in that cloud solutions help provide dynamic infrastructure scalability and flexibility in terms of upfront costs and pricing. From a technology perspective, Amazon and Microsoft have introduced Cloud Data Warehouse tools such as Redshift and Synapse, respectively, to handle large aggregate queries from a variety of sources. These cloud data platforms utilize massive parallel processing, as well as columnar storage/indexing technologies to achieve fast query performance.

Migration and Modernization Issues

Large Tightly Coupled Environment

Many companies today have an assortment of IT systems with different business functions that communicate in concert to perform day-to-day activities. In many cases, the communication/integration paths between systems are large in number and not very tolerant to change (or tightly coupled). As one can imagine, replacing a Data Warehouse within an organization’s technology landscape can be a very complex and time-consuming endeavor, which can pose a significant risk with regard to business continuity and strategy. To help minimize time and risk, take the following steps when considering migration to a modern Data Analytics platform:

1.) Create an inventory of all integration paths to and from the Data Warehouse

• Data Analytics Reporting tools such as Power BI and Tableau
• ETL (Extract, Transform, and Load) Packages
• API / Web Service Endpoints
• Front-End / Application Code
• Linked Server Connections
• Location of Database Backup Files
• Location of Database files if on a SAN (Storage Area Network)
• Availability Groups and Failover Server configuration

2.) Assess legacy hardware and review requirements for a modern data analytics reporting environment. At this point in time, it may be a good analytic strategy for your company to go to the cloud, or maybe it’s best to continue On-Premises servers. Discuss the decision making and strategy internally, and explore costs before committing to a modern analytics platform. In addition, check the feasibility of other systems within your company to connect to the modern environment.

3.) Document security configurations

• Windows / SQL Login Settings
• Firewall and Network Permissions
• Database Object Permissions

4.) Stop new integrations. Any new integrations to your Data Warehouse will only further help complicate your ability to move to a modern analytics platform. Meet with architects and developers to discuss alternate paths and strategy.

5.) Conduct interviews with one or several partner teams/organizations. Meet with other teams that access the Data Warehouse, and share the integration inventory with them. They will be able to review the list appropriately and get any new integrations that you might be unaware of.

Cryptic Code

Oftentimes, data analytics reporting environments contain code routines that are puzzling. Common complaints involve, “I have no idea what this code does,” “This platform/coding language is from the 1980s,” or “Bob wrote this five years ago, left the company, and no one knows how it works or how to use it.” These mysteries will inevitably need to be investigated. Take the following steps to ensure your environment is ready to move forward with a modern analytics platform.

1.) Create an inventory of all stored procedures and code routines that are within the Data Warehouse. An easy way to achieve this is to ensure that all packages and stored procedures get checked-in to Source Control. Source Control platforms such as TFS and GitHub are popular choices for developers today.

2.) Staff personnel that can understand/use the code. This is where you are going to need to roll-up your sleeves and dig in. This will take time and, in many cases, take some specialized expertise and help depending on the nature of the analytics reporting platform. Make sure you are organized, and document your data and analytics findings in an appropriate manner, as it will ease the transition down the road.

3.) Review code for updates in syntax or modern techniques. In many cases, the code itself can change from version to version. Some functions may exist in older versions of SQL but have since been sunset or deprecated as newer versions of the analytics platform become released. In addition, there may be different functions that did not exist in older versions of SQL that can now be implemented to help streamline your efforts and strategy more efficiently.

4.) Map each Stored Procedures/ETL Package to a business process. By going through this effort, you’ll be able to tell which processes are important and which ones are obsolete. In addition, you’ll be able to prioritize which routines are more important than others as it relates to your business.

5.) Re-assess design. Hindsight is always 20/20, but review the design of the current data analytics reporting environment and map out a strategy of how it will fit into your modern environment. It could be that the entire design needs to get reworked, or you can use elements from the current design and port it over to the modern analytics platform. Get the business requirements in conjunction with the current design to seasoned architects to help create your road map going forward.

Building Your Technology Roadmap

Now that an inventory of code exists and business processes and requirements are well understood, it’s time to map out a data analytics plan for your new platform. Take the following steps to streamline the process when developing a new platform.

1.) Choose a technology platform based on business requirements and strategy. Depending on your data strategy and requirements, you may want to explore Big Data solutions, or you may just need a simple upgrade of a transactional Data Warehouse. In any case, whether it be a Big Data solution or Data Warehouse upgrade,review your options for On-Premises solutions/cloud options.

2.) Assess technology capabilities and programming language requirements. Review business requirements, and map out how they will be carried out on the new data platform. Consult with System Architects to translate the business requirements into the technology requirements.

3.) Assess personnel in your technology environment. If the modern platform requires different skills, you may need to train your existing employees or get new employees with the desired skill set.

4.) Create a budget and project strategy. Now that you have defined business goals with your modern analytics platform and have assessed the necessary personnel to carry out development, it’s time to scope out a budget and project strategy. Review strategy with business stakeholders and technology architects for further review.

Looking Ahead

After going through these steps, you should be well-positioned to transition to your modern analytics platform. The most critical steps involved in the transition to a modern analytics platform are the discovery of a detailed inventory of integrations and the formulation of defined business requirements/strategy. You need to ensure you have well-experienced staff leading the helm with these two critical pillars and your organization will be well poised for success to transition to a modern analytics platform. In addition, make sure you have a strong resource that is well experienced in customizing your modern solution to fit your business needs to fulfill your analytics strategy.

About Levvel

You’re going to use technology to change the world. We’re going to help you create it. Whether you are reinventing your company, creating an industry-changing product, or making existing products even better with new technologies—we exist to make your endeavor a success story.

Our experts help unleash your engineering team’s potential. You know that you need to transform your software development lifecycle, and you need to move quickly. We bring seasoned experts to work with you to not only get the processes and tooling right, but to win with the human element of this critical transformation.