The client is a major software provider for home and community-based healthcare providers and Medicaid-managed care payers. The client wanted to modernize its data warehouse with the goal of achieving near real-time data availability for reporting and analytics, improved ability to react quickly to source schema changes, and improved scalability through a cloud-based architecture. The client was seeking an end-to-end cloud solution utilizing Amazon Redshift as the data warehouse given its existing investments in Amazon Web Services (AWS) infrastructure.

Due to the high stakes of the solution selection decision, we recommended a thorough proof-of-concept and assembled a team of subject matter experts in data architecture, data engineering, and DevOps. After reviewing the current data model, we conducted an architecture review and identified and costed potential tool options. Our engineers set up a development environment and ran an eight-week proof-of-concept where we tested both historic and continuous replication data loads to the new warehouse, created infrastructure and automation scripts, and built sample dashboards. The exercise demonstrated that the client’s requirements could be best met by a combination of Amazon Web Service tools – RedShift and Data Migration Services – coupled with Talend, an Extract, Transform, and Load (ETL) tool Anoteros helped select and license on behalf of the client after testing several market-leading ETL tools.

With the client’s approval to move forward, we set up the production environment, managed the conversion of historic data loads, set up monitoring, alert, and reconciliation mechanisms, and created both automated scripts and procedural manuals for steady-state management. After the new warehouse was live, we continued to support the client team, running both the old and new data pipelines in parallel for a few weeks to ensure the new solution was performing as expected. During this phase, we also built out more robust dashboards for monitoring and trained stakeholders on how to use them, before ultimately disconnecting the old data warehouse and pipeline.

Anoteros’ approach to this complex data warehouse migration gave the client confidence to make vendor and tool selection decisions knowing it could meet their needs. Ultimately, they were able to meet their goals with the implementation of the new data solution including near-time replication from the source systems to the data warehouse. Throughout the project, we complemented the in-house data team’s understanding of the existing data model with expertise in cloud-based data warehousing and were able to guide and coach them so that when our project ended, they were able to independently operate and maintain the new environment.