Due to the high stakes of the solution selection decision, we recommended a thorough proof-of-concept and assembled a team of subject matter experts in data architecture, data engineering, and DevOps. After reviewing the current data model, we conducted an architecture review and identified and costed potential tool options. Our engineers set up a development environment and ran an eight-week proof-of-concept where we tested both historic and continuous replication data loads to the new warehouse, created infrastructure and automation scripts, and built sample dashboards. The exercise demonstrated that the client’s requirements could be best met by a combination of Amazon Web Service tools – RedShift and Data Migration Services – coupled with Talend, an Extract, Transform, and Load (ETL) tool Anoteros helped select and license on behalf of the client after testing several market-leading ETL tools.
With the client’s approval to move forward, we set up the production environment, managed the conversion of historic data loads, set up monitoring, alert, and reconciliation mechanisms, and created both automated scripts and procedural manuals for steady-state management. After the new warehouse was live, we continued to support the client team, running both the old and new data pipelines in parallel for a few weeks to ensure the new solution was performing as expected. During this phase, we also built out more robust dashboards for monitoring and trained stakeholders on how to use them, before ultimately disconnecting the old data warehouse and pipeline.