Automating A Product Platform With IaC On AWS Cloud

The platform was purpose-built for media data analytics. The media industry was going through a rapid change with the adoption of streaming technologies and over-the-top (OTT) services. The customer’s Analytics & Engagement platform was hosted in a virtualized environment in a US-based Data Centre and was serving customers in the US, India, Israel, etc.

Key constraints were:

• The core of the platform was the big data engine, which was being provided as a single-tenant SaaS for some customers, and as multi-tenant SaaS for others. This meant different versions for different installs, and a continuing development cycle for each of those installs.
• With a small core product team, the customer was not able to manage the multiple versions, releases and continuing code upgrades.
• The Operations team was using manual (cheat sheets and version docs), for the multiple environments, so scalability/agility in deploying upgrades and accuracy were compromised, especially for larger customer environments that demanded custom offerings
• Lacked scalability in existing production environments

CK conducted a comprehensive study of the existing environment to know its Technology Stack; Infrastructure & Application design; Security, Scalability & Performance criteria

Upon completion of the study, CK came up with the following deliverables.

• Detailed design proposal for an Automated deployment of the infrastructure on AWS
• Identification of automation tools – Hashicorp (Terraform & Packer) Python, etc.
• Automated provisioning of infrastructure & application using Terraform based Infra-As-Code (IaC) templates for the automated deployment of
  • AWS account security & governance best practices and baselines (AWS GuardDuty, Config, IAM Policies, Cloud Trial, Flow Logs, etc.)
  • Network setup & isolation – VPC, Subnets, ACLs, Security Groups, etc.
    Creation of compute resources like EC2 instances, Load Balancers, RDS instances, Elastic File Systems, etc.
  • Installation and configuration of application software post infrastructure creation
  • Customizable Form-based input for the IaC templates
  • Complete redesign of Infrastructure architecture that enabled quick turnaround times in provisioning of newer environments

• A significant, observable (~50%) reduction in time taken for customizing new environments by using editable pre-deployment IaC input form
• >70% reduction in deployment time owing to automation of CI/CD approach
• 100% assurance on definitive and controllable costs for PoC and Dev environments by including IaC-based decommissioning of resources
• Minimized errors on customizable, repeatable & extensible templates with industry-standard best practices

Lessons Learned:

The customer’s project sponsors were not committed to cloud, and were exploring both on-prem and cloud options for their different customers. This caused a lot of delays in finalizing a single set of automation and CI/CD tools for the platform. After initial delays, CK found the project schedules slipping, and requested customer teams to commit to an agreed set of tools