Elastic Project – Shrinking from Elastic to S3

In Blog, Cloud Technology by Jiju Thomas Mathew


One of our portals is seasonal, and we usually pop it to Dynamic on Elastic by October every year and shrink down to static on S3 the subsequent January end. This has been a manual process for the past 5 years. But in 2017 we attempted to automate it with some tweaks to the application. Since we had enhanced the web application with a mobile API linked with Amazon API Gateway, this was easy to some extent.

The application API was enabled with services to initiate the Cloud Shrinking, which just switches some of the keys for enable from true to false, and the website user interface will blank off the user login, coupon search and booking effectively removing all dynamic area.

The total site was leeched using wget (wget --mirror --convert-links --adjust-extension --page-requisites --no-parent [url]). This was further passed through grep and sed to rewrite any internal reference of CloudFront URL to relative URL. Contact forms were removed. Finally pushed the static pages to Amazon S3. The Amazon Route53 hosted zones were updated to point the URL directly to the Amazon S3 hosted website.

All primary instances were backed up as Custom AMI. The database dump was taken with individual tables in separate files as some of the tables are not needed in the next season, but only for archival. The backups were pushed to S3 using aws cli.

The total stack was built using CloudFormation, so the stack delete instruction was issued, and the whole stack was successfully deleted in less than 2 minutes. All helper functions in AWS Lambda are still active and left there, but as the invoking part is the dynamic application, these lambdas will be hibernated till the next season.

The whole operation was orchestrated using a shell script and aws cli from my Dell Vostro running Xubuntu 14.04. At some points, like the database dump and backup, shell snippets were pushed through scp and run using ssh. Though the full operation took close to 20 minutes, mostly the time was waiting for one or the other operation to complete.