Challenges:

• Files will come at any time of the day 24 x 7
• Will be organized into date based folders on bucket level
• Will need to check the stability of file on arrival at the specified Amazon S3 bucket
• Files will vary from between 10 to 900 MB
• Will need to use the same bucket to store the preview files
• Lifecycle rules to be implemented for the preview files to discard after a week
• Keep cost as low as possible without loosing the time value

Initial Solution:

• AWS Lambda with Node.Js 6.10 timed with Amazon Cloud Watch events at specified interval
• Get S3 listing and store it to S3 after checking for repeat status
• Twice same filesize and timestamp means file is stable
• Pass files to On-Demand EC2 from custom Amazon Machine Image
• Using ffmpeg and some shell scripts pick file from Amazon S3, do the process and store it back

After about a week, we started to identify some issues, which were due to the fact that we had overlooked the fine print of “Amazon Instance Hour Billing”. We were launching more or less 30 to 35 instances per day even though they were running just for a minute or two and terminating, we could be billed for as much number of instance hours.

A round of brainstorming was put forth and considering the following points we arrived at a further solid solution.

• AWS Lambda (FaaS) runs on Amazon Linux
• Lambda can run compiled binary if it is statically linked
• Thanks to John Van Sickle (https://johnvansickle.com/ffmpeg/)
• AWS Lambda has ephemeral disk of 512MB
• Lambda can be triggered from S3 events

Final Solution:

• Function from initial solution will pass on any sub 350MB files to a new Function
• New function will do the same process as that of the EC2 using statically linked binary

Outcome:

With over 70 files being processed per day last week through EC2 which was being invoked for almost 35 hours a day, has now come down to about 5 hour perday. Well it means in figures, a reduction of $100 to $15 per month.

Being formal, I would thank Habeeb and Jeevan for the opportunity given to me to make a wide audience aware about Innoval and the AWS initiative of Innoval . My excitement doubled when I heard about the other use cases where “Cloud Bursting”, “Time Data Analysis” and “Big Data as Service” was discussed. These ideas ring more bells that we should concentrate more. Well Saturn was one step ahead with two information, which most, even AWS team did not know, was Lambda now supports C#. Another was Athena! Since we had already used the same for access log analysis we could extend the same for further things.

The hosts “Blue Jeans Networking” deserves an appreciation for the part they have taken forward in supporting the event. Especially Roy and Tanmay who explained their solution and offered the 14 day trial for video conferencing and home computing.

Hope to see more of this, or even host one of ours in #Technopark inviting some of these guys to speak about things.

This year with alerts from the department we were quite concerned about the massive security compromise threats and we had blocked off several country blocks using nginx geoip module and blacklisting. On top of these, we were watching unusual activities.

Like previous seasons, we had the elb logs being written to S3. Normally every year by the end of the season, we try to do a sync of the S3 logs to an EC2 instance, then fireup an instance of mysql to import the whole logs using “LOAD DATA” query. Then drop unwanted colums, and dump the CSV export, bzip and off load to one of our inhouse development system over night to do the sql analysis.

But this year, I got the announcement of Amazon Athena, well myself with a colleague thought we should attempt this and see what we can do. The aws blog itself had an example. Everything was simple, create the SQL View with the pattern matching the log files, and our first query was counting requests grouped by response code, wow 30.52 GB was scanned in 1m 48s. There was no overheads of importing, downloading, querying etc. We just run the query and leave it there, later we can go to the Athena history and view the results. Instantly we were excited, and we tried a couple of more queries. Well thanks to Amazon Athena, we could identify a single IP which had pounded our ELB instance with all sort of junk requests and created about 57K entries in a three minute period all triggering the “HTTP 404”.

There was one issue with Athena, the “Export Results as CSV” was showing some kind of S3 Error which we are pretty sure will be corrected soon by the AWS Team.

Though we were running on AWS, and the mobile app had to be certified from Apple, it was a state of confusion as to getting the IPv6 support for the backend. At the time of starting the development, the IPv6 support for EC2 instances were not available.

After some internal brain storming, we decided to use the Amazon API Gateway, which gave us lot more than we asked for. The standard protection for DOS and extended low latency of the CloudFront were some of them. We can run the application distributed on two domains with a single url provided by the API Gateway, which uses secure transport, and a single KEY management system. This could help us in the near future while we expand the system across an API first architecture backed with microservices.