amazon-dynamo-db-FIFO-Queue

Simulate FIFO Queue with Amazon DynamoDB and NodeJs

In Articles, Cloud Technology by Jiju Thomas Mathew

Introduction ( Disclaimer )

This could be just a concept and has worked for me, which does not imply that it will work for all, and it may even have some serious blunder hidden under which may be found later on during run time.

Why did we think of this

From a supplier we are getting inventory which is of one time use only. Supplier will provide the inventory only on request, and we do not wish to see the inventory is exhausted. Also we wanted to make sure each bach gets finished in the sequence of applying the same. When we were in MySQL / MariaDB this would have been just another table sorted on primary key and a flag status = ENUM(“new”,”used”). But we are in the “serverless” hype and on AWS with DynamoDB.

Since our main project was having a table already deployed, with a STRING partition key (parkey) and STRING sort key (itemtype), I was particular to stick to this concept.

The Design

One row with parkey = “store-keeper” and itemtype = “store-keeper” controlled the queue position, and configuration, further fields were there and the final json is
{
"parkey": "store-keeper",
"itemtype": "store-keeper",
"headAt": 1,
"tailAt": 1,
"minFree": 20
}

To explain things, when we know the exact partition key and sort key, manipulation of row in DynamoDB is very easy1. When we update stock by adding more inventory, the tailAt is incremented with a request for returning the UPDATED_OLD, will be used in the parkey value for itemtype store-inventory and property inventory will contain the “supplier single use item”. When we consume, the headAt is incremented with a request for returning the UPDATED_OLD will be used in the parkey value for itemtype store-inventory for issuing a DynamoDB get-item api call2. Subsequently will be updated using DynamoDB put-item with the added property of “consumed_by”. The inventory structure follows

{
"parkey": "store-inventory: < id > ",
"itemtype": "store-inventory",
"inventory": "our supplier single use item"
}

As soon as the consumption is done, an Inventory Check can be done and if reached minFree level or lower, send intimation to the supplier. For the same we used AWS SES send-templated-email option3. Create your templates using your favourite text (json) editor and update them to the SES console using aws cli4.

Though we referred to DynamoDB and its direct documentation, the code uses a complimentary wrapper which includes the marshaller and unmarshaller. The wrapper is also part of the AWS SDK, which is DynamoDB.DocumentClient

Code – NodeJS module

Link: https://s3.amazonaws.com/documentsandmedia/dynamodb-queue.zip

  1. https://aws.amazon.com/blogs/database/choosing-the-right-dynamodb-partition-key/
  2. https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_GetItem.html
  3. https://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-personalized-email-api.html
  4. https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/ses-template.html

Share this Post

About the Author

Jiju Thomas Mathew

Jiju is a core technology evaluator and an active participant of the FOSS world. During his 30 years of career, he has implemented combinations of Free and Open Source Software to multiple walks of life, which includes Publishing, Information Technology and Cloud.