At Datacoral, we are heavy users of AWS Lambda and other AWS Serverless services because of the power, flexibility and security that we can provide our users by building our architecture on top of these. The benefits of these serverless services include:
- No resource provisioning required; services scale up and down seamlessly based on demand
- Fault tolerance and retry mechanisms are handled by the services themselves
- Services end up being cheaper for most real-world workloads (since machines don’t have to be provisioned for highest levels of traffic)
- Lightweight and fast deployments
- Very tight integration with other cloud services for a simpler developer experience
In this post, we will be discussing two different ways of working with AWS Lambda that are different based on how the individual lambda functions are invoked.
Synchronous Invocations
When AWS Lambda was introduced, it was mainly meant for request/response type of applications, where a process would synchronously invoke a lambda, wait for the lambda to return and then continue with its processing. A common pattern would be to build an HTTP endpoint (typically through AWS API Gateway) and a request to that endpoint would result in the lambda getting invoked. The lambda would get a maximum of 30 seconds to run so the business logic had to complete within that time period. With that being said, most lambdas were still written to be completed within in a few 100s of milliseconds.
Challenges
- Lambda cold starts end up materially impacting performance
- Since lambdas are stateless, there has to be a very performant data store to back the lambda (long read times would eat into the response times).
- Small execution times means that this model is only for APIs, which follow a simple request/response pattern
Advantages
- This is the best understood model of using AWS Lambda, which means that there is a lot of documentation and resources on using this correctly
- It is easier to trace lambda requests, and monitor and debug applications when invoking lambdas synchronously
Asynchronous Invocations
AWS Lambda also supports an invocation model that allows an event to trigger the lambda. The event itself comes from a service like AWS SNS, AWS Kinesis, or AWS SQS and the triggered lambda can run for up to 15 minutes (earlier this used to be 5 minutes). This is a long enough time for complicated business logic and processing reasonable amounts of data.
Fifteen minutes is enough to fetch gigabytes of data from a database such as PostgreSQL, millions of records from an API connector such as Salesforce or loading 100s of millions of rows into a data warehouse such as Redshift. Given the amount of work that can happen in the time a lambda can run, it can be leveraged in data processing pipelines.
Challenges
- Simple tooling does not exist for book-keeping, orchestration and traceability.
- Lambda still need to be stateless and resources available (CPU, Memory, Network, Disk) are still limited
Advantages
- Many complicated operations can be broken down into byte sized chunks, each of which can run within a single lambda invocation, which opens up many use cases to leverage the benefits of AWS Lambda.
- Cold start is not a major problem in this model since lambda start-up times end up being a very small fraction of the overall run time.
Final Thoughts
At Datacoral, we use AWS Lambdas extensively and while synchronous invocations have their place, we are fans of the model where it can run for multiple minutes with our event-driven orchestration managing the end-to-end data flow. This has allowed us to build out micro-batch based connectors on top of AWS Lambda, which have proved to be extremely robust at moving terabytes of data for our customers from different datasources into different data warehouses.
If you want to learn more, drop us a line at hello@datacoral.co, sign up for a free trial or keep in touch with us through our newsletter!
