May 7, 2019
TABLE OF CONTENTS
I think it’s pretty well known these days that Open Source Software (OSS) forms the backbone for many applications we build today. At Levvel, we are hyper-aware of the importance of OSS both internally and for delivering the highest quality solutions for our clients. Whether it be a small bug fix, adding additional functionality, or largescale collaboration, OSS and contributions to OSS are a key part of our success as Technological Partners.
For one of our clients, we built a Scala/Play web application that utilized Redis as both an application cache and priority queue. Redis is a phenomenal choice for many use cases and already existed in the stack for our Caching Layer. For the Priority Queue we decided on Redis over other similar technologies as, aside from already having Redis in the stack, the cost of adding a dedicated MQ solution seemed unnecessary when Redis Sorted Sets met our needs and SLAs perfectly.
The library we were already using for our caching layer was the play-redis wrapper library. This library implemented the Play Cache API interfaces using the incredible rediscala (https://github.com/Ma27/rediscala) Redis client for actual communication to Redis. This meant that Play managed most of the communication with Redis through this wrapper library. We were delighted with play-redis but found ourselves leaning more and more on the underlying rediscala library as the application grew and our needs shifted.
The most critical part of the application we were building involved a batch work submission service which, based on an assigned priority score, scheduled large batch jobs to be executed on a background processing service with limited processing slots. The requirements of the feature fit perfectly with the pattern of a Priority Queue. As mentioned earlier we started by considering several MQ-based systems in the AWS ecosystem where the application resides. We also explored Redis’ built-in Sorted Sets.
As we were building out our workflow, we noticed an issue with the rediscala library: it did not have exposed APIs for the ZPOPMIN or ZPOPMAX Redis Commands which form the basis for the priority POP implementation. There did not appear to be any newer version available with these commands either.
To keep the project going, we decided to reimplement these commands using a combination of other Redis commands within a Redis Transaction call, but this would not serve for a permanent solution. In parallel, we evaluated other libraries which had these commands—of which there weren’t many on the Scala side to our surprise—but this would have incurred a much larger cost in requiring a replacement of the caching layer as well. We decided to dig into the rediscala library itself to see what it would take to surface these commands instead with this in mind.
The code itself is very well written, and the test suite kept confidence high on our ability to make the changes necessary. The change to the library (captured in this pull request: https://github.com/Ma27/rediscala/pull/18) amounted to an earth-shattering 55 lines of code. In isolation this is a small change that would, on the surface, seem inconsequential. This one tiny contribution, however, enabled the removal of well over five times that amount of code (tests included) and thus simplified our integrations, our tests, and the cognitive load necessary to understand what we were doing in our business logic. All the while we were giving back to a library that served us so well in our caching needs thus far.
Senior Engineering Consultant
Matt Edge is a Senior Engineering Consultant at Levvel specializing in full-stack development. His current areas of focus are Scala/Golang, Containerization, and DevOps.
At the end of lunch with a mentee, I used the items on our table to express the fundamental concepts of Kubernetes. Sometime after explaining the purpose of the Kubernetes scheduler, she asked a question I spent the next several weeks thinking about.
API design is crucial, giving structure to application interaction. Given cross-functional teams and applications, development time is reduced with a clear, intuitive way to access data. API development often follows two approaches: REST and GraphQL.
As of June 2018, the state of California passed a new privacy law that could lead to more consequences for US-based companies than the European Union’s General Data Protection Regulation (GDPR). Here's what you need to know and how to be compliant.
Before your data scientists wring value out of your reams of data, it has to be accessible and, on some basic level, coherently arranged. To harness all that brainpower, you need to keep the data wrangling to a minimum. Enter the data lake.