Redis Client for Scala and Priority Queue Optimization: Extending redisscala for ZPOPMIN and ZPOPMAX

Introduction

I think it’s pretty well known these days that Open Source Software (OSS) forms the backbone for many applications we build today. At Levvel, we are hyper-aware of the importance of OSS both internally and for delivering the highest quality solutions for our clients. Whether it be a small bug fix, adding additional functionality, or largescale collaboration, OSS and contributions to OSS are a key part of our success as Technological Partners.

Scala + Redis

For one of our clients, we built a Scala/Play web application that utilized Redis as both an application cache and priority queue. Redis is a phenomenal choice for many use cases and already existed in the stack for our Caching Layer. For the Priority Queue we decided on Redis over other similar technologies as, aside from already having Redis in the stack, the cost of adding a dedicated MQ solution seemed unnecessary when Redis Sorted Sets met our needs and SLAs perfectly.

The library we were already using for our caching layer was the play-redis wrapper library. This library implemented the Play Cache API interfaces using the incredible rediscala (https://github.com/Ma27/rediscala) Redis client for actual communication to Redis. This meant that Play managed most of the communication with Redis through this wrapper library. We were delighted with play-redis but found ourselves leaning more and more on the underlying rediscala library as the application grew and our needs shifted.

Batch Submissions and Priority Queueing

The most critical part of the application we were building involved a batch work submission service which, based on an assigned priority score, scheduled large batch jobs to be executed on a background processing service with limited processing slots. The requirements of the feature fit perfectly with the pattern of a Priority Queue. As mentioned earlier we started by considering several MQ-based systems in the AWS ecosystem where the application resides. We also explored Redis’ built-in Sorted Sets.

Missing Functionality—ZPOPMIN and ZPOPMAX

As we were building out our workflow, we noticed an issue with the rediscala library: it did not have exposed APIs for the ZPOPMIN or ZPOPMAX Redis Commands which form the basis for the priority POP implementation. There did not appear to be any newer version available with these commands either.

To keep the project going, we decided to reimplement these commands using a combination of other Redis commands within a Redis Transaction call, but this would not serve for a permanent solution. In parallel, we evaluated other libraries which had these commands—of which there weren’t many on the Scala side to our surprise—but this would have incurred a much larger cost in requiring a replacement of the caching layer as well. We decided to dig into the rediscala library itself to see what it would take to surface these commands instead with this in mind.

The code itself is very well written, and the test suite kept confidence high on our ability to make the changes necessary. The change to the library (captured in this pull request: https://github.com/Ma27/rediscala/pull/18) amounted to an earth-shattering 55 lines of code. In isolation this is a small change that would, on the surface, seem inconsequential. This one tiny contribution, however, enabled the removal of well over five times that amount of code (tests included) and thus simplified our integrations, our tests, and the cognitive load necessary to understand what we were doing in our business logic. All the while we were giving back to a library that served us so well in our caching needs thus far.