Pioreactor development log #3

🔑 This week we explored key-value databases for storing data, and implemented a solution that is designed to get wiped often. At the end of the day, we got some significant performance improvements!

It was time to grow up. During the months of development, a handful of solutions had been used to solve the following problem: "I have this somewhat-important data point, but don't have access to a datastore to store it in". This problem was almost always present on workers (recall workers, unlike the leader, don't have SQLite installed, and pass all their data to the leader via MQTT). For example, we wanted different processes on a Pioreactor to be aware if a PWM channel was in use, so they don't accidentally use that PWM channel. Another problem was caused by our DAC (digital-to-analog converter) being write-only, but we needed multiple processes to know what the DAC's current state was. 

The solutions we were using ranged from:

• "push the data to MQTT, and we'll retrieve it later". This has a pretty significant network overhead. And if the network was down temporarily, this would cause delays, and circuit-breaker path was needed.
• "save the data to disk as JSON, and read when needed". Not a bad solution, but this has no data integrity guarantees, and is slow for large json files. 
• "create a flag-file that we check for existence". This was our solution for PWM channels: a process should look for a specific file on disk to determine if the corresponding PWM channel was in use. 

It was time to grow up, and pick a proper datastore. Luckily, all our usecases were simple key-value lookups: "what is the state of the DAC's channels?", "is PWM channel X in use?", and so on. So my requirements were to find a key-value store that:

1. Required minimal new installations, and ideally part of my existing Pioreactor tech-stack.
2. Was reasonably fast with little overhead.
3. If not fast, then was thread-safe / multiprocess-safe.
4. Ideally just a key-value store. I don't need indices, or documents, or tables.
5. A technology that is well used and still maintained.

With that in mind, I asked Twitter for their advice. And oh boy, there were a lot of opinions. Some suggested technologies were: Redis, memcache, sqlite3 (with a wrapper to treat it as a key-value store), TinyDB, pickleDB, and shelve. I was seriously close to using sqlite3, as there are some really interesting wrappers for it: diskcache and sqlitedict. However, two reasons gave me pause. The first is that I would need to install SQLite3 on the workers, something that isn't done already. Not a terrible thing, but I wanted to avoid new software installations, if possible. The second reason was that these libraries actually had a pretty large overhead. Now, the way I planned to use my key-value store would be frequent start-ups and closes, checks for membership, and updates. The sqlitedict library would have to do at least two queries for each check plus a query at startup, so potentially a dozen queries every 5 seconds - not to mention it used threads, too. This felt like a pretty significant overhead for my small use cases (admittedly, I didn't actually check the timing on a Raspberry Pi). sqlitedict and diskcache are probably meant to be used by a single processes at startup - not being invoked by multiple smaller processes many times. Don't get me wrong, these libraries are really well designed, but just not for my use case! 

This was new to me, but Unix systems ship with a "database" that predates even SQL, called dbm. dbm is a very straight forward key-value store that uses a flat-file to store its contents and metadata. And wouldn't you know it, Python's standard library has a module to open and access dbm database! The Python module, also called dbm, is very easy to use. So here we have an already implemented "database" on the RPi, and existing Python code to interact with it. And, it's fast enough, too. 

For all of our use cases, we've designed a simple context manager that gives access to the key-value store:

from dbm import ndbm
from contextlib import contextmanager

@contextmanager
def local_intermittent_storage(cache_name):
    try:
        cache = ndbm.open(f"/tmp/{cache_name}", "c")
        yield cache
    finally:
        cache.close()

This allows us to write code like:

with local_intermittent_storage("leds") as led_cache:
    old_state = {channel: float(led_cache.get(channel, 0)) for channel in CHANNELS}

    # update cache
    led_cache[channel] = str(intensity)

    new_state = {channel: float(led_cache.get(channel, 0)) for channel in CHANNELS}

    return new_state, old_state

Mimicking what our hardware does using /tmp

You may have noticed that we store the dbm database in /tmp. What's up with that? We only put things in the cache that follow the "rules" of a Raspberry Pi rebooting: the DAC is cleared on reboot, the PWM channels reset on reboot, etc. So, when the Raspberry Pi boots, the /tmp folder is cleared, along with our cache. Thus our cache mimics what our hardware is doing. This hopefully explains the name local_intermittent_storage.

Another example where we have moved our tech stack on to this cache is our running processes. The Pioreactor codebase keeps duplicate processes from running (it makes no sense to have two Optical Density Reading jobs running, and would only break downstream jobs). At run time of a job, we would ask the operating system for a list of all running processes, and check that list for a job with the same name. This took up to 0.1s, which may not seem like much, but your app being responsive is really important for user satisfaction.

After introducing our cache, jobs will edit their state to the cache, and subsequent jobs will check the cache to see if a duplicate job is running. This is near-instant, meaning we get that 0.1s back. (In the case of a duplicate is detected in the cache, we double check with the "slower" method to be doubly sure). And like our hardware, when the Raspberry Pi reboots, all processes are lost, so our cache should clear ✨