The Future of Kivik

With a stable release of Kivik 2.0 coming soon, it seemed like a good time to share my vision of the Kivik future. This also comes after a brief conversation with a small number of would-be open-source contributors, looking for guidance in contributing to the project. I feel the first step toward that goal is to explain my vision of the future.

Some of what I will discuss here was already mentioned back in 2017 when Kivik was first announced. But as one would expect over 2.5 years, it has become more specific, and some things have changed–not least of all, the current state of Kivik.

Modular design

From the beginning, I’ve had the idea of making Kivik a modular project. This is probably most apparent in the way that drivers are interchangeable, but this modularity can extend to consumers of Kivik, as well.

In the end, my vision is that Kivik may provide a sort of swiss army knife for building CouchDB tools and applications. A few possible applications for Kivik, once this goal is reached, could include:

• The ability to build apps in Go/GopherJS, which use CouchDB for database storage. This is possible today.
• Easily bootstrap CouchDB instnaces. Historically we have had a number of ad-hoc tools to do this, including couchapp and others. None have felt very feature rich or well maintained to me (if I’m wrong, please let me know!)
• Run an in-memory database for testing purposes. With the addition of an HTTP server for Kivik, the application being tested doesn’t even need to be written in Go.
• Dump files from CouchDB to your local disk as JSON or YAML files
• A nice command-line tool to interact with CouchDB from the console, or in shell scripts.
• Build a custom HTTP proxy in Go, to handle authentication or other custom logic.
• And maybe even eventually, with additional backend stores, maybe even put a CouchDB façade on top of MySQL, S3, or DynamoDB, for applications with low performance requirements.

From the chart above, I hope it is apparent how these outcomes, and likely many others, should be possible. But there is a lot to be done, so where to start?

Tooling

The priority for me lately, has been to focus on tooling. I recently finished the kivikmock library to facilitate testing, which has been a big boost to productivity (no longer relying on a Docker install of CouchDB for all tests).

I’ve also recently added the bare essentials to the fsdb driver to allow replicating from an on-disk collection of JSON/YAML files to a live CouchDB server. This has made bootstraping new databases many times easier for me. But there’s much room for improvement.

My next mid-term goal is to build a command-line tool which provides a simple interface to CouchDB and the Filesystem driver. I actually began an experimental version of this project here, but after initial learnings, intend to start from scratch.

Initial project goals include:

• The ability to easily GET and PUT individual documents and attachments.
• The ability to easily query views
• The ability to read and set security objects
• The ability to configure server parameters
• The ability to dump a database via replication to the filesystem
• The ability to bootstrap a database, via replication from the filesystem
• The ability to consume and produce JSON and YAML, translating to JSON when sending to CouchDB

Ideally, the tool should make any manual interaction with CouchDB easier than it presently is with cURL, and also add the option to replicate to/from the filesystem. Naturally, this means the Filesystem driver needs to be fleshed out.

HTTP Server

A longer-term goal is to build a fully-capable HTTP server driven by Kivik. Some basic functionality along these lines already exists in the kivikd package, but it’s far from complete.

Once the Filesystem driver is complete, adding a full HTTP server in front will make it possible to run a stand-alone Kivik server, which could stand in for a CouchDB server for testing or light-weight loads. I find this prospect especially exciting once the Memory driver is complete, as it would allow any developer, in any language, to run a small, self-contained memory-only CouchDB instance for testing, either on their local machine, or in a CI/CD pipeline.

This also has long-term possibilities with the addition of future storage backends for Kivik. Suppose a DynamoDB backend is written, then certain small apps might be able to take advantage of CouchDB’s replication protocol, but using AWS-managed infrastructure, rather than managing their own CouchDB installation.

Other Possibilities

The logical implications of a modular system like Kivik I think are broad. No doubt there are possible applications I have not considered. Rather than documenting all possible use cases, I think the prudent course is to focus on the areas with the greatest possible benefit in the short-term.

Help Wanted

Up to now, this has been driven by my own needs for testing and tooling, and I have been the primary contributor.

But now I’d love to get some greater community involvement, particularly in choosing the area of focus.

If this project intrigues you, here are some areas where some help could go a long way:

• Discussion No coding necessary! If you are a current user of Kivik, or find the future of Kivik promising, your input can help shape the direction. The CLI needs to be designed, and direction on the next features needs to be decided. You can weigh in!
• Documentation I’ve tried to keep things fully documented, but there’s plenty of room for additional documentation, especially with regard to code samples on the wiki.
• Coding If you’re a Go coder, and think this project sounds interesting, your help could be very valuable in bringing this vision to reality.

To get involved, you can submit an issue or stop by the #kivik channel on Gophers Slack.