Craig Weber

I own a bunch of media, and I want to stream it to my devices as easily as with a streaming service, using affordable hardware. Today that is really hard, and I want to briefly lay out why and what could be done to improve it. The problem and solution space is extremely broad, so I'm going to speak abstractly, going into greater depth in specific areas in future posts.

The specific goal of this post is to start a conversation about what problems we face with the current media landscape and what we could do to fix them. I'm laying out my understanding of the problem space and my high-level vision for how it could be fixed in hopes that others (especially those with more expertise in media specifically) will offer their own visions or critique/refine mine.

Problem

As a quick disclaimer, I don't mean to berate any of the existing projects which are actively solving important problems and are dealing with the real-world challenges of shipping software to users (as opposed to dealing in theory, as I am here). These projects evolved organically and had to navigate a bunch problems to provide the hindsight from which we benefit. Shoulders of giants and all that. The folks who started these projects were also undoubtedly learning as they went, leveraging the tools and skills they were familiar with at the time. I'm sure a lot of this may sound arrogant, because the line between arrogance and belief something can be improved is often fine.

The current media landscape is complicated and fragmented. There are dozens of components with little standardization across them, and they frequently try to be all things to everyone, with poor defaults. Ultimately users must be experts in every aspect of media from the intricacies of transcoding to torrenting to VPNs to subtitle management to filesystems to etc.

And for all of that intricacy and the maximalist feature sets, this landscape misses out on some big, obvious-to-me use cases like the ability to easily run these components on "resource constrained devices"¹ like Raspberry Pi or to run them with high availability, or using a standard distributed computing platform like Kubernetes.

Meet the Components

The rest of this document will talk a lot about "components" in the abstract, but to give the reader some idea what I'm talking about, the components in a media system include (not remotely exhaustive):

• media server (how is the media indexed and made available to the clients?)
• media storage (how is the media stored and made available to the server?)
• media acquisition (how do users get their media into the system?)
• subtitle acquisition system (how do users get subtitles for their media?)

Vision

Ultimately, I can imagine a world where these components are incrementally replaced with simpler, more performant, more reliable components that are built with standard tools, practices, and idioms and which can run on a single beefy computer (as today) or distributed across many small, inexpensive computers (which is technically possible today with sufficient effort, expertise, and unfortunate performance/reliability compromises). Over time, we evolve from a disjoint media landscape to a cohesive media platform.

Simplicity

The media platform of the future and its components will be simple to use. They will prefer to do one thing well, eschewing feature creep. They will present few configuration options, and they will choose sensible defaults so lay users can operate them with minimal expertise.

The platform and its components will be minimalist in their featuresets. When considering new features, a full accounting should be undertaken of the costs to support these features, rather than assuming that new features are some unmitigated good. New features must justify their keep.

The components will not be designed to interoperate in arbitrary configurations with other components. Where they are designed to interoperate, they will prefer to be agnostic of each other, instead collaborating through abstract yet well-defined and well-designed interfaces. But mostly they will prefer to not collaborate beyond what is necessary to serve their minimal function.

Standard tools

The entire system, though separated into distinct components, will feel cohesively designed. Contributors to one part of the system will experience minimum friction contributing to other parts of the system.

The components of the future will use the same backend programming languages, the same project structures, the same style conventions and other idioms. Their configuration will be standardized using a single configuration language. They will communicate using standard protocols. They will rely on a common database technology and use the same database migration frameworks. They will leverage a single, common authentication/authorization framework.

The components of the future will also be documented in similar formats so users and developers familiar with one component can easily locate the documentation they need for other components.

Architecture

The components of the future will be built with a backend programming language that is not wasteful of resources (either in its runtime or in the idioms/common-practices of the language and its ecosystem), and the components will be architected with distributed computing in mind to ensure performance and reliability.

Modern distributed computing architectures allow for performance to be increased by taking advantage of the resources available on the computer--rather than designing the system to run in one thread on a single CPU core, the architecture will be designed to run on many CPU cores distributed across an arbitrary number of nodes. This kind of architecture allows for horizontal scaling--you can add more performance/capacity to the system by attaching additional hardware--rather than vertical scaling in which performance is added by replacing existing hardware with upgraded, more performant hardware (rather than replacing a $1K server with a $1200 server, you can plug in a $75 Raspberry Pi alongside your existing hardware).

This same kind of multi-node architecture allows for greater reliability, as failures on one node do not necessarily bring down the whole system. There are far fewer, if any, single points of failure.

Kubernetes as a first class citizen

The components of the future will not require Kubernetes, but they will be packaged for it and they will not make choices that make leveraging Kubernetes difficult. For example, the components of today assume they share a filesystem, which essentially requires a brittle, insecure solution like NFS (or perhaps node selectors) to operationalize for Kubernetes. They also are not packaged for Kubernetes and thus require every operator to figure out for themselves how to run their software on the Kubernetes platform.

How we get there

There is no easy way to get to the ideal end state in any short amount of time. Some of these problems require deep expertise (e.g., real-time transcoding with lightweight hardware). This is a marathon effort. We start by building a distribution consisting of a few core components, packaged with minimal configuration options. From there, we incrementally expand outward to package more components. We also aspire to define good interfaces that allow us to eventually replace individual components with new, standard components.

We version everything in a way that allows for continuous evolution and experimentation.

Call to action

A lot of you probably know more about media generally, the current media landscape specifically, and system administration than I do. What's your vision?