The FOSS repository system is broken (and maybe we can fix it a little)

Where am I coming from?

This is somewhere between a rant and a blog post. I package things in nixpkgs, and I have seen my share of painful packages. Not just in nixpkgs where I am actively contributing, but also other package repositories whenever I go look how other people are dealing with the problems I come across. I do not get paid to package software, I am a volunteer. I package because I want to reasonably trust my computer and keep track of how things work together. Maintaining packages in my repository of choice seems much more productive in that scenario than contributing to a handful of original upstream projects.

This post is partly a response to “How far would hostile distributions go to hurt application developers?”. I am not affiliated with Linux Mint. I do however maintain nix packages developed by theevilskeleton upstream. While I understand and agree with many of their points, I (want to) believe it is possible to get the interaction between upstream and downstream right. In that sense, I want to elaborate why downstream patches are occasionally necessary, why removal may not always be an option, how to avoid misrepresenting upstream, all to foster a healthy relationship.

Terminology used here

Assumptions made

I will in the following assume people are not actively, intentionally hostile. I will further assume writing good software with real-world use is the goal. I understand many upstream developers start a pet project and publish it as best-effort. While this model of software development is very valid upstream, building anything serious on those projects is irresponsible. Suggesting to downstream users (either actual users, or developers of reverse dependencies) such a pet project is stable and maintainable is dangerous and exceedingly hard to clean up later.

Burning bridges harms Users

A package repository is in between an upstream and a user. It is rare users compile upstream software themselves. As such, users need to be able to trust their package repository and the maintainers. Any half-decent maintainer understands their package well enough to also become an upstream contributor when appropriate. Critical bugs, especially security issues, need to be addressed in a reasonable timeframe. If there is no reasonable channel for maintainer and developer to communicate, critical issues will remain unaddressed for users. This means users get a worse experience, and upstream will receive reports from disgruntled users which could have easily been addressed by the package maintainer.

This goes both directions: an upstream pushing potential contributors away, e.g. because of an intrusive CLA, is as damaging to the user experience as a maintainer needlessly causing problems for upstream. This isn’t to say this doesn’t happen. In fact, as we have seen with Fedora and bottles/OBS, or more recently Mint and GNOME calendar, this is a real problem. And the damage has been done, the trust needed for communication between upstream and repository has been burnt away. And users will pay the price.

“Misrepresenting” upstream

Realistically, the only people qualified to represent upstream is upstream themselves. Platforms such as flatpak make this an increasingly enticing option for upstream application developers wanting to publish themselves.

Any package repository will have to make choices outside the control of upstream developers. Be that the compiler, linker, target architecture/OS, feature flags of the package, or even just feature flags of the dependencies to a package. Many open source licenses have terms about not misrepresenting upstream, not suggesting upstream endorsement or similar (e.g. gpl and zlib licenses), often including requirements to make (source code) edits visible to the user. The specifics of this get very blurry very fast once you start asking things such as “the recipe is public, so it is visible?” or “replacing the compiler is not a code change?”. Further, some upstream software may be outright broken in certain scenarios. NixOS as a non-FHS compliant Linux distribution needs to carry many patches just to make programs work at all. I personally believe making software work is a better representation of upstream than (knowingly) packaging something entirely broken.

Safe to say, almost all packages in any repository are in some way different from the canonical upstream version. One of the few exceptions may be if the upstream author doubles as a maintainer in a package repository.

Best we can do as package maintainers is to make it very clear users are to report issues to the maintainers first, with the maintainer relaying the information to upstream as appropriate. Recall maintainers are ideally capable of becoming contributors. This is why.

When upstream makes packaging hard

In an ideal world, we can make && make install and get a working package. Reality is sadly far away from that dream. There are, however, a few things upstreams can do to make packaging easier and less divergent from upstream vision. Many of these things even improve upstream developer experience and onboarding of new contributors!

Dependency Justification

This is super important. If you are an upstream developer looking to improve the situation, please add a dependency justification to your project.

A dependency justification is a list of all the dependencies required to build your project. Crucially, this list also needs to contain reasoning why these dependencies are required and why they are appropriate. Ideally, the dependency justification should contain information about vendor dependencies too, as well as dates when each dependency was last checked (both for fitness, and whether it is even still required at all). If a dependency justification is not viable, chances are you have too many damn dependencies.

Dependency justification helps audit security impact (what pieces of your software are affected if a dependency goes vulnerable?), it helps packagers (what may break if this dependency is updated? What should be tested?), and ultimately it helps users (fewer dependencies = less bandwidth and disk usage for updates).

Modular Dependencies

I understand it is hard to keep dependencies optional. However, the dependency tree of a full package repository such as nixpkgs is incredibly complex. Breaking cyclic dependencies, removing crusty garbage from the 1900s or other such gritty packaging details become much easier if dependencies are (within possibilities) modular, with well-understood effects of what features may be missing in the final build of a package without certain dependencies.

Writing good release notes

Yes, yes, I know documentation is hard. Release notes are crucial not just for users to learn about new features, but also for packagers to learn about how their build recipes need to be adjusted and what to test during an update before it ships to users of a repository. Also, please, if you fix security vulnerabilities, please note the fixed CVEs in your release notes… I wish more upstream release notes had “note to packagers” sections.

Tests

Tests (and continuous integration enforcing these tests) is a good practice in any upstream project. In nixpkgs, we build and execute most package tests if present. This helps to notice issues before they reach users. Regression tests are nice, feature tests are nice. Anything covered in tests won’t randomly explode for users. Even better if green tests are enforced in publicly visible CI on the upstream project. Any test failure in a repository downstream is then immediately obvious as a downstream packaging issue. This eases triage a lot, and helps keep packages stable.

Stale bots

Please never use a stale bot. Stale bots are evil for various reasons. The goal of this entire thing is to make it reasonable to defer triage work to downstream package maintainers. If you still need a stale bot to keep on top of your issues, this whole thing is worthless. You may as well just not care at all and let individual users send their reports into the black hole.

Accountable History

Please never pull a release you have already published, and don’t force-push already published release tags. Any downstream repository will check hashes of the source tree. On hash mismatch (e.g. because we noted down an old hash before force push), downstream packages are left wondering what changed, and whether something malicious is afoot. You wouldn’t force-push your main branch. Please don’t force-push existing releases.

Versioning and Support

Define a versioning schema, and actually stick to it. SemVer is nice: major.minor.patch allows communicating to package maintainers how much stuff will break on a particular update. Bump patch whenever everything is compatible, just with some small non-breaking non-controversial fixes. Bump minor whenever some small behavior changes, and document what exactly changed. Bump major version based on vibes, I guess (like a big rewrite, fancy new feature you are proud of, etc.).

If you have frequent updates, and expect users to care about the specific version, consider date versioning. While date versioning does an awful job communicating stability guarantees, it is much more intuitive and much more obvious whenever a package is hilariously outdated. Users will complain to the package maintainer if they are on version 2017.06 while upstream has published 2026.06 recently.

A rough estimate of how long a release will be considered supported is much appreciated. It makes it obvious to package maintainers they will have to pick up maintenance of the original source code in LTS repositories. It also makes it possible to point to a version string and trivially deny any support upstream. Package maintainers can and will package unsupported crud, but an EOL date or support matrix upstream is often used by repository security teams to enforce package removals. Hint: If some newer release fixes security issues, with fixes not backported to older releases, this does wonders when put into a visible support matrix!

Pre-built Artifacts / Binary Blobs

Just, don’t. You heard of xz. Blobs are a liability packagers barely ever want to deal with. If some prebuilt artifacts are required to build your project, the build system is in some way, shape or form broken and should be fixed. Blobs make reproducible builds unattainable, are a security and compatibility nightmare, and bloat the size of a git repo checkout. If your test suite is so complex you don’t want the test code and test cases in your main development repository, then by all means put the tests in a separate repository, but build them from source. Don’t vendor blobs.

Hacking.md / Building.md

Documentation about how to get started with upstream contributions is nice. Partially to help make it easier for new upstream contributors to do work (e.g. a maintainer becoming a contributor), but also for packagers to understand the “supported” setup they should try to replicate in their package recipe.

Document Standards

Does your package follow uapi configuration file specification? Does it assume gnu coreutils, or is it happy with POSIX? Great! Document it! Documentation about how things should work removes tedious guesswork and makes it much more obvious when weird things start happening. Whether the weirdness is a bug upstream or in the downstream packaging.

Obvious Metadata

License? Authors? Packages need metadata. Especially licenses are hard to get right. Some SPDX identifier does wonders. Also link to the Homepage from your repository / repository from your homepage, as well as a changelog link if it is not just release notes in your git forge of choice. Rust projects have these in their Cargo.toml, but there is no canonical equivalent with ancient crusty C tooling that is still popular. Put it in the README or something, just make it discoverable somehow.

Conclusion

Short of pulling the plug and telling downstream package maintainers to stop packaging something, I very much believe there are ways to make packaging easier and better defined. All the suggested things are tools to make packages easier to write, easier to maintain, and crucially less of a wild west with everyone doing some different unsupported thing. I very much believe packaging can be done responsibly and sustainably. The requirement for that is maintainers taking responsibility for their packages and actually acting as the intermediate they are, while developers make that actually possible.

If you are a developer and need/want help with these things, I am positive you will find packagers willing to help out writing that dependency justification or whatever. Chances are maintainers already did most of the work needed to efficiently maintain your package, you probably just need to ask.