How eBPF Helps GitHub Break Circular Dependencies in Deployments

GitHub faces a unique challenge: it hosts its own source code on github.com, creating a potential circular dependency if the site goes down. Deploying fixes requires access to the same platform that might be unavailable. To solve this, GitHub has adopted eBPF (extended Berkeley Packet Filter) to monitor and block deployment scripts from creating new circular dependencies. This article explains the problem and how eBPF provides a robust solution.

1. What circular dependency does GitHub face when deploying github.com?

GitHub runs its own infrastructure on the same platform it provides to users: github.com. This means that to deploy updates or fix issues on github.com, engineers need access to GitHub itself—for example, to pull source code or release binaries. If an outage makes github.com unavailable, the deployment process cannot proceed, creating a circular dependency. For instance, during a MySQL outage, GitHub might be unable to serve release data, blocking the very scripts needed to fix the database. This self-referential loop is a critical risk that GitHub must carefully manage to ensure platform reliability.

How eBPF Helps GitHub Break Circular Dependencies in Deployments — Source: github.blog

2. What are the three types of circular dependencies that can affect deployments?

In a deployment scenario, circular dependencies fall into three categories. Direct dependencies occur when a deploy script attempts to pull a resource (e.g., an open source tool) from GitHub, but the platform is down. Hidden dependencies arise when a tool already on a machine, during execution, checks GitHub for updates—if GitHub is unreachable, the tool may fail or hang. Transient dependencies happen when a script calls an internal service (like a migrations API), which then tries to fetch something from GitHub, propagating the failure back. Each type represents a distinct way the dependency loop can break a deployment.

3. Why can't GitHub rely solely on code mirrors to solve deployment circular dependencies?

GitHub does maintain a mirror of its source code and pre-built assets for rollback scenarios. However, this only covers the initial fix-forward and rollback cases. The real problem is more subtle: deployment scripts themselves can introduce new circular dependencies during execution. For example, a script might download a binary from GitHub or call an internal service that inadvertently contacts GitHub. These script-level dependencies are not addressed by code mirrors. The onus was previously on each team to manually review their scripts, which proved error-prone and incomplete. A more automated, real-time solution was needed.

4. How does GitHub use eBPF to prevent circular dependencies during deployments?

eBPF allows GitHub to runsandboxed programs in the Linux kernel without changing kernel source code. For deployment safety, GitHub attaches eBPF programs to system calls (like connect() or open()) made by deploy scripts and their child processes. The eBPF probes inspect each call's target—for example, a network connection to an internal service or a file open for a download. If the call attempts to reach a blacklisted endpoint (e.g., GitHub's own servers or other internal services that could create a loop), the eBPF program logs a warning or blocks the call entirely. This selective monitoring and enforcement happen in real time, preventing circular dependencies from forming without requiring script modifications.

5. Can you give an example of a direct circular dependency in a MySQL outage scenario?

Imagine a MySQL outage that makes GitHub unable to serve release metadata. An engineer runs a deploy script on a MySQL node to apply a configuration fix. The script's first step is to pull the latest binary of an open-source tool from GitHub releases. Because GitHub is down, the download fails, and the script cannot continue. This is a direct circular dependency: the fix requires access to the very platform that is out of service. Without eBPF to block or reroute such calls, the incident escalates. With eBPF, the script's attempt to reach GitHub would be blocked, and the script could fall back to a locally cached version or an alternative source, breaking the loop.

6. What is a hidden circular dependency and how can eBPF detect it?

Hidden circular dependencies are more insidious. Suppose a deploy script uses a servicing tool already present on the disk. That tool, when launched, automatically checks GitHub for an update—perhaps by hitting an endpoint like api.github.com/repos/.../releases/latest. If GitHub is unreachable, the tool may wait indefinitely on a timeout or report an error, causing the entire script to hang. Because the dependency is not visible in the script's source code, it can go unnoticed during reviews. eBPF can intercept the tool's network calls and block any that target GitHub's infrastructure, forcing the tool to skip the update check or use a locally stored version, thus preventing the hidden circular dependency from disrupting the deployment.

Tags: