What is Build Pipeline?
A build pipeline is a defined, usually automated workflow that turns a particular source revision and its required inputs into checked software outputs. It prepares the build environment, resolves dependencies, compiles or packages the project, runs the required checks and records the resulting artefacts. A build pipeline can stop with a tested package. It does not have to deploy anything.
The useful question is not simply whether the dashboard is green. It is which revision was built, what inputs were used, which checks actually ran and which output passed them. Those details separate a repeatable engineering process from a script that happens to work on one machine.
Build pipelines sit within the development and automation cluster in the Techie Broadband terminology guide. This article follows the source, runner, checks and artefact, then examines the broadband paths that matter for a UK home lab.
Build pipeline, CI and deployment: what is the difference?
Continuous Integration is the practice of integrating changes frequently and verifying the shared codebase. A build pipeline is one mechanism that performs that verification. Running a pipeline against a long lived isolated branch does not, by itself, mean the team is integrating frequently.
Concept | Main question | Typical outcome |
Build pipeline | Can these inputs produce an acceptable output? | Checked package, image or bundle |
Continuous Integration | Do frequent changes work together? | Feedback on the integrated codebase |
Continuous Delivery | Can this version be released reliably when authorised? | A releasable version and release process |
Continuous Deployment | Do qualifying changes reach production automatically? | An automated production change |
Deployment | What version is installed or running here? | A changed target environment |
Deployment consumes an artefact and changes an environment. A wider delivery workflow can contain both build and deployment jobs, but their responsibilities remain distinct. A production approval, environment configuration, database migration or health check is not implied by a successful compilation.
Not every build compiles a programming language. A static site may render pages and bundle assets; a Python project may create a package; a container build assembles an image. A pipeline can also build firmware or validate a configuration bundle. Define the expected output before selecting the stages.
What goes into a build pipeline?
The source revision is only one input. The result can also depend on package versions, the compiler or runtime, the operating system, build flags, generated files, a container base image and services contacted during the run. A floating dependency or moving image tag can change the result even when the application commit stays the same.
Repository Hosting makes the shared source available to authorised clients. It is the runner that normally checks out a revision and executes the build commands. Storing a commit on the host does not mean it has been built, tested or deployed.
Record the source revision and the version of the pipeline definition used for the run.
Declare dependency and toolchain requirements, including lock files where the ecosystem supports them.
Specify the target operating system and architecture when they affect the output.
Keep build configuration under review and separate sensitive credentials from ordinary source files.
Define the expected artefact, required checks and retention policy before enabling publication.
A trigger starts a run. It might be a push, a pull request, a tag, a schedule or a manual request. Trigger filters matter: a skipped path, branch mismatch or duplicate event can explain a missing or repeated run. A pull request run may build a proposed merge revision rather than the contributor's branch tip, so inspect the actual checked out revision.
Stages, jobs, steps and runners
Names differ between platforms. Microsoft Learn's Azure Pipelines concepts describes stages containing jobs, jobs executed by agents and steps that perform individual tasks or scripts. In that terminology, a stage groups work and a job is an execution boundary. Some control or approval jobs do not need a normal build agent.
Component | Responsibility | Example |
Trigger | Decides when to request a run | Push to an allowed branch |
Orchestrator | Schedules work and evaluates dependencies | Start packaging after required checks |
Runner or agent | Provides execution resources | Linux machine with Python installed |
Job | Groups related execution work | Build an ARM package |
Step | Performs an individual operation | Run the project's test command |
Artefact store | Retains identified run outputs | Store a package and test report |
Do not assume every pipeline is a single line of tasks. Independent jobs can run together; dependent jobs wait for their prerequisites. This dependency graph is commonly a directed acyclic graph, or DAG. GitLab supports dependency based scheduling alongside stage ordering, and GitHub uses job dependencies such as needs. The exact defaults and conditions belong to the chosen platform.
A runner supplies CPU, memory, storage and network access. Separate jobs may execute on separate machines, so a file created in one job is not automatically available in the next. Define the handoff through supported artefact storage, outputs or another deliberate shared service. Passing a filename as a job output does not transfer the file itself.
Steps often run in separate processes. Environment changes made inside one process may not survive into another unless the platform provides a supported mechanism. Likewise, a container's own loopback address refers to that container's network context, not automatically the runner host or a service container.
The infographic below summarises the controlled inputs, build work, required checks and recorded output. Its cache and artefact panels separate two storage roles that are frequently confused.
Artefacts are outputs; caches are shortcuts
An artefact can be a binary, package, container image, static site bundle or configuration archive. Test reports and diagnostic logs are also workflow outputs, although they are not necessarily the deployable product. Give releasable outputs an identity that ties them to their source revision and run.
GitHub's workflow artefact documentation distinguishes retained run outputs and file handoffs from dependency caching. An artefact carries a result forward; a cache helps avoid repeating expensive work. A cache miss should normally make a correctly designed build slower, rather than make it impossible.
Storage role | What it contains | Failure to avoid |
Dependency cache | Reusable downloaded packages or valid intermediate results | Treating a cache as the only copy of an input |
Build artefact | The output associated with a particular run | Deploying an old output left in a directory |
Diagnostic output | Logs, reports and failure evidence | Deleting evidence before investigating |
Release repository | Outputs retained for approved distribution | Assuming temporary run retention is permanent |
A useful cache key reflects inputs that make the cached data compatible, such as the dependency lock file, operating system, architecture and relevant toolchain. Broader fallback keys can restore older material, so the normal dependency resolution or validation step must still make the result correct. Never put credentials in a cache.
A large cache is not automatically faster. Restoring, unpacking and uploading it can cost more than downloading the small set of packages the job needs. Compare cold and warm runs by stage duration, transferred bytes and failure rate. Preserve the ability to build from declared inputs without the cache.
Keep the identity of the output that passed
Where appropriate, build once and promote the same identified output through later environments. Rebuilding separately for staging and production can introduce different dependencies or tooling. Comparing artefact digests helps establish that the bytes being deployed are the bytes previously checked.
This principle has limits. A front end that embeds environment settings during its build may need distinct outputs for distinct environments. Treat each as a separate build with its own recorded inputs and checks, or redesign configuration to be supplied at runtime where suitable. Do not claim two different builds are the same merely because their source commit matches.
A checksum detects a byte difference. It does not prove who built the file, whether its tests were adequate or whether it is safe. An authenticated provenance statement adds evidence about the build process. SLSA's approved version 1.2 build provenance specification records the build definition, builder, resolved dependencies where known and output identity. Verification still needs an expected trusted builder and policy.
What does a successful build actually prove?
A passing result means the configured operations met the configured conditions. It does not prove that all possible defects were tested, that production credentials work or that the application is healthy after installation. Compilation checks a different property from an integration test; a vulnerability scan checks a different property again.
Decide which failures must block publication. An optional diagnostic upload may run even after a test failure so the evidence survives. That does not justify running the releasable package publication step after the same failure. Conditions and settings that allow errors must be deliberate and visible.
Techie Broadband test: a failed check blocks the artefact
We ran a small local packaging experiment on Linux with Python 3.12.14. Three fresh working directories each held a function returning a service port, a unittest check expecting 8080 and a short pipeline script. Python bytecode writing was disabled for these runs so previously cached imports could not hide a source change.
Create router_config.py with service_port() returning 8080.
Create a unittest check asserting that service_port() equals 8080.
Run Python's unittest discovery in a subprocess and save its output as the test log.
If the test process returns a nonzero status, return that failure without copying the releasable file.
If it succeeds, copy router_config.py into the run's artefact directory and calculate its SHA256 hash.
Repeat in a fresh directory, then run a third fresh directory with the returned port changed to 9090.
Run | Observed test result | Observed releasable output |
First clean run: 8080 | Exit status 0 | File copied and hash recorded |
Second clean run: 8080 | Exit status 0 | Identical file hash |
Changed input: 9090 | Exit status 1; assertion failed | No releasable file copied |
The two passing files produced the same SHA256 hash, beginning 0ac044cb23c1fd22. The failing run retained its test log but did not create the releasable file. This demonstrates the separation between collecting failure evidence and publishing a successful output.
The experiment packaged one source file rather than compiling a full application. It did not test a cloud CI service, network speed, dependency security, runner isolation or deployment. Two matching hashes show repeatability for these controlled runs, not universal reproducibility. The important result is that an unacceptable input could not silently become a published package.
Repeatable, reproducible and trustworthy are different
A repeatable process can be executed again using the same documented steps. The Reproducible Builds project uses a stronger criterion: the specified source, environment and instructions allow another party to recreate identical artefact bytes. Automation alone does not meet that criterion.
Timestamps, archive ordering, absolute paths, locale, random values and undeclared network inputs can change bytes. Lock files and pinned tools reduce uncertainty but do not eliminate every source of variation. A pipeline should record enough information to explain its output even when full reproducibility is not yet achievable.
For npm version 11, the npm ci command requires an existing lock file and fails if it conflicts with the package manifest instead of rewriting it. That is a useful dependency control, not a guarantee of complete build reproducibility. Install flags and any dependency scripts still matter.
Docker documents that image tags can move. Pinning a base image by digest identifies particular image content, but that content still needs reviewed updates for security fixes. A permanently pinned vulnerable image is predictable rather than automatically safe. Preserve an update process alongside input control.
Where UK broadband affects a build pipeline
Locate the runner before blaming the broadband connection. With a cloud runner, dependency downloads and most build artefact transfers happen between cloud services. Your home connection carries the source push, dashboard traffic and any files you fetch locally; it does not necessarily carry the runner's package downloads.
With a home lab runner, dependency downloads usually use your downstream connection and cloud artefact uploads use your upstream connection. A fast advertised download rate does not remove a slower upload stage. Techie Broadband's Upload Speed guide explains that outgoing direction and the difference between capacity and achieved throughput.
GitHub's self hosted runner reference specifies outbound HTTPS connections over port 443 for communication with its service, with additional reachable domains depending on the job. A normal outbound runner connection does not require you to expose a build server through router port forwarding. A deployment target or another CI product can have different access requirements.
Runner location | Home broadband carries | Likely constraints to inspect |
Cloud runner | Source push, logs and local downloads | Cloud runner size, queue, registry and quotas |
Home lab runner | Source, dependencies, caches and cloud uploads | Upstream, downstream, CPU, RAM and disk |
Private build host | Traffic along its actual network path | Routing, firewall, registry and shared resources |
Worked broadband calculation: follow both directions
The following figures are original theoretical calculations, not measured ISP results. Assume decimal megabytes, an otherwise idle connection, sustained payload rates equal to the stated rates and no protocol, disk, compression or server delay. Transfer time in seconds equals size in MB multiplied by 8, divided by rate in Mbps.
Transfer | Assumed rate | Ideal transfer time |
Download 600 MB of dependencies | 100 Mbps downstream | 48 seconds |
Upload a 250 MB artefact | 20 Mbps upstream | 100 seconds |
Upload the same 250 MB artefact | 100 Mbps upstream | 20 seconds |
In this illustration, raising upload capacity from 20 to 100 Mbps saves 80 seconds for that upload, an 80% reduction in its transfer time. It does not make compilation five times faster. A cache can avoid some downloads, while a larger cache upload can add new upstream work. Measure what actually crosses the line.
Small serial requests can also make latency important. Shared household traffic, WiFi limitations, loss, congestion, registry limits and storage performance can reduce useful rates. Use stage logs and transfer measurements to distinguish a network wait from CPU work or a runner queue.
Original timing model: elapsed time is not total work
Consider a simplified pipeline with 1 minute of setup, 3 minutes of build work, 4 minutes of unit tests, 5 minutes of analysis and 1 minute of publication. These are illustrative durations selected for the model, not measurements from a vendor or our local experiment.
Assume both checking jobs depend on the completed build, can run independently on available runners and need no extra setup or transfer time. Publication waits for both. Running everything sequentially takes 1 + 3 + 4 + 5 + 1 = 14 minutes. Running the two checks together takes 1 + 3 + max(4, 5) + 1 = 10 minutes.
Scheduling model | Elapsed time | Total modelled execution work |
All operations sequential | 14 minutes | 14 minutes |
Checks run in parallel | 10 minutes | 14 minutes |
Only one check runner available | Up to 14 minutes before other overhead | 14 minutes |
The ideal saving is 4 minutes, about 28.6% of the original elapsed time. The longest dependency chain determines the lower bound: the critical path still includes the 5 minute analysis job. Parallelism has reduced waiting, not removed the work. Real billing can differ by machine type, charging rules and repeated setup.
Optimise the actual critical path. Making the already shorter unit test job faster may not reduce the final completion time while analysis remains slower. Conversely, removing a shared download before both checks can help both branches. Queue delays and artefact handoffs must be included when comparing real runs.
Runner security and supply chain controls
A build runner executes repository code, tools and dependency scripts. Treat that as code execution with whatever access the runner possesses. A private repository does not make every contributor or incoming change trusted, and a container is not a sufficient security policy by itself.
GitHub's secure use guidance recommends minimal token permissions, reviewed workflow changes and pinned action revisions. It also warns that secret redaction is not guaranteed and that untrusted code can compromise persistent self hosted runners. Its guidance supports separating trusted publishing credentials from low trust build work.
Separate untrusted change checks from jobs holding release credentials.
Restrict runner access to production networks, personal files and unrelated services.
Use the minimum required credential permissions and short lived identity where supported.
Review pipeline definitions and third party build components alongside application code.
Keep secrets out of source, caches, output packages and diagnostic logs.
Keep the runner, operating system and toolchain maintained; rebuild clean environments deliberately.
For a home lab, ask what a compromised build could reach. The answer might include a NAS, router administration, backups or other machines on the LAN. Give the runner a narrow network and storage role. Reusing a privileged development computer for arbitrary incoming builds creates a wider trust boundary than the pipeline diagram suggests.
A software bill of materials lists relevant components, while provenance describes production history. Neither proves the software has no vulnerabilities or malicious behaviour. Use those records with checks and an explicit verification policy, rather than treating the existence of a signed file as the end of the assessment.
How to design a useful first pipeline
Define one expected output and the revisions or events that should produce it.
Run the project build and checks locally in a controlled environment first.
Declare the runner requirements, dependencies and toolchain versions.
Add the required checks and make their failure block releasable publication.
Store the output with its source revision, run identity, relevant reports and retention policy.
Verify a deliberately failing change so the failure gate is tested.
Add caching and parallel jobs only after measuring where time is spent.
Keep deployment separate until its approvals, credentials and target health checks are designed.
GitHub Actions, GitLab CI/CD, Azure Pipelines and Jenkins are examples of systems that can orchestrate this work. Their configuration formats, runner models and limits differ. Choose around your repository, required operating systems, trust boundaries, concurrency and maintenance capacity. Product names alone do not decide whether the resulting pipeline is correct.
Cost includes compute time, storage, data transfers and the people maintaining the system. Self hosting adds electricity, updates, hardware and recovery responsibilities. Avoid judging a design only by an advertised free allowance or runner price. Current provider entitlements need checking for the actual account and workload.
Troubleshoot the first failed boundary
Start with the run identity, exact revision and first consequential failure. A downstream job marked skipped may be behaving correctly because an earlier required job failed. Preserve that earlier log, its exit status and the declared runner environment before rerunning anything.
Symptom | Boundary to inspect | Useful next check |
No run created | Trigger and filters | Confirm event, branch and path rules |
Run queued | Scheduling and capacity | Check labels, availability and concurrency |
Dependencies fail | Registry, lock file or access | Read first resolver or connection error |
Works locally only | Environment and undeclared input | Compare versions, paths and clean checkout |
Tests pass despite an error | Conditions and exit status | Check allowed failures and actual test discovery |
Output missing in next job | Artefact handoff | Verify upload, name, download and path |
Wrong package deployed | Output identity | Compare revision, run ID and digest |
Upload slow | Network and storage | Locate runner, transfer direction and payload |
Do not fix a failing required test by allowing the error unless the check is intentionally becoming optional for a justified reason. Confirm that tests were actually discovered: a command can succeed while running no relevant tests. A green status without the expected report is evidence to investigate.
For dependency failures, distinguish authentication, name resolution, certificate validation, rate limiting and a manifest conflict. For intermittent tests, record the environment and inputs before labelling the test flaky. Blind retries can hide a race, resource shortage or dependency incident without correcting it.
When a package exists but cannot run, check its target architecture, required runtime and embedded configuration. When a cache hit causes unexpected behaviour, validate the key and cached contents, then compare with a controlled cold run. Do not confuse a stale output directory with a successfully produced artefact.
Frequently asked questions
Is a build pipeline the same as CI/CD?
No. A build pipeline performs defined build and verification work. CI/CD describes broader practices and delivery processes that may use several pipelines, including separate deployment work.
Does every build pipeline compile code?
No. Some compile binaries; others package code, render static sites, assemble container images or validate configuration. The expected output determines the work.
Does a successful build mean the software is deployed?
No. Deployment needs a separate configured action that changes a target environment. A passing build can finish with an artefact stored for later use.
Can I run a build pipeline in a home lab?
Yes. A suitable runner can execute jobs locally, but you must provide the resources, maintenance, network access and isolation needed for the workload and its trust level.
Is a cache the same as a build artefact?
No. A cache accelerates reusable work; an artefact retains a run output or passes it onward. A correctly designed build should handle an unavailable cache.
Will faster broadband make every build faster?
No. It can shorten transfers on the runner's actual network path. CPU work, tests, storage, queues and cloud service limits can remain the dominant delays.
Key takeaway
A build pipeline is a controlled route from declared inputs to an identified output that has passed defined checks. Keep source identity, execution, caches, artefacts and deployment separate in your reasoning. Test the failure gate, measure the critical path and locate the runner before attributing delays to UK broadband.
Sources and verification
Primary documentation checked on 12 October 2026 included Microsoft Learn, GitHub Docs, GitLab Docs, npm documentation for version 11, Docker's build guidance, the Reproducible Builds definition and the approved SLSA version 1.2 build provenance specification. The three contextual citations link to the most useful structural, artefact and security references. The local test and labelled timing calculations are original Techie Broadband work.
