What is Object Storage?
Object storage is a way of storing data as individually named objects, each with its content and descriptive metadata. An application retrieves an object by its key within a bucket or container, usually through an API over HTTPS. It is useful for backups, photographs, video, software outputs and large datasets, but its access model differs from a shared folder or a disk attached to a computer.
For a UK home lab or engineering team, the important distinction is between where data is kept and how quickly it can travel. A scalable object store does not increase your broadband upload capacity, make an incomplete backup recoverable or automatically provide the file system behaviour an application expects.
This guide places object storage within the wider networking and infrastructure concepts in the Techie Broadband terminology guide, then follows an object from creation to a verified restore.
What is inside an object?
The payload is the actual data: for example, a JPEG, a compressed backup chunk or a CSV dataset. The key is the name an application uses to identify it within the storage namespace. Metadata describes the object, such as its content type, size or creation time. Providers can also support separate tags, version identifiers and other management information.
Component | Illustrative value | What it does |
Bucket | labbackups | Groups objects and provides a policy boundary |
Key | photos/2026/photo.jpg | Identifies the named object within the bucket |
Payload | The JPEG bytes | Contains the actual photograph |
Metadata | Content type: image/jpeg | Helps software interpret the stored data |
Version | Provider assigned identifier | Distinguishes retained versions where supported |
The example names above are illustrative and are not a live storage location. The key is not necessarily a random identifier or a hash. Applications often choose readable names, while content addressed systems may choose names derived from a digest. Uniqueness belongs to the relevant namespace: the same key can appear in different buckets.
AWS describes Amazon S3 objects, buckets and keys in its S3 technical overview. This is a useful concrete implementation of the model, rather than a guarantee that every object storage service behaves identically.
A slash in a key does not always create a directory
In an ordinary flat object namespace, photos/2026/photo.jpg is one object key. A console can display photos and 2026 as folders by interpreting prefixes and delimiters. That presentation does not necessarily give you a directory that can be renamed atomically or permissions that behave like a local file system.
AWS documents this distinction for general purpose S3 buckets. There are also important exceptions: Azure Data Lake Storage can add a hierarchical namespace, and specialised storage products can expose directory operations. Check the actual bucket type and service capabilities before treating a folder view as a file system.
Metadata and tags serve different purposes
Metadata can tell a browser that a payload is an image, how it should be cached or whether it should download as an attachment. User metadata can record an application identifier. Tags often support classification, access policies, lifecycle rules or cost grouping. They are not interchangeable with the object key or its payload.
AWS documents that ordinary user defined S3 metadata is set at upload and changed through a copy operation. Tags have separate operations. Other providers have different update rules, so a tool that changes a label may be changing a tag rather than rewriting the stored metadata. Avoid placing credentials or confidential information in names and descriptive fields.
Object storage, file storage and block storage compared
Model | Access unit | Typical fit |
Object | Named object through an API | Backups, media, datasets and application assets |
File | File in a directory hierarchy | Shared folders and applications expecting file semantics |
Block | Addressed blocks exposed as a volume | Operating system disks and workloads needing disk semantics |
File storage commonly exposes protocols such as SMB or NFS. Block storage gives an operating system a volume on which it can create a file system. Object storage gives software an API for named data. The useful question is which operations your application needs, rather than which model has the most attractive capacity figure.
A mounting tool or gateway can make object storage look like a drive, but it translates operations between different models. Check locking, rename behaviour, random writes, caching and failure recovery. Do not put a conventional database data directory on an object mount simply because the directory is writable. A database explicitly designed to use object storage is a different case.
Cloud storage is a broader term covering multiple storage models. A NAS is a network appliance or system that often presents file shares. S3 is a particular service and API ecosystem. An object store can also run on your own infrastructure: Ceph Object Gateway, for example, exposes S3 and Swift compatible interfaces.
Follow an upload and retrieval through the system
The application chooses a bucket and object key, prepares the payload and obtains credentials or a permitted signed request.
The client connects to the storage endpoint. DNS resolution, network connectivity and TLS establish the route and protected connection.
A PUT request or supported upload workflow sends the data and relevant metadata. The service checks authorisation and validates supported integrity information.
The storage system commits the object according to its implementation and returns the operation result. The application records the key and any version or checksum information it needs.
An authorised GET request retrieves the payload. A HEAD request can inspect metadata without transferring the payload; a listing operation discovers names within the permitted scope.
The receiving application verifies the result and interprets the payload. A successful transfer is only useful if the intended data can actually be consumed.
PUT and GET describe API operations, not the physical arrangement of disks. A distributed store can place replicas or erasure coded fragments across its storage nodes. Clients generally address the logical object rather than selecting individual disks. Placement, redundancy and failure tolerance belong to the implementation and chosen configuration.
The infographic shows the basic object model and API path. It also highlights the two checks that matter after capacity: whether you can restore the intended data and whether your upload rate supports the required workflow.
Whole object writes and partial reads
For the ordinary object model, changing the payload means writing a replacement object or a new version rather than editing arbitrary disk sectors in place. Some services offer specialised append or composition features; these should be treated as specific capabilities. Do not assume a file editor can issue the same operations it would against a local disk.
Reading only part of an object is a separate question. Amazon S3 supports byte range GET requests, which let compatible clients fetch selected ranges and retry smaller sections. A backup application may store many chunks as separate objects to limit the amount it must upload again when source data changes. That chunking belongs to the application design.
Consistency, versions and concurrent writers
Object storage does not automatically mean eventual consistency. AWS documents strong consistency for S3 object writes, reads and listings, including overwrites and deletes. Google Cloud Storage also documents strongly consistent object operations. Consult the specific service contract rather than repeating an outdated generalisation.
Strong consistency does not make a sequence of writes to different keys a single transaction. Two writers can still compete to replace the same name. Applications should use supported preconditions, version identifiers or their own coordination when updates must not overwrite each other silently.
In S3, conditional writes can require that a key does not already exist, or compare an ETag before replacing data. Versioning can retain multiple versions of a key. These mechanisms solve different problems: one prevents an unintended write in the first place, while the other can preserve earlier state after a write succeeds.
A cache can still serve an older response even when the origin store is strongly consistent. If object content is delivered through a Content Delivery Network, its cache policy and invalidation behaviour become a separate part of the update path.
Multipart uploads and integrity checks
Multipart upload divides a large object transfer into separately uploaded parts. The client initiates the upload, sends the parts and completes the operation so the store can assemble the object. Parallel parts can improve use of available capacity, and retrying a failed part can avoid starting the entire transfer again.
AWS documents this process in its multipart upload guide. In S3, uploaded parts continue to occupy billable storage until the upload is completed or aborted. Configure cleanup for abandoned uploads, and confirm completion rather than treating the presence of uploaded parts as a usable final object.
Multipart upload does not multiply your broadband line rate. Too much concurrency can consume CPU, memory, network capacity and request budget without increasing useful throughput. Measure before and after changing it, and use the client's supported retry and backoff behaviour rather than repeatedly hammering a failing endpoint.
An ETag is not a universal checksum
An ETag identifies an object representation for the relevant service. It is not a safe universal assumption that it equals the MD5 digest of a downloaded file. AWS documents differences for multipart uploads and encryption methods. Use the documented checksum algorithm and determine whether the value covers the whole object or is composed from part checksums.
For a controlled recovery test, calculate a trusted digest before uploading and compare it with the downloaded data using the same algorithm. This checks byte integrity. It does not prove that the backup contains every required file, that the database snapshot is consistent or that you can recover the application. Those require separate restore checks.
What limits object storage transfers over UK broadband?
Uploading sends data from your device towards the object store. The available upload speed therefore matters to an initial backup or dataset import. Download capacity matters when bringing the data back. On an asymmetric product these directions can have very different capacities.
For a restore, download speed is one constraint, alongside the storage service, destination disk and the application's ability to decode or rebuild the data. A fast downstream package cannot eliminate archive retrieval delays or the time required to verify a restored dataset.
The complete upload path includes the source disk, backup or transfer software, CPU and encryption, Ethernet or WiFi, switches, router, ONT or modem, access network, ISP path and receiving service. A bottleneck at any relevant stage can limit useful payload throughput. A 1 Gbps Ethernet link can also constrain a multi gigabit setup.
FTTP describes the access technology, not a universal upload ratio. FTTC, cable, mobile, fixed wireless and different fibre products have different characteristics. Check the actual ISP, access network, product and location. Symmetrical capacity can help uploads, but measured sustained throughput still determines whether the job finishes.
Ordinary outbound HTTPS uploads do not require a public static IPv4 address or port forwarding. CGNAT can complicate inbound access to a home hosted object service, but it is not automatically a barrier to uploading to a remote store. VPN routing, firewalls, DNS and service endpoint support can affect the actual connection.
Techie Broadband calculation: a 100 GB initial upload
This is a theoretical calculation, not a measured provider benchmark. Assume 100 GB means 100 billion bytes, the payload is already prepared, the stated upstream rate is available continuously and no time is spent on protocol overhead, retries, other traffic or storage processing. One byte is eight bits.
Time in seconds = payload bytes × 8 ÷ rate in bits per second. For 100 GB at 100 Mbps: 100,000,000,000 × 8 ÷ 100,000,000 = 8,000 seconds, or 2 hours 13 minutes 20 seconds.
Upstream rate | Ideal byte rate | 100 GB transfer time |
20 Mbps | 2.5 MB/s | 11 h 6 min 40 s |
100 Mbps | 12.5 MB/s | 2 h 13 min 20 s |
500 Mbps | 62.5 MB/s | 26 min 40 s |
1 Gbps | 125 MB/s | 13 min 20 s |
The rate in this table is upstream capacity, not the download number used to describe a broadband package. Real completion normally takes longer under these assumptions. Compression or deduplication can reduce the transmitted payload, while changed data, retries and competing uploads can increase it. Measure the bytes actually transferred.
Can the job fit inside a two hour window?
Techie Broadband calculation: a two hour window is 7,200 seconds. Sending 100 GB in that time requires 800,000,000,000 ÷ 7,200 = approximately 111.1 Mbps of sustained payload throughput. A 100 Mbps upstream line cannot meet that target even under ideal conditions.
If we deliberately reserve 20% of line capacity for other traffic, a simple planning model gives 111.1 ÷ 0.8 = approximately 138.9 Mbps of upstream capacity before allowing further headroom for overhead and variation. The 20% is an illustrative planning assumption, not a measured efficiency figure or a service guarantee.
At 80 Mbps of useful sustained payload throughput, the same 100 GB takes 10,000 seconds, or 2 hours 46 minutes 40 seconds. This is why a backup window should be designed around the actual workload and observed transfer rate rather than a product's headline download speed.
An incremental job can be a different workload
If a backup tool genuinely sends only 5 GB of new payload, the ideal time at 100 Mbps is 400 seconds, or 6 minutes 40 seconds. That does not follow merely from editing 5 GB of source files. The tool may rescan, rewrite larger objects, compress, encrypt or deduplicate data in ways that change the transmitted amount.
This gives a useful test: record source changes, uploaded bytes and elapsed time for several ordinary runs. If the transfer payload is unexpectedly large, investigate backup chunking and retention before concluding that more broadband capacity is the only answer.
Why many small objects behave differently
Total bytes alone do not describe the workload. A large sequential object transfer can spend most of its time moving payload. Thousands of tiny objects can spend proportionally more time on request setup, server processing, metadata operations and per object bookkeeping.
Illustrative calculation: if a strictly serial workflow spends 40 milliseconds on each of 10,000 requests, request time alone totals 400 seconds, or 6 minutes 40 seconds, before adding payload transfer. This is a deliberately simplified latency model, not a claim that each request always costs one network round trip.
Connection reuse, concurrency and batching can reduce serial waiting where the application supports them. Packaging small files into an archive can reduce object count, but it changes access and recovery granularity. Retrieving one small source file may then require a larger download. Choose the layout from how data will be read and restored.
Durability is not the same as a recoverable backup
Property | Question to ask |
Durability | Will the storage system preserve the committed data? |
Availability | Can I access the service when I need it? |
Recovery point | How much recent work could I lose? |
Recovery time | How long until the required service works again? |
Restore integrity | Are the recovered bytes and application state usable? |
A highly durable service can faithfully preserve an accidental overwrite or an encrypted backup you can no longer decrypt. Replication may propagate unwanted changes. Versioning, separate recovery credentials, suitable retention and restore testing address risks that redundant storage alone does not solve.
S3 Object Lock protects specified object versions using retention or legal holds where configured. It does not prevent new versions or delete markers from being created. Governance and compliance retention have different bypass rules. Treat immutability as a policy that must be tested against the actual permissions and recovery process.
For backup planning, identify the latest successfully restorable point, not merely the most recent scheduled job. Test recovery when the original device is unavailable. Keep the required encryption keys, application configuration and access procedures recoverable without depending entirely on the machine you are trying to restore.
Access control, encryption and signed URLs
Keep private data private and grant only the read, write, list and delete permissions each identity actually needs.
Use HTTPS for transport and assess the chosen encryption at rest and key management arrangements.
Separate routine upload permissions from authority to remove recovery copies or change retention.
Treat signed URLs as access credentials and avoid exposing them in public logs, analytics or messages.
Audit access and changes, and test expiry, revocation and denied operations using a safe test object.
A presigned URL delegates a particular permitted request for a limited period using the signing identity's authority. It can expire earlier when its underlying credentials expire or are revoked. In S3 it can be reused before expiry, so it is not automatically a single use download token.
Encryption and authorisation solve different problems. A service can decrypt an object for a caller who has permission, so encryption at rest does not excuse a public access policy. Client side encryption changes the recovery requirements again: losing the client key can make otherwise intact stored bytes unusable.
Regions, UK data handling and service choice
A nearby storage endpoint can reduce some network delay, but geographical proximity does not guarantee the fastest route or highest throughput. Test the actual region with representative data, and consider where the application or compute workers run as well as where you sit. Cloud to cloud processing need not pass through your home connection.
For UK organisations handling personal information, a UK server location alone does not settle international transfer obligations. The ICO's restricted transfer guidance explains why the contractual recipient, legal entities and processing arrangements matter. Review the provider agreement, subprocessors, remote access and configured replication alongside location.
This is a storage design consideration, not a claim that choosing one region guarantees compliance. Record what data is stored, where copies are configured to reside and who can access it. Check current organisational requirements before moving production datasets.
Costs, lifecycle policies and compatibility
A storage capacity price is only one part of the bill. Depending on the service, costs can include requests, retrieval, data transfer, minimum storage duration, minimum billable object size, retained versions, replication, monitoring and incomplete uploads. Check current terms for the selected region and class; this guide deliberately does not quote a price that may quickly change.
Archive classes can reduce storage cost while changing the recovery workflow. Some require a restore step before data becomes available, while other colder classes remain directly accessible. Do not equate every archive tier with instant downloads or every infrequent access tier with delayed retrieval.
A lifecycle rule can transition or expire objects and old versions. Test it against retention requirements and restore dependencies. Deleting a manifest, index or encryption metadata can make remaining payload chunks unusable to the backup tool even though the bytes still exist.
S3 compatible means that a service implements some relevant API behaviour, not that every AWS feature, permission model, checksum, retention rule or consistency property is identical. Test the client operations you rely on, including multipart completion, version retrieval and denied deletion.
Object storage can hold software outputs, but a package repository adds publication rules, dependency metadata, version selection and other application behaviour. The distinction matters when comparing a general bucket with dedicated artefact storage.
Troubleshooting a slow or failed transfer
Separate access failure from performance failure. Record the operation, error code, endpoint, time and request identifier without exposing credentials.
For denied requests, check the identity, bucket policy, requested key, region, signature and encryption key permissions. Check signed URL expiry and system clock accuracy where signing is involved.
If a browser application fails but the equivalent authorised client succeeds, inspect CORS and the browser console. CORS is not a replacement for storage authorisation.
For performance testing, use wired Ethernet, verify link negotiation, pause competing traffic and record source disk, CPU and transfer application activity.
Compare a large object with many small objects. Record uploaded bytes, useful throughput, retries and elapsed time against the same region and account.
Change one variable at a time: client concurrency, part size, endpoint, time of day or network. Keep results comparable and retain supported retry behaviour.
For a completed upload that appears missing, verify the exact bucket, key, version and permissions. Check caches and application indexes before assuming a consistency fault.
For a bill or capacity surprise, inspect retained versions, incomplete multipart uploads, replication and lifecycle configuration.
Finish with a restore to a separate destination. Verify trusted checksums where available and test whether the required data or application is usable.
Increasing retries will not fix a permanent permission error, and making a bucket public is not a sensible diagnostic shortcut. A slow transfer through one client also does not prove that the ISP is throttling storage traffic. Use the evidence from the complete path.
Choose the model from the workload
Your requirement | Starting point |
Off site backup or large immutable dataset | Object storage with a supported client and tested recovery |
Shared files with conventional locking and directory behaviour | File storage or a supported gateway with verified semantics |
Operating system disk or conventional database volume | Suitable block storage and file system |
Software package distribution | Repository or registry backed by an appropriate storage layer |
Public media delivery | Object origin plus the required access and caching design |
Before selecting a service, write down the largest object, object count, daily change volume, upload window, recovery target, client compatibility and required retention. Then test a representative upload and restore. That concrete specification is more useful than assuming all cloud storage products are interchangeable.
Frequently asked questions
Is object storage the same as Amazon S3?
No. Object storage is the model. Amazon S3 is one implementation. Other cloud services and systems on your own infrastructure can provide object storage with different APIs and guarantees.
Can I use object storage like a normal shared drive?
A gateway or mounting client can provide a drive interface, but its translation may change locking, caching and update behaviour. Use it only after checking the application's required file semantics.
Does object storage need symmetrical broadband?
No. It can work over asymmetric connections. Symmetrical capacity may shorten uploads when upstream capacity is the bottleneck, but the correct choice depends on measured workload, timing and the rest of the path.
Can object storage contain structured data?
Yes. An object can contain CSV, JSON, database exports or analytical formats as well as photographs and video. Storing a dataset does not by itself provide a transactional database or a query engine.
Is object storage automatically a backup?
No. It becomes part of a backup system when the application captures recoverable data and you configure access, retention, integrity checking and recovery. A synchronised copy that loses deleted files immediately may not provide the recovery point you need.
Why is a restore slower than my speed test?
The restore can include archive preparation, many object requests, throttling, decryption, decompression, disk writes and verification. A speed test measures a different workload and destination.
Key takeaway
Object storage gives applications a scalable way to keep and retrieve named data with metadata. Its usefulness depends on the access model, protection policy and workload, while broadband affects how quickly data reaches or leaves a remote store. Judge the complete system by a successful upload and a tested restore, not by stored capacity alone.
