top of page

What is Data Integrity?

Oct 23
12 min read
What is Data Integrity?

 

Data integrity means preserving data against unintended or unauthorised alteration, loss or destruction, while maintaining the rules and relationships needed for it to remain usable. It matters when data is created, edited, transferred, stored and restored. Integrity controls help establish whether the intended information has survived those operations, rather than merely whether a file exists or a transfer has finished.

 

There are several questions inside that definition. Did the bytes change? Is the required data complete? Are records consistent with the application's rules? Is the reference itself trustworthy? A matching hash can provide strong evidence about bytes, but it cannot decide whether an incorrect figure was entered correctly in the first place.

 

The Techie Broadband terminology guide connects integrity with networking, security and storage. The infographic introduces the main checks and their limits; the practical comparison below shows why one successful check does not answer every integrity question.

 

Data integrity infographic explaining trusted references, content hashes, change detection, validation, recoverable copies and why matching bytes do not prove accuracy

 

What are you trying to preserve?

 

Data integrity does not require data to remain unchanged forever. A legitimate edit should change it. The objective is to preserve the intended state and permit controlled, explainable changes while detecting or preventing changes that should not happen.

 

In security discussions, integrity focuses on protection against improper alteration or destruction. In database and operational discussions, it also includes the structural rules that keep information coherent. The control you need depends on the question you are asking.

 

Question

Property being checked

What the check does not establish by itself

Do these bytes match the expected version?

Content integrity

Whether the expected version is accurate

Are all required files or records present?

Completeness

Whether every value is correct

Do values and relationships satisfy the rules?

Logical consistency

Whether the rules cover every business mistake

Does this content come from the expected source?

Authenticity

Whether that source made a good decision

Can unauthorised people read it?

Confidentiality

Whether the data has remained intact

Can the intended system retrieve and use it?

Availability

Whether the retrieved content is the intended content

 

Accuracy is related but different. A temperature entered with the wrong unit, a mistyped account number or an incorrect quantity can be stored and copied perfectly. Integrity checks need a defined reference, rule or authorised process. Without that context, a claim that data is valid is incomplete.

 

How can data integrity be lost?

 

Failures can begin before a network transfer starts. An application can export the wrong records, a user can overwrite the wrong file or a script can apply an incorrect transformation. Storage, memory, cables and controllers can also introduce faults. Malicious changes add another threat, especially when the person changing the data can also change the evidence used to verify it.

 

Incomplete operations create a different problem. A folder copy may omit a file; a database migration may combine records from inconsistent points in time; a backup may contain the data but lack the credentials, configuration or external assets needed to restore it.

 

The data transfer explanation separates network delivery from application completion. A completed connection does not establish that the intended file set was selected or that the receiving application accepted the data correctly. Integrity needs checks at the points where those mistakes can occur.

 

Checksums, hashes, HMACs and signatures

 

These mechanisms all produce information that can help detect change, but they provide different evidence. Calling every short verification value a checksum can hide an important distinction between accidental-error detection and protection against deliberate replacement.

 

Mechanism

What it can support

Important limit

Error-detecting checksum, such as CRC32

Detecting many accidental changes

Not cryptographic authentication against an attacker

Cryptographic hash, such as SHA-256

Comparing content with a known digest

An untrusted replacement digest can match replacement content

HMAC

Authenticating content using a shared secret key

Anyone holding that key can generate a valid tag

Digital signature

Checking signed content against a public key

Trust depends on the key, signed scope and verification policy

 

A cryptographic hash processes input bytes into a fixed-size digest. SHA-256 produces 256 bits, commonly displayed as 64 hexadecimal characters. Repeating the calculation over identical bytes gives the same result. Comparing the full result with a protected expected digest is useful evidence that the selected bytes match.

 

NIST's hash-function guidance explains collision resistance and related security properties. A digest is not a mathematical guarantee that every possible input has a different output. Treat a suitable cryptographic hash as a security mechanism with defined assumptions, rather than calling it an infallible unique identifier.

 

Why the expected value matters

 

Suppose someone can replace both a download and the hash displayed beside it. The two can match perfectly. The comparison has established agreement with that displayed value, not an independent connection to the original publisher.

 

Protect the reference through an appropriate trusted route. Depending on the task, that could involve a signed release manifest, an independently retained baseline or an authenticated service whose authority you have checked. Simply putting two mutually consistent items in the same writable folder does not make either one trustworthy.

 

HMAC adds a secret to the calculation. A recipient with the appropriate key can verify the authentication tag, but the key must remain protected. A digital signature uses a private signing key and a corresponding public verification key. Obtaining the public key from an untrusted source without checking its identity can undermine the intended assurance.

 

Neither mechanism proves that an old valid message is current or that the signed information is factually correct. Where replay or version confusion matters, the authenticated content and application policy need to identify the intended version or context.

 

What a file-integrity test actually proves

 

Techie Broadband ran a small local test using Python 3.12.14 and SHA-256. The source contained quantity.txt with quantity=10 and a separate notes.txt file. A manifest recorded each relative path, byte count and full hash. The original manifest was retained independently for comparison.

 

The test deliberately separated byte matching, completeness and an artificial application rule: quantity must be zero or greater. These are synthetic examples, not measurements of a storage product, network failure rate or cryptographic attack resistance.

 

Test case

Check result

Practical lesson

Exact copy of both files

Full manifest matched

The selected file set and bytes agreed with the retained baseline

Quantity changed from 10 to 90

File stayed 12 bytes; original hash differed

Equal size does not mean equal content

notes.txt omitted

Remaining file hash matched; full manifest did not

Checking only files that arrived can miss missing files

Changed file accompanied by a new manifest

New manifest matched; original manifest did not

Replacing the reference can hide a change from a comparison against that replacement

Exact copy of quantity=-1 from a separate source

Hash matched its source; application rule failed

A perfectly preserved copy can retain invalid data

 

The change from 10 to 90 also passed the non-negative rule. That rule could reject the negative quantity, but it could not determine whether 90 was an authorised or accurate replacement for 10. Content comparison and application validation answered different questions.

 

In a separate check using a synthetic shared key, an HMAC-SHA-256 tag verified the original quantity file and rejected the modified content against the original tag. This demonstrated the mechanics of the check. It was not a test of key-management security or proof that a real system's secret could not be stolen.

 

A manifest needs scope as well as hashes

 

A useful manifest identifies what must be present, not merely the files a verifier happened to find. Relative paths help distinguish two files with the same name in different directories. Record the intended version or snapshot, and decide whether extra files should also make verification fail.

 

Content hashes alone do not cover every attribute. Ownership, permissions, timestamps, directory structure, links and external dependencies may require separate checks. A filesystem snapshot and a folder of downloaded files may therefore preserve different things even if their ordinary file contents match.

 

Compare the same representation. A compressed archive has different bytes from the extracted content. Different line endings, encodings or an approved metadata change can also produce a mismatch. Establish whether you need identical bytes or equivalence under an explicit transformation before treating every difference as corruption.

 

Logical integrity in databases

 

Databases need more than intact storage blocks. They also need rules describing acceptable values and relationships. Common controls include:

 

  • Required-value rules that reject a missing value where the field must be populated.

  • Primary keys that identify records, and uniqueness rules for values that must not be duplicated.

  • Foreign keys that maintain permitted references between related records.

  • Check constraints and application validation that enforce defined conditions on values.

 

PostgreSQL's constraint documentation describes these mechanisms. The exact definition matters: a foreign key can permit a null value unless a separate rule forbids it, and a check expression can evaluate to null without rejecting the row. Rules must be designed for the intended data model.

 

Transactions can group related changes into an all-or-nothing operation. This helps avoid committing only part of an intended update. It cannot establish that the application chose the correct customer, quantity or business rule, and transaction design still needs appropriate concurrency handling.

 

Physical page checksums address a different layer. For example, PostgreSQL 18 can verify checksummed data pages when they are read, but its checksum protection does not cover every internal structure or temporary file. Check the actual database version and cluster configuration rather than assuming the feature is enabled everywhere.

 

For a live database migration, use its supported consistent export, snapshot or backup process. Hashing files gathered at different moments cannot make the collection represent one consistent application state. After restoration, check the schema, required records, relationships and application behaviour as well as the transferred archive.

 

Integrity during transfer and storage

 

Protection works at defined boundaries. A network protocol, encrypted connection, filesystem and application can each check different data at different stages. Their assurances should be combined without assuming that one layer covers every other layer.

 

TLS provides an authenticated, encrypted channel between its endpoints. TLS 1.3 uses authenticated encryption to detect changes to protected records. That does not validate a wrong file selected before transmission or prevent an authorised application from changing it after receipt.

 

A NAS or server can add storage-level checksums. These can expose damaged blocks when data is read or during an integrity scan, depending on the implementation. Checksum coverage, error reporting and repair behaviour need to be understood for the storage system actually in use.

 

Detection is not the same as repair

 

OpenZFS's scrub and self-healing explanation describes verification of stored blocks and repair where a valid redundant copy is available. Without the necessary good data, identifying a mismatch does not reconstruct the original content.

 

Redundancy therefore matters, but it is not an unlimited guarantee. If all usable copies are damaged, or an application deliberately writes the wrong valid content, the storage system may have no correct alternative to select. A checksum can faithfully protect a bad application write once that write has become the stored state.

 

A scrub is also a different task from restoring an older application version. It checks storage integrity rather than deciding that yesterday's database or document was the one you intended to keep.

 

Backups, synchronisation and recoverability

 

Cloud sync keeps selected state aligned between systems. That can propagate an unwanted edit or deletion. Recoverability depends on the service's retention, versioning and restore behaviour, not simply the number of devices showing the same current content.

 

Keep recovery points that can survive the failure you are planning for. Consider who can modify or delete them, how long useful versions remain and whether the expected hashes or manifests are protected from the same incident. A writable backup and writable verification file under one compromised account may share a failure mode.

 

Test recovery to an isolated destination. Check that required files or records are present, that the restored state is appropriate and that the application can use it. A matching backup-archive hash confirms agreement with the reference for that archive; it does not establish a successful database restore or the presence of every external dependency.

 

Verification also takes time

 

Reading data to calculate its hash uses storage bandwidth and CPU. As a theoretical illustration, reading 1 decimal TB at a constant 200 MB/s takes 1,000,000 MB ÷ 200 = 5,000 seconds, or 1 hour 23 minutes 20 seconds. This excludes hash-processing limits, seeks, competing workloads and other overheads. It is not a measured scan speed.

 

Comparing two copies may require reading both, unless an appropriate trusted reference already exists. Cached data and verification built into a transfer can change the work involved. Do not assume that an integrity check is free, or that it always requires downloading everything again.

 

For UK homes and small offices keeping off-site copies, upload speed affects how quickly new backup data leaves the premises. It does not prove the integrity of that backup. Plan the transfer, verification and restore stages separately, and test the actual recovery path rather than relying only on a broadband speed result.

 

How to verify important data

 

Start by stating what success means. A firmware download, a family photo archive and a database migration need different evidence. The process should connect the selected source with the intended usable result.

 

  1. Identify the intended source, version, file set or consistent application snapshot.

  2. Obtain or create an appropriate reference, and protect it from the changes you need to detect.

  3. Record required content and attributes, including dependencies that live outside the main file set.

  4. Perform the transfer or backup using the supported method, then compare full hashes or verify signatures where appropriate.

  5. Check completeness, permissions, format validity and relevant application rules separately.

  6. Test use or restoration of the result and retain enough evidence to explain what passed and what remains unchecked.

 

For changing source data, use a suitable stable view. Otherwise, a file can change between creating the reference and reading it for comparison, producing a result whose cause is unclear. A consistent snapshot or controlled export helps define which state was actually checked.

 

Keep verification evidence specific: which algorithm, which version, which files, which time and which reference. A green tick without that scope can encourage confidence beyond what the check supports.

 

What to do when an integrity check fails

 

A mismatch is evidence of disagreement. It is not, by itself, evidence that the broadband provider corrupted the file or that a particular disk has failed. Preserve the evidence and investigate the boundaries of the comparison.

 

Observation

Possible explanation

Next useful check

Same size, different hash

Content changed, wrong version or a byte-level transformation

Compare the intended version and exact representation

Every received file passes, but something is missing

Verification omitted the expected file set

Compare paths and counts against the complete manifest

Hash changes between repeated reads

Live edits or an unstable read/storage path

Stop legitimate writers safely and inspect system errors

Valid digest or signature, incorrect contents

Wrong trusted reference, authorised mistake or insufficient rules

Check provenance, version and application validation

Storage checksum errors appear

Damage somewhere in the protected storage path

Inspect storage logs and redundancy before attempting repair

Archive verifies but restore fails

Missing dependencies, permissions, incompatible format or inconsistent backup

Review restore logs and the supported backup procedure

 

Avoid overwriting the only potentially good copy or regenerating the baseline merely to make the error disappear. Preserve logs, manifests and relevant versions. Investigate whether the source, destination or reference changed, and use the storage or application's supported recovery procedure.

 

If the data is actively changing, stabilise it through an appropriate application process before repeating the comparison. For suspected hardware or storage instability, repeated copying alone may create more uncertain copies. Establish which data is trusted before deciding what to replace.

 

Reducing integrity failures

 

The strongest arrangement combines prevention, detection and recovery at the layers where each is useful:

 

  • Restrict who and what can modify important data, signing keys and verification references.

  • Validate inputs and enforce meaningful application and database rules.

  • Use authenticated transfer methods and check stored data with the system's supported integrity mechanisms.

  • Retain protected recovery points and test that the required application state can be restored.

 

Authorised changes should leave an understandable record. Monitoring and audit information help distinguish an expected deployment or edit from a suspicious modification, but logs also need suitable access control and retention.

 

Choose checks according to the consequence of failure. Checking a downloaded installer against a trusted publisher's evidence, scanning a NAS and validating a customer-data import are different jobs. More checks are useful when they close a real gap, not merely when they repeat the same assurance.

 

Key takeaway

 

Data integrity is an ongoing relationship between the intended source, the permitted changes, the stored or transferred content and the result the application needs. A hash is valuable evidence within that relationship. It cannot replace a trusted reference, completeness checks, meaningful rules or recoverable copies.

 

Frequently asked questions

 

Is data integrity the same as data accuracy?

 

No. Accuracy concerns whether information correctly represents what it should describe. Integrity concerns preserving and controlling the intended data and its rules. An incorrect value can be stored and copied without any byte-level change.

 

Does a matching SHA-256 hash prove a file is safe?

 

A full matching SHA-256 digest provides strong evidence that the selected bytes agree with the expected digest under the algorithm's security assumptions. It does not independently establish that the reference is trusted, the content is accurate or software is safe to run.

 

Can a checksum repair corrupted data?

 

A checksum can help detect a disagreement, but it does not normally contain the missing original file. Repair requires an appropriate source of correct data or sufficient redundancy, together with a mechanism that can use it.

 

Does encryption guarantee data integrity?

 

Encryption and integrity are different properties. Authenticated encryption can provide both for the protected content, as in TLS 1.3 records. That protection does not automatically extend to data before it enters or after it leaves the protected boundary.

 

Is synchronisation enough to protect data integrity?

 

Synchronisation can make multiple locations agree with the current state, including an unwanted edit or deletion. Protected recovery points, validation and restore testing address risks that synchronisation alone does not resolve.

 

Why can two visually identical files have different hashes?

 

Their bytes may differ because of line endings, encoding, embedded metadata, compression or another transformation. Check the exact representation and intended version. A visual match is not the same as a byte-for-byte match.

 
 
UK Techie Broadband Providers
bottom of page