Skip to content

hibp-downloader

pypi python build tests license

This is a CLI tool to efficiently download a local copy of the pwned password hash data from the very awesome HIBP pwned passwords api-endpoint using all the good bits; multiprocessing, async-processes, local-caching, content-etags and http2-connection pooling to probably make things as fast as is Pythonly possible.

Features

  • Direct password lookups via the query command — check passwords against the compressed data store with no database or decompression step needed. Fast enough to use behind a web service.
  • Download and store acquired data in gzip compressed format to save on storage and speed up queries.
  • Download the full dataset in under 45 mins (generally CPU bound).
  • Easily resume interrupted download operations into a --data-path without re-clobbering api-source.
  • Only download hash-prefix content blocks when the source content has changed (via content ETAG values); making it easy to periodically sync-up when needed.
  • Ability to generate a single text file with in-order pwned password hash values, similar to PwnedPasswordsDownloader from the awesome HIBP team.
  • Per prefix file metadata in JSON format for easy data reuse by other tooling if required.
  • Standalone validation command to verify the local copy dataset, clean up corrupted or incomplete files, and remove orphaned metadata files.

Install

pipx install hibp-downloader

Usage (download)

screenshot-help.png

Performance

Sample download activity log; host with 32 cores on 500Mbit/s connection. 

...
2024-05-16T10:18:01-0400 | INFO | hibp-downloader | prefix=f80c7 source=[lc:13616 et:3 rc:1002358 ro:25 xx:1] processed=[17836.6MB ~414462H/s] api=[918req/s 17597.4MB] runtime=36.4min
2024-05-16T10:18:02-0400 | INFO | hibp-downloader | prefix=f81af source=[lc:13616 et:3 rc:1002558 ro:25 xx:1] processed=[17840.1MB ~414454H/s] api=[918req/s 17600.9MB] runtime=36.4min
2024-05-16T10:18:02-0400 | INFO | hibp-downloader | prefix=f826f source=[lc:13616 et:3 rc:1002758 ro:25 xx:1] processed=[17843.6MB ~414454H/s] api=[918req/s 17604.4MB] runtime=36.4min
2024-05-16T10:18:03-0400 | INFO | hibp-downloader | prefix=f833f source=[lc:13616 et:3 rc:1002958 ro:25 xx:1] processed=[17847.1MB ~414450H/s] api=[918req/s 17607.9MB] runtime=36.4min

  • 918x requests per second to api.pwnedpasswords.com
  • Log sources are shorthand:
    • lc: local-cache - request-responses handled locally without hitting the network.
    • et: ETag match - request-responses that confirmed our local data was up-to-date and did not require a new download.
    • rc: remote-cache - request-responses that were downloaded to local, but came from the remote-server cache.
    • ro: remote-origin - request-responses that were downloaded to local, and the download needed to be fetched from remote origin source.
    • xx: unknown/failed - request-responses that failed (and successfully retried).
  • ~17GB downloaded in ~36 minutes (full dataset)
  • Approx ~414k hash values received per second
  • Processing in this example appears to be CPU bound, measured traffic around ~160 Mbit/s.

Usage (query)

Query passwords directly against the compressed data store — no decompression, no database import required. This is the recommended approach for any password-checking lookup. screenshot-help.png

Usage (generate)

Generate a single decompressed text file from the data store. If you are generating this to import into a database for lookups, consider using the query command directly instead — it is faster to set up, far easier to maintain, and uses a fraction of the storage. 

hibp-downloader --data-path /path/to/data generate --filename pwned-hashes.txt --hash-type sha1

Usage (validate)

Validate local pwned password files and automatically clean up corrupted data or orphaned metadata files: 

hibp-downloader --data-path /path/to/data validate --hash-type sha1

Project