WebMirror

WebMirror is a fault-tolerant website mirroring and public frontend archiving CLI engine designed to download publicly accessible website resources and reproduce them locally.

WebMirror downloads what is publicly delivered to the browser.

It does NOT recover:

• server-side source code
• PHP source
• Laravel source
• Django source
• Node backend source
• database contents
• .env files
• private APIs
• authentication-protected resources
• private application data

Features

• Isolated, batched downloading process using wget for bulletproof stability
• Automated fault tolerance against process crashes and glibc segmentation faults
• Native Webflow CDN detection and resolution
• Fully resumable downloads with a persistent manifest and staging architecture
• Atomic file replacements to eliminate corruption and duplicates
• Complete JSONL event stream for programmatic interoperability

Supported Website Types

Website	Can mirror?	What you get
HTML/CSS/JS	✅ Excellent	Almost complete frontend
Webflow	✅ Excellent	Rendered pages + public assets
React SPA	✅ Usually	Built JS/CSS + assets
Next.js static export	✅ Excellent	Static output
Next.js SSR	⚠️ Partial	Rendered frontend, not server code
Vue	✅ Usually	Built frontend
Angular	✅ Usually	Built frontend
WordPress	✅ Partial	Rendered pages/assets
PHP	⚠️ Frontend only	HTML generated by PHP
Laravel	⚠️ Frontend only	Rendered HTML/assets
Django	⚠️ Frontend only	Rendered HTML/assets
Node/Express	⚠️ Frontend only	Responses/assets
Rails	⚠️ Frontend only	Rendered pages/assets
Database-driven sites	⚠️ Partial	Publicly rendered data
Login/private application	❌/⚠️	Only accessible public content
API/backend	⚠️ Depends	Public API responses, not source
WebSocket application	⚠️ Limited	Not equivalent to backend
Server-side PHP/Laravel source	❌	Never exposed through normal HTTP

Architecture

flowchart TD
    A[Website URL] --> B[Site Analyzer]
    B --> C[URL Normalizer]
    C --> D[Crawler Queue]
    D --> E[Batch Downloader]
    E --> F[Verification]
    F --> G[Link Rewriter]
    G --> H[Local Mirror]

    D --> I[Persistent Manifest]
    E --> J[Retry Manager]
    E --> K[Network Monitor]
    K --> E

    E --> L[JSONL Events]

Loading

Future Architecture

The CLI engine will eventually power a rich, native Windows application:

Windows Tauri GUI
        ↓
      WSL2
        ↓
WebMirror Python CLI
        ↓
Crawler / Downloader / Verification

Fault Tolerance

A single wget process must never represent the entire mirror. If a naive script encounters a catastrophic error (like exit code 134), the entire mirroring process usually dies.

WebMirror isolates downloads into small batches. If the downloader crashes:

Batch -> wget crash -> engine_error -> preserve state -> retry/requeue -> new process -> continue

Network Resilience

WebMirror implements intelligent network resilience. If you lose internet connectivity mid-mirror, the engine will detect the failure, pause operations, and use exponential backoff to monitor for recovery. Once connectivity is restored, the engine seamlessly resumes exactly where it left off.

Resume / Recovery System

WebMirror persists state using atomic JSON file swaps in a .webmirror/ directory.

Example persistent state:

• manifest.json (The source of truth)
• queue.json
• state.json

If the mirroring process is killed unexpectedly, running the same command again will seamlessly resume the process. It uses strict SHA256 hashing and a staging architecture so that duplicate files are impossible.

Output Structure

A typical output directory will look like this:

example-copy/
├── .webmirror/
│   ├── manifest.json
│   ├── queue.json
│   └── events.jsonl
├── index.html
├── assets/
└── ...

Note

The .webmirror/ directory contains internal engine state and should normally not be committed to Git.

JSONL Protocol

WebMirror guarantees a stable JSONL output stream on stdout, intended for programmatic consumption (like a future GUI). stderr is reserved purely for diagnostic logs.

Example event stream:

{"type":"mirror_started","mirror_id":"...","url":"https://example.com/"}
{"type":"analysis_started"}
{"type":"site_detected","platform":"webflow","confidence":0.95}
{"type":"download_completed","url":"...","local_path":"index.html","bytes":143178}
{"type":"progress","downloaded_files":1,"downloaded_bytes":143178,"queued_files":68}
{"type":"mirror_completed","total_files":351,"total_bytes":87000000,"failed_files":0}

Installation & Usage

git clone <repository>
cd WebMirror

# Verify dependencies
python3 --version
wget --version

# Run mirror
./webmirror mirror \
  --url "https://example.com/" \
  --output "./test-output"

Development and Testing

WebMirror includes test fixtures to prove its fault-tolerance architectures.

# Run glibc segmentation fault injection test
python3 tests/test_fault_injection.py

# Run catastrophic network outage mid-batch test
python3 tests/test_network_resilience.py

Running with Podman

Podman is configured for local CLI development and reproducible testing, but is currently untested in the runtime environment. (Note: The future Windows desktop application will use a Tauri -> WSL2 -> Python CLI architecture, not Docker/Podman.)

Requirements:

• Podman
• Podman Compose

Build the image:

podman compose build

Run a mirror inside the container:

podman compose run --rm webmirror \
  ./webmirror mirror \
  --url "https://example.com/" \
  --output "/mirrors/example"

The downloaded mirror will appear on your host machine inside the ./mirror-output/ directory (which is ignored by Git).

License