CloudHost Web Auditor

A CLI tool for web crawling, HTTP status auditing, and server latency monitoring. Built for system administrators and technical support teams.

---

Why This Tool?

When clients report issues like blank pages, 500 errors, or broken links, support teams need a quick way to diagnose the problem without manually checking every page.

Web Auditor automates this process by:

• Systematically crawling internal pages level by level using BFS.
• Detecting 4xx client errors and 5xx server failures.
• Flagging slow-responding pages (over 3000ms latency).
• Generating CSV reports for further review.
• Providing a clear, color-coded terminal output for quick triage.

Arsitektur Sistem

Berikut adalah tiga diagram utama yang menjelaskan cara kerja dan arsitektur sistem secara visual.

Context Diagram

Diagram ini menunjukkan gambaran paling tinggi dari sistem: siapa saja pihak yang terlibat, data apa yang masuk ke sistem, dan apa yang dihasilkan. Ada empat entitas utama SysAdmin yang menjalankan tool via CLI, server target yang di-crawl, output CSV, dan tampilan terminal.

---

Detail Proses

Diagram ini memecah proses utama menjadi 7 sub-proses yang lebih detail, masing-masing dipetakan langsung ke modul Python yang menjalankannya. Termasuk BFS Queue, Visited URLs set, dan Audit Results sebagai data store internal.

---

System Architecture — VPS + Cloud + Telegram

Diagram ini menjelaskan deployment fisik secara keseluruhan. Web Auditor dijalankan di VPS, melakukan HTTP request ke server target yang ada di cloud, lalu mengirimkan hasil audit (termasuk file CSV) ke tim support melalui Telegram Bot. Cronjob atau Systemd bisa digunakan untuk otomatisasi jadwal scan.

Installation

1. Clone the repository

git clone https://github.com/your-username/web-auditor.git
cd web-auditor

2. Create a virtual environment

python -m venv .venv
source .venv/bin/activate   # Linux/macOS
.venv\Scripts\activate      # Windows

3. Install dependencies

pip install -r requirements.txt

---

Usage

Basic Scan

# Using main.py launcher
python main.py -u https://example.com

# Using Python module
python -m web_auditor -u https://example.com

Full Options

python main.py -u https://client-site.com -d 3 -t 15 -o audit_report.csv --delay 1.0

CLI Parameters

Flag	Long	Type	Default	Description
-u	--start-url	String	(required)	Target website URL to audit
-d	--max-depth	Integer	2	Maximum crawl depth from start URL
-t	--timeout	Integer	10	Request timeout in seconds
-o	--output	String	report.csv	Output CSV file path
	--delay	Float	0.5	Delay between requests (seconds)

---

Terminal Output Preview

---------------- CloudHost Web Auditor v1.0 ----------------

  Target:      https://example.com
  Max Depth:   2
  Timeout:     10s
  Delay:       0.5s
  Output:      report.csv

------------------- 2026-07-07 19:30:00 --------------------

  HEALTHY              | 200 | 145ms  | https://example.com
  HEALTHY              | 200 | 230ms  | https://example.com/about
  BROKEN LINK          | 404 | 120ms  | https://example.com/old-post
  CRITICAL SERVER ERROR| 500 | 5040ms | https://example.com/api/data
  SLOW RESPONSE (>3s)  | 200 | 4200ms | https://example.com/catalog
  TIMEOUT / UNREACHABLE| --  | --     | https://example.com/dead-service

------------------ Live Audit Statistics -------------------

  Total Checked:      6
  Healthy:            2
  Slow:               1
  Broken Links:       1
  Server Errors:      1
  Timeouts:           1

------------------------------------------------------------

---

CSV Report Sample

Source Page	Target URL	Link Type	HTTP Status	Latency (ms)	Diagnosis Status	Recommended Action
--	https://example.com	Entry Point	200	145	HEALTHY	No action required
https://example.com	https://example.com/about	Internal	200	230	HEALTHY	No action required
https://example.com/blog	https://example.com/old-post	Internal	404	120	BROKEN LINK	Check CMS trash or update permalink
https://example.com/shop	https://example.com/api/cart	Internal	502	5040	BAD GATEWAY	Check PHP-FPM error log and server load

---

Error Classification

Category	HTTP Codes	Label
Healthy	200 (latency < 3s)	HEALTHY
Slow Response	200 (latency >= 3s)	SLOW RESPONSE
Broken Link	400, 403, 404	BROKEN LINK
Server Error	500, 502, 503, 504	CRITICAL SERVER ERROR
Timeout	Connection failure	TIMEOUT / UNREACHABLE

---

Features

• BFS Algorithm: Ensures complete level-by-level coverage of the website.
• Domain-Aware Filtering: Crawls internal links deeply, but only checks the status of external links.
• Graceful Interruption: Press Ctrl+C to stop the scan and still generate a partial report.
• Connection Pooling: Uses requests.Session for efficient HTTP connection handling.
• Rate Limiting: Configurable delay to prevent overwhelming the target servers.
• Custom User-Agent: Identifies itself clearly as CloudHost-Support-Auditor/1.0.

---

Project Structure

scrap-bash/
├── main.py                 # Simple launcher script
├── requirements.txt        # Python dependencies
├── README.md               # Documentation
├── web_auditor/            # Core package
│   ├── __init__.py         # Package init + Windows encoding fix
│   ├── __main__.py         # Entry point (python -m web_auditor)
│   ├── cli.py              # Argparse CLI setup & signal handling
│   ├── constants.py        # Shared constants & Rich console setup
│   ├── crawler.py          # BFS web crawler engine
│   ├── diagnosis.py        # HTTP status diagnosis logic
│   ├── models.py           # Data models (AuditResult, CrawlerStats)
│   ├── reporter.py         # CSV report generation & terminal output
│   └── url_utils.py        # URL normalization & link extraction
├── docs/                   # Architecture diagrams (.drawio)
│   ├── 1-DFD-Level-0-Context.drawio
│   ├── 2-DFD-Level-1-Detail.drawio
│   ├── 3-System-Architecture.drawio
│   └── 4-ERD-Entity-Relationship.drawio
├── penjelasan.md           # Project explanation in Indonesian
├── work.md                 # Technical specifications
└── report.csv              # Generated audit report