nCore Leaderboard Scraper (Turbo-Go 1.26+)

A hyper-performance, zero-allocation Go scraper designed to saturate a Gigabit connection while extracting user ranks from ~1.8M nCore profiles in under 10 minutes.

🚀 Performance Specs

Throughput: 5,000+ Requests Per Second (RPS) on a 1Gbps line.
CPU Efficiency: Multi-threaded worker pool utilizing all available cores with GOMAXPROCS.
Zero-Allocation Parsing: Raw-byte signature scanning instead of heavy DOM parsing (100x faster than goquery).
Network Stack: Powered by fasthttp for maximum connection reuse and zero-allocation networking.
Storage: Instant O(log N) lookups via SQLite 3 with WAL-mode transactions and indexed ranks.

🛠️ Features

Universal Parser: Targeted signature scanning (profil_jobb_elso2) ensuring 100% accuracy.
Real-Time Telemetry: Live CLI status bar with Progress, RPS, Found count, and ETA.
Robust Networking: Automated 3-stage exponential backoff retries for transient errors.
Security Monitoring: Active detection of session expiration or rate-limiting (Login page detection).
Dual-Mode Output: Sorted results saved to a professional-grade leaderboard.db.

📦 Requirements

Go 1.26+ (Uses modern slices, cmp, and atomic types).
Just (Case-sensitive command runner).
AVX2-capable CPU (Targeted via GOAMD64=v3 build flags).

📥 Setup

Clone and install dependencies:

git clone https://github.com/skidoodle/ncore-leaderboard
cd ncore-leaderboard
just tidy

Configure your credentials in .env.local or .env:

NICK=your_username
PASS=your_pass_cookie_value

🎮 Usage

Build & Run

just run

Build Only (Optimized)

Produces a stripped, architecture-targeted ncore-leaderboard.exe.

just build

Data Exploration

Once scraping is complete, the results are in leaderboard.db. You can query it instantly:

# Show top 25 users
just top 25

# Query a specific rank
just query 1066

🏗️ Architecture details

Concurrency: Managed by a buffered job channel and 1,000+ goroutine workers.
Memory: Results are stored in-memory using uint32 to halve RAM footprint before being flushed to SQLite in a single atomic transaction.
Indexing: Ranks and Profile IDs are indexed on disk for sub-millisecond leaderboard queries.

⚖️ License

GPL-3.0 License. Built for performance, speed, and precision.

2.5 KiB Raw Blame History