diff --git a/readme.md b/readme.md index dd2ce4a..b81b9cc 100644 --- a/readme.md +++ b/readme.md @@ -1,72 +1,60 @@ -# nCore Leaderboard Scraper (Turbo-Go 1.26+) +# ncore-leaderboard -A hyper-performance, zero-allocation Go scraper designed to saturate a Gigabit connection while extracting user ranks from ~1.8M nCore profiles in under 10 minutes. +This is a utility for scraping user ranks from nCore profiles and storing +the results in a SQLite database for sorting and analysis. It is designed +for efficient data collection using concurrent workers and low-level +parsing. -## 🚀 Performance Specs +## Requirements -- **Throughput:** 5,000+ Requests Per Second (RPS) on a 1Gbps line. -- **CPU Efficiency:** Multi-threaded worker pool utilizing all available cores with `GOMAXPROCS`. -- **Zero-Allocation Parsing:** Raw-byte signature scanning instead of heavy DOM parsing (100x faster than `goquery`). -- **Network Stack:** Powered by `fasthttp` for maximum connection reuse and zero-allocation networking. -- **Storage:** Instant O(log N) lookups via **SQLite 3** with WAL-mode transactions and indexed ranks. +- Go 1.26 or later +- just (optional, for automated builds) +- sqlite3 (optional, for CLI queries) -## 🛠️ Features +## Installation -- **Universal Parser:** Targeted signature scanning (`profil_jobb_elso2`) ensuring 100% accuracy. -- **Real-Time Telemetry:** Live CLI status bar with Progress, RPS, Found count, and ETA. -- **Robust Networking:** Automated 3-stage exponential backoff retries for transient errors. -- **Security Monitoring:** Active detection of session expiration or rate-limiting (Login page detection). -- **Dual-Mode Output:** Sorted results saved to a professional-grade `leaderboard.db`. +Clone the repository and install the necessary Go modules: -## 📦 Requirements + git clone https://github.com/skidoodle/ncore-leaderboard + cd ncore-leaderboard + go mod tidy -- **Go 1.26+** (Uses modern `slices`, `cmp`, and `atomic` types). -- **Just** (Case-sensitive command runner). -- **AVX2-capable CPU** (Targeted via `GOAMD64=v3` build flags). +## Usage -## 📥 Setup +Create a `.env` file in the project root with your credentials: -1. Clone and install dependencies: - ```powershell - git clone https://github.com/skidoodle/ncore-leaderboard - cd ncore-leaderboard - just tidy - ``` + NICK=username + PASS=cookie_pass -2. Configure your credentials in `.env.local` or `.env`: - ```env - NICK=your_username - PASS=your_pass_cookie_value - ``` +Build and execute the program using the provided justfile: -## 🎮 Usage + just run -### Build & Run -```powershell -just run -``` +Alternatively, build the binary manually: -### Build Only (Optimized) -Produces a stripped, architecture-targeted `ncore-leaderboard.exe`. -```powershell -just build -``` + go build -o ncore-leaderboard main.go + ./ncore-leaderboard -### Data Exploration -Once scraping is complete, the results are in `leaderboard.db`. You can query it instantly: -```powershell -# Show top 25 users -just top 25 +The program processes the profile range defined in `main.go` and produces +a `leaderboard.db` file. -# Query a specific rank -just query 1066 -``` +## Data Analysis -## 🏗️ Architecture details +The results are stored in a SQLite database with indices on rank and ID. +You can query the data using any standard SQLite client or the following +commands: -- **Concurrency:** Managed by a buffered job channel and 1,000+ goroutine workers. -- **Memory:** Results are stored in-memory using `uint32` to halve RAM footprint before being flushed to SQLite in a single atomic transaction. -- **Indexing:** Ranks and Profile IDs are indexed on disk for sub-millisecond leaderboard queries. + just top 10 # List the top 10 users by rank + just query 1066 # Find the user at a specific rank -## ⚖️ License -GPL-3.0 License. Built for performance, speed, and precision. +## Implementation Details + +The scraper utilizes a worker pool to manage concurrent HTTP requests via +fasthttp. It performs raw byte signature scanning on HTML responses to +extract data without the overhead of a DOM parser. Results are committed +to SQLite in a single transaction to ensure atomicity and speed. + +## License + +This software is released under the GNU General Public License v3.0. +See the license file for details.