mirror of
https://github.com/skidoodle/ncore-leaderboard.git
synced 2026-04-27 23:57:36 +02:00
This commit is contained in:
@@ -1,72 +1,60 @@
|
||||
# nCore Leaderboard Scraper (Turbo-Go 1.26+)
|
||||
# ncore-leaderboard
|
||||
|
||||
A hyper-performance, zero-allocation Go scraper designed to saturate a Gigabit connection while extracting user ranks from ~1.8M nCore profiles in under 10 minutes.
|
||||
This is a utility for scraping user ranks from nCore profiles and storing
|
||||
the results in a SQLite database for sorting and analysis. It is designed
|
||||
for efficient data collection using concurrent workers and low-level
|
||||
parsing.
|
||||
|
||||
## 🚀 Performance Specs
|
||||
## Requirements
|
||||
|
||||
- **Throughput:** 5,000+ Requests Per Second (RPS) on a 1Gbps line.
|
||||
- **CPU Efficiency:** Multi-threaded worker pool utilizing all available cores with `GOMAXPROCS`.
|
||||
- **Zero-Allocation Parsing:** Raw-byte signature scanning instead of heavy DOM parsing (100x faster than `goquery`).
|
||||
- **Network Stack:** Powered by `fasthttp` for maximum connection reuse and zero-allocation networking.
|
||||
- **Storage:** Instant O(log N) lookups via **SQLite 3** with WAL-mode transactions and indexed ranks.
|
||||
- Go 1.26 or later
|
||||
- just (optional, for automated builds)
|
||||
- sqlite3 (optional, for CLI queries)
|
||||
|
||||
## 🛠️ Features
|
||||
## Installation
|
||||
|
||||
- **Universal Parser:** Targeted signature scanning (`profil_jobb_elso2`) ensuring 100% accuracy.
|
||||
- **Real-Time Telemetry:** Live CLI status bar with Progress, RPS, Found count, and ETA.
|
||||
- **Robust Networking:** Automated 3-stage exponential backoff retries for transient errors.
|
||||
- **Security Monitoring:** Active detection of session expiration or rate-limiting (Login page detection).
|
||||
- **Dual-Mode Output:** Sorted results saved to a professional-grade `leaderboard.db`.
|
||||
Clone the repository and install the necessary Go modules:
|
||||
|
||||
## 📦 Requirements
|
||||
|
||||
- **Go 1.26+** (Uses modern `slices`, `cmp`, and `atomic` types).
|
||||
- **Just** (Case-sensitive command runner).
|
||||
- **AVX2-capable CPU** (Targeted via `GOAMD64=v3` build flags).
|
||||
|
||||
## 📥 Setup
|
||||
|
||||
1. Clone and install dependencies:
|
||||
```powershell
|
||||
git clone https://github.com/skidoodle/ncore-leaderboard
|
||||
cd ncore-leaderboard
|
||||
just tidy
|
||||
```
|
||||
go mod tidy
|
||||
|
||||
2. Configure your credentials in `.env.local` or `.env`:
|
||||
```env
|
||||
NICK=your_username
|
||||
PASS=your_pass_cookie_value
|
||||
```
|
||||
## Usage
|
||||
|
||||
## 🎮 Usage
|
||||
Create a `.env` file in the project root with your credentials:
|
||||
|
||||
### Build & Run
|
||||
```powershell
|
||||
just run
|
||||
```
|
||||
NICK=username
|
||||
PASS=cookie_pass
|
||||
|
||||
### Build Only (Optimized)
|
||||
Produces a stripped, architecture-targeted `ncore-leaderboard.exe`.
|
||||
```powershell
|
||||
just build
|
||||
```
|
||||
Build and execute the program using the provided justfile:
|
||||
|
||||
### Data Exploration
|
||||
Once scraping is complete, the results are in `leaderboard.db`. You can query it instantly:
|
||||
```powershell
|
||||
# Show top 25 users
|
||||
just top 25
|
||||
just run
|
||||
|
||||
# Query a specific rank
|
||||
just query 1066
|
||||
```
|
||||
Alternatively, build the binary manually:
|
||||
|
||||
## 🏗️ Architecture details
|
||||
go build -o ncore-leaderboard main.go
|
||||
./ncore-leaderboard
|
||||
|
||||
- **Concurrency:** Managed by a buffered job channel and 1,000+ goroutine workers.
|
||||
- **Memory:** Results are stored in-memory using `uint32` to halve RAM footprint before being flushed to SQLite in a single atomic transaction.
|
||||
- **Indexing:** Ranks and Profile IDs are indexed on disk for sub-millisecond leaderboard queries.
|
||||
The program processes the profile range defined in `main.go` and produces
|
||||
a `leaderboard.db` file.
|
||||
|
||||
## ⚖️ License
|
||||
GPL-3.0 License. Built for performance, speed, and precision.
|
||||
## Data Analysis
|
||||
|
||||
The results are stored in a SQLite database with indices on rank and ID.
|
||||
You can query the data using any standard SQLite client or the following
|
||||
commands:
|
||||
|
||||
just top 10 # List the top 10 users by rank
|
||||
just query 1066 # Find the user at a specific rank
|
||||
|
||||
## Implementation Details
|
||||
|
||||
The scraper utilizes a worker pool to manage concurrent HTTP requests via
|
||||
fasthttp. It performs raw byte signature scanning on HTML responses to
|
||||
extract data without the overhead of a DOM parser. Results are committed
|
||||
to SQLite in a single transaction to ensure atomicity and speed.
|
||||
|
||||
## License
|
||||
|
||||
This software is released under the GNU General Public License v3.0.
|
||||
See the license file for details.
|
||||
|
||||
Reference in New Issue
Block a user