docs: simplify README to a minimal and functional format
Go / build (push) Has been cancelled

This commit is contained in:
2026-03-23 22:58:28 +01:00
parent b4e396aaf0
commit 74123f49f1
+42 -54
View File
@@ -1,72 +1,60 @@
# nCore Leaderboard Scraper (Turbo-Go 1.26+)
# ncore-leaderboard
A hyper-performance, zero-allocation Go scraper designed to saturate a Gigabit connection while extracting user ranks from ~1.8M nCore profiles in under 10 minutes.
This is a utility for scraping user ranks from nCore profiles and storing
the results in a SQLite database for sorting and analysis. It is designed
for efficient data collection using concurrent workers and low-level
parsing.
## 🚀 Performance Specs
## Requirements
- **Throughput:** 5,000+ Requests Per Second (RPS) on a 1Gbps line.
- **CPU Efficiency:** Multi-threaded worker pool utilizing all available cores with `GOMAXPROCS`.
- **Zero-Allocation Parsing:** Raw-byte signature scanning instead of heavy DOM parsing (100x faster than `goquery`).
- **Network Stack:** Powered by `fasthttp` for maximum connection reuse and zero-allocation networking.
- **Storage:** Instant O(log N) lookups via **SQLite 3** with WAL-mode transactions and indexed ranks.
- Go 1.26 or later
- just (optional, for automated builds)
- sqlite3 (optional, for CLI queries)
## 🛠️ Features
## Installation
- **Universal Parser:** Targeted signature scanning (`profil_jobb_elso2`) ensuring 100% accuracy.
- **Real-Time Telemetry:** Live CLI status bar with Progress, RPS, Found count, and ETA.
- **Robust Networking:** Automated 3-stage exponential backoff retries for transient errors.
- **Security Monitoring:** Active detection of session expiration or rate-limiting (Login page detection).
- **Dual-Mode Output:** Sorted results saved to a professional-grade `leaderboard.db`.
Clone the repository and install the necessary Go modules:
## 📦 Requirements
- **Go 1.26+** (Uses modern `slices`, `cmp`, and `atomic` types).
- **Just** (Case-sensitive command runner).
- **AVX2-capable CPU** (Targeted via `GOAMD64=v3` build flags).
## 📥 Setup
1. Clone and install dependencies:
```powershell
git clone https://github.com/skidoodle/ncore-leaderboard
cd ncore-leaderboard
just tidy
```
go mod tidy
2. Configure your credentials in `.env.local` or `.env`:
```env
NICK=your_username
PASS=your_pass_cookie_value
```
## Usage
## 🎮 Usage
Create a `.env` file in the project root with your credentials:
NICK=username
PASS=cookie_pass
Build and execute the program using the provided justfile:
### Build & Run
```powershell
just run
```
### Build Only (Optimized)
Produces a stripped, architecture-targeted `ncore-leaderboard.exe`.
```powershell
just build
```
Alternatively, build the binary manually:
### Data Exploration
Once scraping is complete, the results are in `leaderboard.db`. You can query it instantly:
```powershell
# Show top 25 users
just top 25
go build -o ncore-leaderboard main.go
./ncore-leaderboard
# Query a specific rank
just query 1066
```
The program processes the profile range defined in `main.go` and produces
a `leaderboard.db` file.
## 🏗️ Architecture details
## Data Analysis
- **Concurrency:** Managed by a buffered job channel and 1,000+ goroutine workers.
- **Memory:** Results are stored in-memory using `uint32` to halve RAM footprint before being flushed to SQLite in a single atomic transaction.
- **Indexing:** Ranks and Profile IDs are indexed on disk for sub-millisecond leaderboard queries.
The results are stored in a SQLite database with indices on rank and ID.
You can query the data using any standard SQLite client or the following
commands:
## ⚖️ License
GPL-3.0 License. Built for performance, speed, and precision.
just top 10 # List the top 10 users by rank
just query 1066 # Find the user at a specific rank
## Implementation Details
The scraper utilizes a worker pool to manage concurrent HTTP requests via
fasthttp. It performs raw byte signature scanning on HTML responses to
extract data without the overhead of a DOM parser. Results are committed
to SQLite in a single transaction to ensure atomicity and speed.
## License
This software is released under the GNU General Public License v3.0.
See the license file for details.