ncore-leaderboard

This is a utility for scraping user ranks from nCore profiles and storing the results in a SQLite database for sorting and analysis. It is designed for efficient data collection using concurrent workers and low-level parsing.

Requirements

Go 1.26 or later
just (optional, for automated builds)
sqlite3 (optional, for CLI queries)

Installation

Clone the repository and install the necessary Go modules:

git clone https://github.com/skidoodle/ncore-leaderboard
cd ncore-leaderboard
go mod tidy

Usage

Create a .env file in the project root with your credentials:

NICK=username
PASS=cookie_pass

Build and execute the program using the provided justfile:

just run

Alternatively, build the binary manually:

go build -o ncore-leaderboard main.go
./ncore-leaderboard

The program processes the profile range defined in main.go and produces a leaderboard.db file.

Data Analysis

The results are stored in a SQLite database with indices on rank and ID. You can query the data using any standard SQLite client or the following commands:

just top 10        # List the top 10 users by rank
just query 1066    # Find the user at a specific rank

Implementation Details

The scraper utilizes a worker pool to manage concurrent HTTP requests via fasthttp. It performs raw byte signature scanning on HTML responses to extract data without the overhead of a DOM parser. Results are committed to SQLite in a single transaction to ensure atomicity and speed.

License

This software is released under the GNU General Public License v3.0. See the license file for details.

1.6 KiB Raw Permalink Blame History