mirror of
https://github.com/skidoodle/ncore-leaderboard.git
synced 2026-04-28 08:07:35 +02:00
61 lines
1.6 KiB
Markdown
61 lines
1.6 KiB
Markdown
# ncore-leaderboard
|
|
|
|
This is a utility for scraping user ranks from nCore profiles and storing
|
|
the results in a SQLite database for sorting and analysis. It is designed
|
|
for efficient data collection using concurrent workers and low-level
|
|
parsing.
|
|
|
|
## Requirements
|
|
|
|
- Go 1.26 or later
|
|
- just (optional, for automated builds)
|
|
- sqlite3 (optional, for CLI queries)
|
|
|
|
## Installation
|
|
|
|
Clone the repository and install the necessary Go modules:
|
|
|
|
git clone https://github.com/skidoodle/ncore-leaderboard
|
|
cd ncore-leaderboard
|
|
go mod tidy
|
|
|
|
## Usage
|
|
|
|
Create a `.env` file in the project root with your credentials:
|
|
|
|
NICK=username
|
|
PASS=cookie_pass
|
|
|
|
Build and execute the program using the provided justfile:
|
|
|
|
just run
|
|
|
|
Alternatively, build the binary manually:
|
|
|
|
go build -o ncore-leaderboard main.go
|
|
./ncore-leaderboard
|
|
|
|
The program processes the profile range defined in `main.go` and produces
|
|
a `leaderboard.db` file.
|
|
|
|
## Data Analysis
|
|
|
|
The results are stored in a SQLite database with indices on rank and ID.
|
|
You can query the data using any standard SQLite client or the following
|
|
commands:
|
|
|
|
just top 10 # List the top 10 users by rank
|
|
just query 1066 # Find the user at a specific rank
|
|
|
|
## Implementation Details
|
|
|
|
The scraper utilizes a worker pool to manage concurrent HTTP requests via
|
|
fasthttp. It performs raw byte signature scanning on HTML responses to
|
|
extract data without the overhead of a DOM parser. Results are committed
|
|
to SQLite in a single transaction to ensure atomicity and speed.
|
|
|
|
## License
|
|
|
|
This software is released under the GNU General Public License v3.0.
|
|
See the license file for details.
|