nCore Profile Scraper

This is a Go program for scraping and sorting user profile data from nCore, saving results to a CSV file.

Key Features

Concurrent Scraping: Fast, parallel processing of profiles.
Quicksort Algorithm: Efficient sorting by attributes.
Batch Writing: Saves data incrementally to reduce memory usage.

Setup

Clone the repository and install dependencies:

git clone https://github.com/skidoodle/ncore-leaderboard
cd ncore-leaderboard
go mod tidy

Create a .env file with your credentials:
```
NICK=your_username
PASS=your_pass
```

Usage

Run the scraper: bash go run main.go

Scrapes profiles from the configured range.
Outputs sorted data to output.log in CSV format.

Configuration

Edit these parameters in main.go as needed:

startProfile, endProfile: Profile ID range.
concurrency: Number of concurrent requests.
outputFile: Output file name.
writeBatch: Profiles processed per save.

Output Format

The CSV file output.log contains:

Profile URL
Attribute Value (e.g., rank)

License

This project is licensed under the GPL-3.0 License.

1.2 KiB Raw Permalink Blame History