mirror of
https://gitlab.com/MrFry/mrfrys-node-server
synced 2025-04-01 20:24:18 +02:00
325 lines
16 KiB
Markdown
Executable file
325 lines
16 KiB
Markdown
Executable file
Modular node server written in typescript with express js. Serves two frontends and a userscript,
|
|
which together with this server are designed to collect questions from moodle/elearning sites.
|
|
|
|
## Requirements
|
|
|
|
* Node v19.8.1 (and npm)
|
|
* sqlite
|
|
* Linux (not tested on windows)
|
|
* Cloudflare is highly recommended to hide host IP, and use its other services
|
|
|
|
### Other requirements
|
|
|
|
The server doesn't have high hardware requirements, but users can use it in seasonally, and in
|
|
bursts. This causes it to consume very low resources in most of the time, but can spike up quite a
|
|
bit for a few minutes. This is because solvable test are commonly scheduled to specific time
|
|
intervals (exams, end of term tests). This also depends on the number of users using it, and the
|
|
question db size.
|
|
|
|
It used to run fine on a raspberry pi 3 B+, but needed a faster storage than an SD card. Later the
|
|
pi was switched, and ran flawlessly on a Core2 Duo 3GHz, 4GB RAM + SSD. With larger question
|
|
databases the pi might not be enough.
|
|
|
|
The server utilizes multiple CPU cores, long running operations are ran in separate threads. Because
|
|
of the implementation, the more cores a CPU has, the server uses more memory, but able to run more
|
|
threads, and serve more requests at once. The used cores can be limited with environment variables
|
|
(detailed below, `NS_THREAD_COUNT`).
|
|
|
|
To set up and perform maintenance tasks you will need basic linux, sqlite, javascript and cloudflare
|
|
knowledge, a registered domain (or subdomain on some domain),
|
|
|
|
## Terminology
|
|
|
|
| Name | Description |
|
|
| --- | --- |
|
|
| Question database | A JSON file, array of saved subjects wich have a Name, and Questions array |
|
|
| Peer to peer functionality | The ability to share question databases and users with other instances of this server |
|
|
| Peer | Another instance of this server, with peer to peer functionality set up |
|
|
| Registered peer | A peer wich have `pw` and optionally `publicKey` specified in `peers.json` |
|
|
| User \#1 | The first user created, admin of the server |
|
|
| Submodule | A server submodule, dynamically loaded, and stored in `./src/modules/` |
|
|
| Git module | Git modules stored in `./submodules` |
|
|
|
|
## Setup
|
|
|
|
Run `./scripts/setup.sh`, then `npm run dev` for development, or `npm run start` for prod.
|
|
|
|
On the first run there will be a number of errors, that some files weren't found, or some
|
|
environment variables needed to be specified. Please create them according to the messages, these
|
|
are necessary for the server to function.
|
|
|
|
There will be also a lot of information about files and other necessary things being created. **Please
|
|
read them very carefully, you should know about what was created!**
|
|
|
|
### Setup notes and tips
|
|
|
|
* **Gitlab specific notes**: You need a gitlab account with SSH keys set up in order to be able to
|
|
use SSH for cloning this (and any) project. This includes git modules of this project. To use HTTPS
|
|
set the `CLONE_WITH_HTTPS` environment variable before running `setup.sh`. To set up SSH keys:
|
|
https://docs.gitlab.com/ee/user/ssh.html
|
|
* You either need to specify a `DOMAIN` env var, or write the domain to `./data/domain`. This is
|
|
necesarry to make xmlhttp requests and some redirects work in: statically built HTML files (by
|
|
next.js), userscript, and the server itself. Domain is for example: 'qmining.com', without
|
|
'http://' and '/'-s
|
|
* By default the server redirects all HTTP traffic to HTTPS. To disable this use
|
|
`NS_NO_HTTPS_FORCE`
|
|
* The server launches a thread for each CPU core. This could be an overkill on 4+ cored CPU-s. Use
|
|
`NS_THREAD_COUNT` to restrict the number of threads, and potentially reduce memory usage of the server.
|
|
* The setup script can be also used to update and rebuild all git modules if ran after the initial
|
|
setup
|
|
|
|
## Web server paths
|
|
|
|
There are different routes assigned to different submodules. You can see these in detail in the
|
|
files `./src/modules.json`.
|
|
|
|
## Peer to peer
|
|
|
|
This server implements P2P functionality. It can fetch question databases and users from other
|
|
server instances, and merge the response data to its own databases. The server also instantly sends
|
|
new questions received from users, new users and uploaded user files to all registered peers. The
|
|
sync feature should be used for initialization and rarely for catching up, since important data is
|
|
received instantly.
|
|
|
|
To setup P2P functionality you have to create a few files in `./data/p2p`:
|
|
|
|
* `selfInfo.json`: information of this peer. Required:
|
|
```json
|
|
{
|
|
"name": "any name you choose",
|
|
"contact": "contact to server administrator (irc server, e-mail, anything)",
|
|
"host": "server host (like 'qmining.com', without 'http(s):// and /-s')",
|
|
"port": "server port, number"
|
|
}
|
|
```
|
|
|
|
* `peers.json`: List of registered peers. An array, with objects same as above, and:
|
|
```json
|
|
{
|
|
"pw": "password to the host, so the server can log in there. Please use a dedicated password, that only the server uses!",
|
|
"publicKey": "public key of the server for encryption",
|
|
"http": "boolean, if true then http will be used to communicate instead of https"
|
|
}
|
|
```
|
|
Public key is optional, but needed to encrypt and add the users database in the response, so they
|
|
can be synced too.
|
|
|
|
New keys will be added during certain actions, such as: `sessionCookie` and `last${key}Sync`
|
|
|
|
### Using `/syncp2pdata`
|
|
|
|
To sync user \#1 should perform a get request to `/syncp2pdata`.
|
|
|
|
Upon syncing data or having a peer request data from your server there will be new entries in
|
|
`./data/p2p/thirdPartyPeers.json`. In this file you can review the peers, see their contact and
|
|
host, and you can add them to your `peers.json` file. `thirdPartyPeers.json` doesn't contain the
|
|
public key, you can obtain it (and some further information) by calling the peers `/api/selfinfo`
|
|
EP.
|
|
|
|
`/syncp2pdata` accepts a few query parameters:
|
|
|
|
| Name | Type | Description |
|
|
| --- | --- | --- |
|
|
| allTime | boolean | If it should ignore the last sync date, and get all data |
|
|
| users | boolean | If it should sync users |
|
|
| questions | boolean | If it should sync questions |
|
|
|
|
If `users` and `questions` are all missing, then everything will be synced. The last sync date is
|
|
stored separately for them.
|
|
|
|
## Maintenance
|
|
|
|
The server doesn't require that much maintenance, but you are advised to:
|
|
|
|
* Check the chat messages, the users can contact you there through the website
|
|
* Check the received messages on the platform you provided in the p2p information "contact" field
|
|
* Check and moderate the forum(s) of the page
|
|
* Sync the server with other peers (this can be automated)
|
|
* Check `./data/p2p/thirdPartyPeers.json` file for new peers, and decide if you should use them
|
|
* Regularly check if there is an update to this server and git modules, and update them
|
|
* Watch out for directories that can get big:
|
|
* `./stats`: server statistics and logs
|
|
* `./data/dbs/backup`: backup of databases
|
|
* `./public/backs`: backup of question databases
|
|
* `./public/userFiles`: files shared by users
|
|
* `./public/savedQuestions`: unanswered questions saved by the userscript
|
|
* Make regular backups of important data:
|
|
* `./data/dbs`: user and messages database
|
|
* `./data/p2p`: p2p data, and public/private keys
|
|
* `./public/questionDbs`: question dbs
|
|
* `./public/questionDbs.json`: information about question db-s
|
|
* `./public/userFiles`: files shared by users
|
|
* `./public/forum`: forum entries and comments
|
|
* `./public/savedQuestions`: unanswered questions saved by the userscript
|
|
* Most files not tracked by git
|
|
|
|
## Server maintenance utils
|
|
|
|
These scripts can be found in `./src/standaloneUtils`.
|
|
|
|
| Name | Description |
|
|
| --- | --- |
|
|
| serverMaintenenceUtils.js | Designed to be a single script to collect all these utils. Not done yet |
|
|
| dbSetup.js | Sets up the database |
|
|
| rmDuplicates.js | Removes duplicates (questions and subjects) from question db-s. Can merge db-s. This is extremely CPU intensive |
|
|
|
|
The other undocumented scripts are rather old, and should be checked if they work, and if they are
|
|
needed at all
|
|
|
|
## Server usage statistics
|
|
|
|
The real time log can be found in `./stats/(v)logs`. Each folder contains rotated logs, one file per
|
|
day. In `log` only the most important events are logged, in `vlogs` every request is logged.
|
|
|
|
There are other files in `./stats/`, the contents of them is explained in the detailed file
|
|
structure below.
|
|
|
|
The script `./scripts/serverStats.js` pretty prints server statistics. The first argument should be
|
|
the server root directory. The second one is optional: a negative number, which shifts the
|
|
statistics from the current day. For ex.: if its -4, it displays stats from 4 days before now.
|
|
|
|
`./scripts/exportStats.js` exports data configured in `exportStats.js`-s first few line to .csv. The
|
|
result can be found in the directory it was ran.
|
|
|
|
## Environment variables
|
|
|
|
### For the server
|
|
|
|
Set these before launching the server (`VAR="" npm run start`)
|
|
|
|
| Name | Type | Description |
|
|
| --- | --- | --- |
|
|
| PORT | number | The port the http server should run on |
|
|
| DOMAIN | string | The domain that the server should use for redirects, cookies, etc. ex.: `qmining.com`, without 'https://', and '/'-s. If not specified `./data/domain` will be used |
|
|
| NS_NO_HTTPS_FORCE | boolean | Disables automatic redirects from http to https |
|
|
| NS_THREAD_COUNT | number | Nubmer of CPU cores to use |
|
|
| NS_SQL_DEBUG_LOG | boolean | If the SQL queries should be logged |
|
|
| NS_NOUSER | boolean | If the authorization should be skipped (for testing) |
|
|
| NS_NOLOG | boolean | If logging should be skipped |
|
|
|
|
### For the setup script (`./scripts/setup.sh`)
|
|
|
|
Set these before starting the setup (`VAR="" ./scripts/setup.sh`)
|
|
|
|
| Name | Type | Description |
|
|
| --- | --- | --- |
|
|
| DOMAIN | string | The domain that the server should use for redirects, cookies, etc. ex.: `qmining.com`, without 'https://', and '/'-s. If not specified `./data/domain` will be used |
|
|
| CLONE_WITH_HTTPS | boolean | By default git clones with ssh, which can cause troubles. Set this to use https |
|
|
|
|
## npm scripts
|
|
|
|
| Name | Description |
|
|
| --- | --- |
|
|
| npm run start | Runs server in prod mode. Requires build files in `./dist` |
|
|
| npm run dev | Re-builds and runs server in development mode with debugger attached |
|
|
| npm run build | Builds the server with tsc |
|
|
| npm run export | Same as build |
|
|
| npm run test | Runs jest tests |
|
|
| npm run test-debug | Runs jest tests in watch mode with debugger attached |
|
|
|
|
## Git modules
|
|
|
|
More details about the specific git module should be found in its directory: `./submodules/`
|
|
|
|
# qmining-page
|
|
|
|
Next.js frontend for the server. The server serves its static exported html files. The frontend
|
|
contains many features, including chat using sockets, forum, question database viewer and other
|
|
pages detailing information about the userscript
|
|
|
|
https://gitlab.com/MrFry/qmining-page
|
|
|
|
# qmining-data-editor
|
|
|
|
Next.js frontend designed to give the users the ability, to edit the question dbs. Users can add new
|
|
questions, edit existing ones, delete invalid questions and many more. The server saves questions
|
|
and its possible answers during the users are solving online tests. These are saved, and can be
|
|
viewed in the data editor, and the correct answers can be choosen, and saved in the question
|
|
databases.
|
|
|
|
https://gitlab.com/MrFry/qmining-data-editor
|
|
|
|
# moodle-test-userscript
|
|
|
|
Javasript userscript to be run in tampermonkey or similar userscript handler. It's job is to collect
|
|
questions during tests, and on test review pages, and send it to the server. During the script also
|
|
asks the server, if there is a solution saved for the current question, and if yes, it displays it.
|
|
|
|
https://gitlab.com/MrFry/moodle-test-userscript
|
|
|
|
## Detailed file structure
|
|
```
|
|
.
|
|
├── data/ server specific data files not tracked by git
|
|
│ ├── admins.json forum admins. should be removed and should use admin column from user db
|
|
│ ├── httpsfiles.json file including privkeyFile, fullchainFile, chainFile for https functionality
|
|
│ ├── apiRootRedirectTo url where domain/api should redirect to
|
|
│ ├── dbs/ directory for databases, and for their backups
|
|
│ ├── domain the domain the server is hosted on. Used when `DOMAIN` env var is empty
|
|
│ ├── f/ user files received TODO: check if this is needed
|
|
│ ├── links.json urls for irc, patreon and donate
|
|
│ ├── nolog ids of users separated by new lines to ignore on logging
|
|
│ ├── p2p/ p2p data
|
|
│ │ ├── key.priv private key generated on first server run
|
|
│ │ ├── key.pub public key generated on first server run
|
|
│ │ ├── peers.json peers to check on sync
|
|
│ │ ├── selfInfo.json p2p information of this server instance
|
|
│ │ └── thirdPartyPeers.json third party peers encountered
|
|
│ ├── statExclude.json array of strings. If included in url then excluded from stats
|
|
│ └── testUsers.json test users. received data won't get added to question dbs
|
|
├── dist/ build .js files
|
|
├── extraSubmodules/ extra submodules not tracked by git
|
|
├── nextStatic/ exported static html files generated by next.js
|
|
├── scripts scripts for server maintenance
|
|
│ ├── exportStats.js exports statistics to csv
|
|
│ ├── postBuild.sh runs after every build
|
|
│ ├── runDocker.sh runs docker
|
|
│ ├── serverStats.js pretty prints server statistics
|
|
│ └── setup.sh sets up the server before first run
|
|
├── src server source files
|
|
├── stats statistics files
|
|
│ ├── askedQuestions received data on /ask, text file
|
|
│ ├── recdata received data on /isAdding
|
|
│ ├── recievedQuestions same as recdata, should be removed
|
|
│ ├── dailyDataCount daily data count, text file
|
|
│ ├── dataEdits data editor activity log
|
|
│ ├── logs generic logs per day. latest is 'log'
|
|
│ ├── vlogs detailed logs per day. latest is 'log'
|
|
│ ├── qvote/ quick vote data directory
|
|
│ ├── registeredScripts.json registered scripts JSON
|
|
│ ├── stats website accesses per path
|
|
│ ├── idstats test solving statistics per user
|
|
│ ├── idvstats test solving statistics per day per user
|
|
│ ├── ustats website accesses count per user
|
|
│ ├── uvstats website accesses count per day per user
|
|
│ └── vstats website accessed paths per day
|
|
├── submodules git modules
|
|
│ ├── moodle-test-userscript userscript
|
|
│ ├── qmining-data-editor data editor frontend
|
|
│ └── qmining-page qmining frontend
|
|
├── testingTools testing tools for the server
|
|
├── defaultPublicFiles static public files that the frontends use, like images
|
|
└── public/ public directories of the server
|
|
├── backs/ question database backups
|
|
├── chatFiles/ files sent on chat
|
|
├── contacts.json contacts displayed on the /contact page
|
|
├── forum/ forum contents
|
|
├── forumFiles/ files uploaded to forums
|
|
├── moodle-test-userscript link to the userscript, it fetches updates from this path
|
|
├── motd motto of the day
|
|
├── questionDbs/ directory of the question db-s
|
|
├── questionDbs.json question db-s information
|
|
├── savedQuestions/ un-answered questions for dataeditor, saved from test pages
|
|
└── userFiles/ files shared by users
|
|
```
|
|
|
|
## Related repositories
|
|
|
|
* Frontend: https://gitlab.com/MrFry/qmining-page
|
|
* Dataeditor: https://gitlab.com/MrFry/qmining-data-editor
|
|
* Userscript: https://gitlab.com/MrFry/moodle-test-userscript
|
|
* Discord bot: https://gitlab.com/MrFry/qmining-discord-bot
|
|
|
|
## License
|
|
|
|
GNU General Public License, version 3
|