albert/mrfrys-node-server
1
0
Fork 0
mirror of https://gitlab.com/MrFry/mrfrys-node-server synced 2025-04-01 20:24:18 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions
mrfrys-node-server/README.md

233 lines
12 KiB
Markdown
Executable file
Raw Blame History

Modular node server written in typescript with express js. Serves two frontends and a userscript,
which together with this server are designed to collect questions from moodle/elearning sites.
## Requirements
* Node v19.8.1 (and npm)
* sqlite
* Linux (not tested on windows)
* Cloudflare is highly recommended to hide host IP, and use its other services
### Hardware requirements
The server doesn't have high hardware requirements, but users can use it in seasonally, and in
bursts. This causes it to consume very low resources in most of the time, but can spike up quite a
bit for a few minutes. This is because solvable test are commonly scheduled to specific time
intervals (exams, end of term tests). This also depends on the number of users using it, and the
question db size.
It used to run fine on a raspberry pi 3 B+, but needed a faster storage than an SD card. Later the
pi was switched, and ran flawlessly on a Core2 Duo 3GHz, 4GB RAM + SSD. With larger question
databases the pi might not be enough.
The server utilizes multiple CPU cores, long running operations are ran in separate threads. Because
of the implementation, the more cores a CPU has, the server uses more memory, but able to run more
threads, and serve more requests at once. The used cores can be limited with environment variables
(detailed below).
## Terminology
| Name | Description |
| --- | --- |
| Question database | A JSON file, array of saved subjects wich have a Name, and Questions array |
| Peer to peer functionality | The ability to share question databases and users with other instances of this server |
| Peer | Another instance of this server, with peer to peer functionality set up |
| User \#1 | The first user created, admin of the server |
## Setup
Run `./scripts/setup.sh`, then `npm run dev` for development, or `npm run start` for prod
On the first run there will be a number of errors, that some files weren't found. Please create them
according to the messages, these are necessary for the server to function.
There will be also a lot of information about files and other necessary things being created. **Please
read them very carefully, you should know about what was created!**
## Web server paths
There are different routes assigned to different modules. You can see these in detail in the files
`./src/modules.json`.
## Peer to peer
This server implements P2P functionality. It can fetch question databases and users from other
server instances, and merge the response data to its own databases
To setup P2P functionality you have to create a few files in `./data/p2p`:
* `selfInfo.json`: information of this peer. Required:
```
{
"name": "Any name you choose",
"contact": "contact to server administrator (irc server, e-mail, anything)",
"host": "server host (like somesite.com, without 'http(s)://')",
"port": "server port, number"
}
```
* `peers.json` : an array, with objects same as above, and `{ publicKey: "public key of the server"
}`. Public key is used to encrypt the users database in the response, so they can be synced too.
Extra configuration: HTTP and pw! TODO
Uppon syncing data or having a peer request data from your server there will be new entries in
`./data/p2p/thirdPartyPeers.json`. Here you can review the peers, see their contact and host, and if
you choose you can add them to your `peers.json` file. `thirdPartyPeers.json` should also contain
the public key
To start syncing user \#1 should perform a get request to `/syncp2pdata`
## Maintenance
The server doesn't require that much maintenance, but you are advised to:
* Check the chat messages, the users can contact you there through the website
* Check the received messages on the platform you provided in the p2p information "contact" field
* Check and moderate the forum(s) of the page
* Sync the server with other peers (this can be automated)
* Check `./data/p2p/thirdPartyPeers.json` file for new peers, and decide if you should use them
* Regularly check if there is an update to this server and submodules, and update them (using git)
* Watch out for directories that can get big:
* `./stats`: server statistics and logs
* `./data/dbs/backup`: backup of databases
* `./publicDirs/qminingPublic/backs`: backup of question databases
* `./publicDirs/qminingPublic/userFiles`: files shared by users
* `./publicDirs/qminingPublic/savedQuestions`: unanswered questions saved by the userscript
* Make regular backups of important data:
* `./data/dbs`: user and messages database
* `./data/p2p`: p2p data, and public/private keys
* `./publicDirs/qminingPublic/questionDbs`: question dbs
* `./publicDirs/qminingPublic/questionDbs.json`: information about question db-s
* `./publicDirs/qminingPublic/userFiles`: files shared by users
* `./publicDirs/qminingPublic/forum`: forum entries and comments
* `./publicDirs/qminingPublic/savedQuestions`: unanswered questions saved by the userscript
* Most files not tracked by git
## Server usage statistics
The real time log can be found in `./stats/(v)logs`. Each folder contains rotated logs, one file per
day. In `log` only the most important events are logged, in `vlogs` every request is logged.
There are other files in `./stats/`, the contents of them is explained in the detailed file
structure below.
The script `./scripts/serverStats.js` pretty prints server statistics. The first argument should be
the server root directory. The second one is optional: a negative number, which shifts the
statistics from the current day. For ex.: if its -4, it displays stats from 4 days before now.
`./scripts/exportStats.js` exports data configured in `exportStats.js`-s first few line to .csv. The
result can be found in the directory it was ran.
## Server maintenance utils
These scripts can be found in `./src/standaloneUtils`.
| Name | Description |
| --- | --- |
| serverMaintenenceUtils.js | Designed to be a single script to collect all these utils. Not done yet |
| dbSetup.js | Sets up the database |
| rmDuplicates.js | Removes duplicates (questions and subjects) from question db-s. Can merge db-s. This is extremely CPU intensive |
The other undocumented scripts are rather old, and should be checked if they work, and if they are
needed at all
## Environment variables
| Name | Type | Description |
| --- | --- | --- |
| PORT | number | The port the http server should run on |
| NS_THREAD_COUNT | number | Nubmer of CPU cores to use |
| NS_NOUSER | boolean | If the authorization should be skipped (for testing) |
| NS_NO_HTTPS_FORCE | boolean | Disables automatic redirects from http to https |
| NS_LOGLEVEL | number | Debug log level, 0 is the least verbose |
| NS_NOLOG | boolean | If logging should be skipped |
| NS_SQL_DEBUG_LOG | boolean | If the SQL queries should be logged |
## npm scripts
| Name | Description |
| --- | --- |
| npm run start | Runs server in prod mode. Requires build files in `./dist` |
| npm run dev | Re-builds and runs server in development mode with debugger attached |
| npm run build | Builds the server with tsc |
| npm run export | Same as build |
| npm run test | Runs jest tests |
| npm run test-debug | Runs jest tests in watch mode with debugger attached |
## Detailed file structure
```
.
├── data/ server specific data files not tracked by git
│ ├── admins.json forum admins. should be removed and should use admin column from user db
│ ├── apiRootRedirectTo url where domain/api should redirect to
│ ├── dbs/ directory for databases, and for their backups
│ ├── domain the domain the server is hosted on
│ ├── donateURL url where the donate button should take to TODO: check if this is needed
│ ├── f/ user files received TODO: check if this is needed
│ ├── links.json urls for irc, patreon and donate
│ ├── nolog ids of users separated by new lines to ignore on logging
│ ├── p2p/ p2p data
│ │ ├── key.priv private key generated on first server run
│ │ ├── key.pub public key generated on first server run
│ │ ├── peers.json peers to check on sync
│ │ ├── selfInfo.json p2p information of this server instance
│ │ └── thirdPartyPeers.json third party peers encountered
│ ├── statExclude.json array of strings. If included in url then excluded from stats
│ └── testUsers.json test users. received data won't get added to question dbs
├── dist/ build .js files
├── extraSubmodules/ extra submodules not tracked by git
├── nextStatic/ exported static html files generated by next.js
├── scripts scripts for server maintenance
│ ├── exportStats.js exports statistics to csv
│ ├── postBuild.sh runs after every build
│ ├── runDocker.sh runs docker
│ ├── serverStats.js pretty prints server statistics
│ └── setup.sh sets up the server before first run
├── src server source files
├── stats statistics files
│ ├── askedQuestions received data on /ask, text file
│ ├── recdata received data on /isAdding
│ ├── recievedQuestions same as recdata, should be removed
│ ├── dailyDataCount daily data count, text file
│ ├── dataEdits data editor activity log
│ ├── logs generic logs per day. latest is 'log'
│ ├── vlogs detailed logs per day. latest is 'log'
│ ├── qvote/ quick vote data directory
│ ├── registeredScripts.json registered scripts JSON
│ ├── stats website accesses per path
│ ├── idstats test solving statistics per user
│ ├── idvstats test solving statistics per day per user
│ ├── ustats website accesses count per user
│ ├── uvstats website accesses count per day per user
│ └── vstats website accessed paths per day
├── submodules git submodules
│ ├── moodle-test-userscript userscript
│ ├── qmining-data-editor data editor frontend
│ └── qmining-page qmining frontend
├── testingTools testing tools for the server
└── publicDirs/ public directories of the server, mostly available on the domain root
└── qminingPublic/ qmining module public path (modules: qmining, dataeditor, api)
├── backs/ question database backups
├── chatFiles/ files sent on chat
├── contacts.json contacts displayed on the /contact page
├── forum/ forum contents
├── forumFiles/ files uploaded to forums
├── moodle-test-userscript link to the userscript, it fetches updates from this path
├── motd motto of the day
├── questionDbs/ directory of the question db-s
├── questionDbs.json question db-s information
├── savedQuestions/ un-answered questions for dataeditor, saved from test pages
└── userFiles/ files shared by users
```
## Related repositories
* Frontend: https://gitlab.com/MrFry/qmining-page
* Dataeditor: https://gitlab.com/MrFry/qmining-data-editor
* Userscript: https://gitlab.com/MrFry/moodle-test-userscript
* Discord bot: https://gitlab.com/MrFry/qmining-discord-bot
## License
GNU General Public License, version 3
Reference in a new issue View git blame Copy permalink