mirror of
https://gitlab.com/MrFry/mrfrys-node-server
synced 2025-04-01 20:24:18 +02:00
readme update, removed unnecesarry parts from setup, validation token null check
This commit is contained in:
mrfry
parent
cbc536483c
commit
d09b74ac4a
4 changed files with 86 additions and 30 deletions
109
README.md
109
README.md
|
|
@ -1,22 +1,53 @@
|
|||
# Express.js server
|
||||
Modular node server written in typescript with express js. Serves two frontends and a userscript,
|
||||
which together with this server are designed to collect questions from moodle/elearning sites.
|
||||
|
||||
## Requirements
|
||||
|
||||
* Node v19.8.1 (and npm)
|
||||
* sqlite
|
||||
* Linux (not tested on windows)
|
||||
* Cloudflare is highly recommended to hide host IP, and use its other services
|
||||
|
||||
### Hardware requirements
|
||||
|
||||
The server doesn't have high hardware requirements, but users can use it in seasonally, and in
|
||||
bursts. This causes it to consume very low resources in most of the time, but can spike up quite a
|
||||
bit for a few minutes. This is because solvable test are commonly scheduled to specific time
|
||||
intervals (exams, end of term tests). This also depends on the number of users using it, and the
|
||||
question db size.
|
||||
|
||||
It used to run fine on a raspberry pi 3 B+, but needed a faster storage than an SD card. Later the
|
||||
pi was switched, and ran flawlessly on a Core2 Duo 3GHz, 4GB RAM + SSD. With larger question
|
||||
databases the pi might not be enough.
|
||||
|
||||
The server utilizes multiple CPU cores, long running operations are ran in separate threads. Because
|
||||
of the implementation, the more cores a CPU has, the server uses more memory, but able to run more
|
||||
threads, and serve more requests at once. The used cores can be limited with environment variables
|
||||
(detailed below).
|
||||
|
||||
## Terminology
|
||||
|
||||
| Name | Description |
|
||||
| --- | --- |
|
||||
| Question database | A JSON file, array of saved subjects wich have a Name, and Questions array |
|
||||
| peer to peer functionality | The ability to share question databases and users with other instances of this server |
|
||||
| Peer | Another instance of this server, with peer to peer functionality set up |
|
||||
| User #1 | The first user created, admin of the server |
|
||||
|
||||
## Setup
|
||||
|
||||
Run `./scripts/setup.sh`, then `npm run dev` for development, or `npm run start` for prod
|
||||
|
||||
On the first run there will be a number of errors, that some files weren't found. Please create them
|
||||
according to the messages, these are necessary for the server to function
|
||||
according to the messages, these are necessary for the server to function.
|
||||
|
||||
## Terminology
|
||||
| Name | Description |
|
||||
| --- | --- |
|
||||
| Question database | a JSON file, array of saved subjects wich have a Name, and Questions array |
|
||||
| Peer | another instance of this server, with p2p set up |
|
||||
There will be also a lot of information about files and other necessary things being created. **Please
|
||||
read them very carefully, you should know about what was created!**
|
||||
|
||||
## P2P setup
|
||||
## Peer to peer
|
||||
|
||||
This server implements P2P functionality. It can fetch data from other server instances, and merge
|
||||
the response data to its own database
|
||||
This server implements P2P functionality. It can fetch question databases and users from other
|
||||
server instances, and merge the response data to its own databases
|
||||
|
||||
To setup P2P functionality you have to create a few files in `./data/p2p`:
|
||||
|
||||
|
|
@ -30,14 +61,17 @@ To setup P2P functionality you have to create a few files in `./data/p2p`:
|
|||
}
|
||||
```
|
||||
|
||||
* `peers.json` : an array, with objects same as above, and `{ publicKey: "public key of the server" }`. Public key is needed to recieve users too.
|
||||
* `peers.json` : an array, with objects same as above, and `{ publicKey: "public key of the server"
|
||||
}`. Public key is used to encrypt the users database in the response, so they can be synced too.
|
||||
|
||||
Uppon syncing data or having a peer request data from your server there will be new entries in
|
||||
`./data/p2p/thirdPartyPeers.json`. Here you can review the peers, see their contact and host, and if
|
||||
you choose you can add them to your `peers.json` file. `thirdPartyPeers.json` should also contain
|
||||
the public key
|
||||
|
||||
## Maintenence
|
||||
To start syncing user #1 should perform a get request to `/syncp2pdata`
|
||||
|
||||
## Maintenance
|
||||
|
||||
The server doesn't require that much maintenance, but you are advised to:
|
||||
|
||||
|
|
@ -45,12 +79,12 @@ The server doesn't require that much maintenance, but you are advised to:
|
|||
* Check the received messages on the platform you provided in the p2p information "contact" field
|
||||
* Check and moderate the forum(s) of the page
|
||||
* Sync the server with other peers (this can be automated)
|
||||
* Check `./data/p2p/thirdPartyPeers.json` file for new peers, and decide if you should use them
|
||||
* Regularly check if there is an update to this server and submodules, and update them (using git)
|
||||
* Watch out for directories that can get big:
|
||||
* `./stats`: server statistics and logs
|
||||
* `./data/dbs/backup`: backup of databases
|
||||
* `./publicDirs/qminingPublic/backs`: backup of question databases
|
||||
* `./publicDirs/qminingPublic/backs`: backup of question databases
|
||||
* `./publicDirs/qminingPublic/userFiles`: files shared by users
|
||||
* `./publicDirs/qminingPublic/savedQuestions`: unanswered questions saved by the userscript
|
||||
* Make regular backups of important data:
|
||||
|
|
@ -68,6 +102,9 @@ The server doesn't require that much maintenance, but you are advised to:
|
|||
The real time log can be found in `./stats/(v)logs`. Each folder contains rotated logs, one file per
|
||||
day. In `log` only the most important events are logged, in `vlogs` every request is logged.
|
||||
|
||||
There are other files in `./stats/`, the contents of them is explained in the detailed file
|
||||
structure below.
|
||||
|
||||
The script `./scripts/serverStats.js` pretty prints server statistics. The first argument should be
|
||||
the server root directory. The second one is optional: a negative number, which shifts the
|
||||
statistics from the current day. For ex.: if its -4, it displays stats from 4 days before now.
|
||||
|
|
@ -75,44 +112,66 @@ statistics from the current day. For ex.: if its -4, it displays stats from 4 da
|
|||
`./scripts/exportStats.js` exports data configured in `exportStats.js`-s first few line to .csv. The
|
||||
result can be found in the directory it was ran.
|
||||
|
||||
## Server maintenence utils
|
||||
## Server maintenance utils
|
||||
|
||||
These scripts can be found in `./src/standaloneUtils`.
|
||||
|
||||
| name | description |
|
||||
| Name | Description |
|
||||
| --- | --- |
|
||||
| serverMaintenenceUtils.js | Designed to be a single script to collect all these utils. Not done yet |
|
||||
| dbSetup.js | Sets up the database |
|
||||
| rmDuplicates.js | Removes duplicates (questions and subjects) from question db-s. Can merge db-s |
|
||||
| rmDuplicates.js | Removes duplicates (questions and subjects) from question db-s. Can merge db-s. This is extremely CPU intensive |
|
||||
|
||||
The other undocumented scripts are rather old, and should be checked if they work, and if they are
|
||||
needed at all
|
||||
|
||||
## Detailed file structure
|
||||
## Environment variables
|
||||
|
||||
| Name | Type | Description |
|
||||
| --- | --- | --- |
|
||||
| PORT | number | The port the http server should run on |
|
||||
| NS_THREAD_COUNT | number | Nubmer of CPU cores to use |
|
||||
| NS_NOUSER | boolean | If the authorization should be skipped (for testing) |
|
||||
| NS_DEVEL | boolean | Developemnt mode. Now it only disables automatic redirects from http to https |
|
||||
| NS_LOGLEVEL | number | Debug log level, 0 is the least verbose |
|
||||
| NS_NOLOG | boolean | If logging should be skipped |
|
||||
| NS_SQL_DEBUG_LOG | boolean | If the SQL queries should be logged |
|
||||
|
||||
## npm scripts
|
||||
|
||||
| Name | Description |
|
||||
| --- | --- |
|
||||
| npm run start | Runs server in prod mode. Requires build files in `./dist` |
|
||||
| npm run dev | Re-builds and runs server in development mode with debugger attached |
|
||||
| npm run build | Builds the server with tsc |
|
||||
| npm run export | Same as build |
|
||||
| npm run test | Runs jest tests |
|
||||
| npm run test-debug | Runs jest tests in watch mode with debugger attached |
|
||||
|
||||
## Detailed file structure
|
||||
```
|
||||
.
|
||||
├── data/ server specific data files not tracked by git
|
||||
│ ├── admins.json forum admins. should be removed and should use admin column from user db
|
||||
│ ├── apiRootRedirectTo url where domain/api should redirect to
|
||||
│ ├── dbs/ directory for databases
|
||||
│ ├── dbs/ directory for databases, and for their backups
|
||||
│ ├── domain the domain the server is hosted on
|
||||
│ ├── donateURL url where the donate button should take to TODO: check if this is needed
|
||||
│ ├── f/ user files recieved TODO: check if this is needed
|
||||
│ ├── f/ user files received TODO: check if this is needed
|
||||
│ ├── links.json urls for irc, patreon and donate
|
||||
│ ├── nolog ids of users separated by new lines to ignore on logging
|
||||
│ ├── p2p/ p2p data
|
||||
│ │ ├── key.priv private key generated on first server run
|
||||
│ │ ├── key.pub public key generated on first server run
|
||||
│ │ ├── peers.json peers to check on sync
|
||||
│ │ ├── selfInfo.json p2p information of thish server instance
|
||||
│ │ ├── selfInfo.json p2p information of this server instance
|
||||
│ │ └── thirdPartyPeers.json third party peers encountered
|
||||
│ ├── statExclude.json array of strings. If included in url then excluded from stats
|
||||
│ └── testUsers.json test users. revieved data won't get added to question dbs
|
||||
│ └── testUsers.json test users. received data won't get added to question dbs
|
||||
├── dist/ build .js files
|
||||
├── extraSubmodules/ extra submodules not tracked by git
|
||||
├── nextStatic/ exported statis html files generated by next.js
|
||||
├── scripts scripts for server maintanance
|
||||
├── nextStatic/ exported static html files generated by next.js
|
||||
├── scripts scripts for server maintenance
|
||||
│ ├── exportStats.js exports statistics to csv
|
||||
│ ├── postBuild.sh runs after every build
|
||||
│ ├── runDocker.sh runs docker
|
||||
|
|
@ -120,8 +179,8 @@ needed at all
|
|||
│ └── setup.sh sets up the server before first run
|
||||
├── src server source files
|
||||
├── stats statistics files
|
||||
│ ├── askedQuestions recieved data on /ask, text file
|
||||
│ ├── recdata recieved data on /isAdding
|
||||
│ ├── askedQuestions received data on /ask, text file
|
||||
│ ├── recdata received data on /isAdding
|
||||
│ ├── recievedQuestions same as recdata, should be removed
|
||||
│ ├── dailyDataCount daily data count, text file
|
||||
│ ├── dataEdits data editor activity log
|
||||
|
|
|
|||
|
|
@ -83,10 +83,7 @@ setupJson stats/stats
|
|||
setupJson stats/vstats
|
||||
setupJson stats/idstats
|
||||
setupJson stats/idvstats
|
||||
setupJson publicDirs/qminingPublic/userSpecificMotd.json
|
||||
setupJson publicDirs/qminingPublic/news.json
|
||||
|
||||
touch publicDirs/qminingPublic/version
|
||||
touch publicDirs/qminingPublic/motd
|
||||
|
||||
if [ -z "$(ls -A ./data/dbs)" ]; then
|
||||
|
|
|
|||
|
|
@ -325,7 +325,7 @@ function setup(data: SubmoduleData): Submodule {
|
|||
id: +userid,
|
||||
})
|
||||
|
||||
if (specifiedUser.length === 0) {
|
||||
if (specifiedUser.length === 0 || !specifiedUser[0]) {
|
||||
res.json({
|
||||
result: 'nouserid',
|
||||
msg: 'couldnt find user',
|
||||
|
|
|
|||
|
|
@ -1 +1 @@
|
|||
Subproject commit dfc5c93022b5c1a665e73e351a39fe6b8969d46c
|
||||
Subproject commit f82ecaa6d00d4f0625fb678c2c645a6ed3d79c03
|
||||
Loading…
Add table
Add a link
Reference in a new issue