From d09b74ac4a9efe6503c0ce54c2dd863efa03b367 Mon Sep 17 00:00:00 2001
From: mrfry <mr.fry@tutanota.com>
Date: Tue, 4 Apr 2023 18:10:01 +0200
Subject: [PATCH] readme update, removed unnecesarry parts from setup,
validation token null check
---
README.md | 109 ++++++++++++++-----
scripts/setup.sh | 3 -
src/modules/api/submodules/userManagement.ts | 2 +-
submodules/qmining-page | 2 +-
4 files changed, 86 insertions(+), 30 deletions(-)
diff --git a/README.md b/README.md
index ad62ce4..225d83a 100755
--- a/README.md
+++ b/README.md
@@ -1,22 +1,53 @@
-# Express.js server
+Modular node server written in typescript with express js. Serves two frontends and a userscript,
+which together with this server are designed to collect questions from moodle/elearning sites.
+
+## Requirements
+
+* Node v19.8.1 (and npm)
+* sqlite
+* Linux (not tested on windows)
+* Cloudflare is highly recommended to hide host IP, and use its other services
+
+### Hardware requirements
+
+The server doesn't have high hardware requirements, but users can use it in seasonally, and in
+bursts. This causes it to consume very low resources in most of the time, but can spike up quite a
+bit for a few minutes. This is because solvable test are commonly scheduled to specific time
+intervals (exams, end of term tests). This also depends on the number of users using it, and the
+question db size.
+
+It used to run fine on a raspberry pi 3 B+, but needed a faster storage than an SD card. Later the
+pi was switched, and ran flawlessly on a Core2 Duo 3GHz, 4GB RAM + SSD. With larger question
+databases the pi might not be enough.
+
+The server utilizes multiple CPU cores, long running operations are ran in separate threads. Because
+of the implementation, the more cores a CPU has, the server uses more memory, but able to run more
+threads, and serve more requests at once. The used cores can be limited with environment variables
+(detailed below).
+
+## Terminology
+
+ | Name | Description |
+ | --- | --- |
+ | Question database | A JSON file, array of saved subjects wich have a Name, and Questions array |
+ | peer to peer functionality | The ability to share question databases and users with other instances of this server |
+ | Peer | Another instance of this server, with peer to peer functionality set up |
+ | User #1 | The first user created, admin of the server |
## Setup
Run `./scripts/setup.sh`, then `npm run dev` for development, or `npm run start` for prod
On the first run there will be a number of errors, that some files weren't found. Please create them
-according to the messages, these are necessary for the server to function
+according to the messages, these are necessary for the server to function.
-## Terminology
- | Name | Description |
- | --- | --- |
- | Question database | a JSON file, array of saved subjects wich have a Name, and Questions array |
- | Peer | another instance of this server, with p2p set up |
+There will be also a lot of information about files and other necessary things being created. **Please
+read them very carefully, you should know about what was created!**
-## P2P setup
+## Peer to peer
-This server implements P2P functionality. It can fetch data from other server instances, and merge
-the response data to its own database
+This server implements P2P functionality. It can fetch question databases and users from other
+server instances, and merge the response data to its own databases
To setup P2P functionality you have to create a few files in `./data/p2p`:
@@ -30,14 +61,17 @@ To setup P2P functionality you have to create a few files in `./data/p2p`:
}
```
- * `peers.json` : an array, with objects same as above, and `{ publicKey: "public key of the server" }`. Public key is needed to recieve users too.
+ * `peers.json` : an array, with objects same as above, and `{ publicKey: "public key of the server"
+ }`. Public key is used to encrypt the users database in the response, so they can be synced too.
Uppon syncing data or having a peer request data from your server there will be new entries in
`./data/p2p/thirdPartyPeers.json`. Here you can review the peers, see their contact and host, and if
you choose you can add them to your `peers.json` file. `thirdPartyPeers.json` should also contain
the public key
-## Maintenence
+To start syncing user #1 should perform a get request to `/syncp2pdata`
+
+## Maintenance
The server doesn't require that much maintenance, but you are advised to:
@@ -45,12 +79,12 @@ The server doesn't require that much maintenance, but you are advised to:
* Check the received messages on the platform you provided in the p2p information "contact" field
* Check and moderate the forum(s) of the page
* Sync the server with other peers (this can be automated)
+ * Check `./data/p2p/thirdPartyPeers.json` file for new peers, and decide if you should use them
* Regularly check if there is an update to this server and submodules, and update them (using git)
* Watch out for directories that can get big:
* `./stats`: server statistics and logs
* `./data/dbs/backup`: backup of databases
* `./publicDirs/qminingPublic/backs`: backup of question databases
- * `./publicDirs/qminingPublic/backs`: backup of question databases
* `./publicDirs/qminingPublic/userFiles`: files shared by users
* `./publicDirs/qminingPublic/savedQuestions`: unanswered questions saved by the userscript
* Make regular backups of important data:
@@ -68,6 +102,9 @@ The server doesn't require that much maintenance, but you are advised to:
The real time log can be found in `./stats/(v)logs`. Each folder contains rotated logs, one file per
day. In `log` only the most important events are logged, in `vlogs` every request is logged.
+There are other files in `./stats/`, the contents of them is explained in the detailed file
+structure below.
+
The script `./scripts/serverStats.js` pretty prints server statistics. The first argument should be
the server root directory. The second one is optional: a negative number, which shifts the
statistics from the current day. For ex.: if its -4, it displays stats from 4 days before now.
@@ -75,44 +112,66 @@ statistics from the current day. For ex.: if its -4, it displays stats from 4 da
`./scripts/exportStats.js` exports data configured in `exportStats.js`-s first few line to .csv. The
result can be found in the directory it was ran.
-## Server maintenence utils
+## Server maintenance utils
These scripts can be found in `./src/standaloneUtils`.
- | name | description |
+ | Name | Description |
| --- | --- |
| serverMaintenenceUtils.js | Designed to be a single script to collect all these utils. Not done yet |
| dbSetup.js | Sets up the database |
- | rmDuplicates.js | Removes duplicates (questions and subjects) from question db-s. Can merge db-s |
+ | rmDuplicates.js | Removes duplicates (questions and subjects) from question db-s. Can merge db-s. This is extremely CPU intensive |
The other undocumented scripts are rather old, and should be checked if they work, and if they are
needed at all
-## Detailed file structure
+## Environment variables
+ | Name | Type | Description |
+ | --- | --- | --- |
+ | PORT | number | The port the http server should run on |
+ | NS_THREAD_COUNT | number | Nubmer of CPU cores to use |
+ | NS_NOUSER | boolean | If the authorization should be skipped (for testing) |
+ | NS_DEVEL | boolean | Developemnt mode. Now it only disables automatic redirects from http to https |
+ | NS_LOGLEVEL | number | Debug log level, 0 is the least verbose |
+ | NS_NOLOG | boolean | If logging should be skipped |
+ | NS_SQL_DEBUG_LOG | boolean | If the SQL queries should be logged |
+
+## npm scripts
+
+ | Name | Description |
+ | --- | --- |
+ | npm run start | Runs server in prod mode. Requires build files in `./dist` |
+ | npm run dev | Re-builds and runs server in development mode with debugger attached |
+ | npm run build | Builds the server with tsc |
+ | npm run export | Same as build |
+ | npm run test | Runs jest tests |
+ | npm run test-debug | Runs jest tests in watch mode with debugger attached |
+
+## Detailed file structure
```
.
├── data/ server specific data files not tracked by git
│ ├── admins.json forum admins. should be removed and should use admin column from user db
│ ├── apiRootRedirectTo url where domain/api should redirect to
-│ ├── dbs/ directory for databases
+│ ├── dbs/ directory for databases, and for their backups
│ ├── domain the domain the server is hosted on
│ ├── donateURL url where the donate button should take to TODO: check if this is needed
-│ ├── f/ user files recieved TODO: check if this is needed
+│ ├── f/ user files received TODO: check if this is needed
│ ├── links.json urls for irc, patreon and donate
│ ├── nolog ids of users separated by new lines to ignore on logging
│ ├── p2p/ p2p data
│ │ ├── key.priv private key generated on first server run
│ │ ├── key.pub public key generated on first server run
│ │ ├── peers.json peers to check on sync
-│ │ ├── selfInfo.json p2p information of thish server instance
+│ │ ├── selfInfo.json p2p information of this server instance
│ │ └── thirdPartyPeers.json third party peers encountered
│ ├── statExclude.json array of strings. If included in url then excluded from stats
-│ └── testUsers.json test users. revieved data won't get added to question dbs
+│ └── testUsers.json test users. received data won't get added to question dbs
├── dist/ build .js files
├── extraSubmodules/ extra submodules not tracked by git
-├── nextStatic/ exported statis html files generated by next.js
-├── scripts scripts for server maintanance
+├── nextStatic/ exported static html files generated by next.js
+├── scripts scripts for server maintenance
│ ├── exportStats.js exports statistics to csv
│ ├── postBuild.sh runs after every build
│ ├── runDocker.sh runs docker
@@ -120,8 +179,8 @@ needed at all
│ └── setup.sh sets up the server before first run
├── src server source files
├── stats statistics files
-│ ├── askedQuestions recieved data on /ask, text file
-│ ├── recdata recieved data on /isAdding
+│ ├── askedQuestions received data on /ask, text file
+│ ├── recdata received data on /isAdding
│ ├── recievedQuestions same as recdata, should be removed
│ ├── dailyDataCount daily data count, text file
│ ├── dataEdits data editor activity log
diff --git a/scripts/setup.sh b/scripts/setup.sh
index 6ffb487..6672182 100755
--- a/scripts/setup.sh
+++ b/scripts/setup.sh
@@ -83,10 +83,7 @@ setupJson stats/stats
setupJson stats/vstats
setupJson stats/idstats
setupJson stats/idvstats
-setupJson publicDirs/qminingPublic/userSpecificMotd.json
-setupJson publicDirs/qminingPublic/news.json
-touch publicDirs/qminingPublic/version
touch publicDirs/qminingPublic/motd
if [ -z "$(ls -A ./data/dbs)" ]; then
diff --git a/src/modules/api/submodules/userManagement.ts b/src/modules/api/submodules/userManagement.ts
index b2b0e06..890d87b 100644
--- a/src/modules/api/submodules/userManagement.ts
+++ b/src/modules/api/submodules/userManagement.ts
@@ -325,7 +325,7 @@ function setup(data: SubmoduleData): Submodule {
id: +userid,
})
- if (specifiedUser.length === 0) {
+ if (specifiedUser.length === 0 || !specifiedUser[0]) {
res.json({
result: 'nouserid',
msg: 'couldnt find user',
diff --git a/submodules/qmining-page b/submodules/qmining-page
index dfc5c93..f82ecaa 160000
--- a/submodules/qmining-page
+++ b/submodules/qmining-page
@@ -1 +1 @@
-Subproject commit dfc5c93022b5c1a665e73e351a39fe6b8969d46c
+Subproject commit f82ecaa6d00d4f0625fb678c2c645a6ed3d79c03