readme update, removed unnecesarry parts from setup, validation token null check

README.md

@@ -1,22 +1,53 @@
-# Express.js server
+Modular node server written in typescript with express js. Serves two frontends and a userscript,
+which together with this server are designed to collect questions from moodle/elearning sites.
+
+## Requirements
+
+* Node v19.8.1 (and npm)
+* sqlite
+* Linux (not tested on windows)
+* Cloudflare is highly recommended to hide host IP, and use its other services
+
+### Hardware requirements
+
+The server doesn't have high hardware requirements, but users can use it in seasonally, and in
+bursts. This causes it to consume very low resources in most of the time, but can spike up quite a
+bit for a few minutes. This is because solvable test are commonly scheduled to specific time
+intervals (exams, end of term tests). This also depends on the number of users using it, and the
+question db size.
+
+It used to run fine on a raspberry pi 3 B+, but needed a faster storage than an SD card. Later the
+pi was switched, and ran flawlessly on a Core2 Duo 3GHz, 4GB RAM + SSD. With larger question
+databases the pi might not be enough.
+
+The server utilizes multiple CPU cores, long running operations are ran in separate threads. Because
+of the implementation, the more cores a CPU has, the server uses more memory, but able to run more
+threads, and serve more requests at once. The used cores can be limited with environment variables
+(detailed below).
+
+## Terminology
+
+ | Name | Description |
+ | --- | --- |
+ | Question database | A JSON file, array of saved subjects wich have a Name, and Questions array |
+ | peer to peer functionality | The ability to share question databases and users with other instances of this server |
+ | Peer | Another instance of this server, with peer to peer functionality set up |
+ | User #1 | The first user created, admin of the server |
 
 ## Setup
 
 Run `./scripts/setup.sh`, then `npm run dev` for development, or `npm run start` for prod
 
 On the first run there will be a number of errors, that some files weren't found. Please create them
-according to the messages, these are necessary for the server to function
+according to the messages, these are necessary for the server to function.
 
-## Terminology
+There will be also a lot of information about files and other necessary things being created. **Please
- | Name | Description |
+read them very carefully, you should know about what was created!**
- | --- | --- |
- | Question database | a JSON file, array of saved subjects wich have a Name, and Questions array |
- | Peer | another instance of this server, with p2p set up |
 
-## P2P setup
+## Peer to peer
 
-This server implements P2P functionality. It can fetch data from other server instances, and merge
+This server implements P2P functionality. It can fetch question databases and users from other
-the response data to its own database
+server instances, and merge the response data to its own databases
 
 To setup P2P functionality you have to create a few files in `./data/p2p`:
 
@@ -30,14 +61,17 @@ To setup P2P functionality you have to create a few files in `./data/p2p`:
     }
     ```
 
- * `peers.json` : an array, with objects same as above, and `{ publicKey: "public key of the server" }`. Public key is needed to recieve users too.
+ * `peers.json` : an array, with objects same as above, and `{ publicKey: "public key of the server"
+ }`. Public key is used to encrypt the users database in the response, so they can be synced too.
 
 Uppon syncing data or having a peer request data from your server there will be new entries in
 `./data/p2p/thirdPartyPeers.json`. Here you can review the peers, see their contact and host, and if
 you choose you can add them to your `peers.json` file. `thirdPartyPeers.json` should also contain
 the public key
 
-## Maintenence
+To start syncing user #1 should perform a get request to `/syncp2pdata`
+
+## Maintenance
 
 The server doesn't require that much maintenance, but you are advised to:
 
@@ -45,12 +79,12 @@ The server doesn't require that much maintenance, but you are advised to:
  * Check the received messages on the platform you provided in the p2p information "contact" field
  * Check and moderate the forum(s) of the page
  * Sync the server with other peers (this can be automated)
+ * Check `./data/p2p/thirdPartyPeers.json` file for new peers, and decide if you should use them
  * Regularly check if there is an update to this server and submodules, and update them (using git)
  * Watch out for directories that can get big:
    * `./stats`: server statistics and logs
    * `./data/dbs/backup`: backup of databases
    * `./publicDirs/qminingPublic/backs`: backup of question databases
-   * `./publicDirs/qminingPublic/backs`: backup of question databases
    * `./publicDirs/qminingPublic/userFiles`: files shared by users
    * `./publicDirs/qminingPublic/savedQuestions`: unanswered questions saved by the userscript
  * Make regular backups of important data:
@@ -68,6 +102,9 @@ The server doesn't require that much maintenance, but you are advised to:
 The real time log can be found in `./stats/(v)logs`. Each folder contains rotated logs, one file per
 day. In `log` only the most important events are logged, in `vlogs` every request is logged.
 
+There are other files in `./stats/`, the contents of them is explained in the detailed file
+structure below.
+
 The script `./scripts/serverStats.js` pretty prints server statistics. The first argument should be
 the server root directory. The second one is optional: a negative number, which shifts the
 statistics from the current day. For ex.: if its -4, it displays stats from 4 days before now.
@@ -75,44 +112,66 @@ statistics from the current day. For ex.: if its -4, it displays stats from 4 da
 `./scripts/exportStats.js` exports data configured in `exportStats.js`-s first few line to .csv. The
 result can be found in the directory it was ran.
 
-## Server maintenence utils
+## Server maintenance utils
 
 These scripts can be found in `./src/standaloneUtils`.
 
- | name | description |
+ | Name | Description |
  | --- | --- |
  | serverMaintenenceUtils.js | Designed to be a single script to collect all these utils. Not done yet |
  | dbSetup.js | Sets up the database |
- | rmDuplicates.js | Removes duplicates (questions and subjects) from question db-s. Can merge db-s |
+ | rmDuplicates.js | Removes duplicates (questions and subjects) from question db-s. Can merge db-s. This is extremely CPU intensive |
 
 The other undocumented scripts are rather old, and should be checked if they work, and if they are
 needed at all
 
-## Detailed file structure
+## Environment variables
 
+ | Name | Type | Description |
+ | --- | --- | --- |
+ | PORT | number | The port the http server should run on |
+ | NS_THREAD_COUNT | number | Nubmer of CPU cores to use |
+ | NS_NOUSER | boolean | If the authorization should be skipped (for testing) |
+ | NS_DEVEL | boolean | Developemnt mode. Now it only disables automatic redirects from http to https |
+ | NS_LOGLEVEL | number | Debug log level, 0 is the least verbose |
+ | NS_NOLOG | boolean | If logging should be skipped |
+ | NS_SQL_DEBUG_LOG | boolean | If the SQL queries should be logged |
+
+## npm scripts
+
+ | Name | Description |
+ | --- | --- |
+ | npm run start | Runs server in prod mode. Requires build files in `./dist` |
+ | npm run dev | Re-builds and runs server in development mode with debugger attached |
+ | npm run build | Builds the server with tsc |
+ | npm run export | Same as build |
+ | npm run test | Runs jest tests |
+ | npm run test-debug | Runs jest tests in watch mode with debugger attached |
+
+## Detailed file structure
 ```
 .
 ├── data/                          server specific data files not tracked by git
 │   ├── admins.json                forum admins. should be removed and should use admin column from user db
 │   ├── apiRootRedirectTo          url where domain/api should redirect to
-│   ├── dbs/                       directory for databases
+│   ├── dbs/                       directory for databases, and for their backups
 │   ├── domain                     the domain the server is hosted on
 │   ├── donateURL                  url where the donate button should take to TODO: check if this is needed
-│   ├── f/                         user files recieved TODO: check if this is needed
+│   ├── f/                         user files received TODO: check if this is needed
 │   ├── links.json                 urls for irc, patreon and donate
 │   ├── nolog                      ids of users separated by new lines to ignore on logging
 │   ├── p2p/                       p2p data
 │   │   ├── key.priv               private key generated on first server run
 │   │   ├── key.pub                public key generated on first server run
 │   │   ├── peers.json             peers to check on sync
-│   │   ├── selfInfo.json          p2p information of thish server instance
+│   │   ├── selfInfo.json          p2p information of this server instance
 │   │   └── thirdPartyPeers.json   third party peers encountered
 │   ├── statExclude.json           array of strings. If included in url then excluded from stats
-│   └── testUsers.json             test users. revieved data won't get added to question dbs
+│   └── testUsers.json             test users. received data won't get added to question dbs
 ├── dist/                          build .js files
 ├── extraSubmodules/               extra submodules not tracked by git
-├── nextStatic/                    exported statis html files generated by next.js
+├── nextStatic/                    exported static html files generated by next.js
-├── scripts                        scripts for server maintanance
+├── scripts                        scripts for server maintenance
 │   ├── exportStats.js             exports statistics to csv
 │   ├── postBuild.sh               runs after every build
 │   ├── runDocker.sh               runs docker
@@ -120,8 +179,8 @@ needed at all
 │   └── setup.sh                   sets up the server before first run
 ├── src                            server source files
 ├── stats                          statistics files
-│   ├── askedQuestions             recieved data on /ask, text file
+│   ├── askedQuestions             received data on /ask, text file
-│   ├── recdata                    recieved data on /isAdding
+│   ├── recdata                    received data on /isAdding
 │   ├── recievedQuestions          same as recdata, should be removed
 │   ├── dailyDataCount             daily data count, text file
 │   ├── dataEdits                  data editor activity log

scripts/setup.sh

@@ -83,10 +83,7 @@ setupJson stats/stats
 setupJson stats/vstats
 setupJson stats/idstats
 setupJson stats/idvstats
-setupJson publicDirs/qminingPublic/userSpecificMotd.json
-setupJson publicDirs/qminingPublic/news.json
 
-touch publicDirs/qminingPublic/version
 touch publicDirs/qminingPublic/motd
 
 if [ -z "$(ls -A ./data/dbs)" ]; then

src/modules/api/submodules/userManagement.ts

@@ -325,7 +325,7 @@ function setup(data: SubmoduleData): Submodule {
                     id: +userid,
                 })
 
-                if (specifiedUser.length === 0) {
+                if (specifiedUser.length === 0 || !specifiedUser[0]) {
                     res.json({
                         result: 'nouserid',
                         msg: 'couldnt find user',

submodules/qmining-page

@@ -1 +1 @@
-Subproject commit dfc5c93022b5c1a665e73e351a39fe6b8969d46c
+Subproject commit f82ecaa6d00d4f0625fb678c2c645a6ed3d79c03