diff --git a/README.md b/README.md index ad62ce4..225d83a 100755 --- a/README.md +++ b/README.md @@ -1,22 +1,53 @@ -# Express.js server +Modular node server written in typescript with express js. Serves two frontends and a userscript, +which together with this server are designed to collect questions from moodle/elearning sites. + +## Requirements + +* Node v19.8.1 (and npm) +* sqlite +* Linux (not tested on windows) +* Cloudflare is highly recommended to hide host IP, and use its other services + +### Hardware requirements + +The server doesn't have high hardware requirements, but users can use it in seasonally, and in +bursts. This causes it to consume very low resources in most of the time, but can spike up quite a +bit for a few minutes. This is because solvable test are commonly scheduled to specific time +intervals (exams, end of term tests). This also depends on the number of users using it, and the +question db size. + +It used to run fine on a raspberry pi 3 B+, but needed a faster storage than an SD card. Later the +pi was switched, and ran flawlessly on a Core2 Duo 3GHz, 4GB RAM + SSD. With larger question +databases the pi might not be enough. + +The server utilizes multiple CPU cores, long running operations are ran in separate threads. Because +of the implementation, the more cores a CPU has, the server uses more memory, but able to run more +threads, and serve more requests at once. The used cores can be limited with environment variables +(detailed below). + +## Terminology + + | Name | Description | + | --- | --- | + | Question database | A JSON file, array of saved subjects wich have a Name, and Questions array | + | peer to peer functionality | The ability to share question databases and users with other instances of this server | + | Peer | Another instance of this server, with peer to peer functionality set up | + | User #1 | The first user created, admin of the server | ## Setup Run `./scripts/setup.sh`, then `npm run dev` for development, or `npm run start` for prod On the first run there will be a number of errors, that some files weren't found. Please create them -according to the messages, these are necessary for the server to function +according to the messages, these are necessary for the server to function. -## Terminology - | Name | Description | - | --- | --- | - | Question database | a JSON file, array of saved subjects wich have a Name, and Questions array | - | Peer | another instance of this server, with p2p set up | +There will be also a lot of information about files and other necessary things being created. **Please +read them very carefully, you should know about what was created!** -## P2P setup +## Peer to peer -This server implements P2P functionality. It can fetch data from other server instances, and merge -the response data to its own database +This server implements P2P functionality. It can fetch question databases and users from other +server instances, and merge the response data to its own databases To setup P2P functionality you have to create a few files in `./data/p2p`: @@ -30,14 +61,17 @@ To setup P2P functionality you have to create a few files in `./data/p2p`: } ``` - * `peers.json` : an array, with objects same as above, and `{ publicKey: "public key of the server" }`. Public key is needed to recieve users too. + * `peers.json` : an array, with objects same as above, and `{ publicKey: "public key of the server" + }`. Public key is used to encrypt the users database in the response, so they can be synced too. Uppon syncing data or having a peer request data from your server there will be new entries in `./data/p2p/thirdPartyPeers.json`. Here you can review the peers, see their contact and host, and if you choose you can add them to your `peers.json` file. `thirdPartyPeers.json` should also contain the public key -## Maintenence +To start syncing user #1 should perform a get request to `/syncp2pdata` + +## Maintenance The server doesn't require that much maintenance, but you are advised to: @@ -45,12 +79,12 @@ The server doesn't require that much maintenance, but you are advised to: * Check the received messages on the platform you provided in the p2p information "contact" field * Check and moderate the forum(s) of the page * Sync the server with other peers (this can be automated) + * Check `./data/p2p/thirdPartyPeers.json` file for new peers, and decide if you should use them * Regularly check if there is an update to this server and submodules, and update them (using git) * Watch out for directories that can get big: * `./stats`: server statistics and logs * `./data/dbs/backup`: backup of databases * `./publicDirs/qminingPublic/backs`: backup of question databases - * `./publicDirs/qminingPublic/backs`: backup of question databases * `./publicDirs/qminingPublic/userFiles`: files shared by users * `./publicDirs/qminingPublic/savedQuestions`: unanswered questions saved by the userscript * Make regular backups of important data: @@ -68,6 +102,9 @@ The server doesn't require that much maintenance, but you are advised to: The real time log can be found in `./stats/(v)logs`. Each folder contains rotated logs, one file per day. In `log` only the most important events are logged, in `vlogs` every request is logged. +There are other files in `./stats/`, the contents of them is explained in the detailed file +structure below. + The script `./scripts/serverStats.js` pretty prints server statistics. The first argument should be the server root directory. The second one is optional: a negative number, which shifts the statistics from the current day. For ex.: if its -4, it displays stats from 4 days before now. @@ -75,44 +112,66 @@ statistics from the current day. For ex.: if its -4, it displays stats from 4 da `./scripts/exportStats.js` exports data configured in `exportStats.js`-s first few line to .csv. The result can be found in the directory it was ran. -## Server maintenence utils +## Server maintenance utils These scripts can be found in `./src/standaloneUtils`. - | name | description | + | Name | Description | | --- | --- | | serverMaintenenceUtils.js | Designed to be a single script to collect all these utils. Not done yet | | dbSetup.js | Sets up the database | - | rmDuplicates.js | Removes duplicates (questions and subjects) from question db-s. Can merge db-s | + | rmDuplicates.js | Removes duplicates (questions and subjects) from question db-s. Can merge db-s. This is extremely CPU intensive | The other undocumented scripts are rather old, and should be checked if they work, and if they are needed at all -## Detailed file structure +## Environment variables + | Name | Type | Description | + | --- | --- | --- | + | PORT | number | The port the http server should run on | + | NS_THREAD_COUNT | number | Nubmer of CPU cores to use | + | NS_NOUSER | boolean | If the authorization should be skipped (for testing) | + | NS_DEVEL | boolean | Developemnt mode. Now it only disables automatic redirects from http to https | + | NS_LOGLEVEL | number | Debug log level, 0 is the least verbose | + | NS_NOLOG | boolean | If logging should be skipped | + | NS_SQL_DEBUG_LOG | boolean | If the SQL queries should be logged | + +## npm scripts + + | Name | Description | + | --- | --- | + | npm run start | Runs server in prod mode. Requires build files in `./dist` | + | npm run dev | Re-builds and runs server in development mode with debugger attached | + | npm run build | Builds the server with tsc | + | npm run export | Same as build | + | npm run test | Runs jest tests | + | npm run test-debug | Runs jest tests in watch mode with debugger attached | + +## Detailed file structure ``` . ├── data/ server specific data files not tracked by git │ ├── admins.json forum admins. should be removed and should use admin column from user db │ ├── apiRootRedirectTo url where domain/api should redirect to -│ ├── dbs/ directory for databases +│ ├── dbs/ directory for databases, and for their backups │ ├── domain the domain the server is hosted on │ ├── donateURL url where the donate button should take to TODO: check if this is needed -│ ├── f/ user files recieved TODO: check if this is needed +│ ├── f/ user files received TODO: check if this is needed │ ├── links.json urls for irc, patreon and donate │ ├── nolog ids of users separated by new lines to ignore on logging │ ├── p2p/ p2p data │ │ ├── key.priv private key generated on first server run │ │ ├── key.pub public key generated on first server run │ │ ├── peers.json peers to check on sync -│ │ ├── selfInfo.json p2p information of thish server instance +│ │ ├── selfInfo.json p2p information of this server instance │ │ └── thirdPartyPeers.json third party peers encountered │ ├── statExclude.json array of strings. If included in url then excluded from stats -│ └── testUsers.json test users. revieved data won't get added to question dbs +│ └── testUsers.json test users. received data won't get added to question dbs ├── dist/ build .js files ├── extraSubmodules/ extra submodules not tracked by git -├── nextStatic/ exported statis html files generated by next.js -├── scripts scripts for server maintanance +├── nextStatic/ exported static html files generated by next.js +├── scripts scripts for server maintenance │ ├── exportStats.js exports statistics to csv │ ├── postBuild.sh runs after every build │ ├── runDocker.sh runs docker @@ -120,8 +179,8 @@ needed at all │ └── setup.sh sets up the server before first run ├── src server source files ├── stats statistics files -│ ├── askedQuestions recieved data on /ask, text file -│ ├── recdata recieved data on /isAdding +│ ├── askedQuestions received data on /ask, text file +│ ├── recdata received data on /isAdding │ ├── recievedQuestions same as recdata, should be removed │ ├── dailyDataCount daily data count, text file │ ├── dataEdits data editor activity log diff --git a/scripts/setup.sh b/scripts/setup.sh index 6ffb487..6672182 100755 --- a/scripts/setup.sh +++ b/scripts/setup.sh @@ -83,10 +83,7 @@ setupJson stats/stats setupJson stats/vstats setupJson stats/idstats setupJson stats/idvstats -setupJson publicDirs/qminingPublic/userSpecificMotd.json -setupJson publicDirs/qminingPublic/news.json -touch publicDirs/qminingPublic/version touch publicDirs/qminingPublic/motd if [ -z "$(ls -A ./data/dbs)" ]; then diff --git a/src/modules/api/submodules/userManagement.ts b/src/modules/api/submodules/userManagement.ts index b2b0e06..890d87b 100644 --- a/src/modules/api/submodules/userManagement.ts +++ b/src/modules/api/submodules/userManagement.ts @@ -325,7 +325,7 @@ function setup(data: SubmoduleData): Submodule { id: +userid, }) - if (specifiedUser.length === 0) { + if (specifiedUser.length === 0 || !specifiedUser[0]) { res.json({ result: 'nouserid', msg: 'couldnt find user', diff --git a/submodules/qmining-page b/submodules/qmining-page index dfc5c93..f82ecaa 160000 --- a/submodules/qmining-page +++ b/submodules/qmining-page @@ -1 +1 @@ -Subproject commit dfc5c93022b5c1a665e73e351a39fe6b8969d46c +Subproject commit f82ecaa6d00d4f0625fb678c2c645a6ed3d79c03