From c00552abf2cfdf53160339289b96490b427c08d8 Mon Sep 17 00:00:00 2001 From: mrfry Date: Mon, 3 Apr 2023 17:21:28 +0200 Subject: [PATCH] readme update --- README.md | 155 ++++++++++++++++++++++++++++++++++++++++ scripts/start.sh | 4 -- src/utils/files.ts | 3 +- submodules/qmining-page | 2 +- 4 files changed, 158 insertions(+), 6 deletions(-) delete mode 100755 scripts/start.sh diff --git a/README.md b/README.md index e3bef0c..41104fa 100755 --- a/README.md +++ b/README.md @@ -3,3 +3,158 @@ ## Setup Run `./scripts/setup.sh`, then `npm run dev` for development, or `npm run start` for prod + +On the first run there will be a number of errors, that some files weren't found. Please create them +according to the messages, these are necessary for the server to function + +## Terminology + | Name | Description | + | --- | --- | + | Question database | a JSON file, array of saved subjects wich have a Name, and Questions array | + | Peer | another instance of this server, with p2p set up | + +## P2P setup + +This server implements P2P functionality. It can fetch data from other server instances, and merge +the response data to its own database + +To setup P2P functionality you have to create a few files in `./data/p2p`: + + * `selfInfo.json`: information of this peer. Required: + ``` + { + name: "Any name you choose", + contact: "contact to server administrator (irc server, e-mail, anything)", + host: "server host (like somesite.com, without 'http(s)://')", + port: "server port, number", + } + ``` + + * `peers.json` : an array, with objects same as above, and `{ publicKey: "public key of the server" }`. Public key is needed to recieve users too. + +Uppon syncing data or having a peer request data from your server there will be new entries in +`./data/p2p/thirdPartyPeers.json`. Here you can review the peers, see their contact and host, and if +you choose you can add them to your `peers.json` file. `thirdPartyPeers.json` should also contain +the public key + +## Maintenence + +The server doesn't require that much maintenance, but you are advised to: + + * Check the chat messages, the users can contact you there through the website + * Check the received messages on the platform you provided in the p2p information "contact" field + * Check and moderate the forum(s) of the page + * Sync the server with other peers (this can be automated) + * Regularly check if there is an update to this server and submodules, and update them (using git) + * Watch out for directories that can get big: + * `./stats`: server statistics and logs + * `./data/dbs/backup`: backup of databases + * `./publicDirs/qminingPublic/backs`: backup of question databases + * `./publicDirs/qminingPublic/backs`: backup of question databases + * `./publicDirs/qminingPublic/userFiles`: files shared by users + * `./publicDirs/qminingPublic/savedQuestions`: unanswered questions saved by the userscript + * Make regular backups of important data: + * `./data/dbs`: user and messages database + * `./data/p2p`: p2p data, and public/private keys + * `./publicDirs/qminingPublic/questionDbs`: question dbs + * `./publicDirs/qminingPublic/questionDbs.json`: information about question db-s + * `./publicDirs/qminingPublic/userFiles`: files shared by users + * `./publicDirs/qminingPublic/forum`: forum entries and comments + * `./publicDirs/qminingPublic/savedQuestions`: unanswered questions saved by the userscript + * Most files not tracked by git + +## Server usage statistics + +The real time log can be found in `./stats/(v)logs`. Each folder contains rotated logs, one file per +day. In `log` only the most important events are logged, in `vlogs` every request is logged. + +The script `./scripts/serverStats.js` pretty prints server statistics. The first argument should be +the server root directory. The second one is optional: a negative number, which shifts the +statistics from the current day. For ex.: if its -4, it displays stats from 4 days before now. + +`./scripts/exportStats.js` exports data configured in `exportStats.js`-s first few line to .csv. The +result can be found in the directory it was ran. + +## Server maintenence utils + +These scripts can be found in `./src/standaloneUtils`. + + | name | description | + | --- | --- | + | serverMaintenenceUtils.js | Designed to be a single script to collect all these utils. Not done yet | + | dbSetup.js | Sets up the database | + | rmDuplicates.js | Removes duplicates (questions and subjects) from question db-s. Can merge db-s | + +The other undocumented scripts are rather old, and should be checked if they work, and if they are +needed at all + +## Detailed file structure + +``` +. +├── data/ server specific data files not tracked by git +│ ├── admins.json forum admins. should be removed and should use admin column from user db +│ ├── apiRootRedirectTo url where domain/api should redirect to +│ ├── dbs/ directory for databases +│ ├── domain the domain the server is hosted on +│ ├── donateURL url where the donate button should take to TODO: check if this is needed +│ ├── f/ user files recieved TODO: check if this is needed +│ ├── links.json urls for irc, patreon and donate +│ ├── nolog ids of users separated by new lines to ignore on logging +│ ├── p2p/ p2p data +│ │ ├── key.priv private key generated on first server run +│ │ ├── key.pub public key generated on first server run +│ │ ├── peers.json peers to check on sync +│ │ ├── selfInfo.json p2p information of thish server instance +│ │ └── thirdPartyPeers.json third party peers encountered +│ ├── statExclude.json array of strings. If included in url then excluded from stats +│ └── testUsers.json test users. revieved data won't get added to question dbs +├── dist/ build .js files +├── extraSubmodules/ extra submodules not tracked by git +├── nextStatic/ exported statis html files generated by next.js +├── scripts scripts for server maintanance +│ ├── exportStats.js exports statistics to csv +│ ├── postBuild.sh runs after every build +│ ├── runDocker.sh runs docker +│ ├── serverStats.js pretty prints server statistics +│ └── setup.sh sets up the server before first run +├── src server source files +├── stats statistics files +│ ├── askedQuestions recieved data on /ask, text file +│ ├── recdata recieved data on /isAdding +│ ├── recievedQuestions same as recdata, should be removed +│ ├── dailyDataCount daily data count, text file +│ ├── dataEdits data editor activity log +│ ├── logs generic logs per day. latest is 'log' +│ ├── vlogs detailed logs per day. latest is 'log' +│ ├── qvote/ quick vote data directory +│ ├── registeredScripts.json registered scripts JSON +│ ├── stats website accesses per path +│ ├── idstats test solving statistics per user +│ ├── idvstats test solving statistics per day per user +│ ├── ustats website accesses count per user +│ ├── uvstats website accesses count per day per user +│ └── vstats website accessed paths per day +├── submodules git submodules +│ ├── moodle-test-userscript userscript +│ ├── qmining-data-editor data editor frontend +│ └── qmining-page qmining frontend +├── testingTools testing tools for the server +└── publicDirs/ public directories of the server, mostly available on the domain root + └── qminingPublic/ qmining module public path (modules: qmining, dataeditor, api) + ├── backs/ question database backups + ├── chatFiles/ files sent on chat + ├── contacts.json contacts displayed on the /contact page + ├── forum/ forum contents + ├── forumFiles/ files uploaded to forums + ├── moodle-test-userscript link to the userscript, it fetches updates from this path + ├── motd motto of the day + ├── questionDbs/ directory of the question db-s + ├── questionDbs.json question db-s information + ├── savedQuestions/ un-answered questions for dataeditor, saved from test pages + └── userFiles/ files shared by users +``` + +## License + +GNU General Public License, version 3 diff --git a/scripts/start.sh b/scripts/start.sh deleted file mode 100755 index 3cca89e..0000000 --- a/scripts/start.sh +++ /dev/null @@ -1,4 +0,0 @@ -#!/bin/bash - -scripts/make.sh -npm run start > /dev/null diff --git a/src/utils/files.ts b/src/utils/files.ts index ee44057..02e70c7 100644 --- a/src/utils/files.ts +++ b/src/utils/files.ts @@ -171,7 +171,8 @@ export const files = { path: 'data/testUsers.json', defaultValue: JSON.stringify({ userIds: [] }), schema: TestUsersSchema, - description: 'test users, which are excluded from stats', + description: + 'test users, recieved data from them wont get added to question dbs', }, // -------------------------------------------------------------------------------- diff --git a/submodules/qmining-page b/submodules/qmining-page index 5c779a6..dfc5c93 160000 --- a/submodules/qmining-page +++ b/submodules/qmining-page @@ -1 +1 @@ -Subproject commit 5c779a657dc0543cef2abc886057e7828ce711f8 +Subproject commit dfc5c93022b5c1a665e73e351a39fe6b8969d46c