Express.js server

Setup

Run ./scripts/setup.sh, then npm run dev for development, or npm run start for prod

On the first run there will be a number of errors, that some files weren't found. Please create them according to the messages, these are necessary for the server to function

Terminology

Name	Description
Question database	a JSON file, array of saved subjects wich have a Name, and Questions array
Peer	another instance of this server, with p2p set up

P2P setup

This server implements P2P functionality. It can fetch data from other server instances, and merge the response data to its own database

To setup P2P functionality you have to create a few files in ./data/p2p:

• selfInfo.json: information of this peer. Required:

  {
    name: "Any name you choose",
    contact: "contact to server administrator (irc server, e-mail, anything)",
    host: "server host (like somesite.com, without 'http(s)://')",
    port: "server port, number",
  }
  

• peers.json : an array, with objects same as above, and { publicKey: "public key of the server" }. Public key is needed to recieve users too.

Uppon syncing data or having a peer request data from your server there will be new entries in ./data/p2p/thirdPartyPeers.json. Here you can review the peers, see their contact and host, and if you choose you can add them to your peers.json file. thirdPartyPeers.json should also contain the public key

Maintenence

The server doesn't require that much maintenance, but you are advised to:

• Check the chat messages, the users can contact you there through the website
• Check the received messages on the platform you provided in the p2p information "contact" field
• Check and moderate the forum(s) of the page
• Sync the server with other peers (this can be automated)
• Regularly check if there is an update to this server and submodules, and update them (using git)
• Watch out for directories that can get big:
  • ./stats: server statistics and logs
  • ./data/dbs/backup: backup of databases
  • ./publicDirs/qminingPublic/backs: backup of question databases
  • ./publicDirs/qminingPublic/backs: backup of question databases
  • ./publicDirs/qminingPublic/userFiles: files shared by users
  • ./publicDirs/qminingPublic/savedQuestions: unanswered questions saved by the userscript
• Make regular backups of important data:
  • ./data/dbs: user and messages database
  • ./data/p2p: p2p data, and public/private keys
  • ./publicDirs/qminingPublic/questionDbs: question dbs
  • ./publicDirs/qminingPublic/questionDbs.json: information about question db-s
  • ./publicDirs/qminingPublic/userFiles: files shared by users
  • ./publicDirs/qminingPublic/forum: forum entries and comments
  • ./publicDirs/qminingPublic/savedQuestions: unanswered questions saved by the userscript
  • Most files not tracked by git

Server usage statistics

The real time log can be found in ./stats/(v)logs. Each folder contains rotated logs, one file per day. In log only the most important events are logged, in vlogs every request is logged.

The script ./scripts/serverStats.js pretty prints server statistics. The first argument should be the server root directory. The second one is optional: a negative number, which shifts the statistics from the current day. For ex.: if its -4, it displays stats from 4 days before now.

./scripts/exportStats.js exports data configured in exportStats.js-s first few line to .csv. The result can be found in the directory it was ran.

Server maintenence utils

These scripts can be found in ./src/standaloneUtils.

name	description
serverMaintenenceUtils.js	Designed to be a single script to collect all these utils. Not done yet
dbSetup.js	Sets up the database
rmDuplicates.js	Removes duplicates (questions and subjects) from question db-s. Can merge db-s

The other undocumented scripts are rather old, and should be checked if they work, and if they are needed at all

Detailed file structure

.
├── data/                          server specific data files not tracked by git
│   ├── admins.json                forum admins. should be removed and should use admin column from user db
│   ├── apiRootRedirectTo          url where domain/api should redirect to
│   ├── dbs/                       directory for databases
│   ├── domain                     the domain the server is hosted on
│   ├── donateURL                  url where the donate button should take to TODO: check if this is needed
│   ├── f/                         user files recieved TODO: check if this is needed
│   ├── links.json                 urls for irc, patreon and donate
│   ├── nolog                      ids of users separated by new lines to ignore on logging
│   ├── p2p/                       p2p data
│   │   ├── key.priv               private key generated on first server run
│   │   ├── key.pub                public key generated on first server run
│   │   ├── peers.json             peers to check on sync
│   │   ├── selfInfo.json          p2p information of thish server instance
│   │   └── thirdPartyPeers.json   third party peers encountered
│   ├── statExclude.json           array of strings. If included in url then excluded from stats
│   └── testUsers.json             test users. revieved data won't get added to question dbs
├── dist/                          build .js files
├── extraSubmodules/               extra submodules not tracked by git
├── nextStatic/                    exported statis html files generated by next.js
├── scripts                        scripts for server maintanance
│   ├── exportStats.js             exports statistics to csv
│   ├── postBuild.sh               runs after every build
│   ├── runDocker.sh               runs docker
│   ├── serverStats.js             pretty prints server statistics
│   └── setup.sh                   sets up the server before first run
├── src                            server source files
├── stats                          statistics files
│   ├── askedQuestions             recieved data on /ask, text file
│   ├── recdata                    recieved data on /isAdding
│   ├── recievedQuestions          same as recdata, should be removed
│   ├── dailyDataCount             daily data count, text file
│   ├── dataEdits                  data editor activity log
│   ├── logs                       generic logs per day. latest is 'log'
│   ├── vlogs                      detailed logs per day. latest is 'log'
│   ├── qvote/                     quick vote data directory
│   ├── registeredScripts.json     registered scripts JSON
│   ├── stats                      website accesses per path
│   ├── idstats                    test solving statistics per user
│   ├── idvstats                   test solving statistics per day per user
│   ├── ustats                     website accesses count per user
│   ├── uvstats                    website accesses count per day per user
│   └── vstats                     website accessed paths per day
├── submodules                     git submodules
│   ├── moodle-test-userscript     userscript
│   ├── qmining-data-editor        data editor frontend
│   └── qmining-page               qmining frontend
├── testingTools                   testing tools for the server
└── publicDirs/                    public directories of the server, mostly available on the domain root
    └── qminingPublic/             qmining module public path (modules: qmining, dataeditor, api)
        ├── backs/                 question database backups
        ├── chatFiles/             files sent on chat
        ├── contacts.json          contacts displayed on the /contact page
        ├── forum/                 forum contents
        ├── forumFiles/            files uploaded to forums
        ├── moodle-test-userscript link to the userscript, it fetches updates from this path
        ├── motd                   motto of the day
        ├── questionDbs/           directory of the question db-s
        ├── questionDbs.json       question db-s information
        ├── savedQuestions/        un-answered questions for dataeditor, saved from test pages
        └── userFiles/             files shared by users

License

GNU General Public License, version 3