Modular node server written in typescript with express js. Serves two frontends and a userscript, which together with this server are designed to collect questions from moodle/elearning sites.

Requirements

• Node v19.8.1 (and npm)
• sqlite
• Linux (not tested on windows)
• Cloudflare is highly recommended to hide host IP, and use its other services

Other requirements

The server doesn't have high hardware requirements, but users can use it in seasonally, and in bursts. This causes it to consume very low resources in most of the time, but can spike up quite a bit for a few minutes. This is because solvable test are commonly scheduled to specific time intervals (exams, end of term tests). This also depends on the number of users using it, and the question db size.

It used to run fine on a raspberry pi 3 B+, but needed a faster storage than an SD card. Later the pi was switched, and ran flawlessly on a Core2 Duo 3GHz, 4GB RAM + SSD. With larger question databases the pi might not be enough.

The server utilizes multiple CPU cores, long running operations are ran in separate threads. Because of the implementation, the more cores a CPU has, the server uses more memory, but able to run more threads, and serve more requests at once. The used cores can be limited with environment variables (detailed below, NS_THREAD_COUNT).

To set up and perform maintenance tasks you will need basic linux, sqlite, javascript and cloudflare knowledge, a registered domain (or subdomain on some domain),

Terminology

Name	Description
Question database	A JSON file, array of saved subjects wich have a Name, and Questions array
Peer to peer functionality	The ability to share question databases and users with other instances of this server
Peer	Another instance of this server, with peer to peer functionality set up
Registered peer	A peer wich have pw and optionally publicKey specified in peers.json
User #1	The first user created, admin of the server
Submodule	A server submodule, dynamically loaded, and stored in ./src/modules/
Git module	Git modules stored in ./submodules

Setup

Run ./scripts/setup.sh, then npm run dev for development, or npm run start for prod.

On the first run there will be a number of errors, that some files weren't found, or some environment variables needed to be specified. Please create them according to the messages, these are necessary for the server to function.

There will be also a lot of information about files and other necessary things being created. Please read them very carefully, you should know about what was created!

Setup notes and tips

• Gitlab specific notes: You need a gitlab account with SSH keys set up in order to be able to use SSH for cloning this (and any) project. This includes git modules of this project. To use HTTPS set the CLONE_WITH_HTTPS environment variable before running setup.sh. To set up SSH keys: https://docs.gitlab.com/ee/user/ssh.html
• You either need to specify a DOMAIN env var, or write the domain to ./data/domain. This is necesarry to make xmlhttp requests and some redirects work in: statically built HTML files (by next.js), userscript, and the server itself. Domain is for example: 'qmining.com', without 'http://' and '/'-s
• By default the server redirects all HTTP traffic to HTTPS. To disable this use NS_NO_HTTPS_FORCE
• The server launches a thread for each CPU core. This could be an overkill on 4+ cored CPU-s. Use NS_THREAD_COUNT to restrict the number of threads, and potentially reduce memory usage of the server.
• The setup script can be also used to update and rebuild all git modules if ran after the initial setup

Web server paths

There are different routes assigned to different submodules. You can see these in detail in the files ./src/modules.json.

Peer to peer

This server implements P2P functionality. It can fetch question databases and users from other server instances, and merge the response data to its own databases. The server also instantly sends new questions received from users, new users and uploaded user files to all registered peers. The sync feature should be used for initialization and rarely for catching up, since important data is received instantly.

To setup P2P functionality you have to create a few files in ./data/p2p:

• selfInfo.json: information of this peer. Required:

  {
      "name": "any name you choose",
      "contact": "contact to server administrator (irc server, e-mail, anything)",
      "host": "server host (like 'qmining.com', without 'http(s):// and /-s')",
      "port": "server port, number"
  }
  

• peers.json: List of registered peers. An array, with objects same as above, and:

  {
      "pw": "password to the host, so the server can log in there. Please use a dedicated password, that only the server uses!",
      "publicKey": "public key of the server for encryption",
      "http": "boolean, if true then http will be used to communicate instead of https"
  }
  

  Public key is optional, but needed to encrypt and add the users database in the response, so they can be synced too.

  New keys will be added during certain actions, such as: sessionCookie and last${key}Sync

Using /syncp2pdata

To sync user #1 should perform a get request to /syncp2pdata.

Upon syncing data or having a peer request data from your server there will be new entries in ./data/p2p/thirdPartyPeers.json. In this file you can review the peers, see their contact and host, and you can add them to your peers.json file. thirdPartyPeers.json doesn't contain the public key, you can obtain it (and some further information) by calling the peers /api/selfinfo EP.

/syncp2pdata accepts a few query parameters:

Name	Type	Description
allTime	boolean	If it should ignore the last sync date, and get all data
users	boolean	If it should sync users
questions	boolean	If it should sync questions

If users and questions are all missing, then everything will be synced. The last sync date is stored separately for them.

Maintenance

The server doesn't require that much maintenance, but you are advised to:

• Check the chat messages, the users can contact you there through the website
• Check the received messages on the platform you provided in the p2p information "contact" field
• Check and moderate the forum(s) of the page
• Sync the server with other peers (this can be automated)
• Check ./data/p2p/thirdPartyPeers.json file for new peers, and decide if you should use them
• Regularly check if there is an update to this server and git modules, and update them
• Watch out for directories that can get big:
  • ./stats: server statistics and logs
  • ./data/dbs/backup: backup of databases
  • ./public/backs: backup of question databases
  • ./public/userFiles: files shared by users
  • ./public/savedQuestions: unanswered questions saved by the userscript
• Make regular backups of important data:
  • ./data/dbs: user and messages database
  • ./data/p2p: p2p data, and public/private keys
  • ./public/questionDbs: question dbs
  • ./public/questionDbs.json: information about question db-s
  • ./public/userFiles: files shared by users
  • ./public/forum: forum entries and comments
  • ./public/savedQuestions: unanswered questions saved by the userscript
  • Most files not tracked by git

Server maintenance utils

These scripts can be found in ./src/standaloneUtils.

Name	Description
serverMaintenenceUtils.js	Designed to be a single script to collect all these utils. Not done yet
dbSetup.js	Sets up the database
rmDuplicates.js	Removes duplicates (questions and subjects) from question db-s. Can merge db-s. This is extremely CPU intensive

The other undocumented scripts are rather old, and should be checked if they work, and if they are needed at all

Server usage statistics

The real time log can be found in ./stats/(v)logs. Each folder contains rotated logs, one file per day. In log only the most important events are logged, in vlogs every request is logged.

There are other files in ./stats/, the contents of them is explained in the detailed file structure below.

The script ./scripts/serverStats.js pretty prints server statistics. The first argument should be the server root directory. The second one is optional: a negative number, which shifts the statistics from the current day. For ex.: if its -4, it displays stats from 4 days before now.

./scripts/exportStats.js exports data configured in exportStats.js-s first few line to .csv. The result can be found in the directory it was ran.

Environment variables

For the server

Set these before launching the server (VAR="" npm run start)

Name	Type	Description
PORT	number	The port the http server should run on
DOMAIN	string	The domain that the server should use for redirects, cookies, etc. ex.: qmining.com, without 'https://', and '/'-s. If not specified ./data/domain will be used
NS_NO_HTTPS_FORCE	boolean	Disables automatic redirects from http to https
NS_THREAD_COUNT	number	Nubmer of CPU cores to use
NS_SQL_DEBUG_LOG	boolean	If the SQL queries should be logged
NS_NOUSER	boolean	If the authorization should be skipped (for testing)
NS_NOLOG	boolean	If logging should be skipped

For the setup script (./scripts/setup.sh)

Set these before starting the setup (VAR="" ./scripts/setup.sh)

Name	Type	Description
DOMAIN	string	The domain that the server should use for redirects, cookies, etc. ex.: qmining.com, without 'https://', and '/'-s. If not specified ./data/domain will be used
CLONE_WITH_HTTPS	boolean	By default git clones with ssh, which can cause troubles. Set this to use https

npm scripts

Name	Description
npm run start	Runs server in prod mode. Requires build files in ./dist
npm run dev	Re-builds and runs server in development mode with debugger attached
npm run build	Builds the server with tsc
npm run export	Same as build
npm run test	Runs jest tests
npm run test-debug	Runs jest tests in watch mode with debugger attached

Git modules

More details about the specific git module should be found in its directory: ./submodules/

qmining-page

Next.js frontend for the server. The server serves its static exported html files. The frontend contains many features, including chat using sockets, forum, question database viewer and other pages detailing information about the userscript

https://gitlab.com/MrFry/qmining-page

qmining-data-editor

Next.js frontend designed to give the users the ability, to edit the question dbs. Users can add new questions, edit existing ones, delete invalid questions and many more. The server saves questions and its possible answers during the users are solving online tests. These are saved, and can be viewed in the data editor, and the correct answers can be choosen, and saved in the question databases.

https://gitlab.com/MrFry/qmining-data-editor

moodle-test-userscript

Javasript userscript to be run in tampermonkey or similar userscript handler. It's job is to collect questions during tests, and on test review pages, and send it to the server. During the script also asks the server, if there is a solution saved for the current question, and if yes, it displays it.

https://gitlab.com/MrFry/moodle-test-userscript

Detailed file structure

.
├── data/                          server specific data files not tracked by git
│   ├── admins.json                forum admins. should be removed and should use admin column from user db
│   ├── httpsfiles.json            file including privkeyFile, fullchainFile, chainFile for https functionality
│   ├── apiRootRedirectTo          url where domain/api should redirect to
│   ├── dbs/                       directory for databases, and for their backups
│   ├── domain                     the domain the server is hosted on. Used when `DOMAIN` env var is empty
│   ├── f/                         user files received TODO: check if this is needed
│   ├── links.json                 urls for irc, patreon and donate
│   ├── nolog                      ids of users separated by new lines to ignore on logging
│   ├── p2p/                       p2p data
│   │   ├── key.priv               private key generated on first server run
│   │   ├── key.pub                public key generated on first server run
│   │   ├── peers.json             peers to check on sync
│   │   ├── selfInfo.json          p2p information of this server instance
│   │   └── thirdPartyPeers.json   third party peers encountered
│   ├── statExclude.json           array of strings. If included in url then excluded from stats
│   └── testUsers.json             test users. received data won't get added to question dbs
├── dist/                          build .js files
├── extraSubmodules/               extra submodules not tracked by git
├── nextStatic/                    exported static html files generated by next.js
├── scripts                        scripts for server maintenance
│   ├── exportStats.js             exports statistics to csv
│   ├── postBuild.sh               runs after every build
│   ├── runDocker.sh               runs docker
│   ├── serverStats.js             pretty prints server statistics
│   └── setup.sh                   sets up the server before first run
├── src                            server source files
├── stats                          statistics files
│   ├── askedQuestions             received data on /ask, text file
│   ├── recdata                    received data on /isAdding
│   ├── recievedQuestions          same as recdata, should be removed
│   ├── dailyDataCount             daily data count, text file
│   ├── dataEdits                  data editor activity log
│   ├── logs                       generic logs per day. latest is 'log'
│   ├── vlogs                      detailed logs per day. latest is 'log'
│   ├── qvote/                     quick vote data directory
│   ├── registeredScripts.json     registered scripts JSON
│   ├── stats                      website accesses per path
│   ├── idstats                    test solving statistics per user
│   ├── idvstats                   test solving statistics per day per user
│   ├── ustats                     website accesses count per user
│   ├── uvstats                    website accesses count per day per user
│   └── vstats                     website accessed paths per day
├── submodules                     git modules
│   ├── moodle-test-userscript     userscript
│   ├── qmining-data-editor        data editor frontend
│   └── qmining-page               qmining frontend
├── testingTools                   testing tools for the server
├── defaultPublicFiles             static public files that the frontends use, like images
└── public/                        public directories of the server
    ├── backs/                     question database backups
    ├── chatFiles/                 files sent on chat
    ├── contacts.json              contacts displayed on the /contact page
    ├── forum/                     forum contents
    ├── forumFiles/                files uploaded to forums
    ├── moodle-test-userscript     link to the userscript, it fetches updates from this path
    ├── motd                       motto of the day
    ├── questionDbs/               directory of the question db-s
    ├── questionDbs.json           question db-s information
    ├── savedQuestions/            un-answered questions for dataeditor, saved from test pages
    └── userFiles/                 files shared by users

Related repositories

• Frontend: https://gitlab.com/MrFry/qmining-page
• Dataeditor: https://gitlab.com/MrFry/qmining-data-editor
• Userscript: https://gitlab.com/MrFry/moodle-test-userscript
• Discord bot: https://gitlab.com/MrFry/qmining-discord-bot

License

GNU General Public License, version 3