docs(migrations): Some documentation ++

Migrations allow us to keep track of database changes and upgrading databases if needed.

.woodpecker/lint.yml

@@ -1,6 +1,6 @@
 pipeline:
   lint:
-    image: python:alpine
+    image: python:slim
     commands:
       - pip install black
       - black --check --line-length 120 --target-version=py37 .

.woodpecker/test.yml

@@ -0,0 +1,21 @@
+pipeline:
+  install:
+    image: python:${PYTHON}-slim
+    commands:
+      - python -m venv --clear venv
+      - export PATH=venv/bin:$PATH
+      - python -m pip install --upgrade pip
+      - pip install -v ".[tests]"
+  test:
+    image: python:${PYTHON}-slim
+    commands:
+      - export PATH=venv/bin:$PATH
+      - python -m pytest
+
+
+matrix:
+  PYTHON:
+    - 3.10
+    - 3.9
+    - 3.8
+    - 3.7

README.md

@@ -3,14 +3,17 @@
 
 This is the backend of the Flaschengeist.
 
-## Installation
-### Requirements
-- `mysql` or `mariadb`
-    - maybe `libmariadb` development files[1]
-- python 3.7+
+# Installation
+## Main package
+### System dependencies
+- **python 3.7+**
+- Database (MySQL / mariadb by default)
 
-[1] By default Flaschengeist uses mysql as database backend, if you are on Windows Flaschengeist uses `PyMySQL`, but on
-Linux / Mac the faster `mysqlclient` is used, if it is not already installed installing from pypi requires the
+By default Flaschengeist uses mysql as database backend,
+if you are on Windows Flaschengeist uses `PyMySQL`, which does not require any other system packages.
+
+But on Linux / Mac / *nix the faster `mysqlclient` is used,
+if it is not already installed, installing from PyPi requires the
 development files for `libmariadb` to be present on your system.
 
 ### Install python files
@@ -22,18 +25,34 @@ or if you want to also run the tests:
 
     pip3 install --user ".[ldap,test]"
 
-You will also need a MySQL driver, recommended drivers are
-- `mysqlclient`
-- `PyMySQL`
+You will also need a MySQL driver, by default one of this is installed:
+- `mysqlclient` (non Windows)
+- `PyMySQL` (on Windows)
 
-`setup.py` will try to install a matching driver.
+#### Hint on MySQL driver on Windows:
+If you want to use `mysqlclient` instead of `PyMySQL` (performance?) you have to follow [this guide](https://www.radishlogic.com/coding/python-3/installing-mysqldb-for-python-3-in-windows/)
 
-#### Windows
-Same as above, but if you want to use `mysqlclient` instead of `PyMySQL` (performance?) you have to follow this guide:
+### Install database
+The user needs to have full permissions to the database.
+If not you need to create user and database manually do (or similar on Windows):
 
-https://www.radishlogic.com/coding/python-3/installing-mysqldb-for-python-3-in-windows/
+    (
+        echo "CREATE DATABASE flaschengeist;"
+        echo "CREATE USER 'flaschengeist'@'localhost' IDENTIFIED BY 'flaschengeist';"
+        echo "GRANT ALL PRIVILEGES ON flaschengeist.* TO 'flaschengeist'@'localhost';"
+        echo "FLUSH PRIVILEGES;"
+    ) | sudo mysql
 
-### Configuration
+Then you can install the database tables, this will update all tables from core + all enabled plugins.
+
+    $ flaschengeist db upgrade heads
+
+## Plugins
+To only upgrade one plugin:
+
+    $ flaschengeist db upgrade events@head
+
+## Configuration
 Configuration is done within the a `flaschengeist.toml`file, you can copy the one located inside the module path
 (where flaschegeist is installed) or create an empty one and place it inside either:
 1. `~/.config/`
@@ -54,21 +73,6 @@ So you have to configure one of the following options to call flaschengeists CRO
   - Pros: Guaranteed execution interval, no impact on user experience (at least if you do not limit wsgi worker threads)
   - Cons: Uses one of the webserver threads while executing
 
-### Database installation
-The user needs to have full permissions to the database.
-If not you need to create user and database manually do (or similar on Windows):
-
-    (
-        echo "CREATE DATABASE flaschengeist;"
-        echo "CREATE USER 'flaschengeist'@'localhost' IDENTIFIED BY 'flaschengeist';"
-        echo "GRANT ALL PRIVILEGES ON flaschengeist.* TO 'flaschengeist'@'localhost';"
-        echo "FLUSH PRIVILEGES;"
-    ) | sudo mysql
-
-Then you can install the database tables and initial entries:
-
-    $ flaschengeist install
-
 ### Run
     $ flaschengeist run
 or with debug messages:

flaschengeist/app.py

@@ -101,10 +101,11 @@ def create_app(test_config=None, cli=False):
     CORS(app)
 
     with app.app_context():
-        from flaschengeist.database import db
+        from flaschengeist.database import db, migrate
 
         configure_app(app, test_config, cli)
         db.init_app(app)
+        migrate.init_app(app, db, compare_type=True)
         __load_plugins(app)
 
     @app.route("/", methods=["GET"])

flaschengeist/controller/userController.py

@@ -101,7 +101,7 @@ def set_roles(user: User, roles: list[str], create=False):
     Raises:
         BadRequest if invalid arguments given or not all roles found while *create* is set to false
     """
-    from roleController import create_role
+    from .roleController import create_role
 
     if not isinstance(roles, list) and any([not isinstance(r, str) for r in roles]):
         raise BadRequest("Invalid role name")
@@ -149,7 +149,7 @@ def get_user_by_role(role: Role):
     return User.query.join(User.roles_).filter_by(role_id=role.id).all()
 
 
-def get_user(uid, deleted=False):
+def get_user(uid, deleted=False) -> User:
     """Get an user by userid from database
     Args:
         uid: Userid to search for

flaschengeist/database.py

@@ -1,3 +1,6 @@
+import os
+from flask import current_app
+from flask_migrate import Migrate
 from flask_sqlalchemy import SQLAlchemy
 from sqlalchemy import MetaData
 
@@ -14,6 +17,26 @@ metadata = MetaData(
 
 
 db = SQLAlchemy(metadata=metadata)
+migrate = Migrate()
+
+
+@migrate.configure
+def configure_alembic(config):
+    # Load migration paths from plugins
+    migrations = [str(p.migrations_path) for p in current_app.config["FG_PLUGINS"].values() if p and p.migrations_path]
+    if len(migrations) > 0:
+        # Get configured paths
+        paths = config.get_main_option("version_locations")
+        # Get configured path seperator
+        sep = config.get_main_option("version_path_separator", "os")
+        if paths:
+            # Insert configured paths at the front, before plugin migrations
+            migrations.insert(0, config.get_main_option("version_locations"))
+        sep = os.pathsep if sep == "os" else " " if sep == "space" else sep
+        # write back seperator (we changed it if neither seperator nor locations were specified)
+        config.set_main_option("version_path_separator", sep)
+        config.set_main_option("version_locations", sep.join(migrations))
+    return config
 
 
 def case_sensitive(s):

flaschengeist/models/__init__.py

@@ -1,7 +1,7 @@
 import sys
 import datetime
 
-from sqlalchemy import BigInteger
+from sqlalchemy import BigInteger, util
 from sqlalchemy.dialects import mysql, sqlite
 from sqlalchemy.types import DateTime, TypeDecorator
 
@@ -48,7 +48,11 @@ class Serial(TypeDecorator):
     """Same as MariaDB Serial used for IDs"""
 
     cache_ok = True
-    impl = BigInteger().with_variant(mysql.BIGINT(unsigned=True), "mysql").with_variant(sqlite.INTEGER, "sqlite")
+    impl = BigInteger().with_variant(mysql.BIGINT(unsigned=True), "mysql").with_variant(sqlite.INTEGER(), "sqlite")
+
+    # https://alembic.sqlalchemy.org/en/latest/autogenerate.html?highlight=custom%20column#affecting-the-rendering-of-types-themselves
+    def __repr__(self) -> str:
+        return util.generic_repr(self)
 
 
 class UtcDateTime(TypeDecorator):
@@ -85,3 +89,7 @@ class UtcDateTime(TypeDecorator):
                 value = value.astimezone(datetime.timezone.utc)
             value = value.replace(tzinfo=datetime.timezone.utc)
         return value
+
+    # https://alembic.sqlalchemy.org/en/latest/autogenerate.html?highlight=custom%20column#affecting-the-rendering-of-types-themselves
+    def __repr__(self) -> str:
+        return util.generic_repr(self)

flaschengeist/plugins/balance/__init__.py

@@ -3,6 +3,7 @@
 Extends users plugin with balance functions
 """
 
+import pathlib
 from flask import Blueprint, current_app
 from werkzeug.local import LocalProxy
 from werkzeug.exceptions import NotFound
@@ -67,6 +68,8 @@ class BalancePlugin(Plugin):
         super(BalancePlugin, self).__init__(config)
         from . import routes
 
+        self.migrations_path = (pathlib.Path(__file__).parent / "migrations").resolve()
+
         @plugins_loaded
         def post_loaded(*args, **kwargs):
             if config.get("allow_service_debit", False) and "events" in current_app.config["FG_PLUGINS"]:

flaschengeist/plugins/balance/migrations/f07df84f7a95_initial_balance_migration.py

@@ -0,0 +1,47 @@
+"""Initial balance migration
+
+Revision ID: f07df84f7a95
+Revises: 
+Create Date: 2021-12-19 21:12:53.192267
+
+"""
+from alembic import op
+import sqlalchemy as sa
+import flaschengeist
+
+
+# revision identifiers, used by Alembic.
+revision = "f07df84f7a95"
+down_revision = None
+branch_labels = ("balance",)
+depends_on = "d3026757c7cb"
+
+
+def upgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.create_table(
+        "balance_transaction",
+        sa.Column("receiver_id", flaschengeist.models.Serial(), nullable=True),
+        sa.Column("sender_id", flaschengeist.models.Serial(), nullable=True),
+        sa.Column("author_id", flaschengeist.models.Serial(), nullable=False),
+        sa.Column("id", flaschengeist.models.Serial(), nullable=False),
+        sa.Column("time", flaschengeist.models.UtcDateTime(), nullable=False),
+        sa.Column("amount", sa.Numeric(precision=5, scale=2, asdecimal=False), nullable=False),
+        sa.Column("reversal_id", flaschengeist.models.Serial(), nullable=True),
+        sa.ForeignKeyConstraint(["author_id"], ["user.id"], name=op.f("fk_balance_transaction_author_id_user")),
+        sa.ForeignKeyConstraint(["receiver_id"], ["user.id"], name=op.f("fk_balance_transaction_receiver_id_user")),
+        sa.ForeignKeyConstraint(
+            ["reversal_id"],
+            ["balance_transaction.id"],
+            name=op.f("fk_balance_transaction_reversal_id_balance_transaction"),
+        ),
+        sa.ForeignKeyConstraint(["sender_id"], ["user.id"], name=op.f("fk_balance_transaction_sender_id_user")),
+        sa.PrimaryKeyConstraint("id", name=op.f("pk_balance_transaction")),
+    )
+    # ### end Alembic commands ###
+
+
+def downgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_table("balance_transaction")
+    # ### end Alembic commands ###

flaschengeist/plugins/pricelist/__init__.py

@@ -1,5 +1,6 @@
 """Pricelist plugin"""
 
+import pathlib
 from flask import Blueprint, jsonify, request, current_app
 from werkzeug.local import LocalProxy
 from werkzeug.exceptions import BadRequest, Forbidden, NotFound, Unauthorized
@@ -24,6 +25,7 @@ class PriceListPlugin(Plugin):
 
     def __init__(self, cfg):
         super().__init__(cfg)
+        self.migrations_path = (pathlib.Path(__file__).parent / "migrations").resolve()
         config = {"discount": 0}
         config.update(cfg)
 

flaschengeist/plugins/pricelist/migrations/7d9d306be676_initial_pricelist_migration.py

@@ -0,0 +1,141 @@
+"""Initial pricelist migration
+
+Revision ID: 7d9d306be676
+Revises: 
+Create Date: 2021-12-19 21:43:30.203811
+
+"""
+from alembic import op
+import sqlalchemy as sa
+import flaschengeist
+
+
+# revision identifiers, used by Alembic.
+revision = "7d9d306be676"
+down_revision = None
+branch_labels = ("pricelist",)
+depends_on = "d3026757c7cb"
+
+
+def upgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.create_table(
+        "drink_extra_ingredient",
+        sa.Column("id", flaschengeist.models.Serial(), nullable=False),
+        sa.Column("name", sa.String(length=30), nullable=False),
+        sa.Column("price", sa.Numeric(precision=5, scale=2, asdecimal=False), nullable=True),
+        sa.PrimaryKeyConstraint("id", name=op.f("pk_drink_extra_ingredient")),
+        sa.UniqueConstraint("name", name=op.f("uq_drink_extra_ingredient_name")),
+    )
+    op.create_table(
+        "drink_tag",
+        sa.Column("id", flaschengeist.models.Serial(), nullable=False),
+        sa.Column("name", sa.String(length=30), nullable=False),
+        sa.Column("color", sa.String(length=7), nullable=False),
+        sa.PrimaryKeyConstraint("id", name=op.f("pk_drink_tag")),
+        sa.UniqueConstraint("name", name=op.f("uq_drink_tag_name")),
+    )
+    op.create_table(
+        "drink_type",
+        sa.Column("id", flaschengeist.models.Serial(), nullable=False),
+        sa.Column("name", sa.String(length=30), nullable=False),
+        sa.PrimaryKeyConstraint("id", name=op.f("pk_drink_type")),
+        sa.UniqueConstraint("name", name=op.f("uq_drink_type_name")),
+    )
+    op.create_table(
+        "drink",
+        sa.Column("id", flaschengeist.models.Serial(), nullable=False),
+        sa.Column("article_id", sa.String(length=64), nullable=True),
+        sa.Column("package_size", sa.Integer(), nullable=True),
+        sa.Column("name", sa.String(length=60), nullable=False),
+        sa.Column("volume", sa.Numeric(precision=5, scale=2, asdecimal=False), nullable=True),
+        sa.Column("cost_per_volume", sa.Numeric(precision=5, scale=3, asdecimal=False), nullable=True),
+        sa.Column("cost_per_package", sa.Numeric(precision=5, scale=3, asdecimal=False), nullable=True),
+        sa.Column("receipt", sa.PickleType(), nullable=True),
+        sa.Column("type_id", flaschengeist.models.Serial(), nullable=True),
+        sa.Column("image_id", flaschengeist.models.Serial(), nullable=True),
+        sa.ForeignKeyConstraint(["image_id"], ["image.id"], name=op.f("fk_drink_image_id_image")),
+        sa.ForeignKeyConstraint(["type_id"], ["drink_type.id"], name=op.f("fk_drink_type_id_drink_type")),
+        sa.PrimaryKeyConstraint("id", name=op.f("pk_drink")),
+    )
+    op.create_table(
+        "drink_ingredient",
+        sa.Column("id", flaschengeist.models.Serial(), nullable=False),
+        sa.Column("volume", sa.Numeric(precision=5, scale=2, asdecimal=False), nullable=False),
+        sa.Column("ingredient_id", flaschengeist.models.Serial(), nullable=True),
+        sa.ForeignKeyConstraint(["ingredient_id"], ["drink.id"], name=op.f("fk_drink_ingredient_ingredient_id_drink")),
+        sa.PrimaryKeyConstraint("id", name=op.f("pk_drink_ingredient")),
+    )
+    op.create_table(
+        "drink_price_volume",
+        sa.Column("id", flaschengeist.models.Serial(), nullable=False),
+        sa.Column("drink_id", flaschengeist.models.Serial(), nullable=True),
+        sa.Column("volume", sa.Numeric(precision=5, scale=2, asdecimal=False), nullable=True),
+        sa.ForeignKeyConstraint(["drink_id"], ["drink.id"], name=op.f("fk_drink_price_volume_drink_id_drink")),
+        sa.PrimaryKeyConstraint("id", name=op.f("pk_drink_price_volume")),
+    )
+    op.create_table(
+        "drink_x_tag",
+        sa.Column("drink_id", flaschengeist.models.Serial(), nullable=True),
+        sa.Column("tag_id", flaschengeist.models.Serial(), nullable=True),
+        sa.ForeignKeyConstraint(["drink_id"], ["drink.id"], name=op.f("fk_drink_x_tag_drink_id_drink")),
+        sa.ForeignKeyConstraint(["tag_id"], ["drink_tag.id"], name=op.f("fk_drink_x_tag_tag_id_drink_tag")),
+    )
+    op.create_table(
+        "drink_x_type",
+        sa.Column("drink_id", flaschengeist.models.Serial(), nullable=True),
+        sa.Column("type_id", flaschengeist.models.Serial(), nullable=True),
+        sa.ForeignKeyConstraint(["drink_id"], ["drink.id"], name=op.f("fk_drink_x_type_drink_id_drink")),
+        sa.ForeignKeyConstraint(["type_id"], ["drink_type.id"], name=op.f("fk_drink_x_type_type_id_drink_type")),
+    )
+    op.create_table(
+        "drink_ingredient_association",
+        sa.Column("id", flaschengeist.models.Serial(), nullable=False),
+        sa.Column("volume_id", flaschengeist.models.Serial(), nullable=True),
+        sa.Column("_drink_ingredient_id", flaschengeist.models.Serial(), nullable=True),
+        sa.Column("_extra_ingredient_id", flaschengeist.models.Serial(), nullable=True),
+        sa.ForeignKeyConstraint(
+            ["_drink_ingredient_id"],
+            ["drink_ingredient.id"],
+            name=op.f("fk_drink_ingredient_association__drink_ingredient_id_drink_ingredient"),
+        ),
+        sa.ForeignKeyConstraint(
+            ["_extra_ingredient_id"],
+            ["drink_extra_ingredient.id"],
+            name=op.f("fk_drink_ingredient_association__extra_ingredient_id_drink_extra_ingredient"),
+        ),
+        sa.ForeignKeyConstraint(
+            ["volume_id"],
+            ["drink_price_volume.id"],
+            name=op.f("fk_drink_ingredient_association_volume_id_drink_price_volume"),
+        ),
+        sa.PrimaryKeyConstraint("id", name=op.f("pk_drink_ingredient_association")),
+    )
+    op.create_table(
+        "drink_price",
+        sa.Column("id", flaschengeist.models.Serial(), nullable=False),
+        sa.Column("price", sa.Numeric(precision=5, scale=2, asdecimal=False), nullable=True),
+        sa.Column("volume_id", flaschengeist.models.Serial(), nullable=True),
+        sa.Column("public", sa.Boolean(), nullable=True),
+        sa.Column("description", sa.String(length=30), nullable=True),
+        sa.ForeignKeyConstraint(
+            ["volume_id"], ["drink_price_volume.id"], name=op.f("fk_drink_price_volume_id_drink_price_volume")
+        ),
+        sa.PrimaryKeyConstraint("id", name=op.f("pk_drink_price")),
+    )
+    # ### end Alembic commands ###
+
+
+def downgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_table("drink_price")
+    op.drop_table("drink_ingredient_association")
+    op.drop_table("drink_x_type")
+    op.drop_table("drink_x_tag")
+    op.drop_table("drink_price_volume")
+    op.drop_table("drink_ingredient")
+    op.drop_table("drink")
+    op.drop_table("drink_type")
+    op.drop_table("drink_tag")
+    op.drop_table("drink_extra_ingredient")
+    # ### end Alembic commands ###

migrations/alembic.ini

@@ -0,0 +1,52 @@
+# A generic, single database configuration.
+
+[alembic]
+# template used to generate migration files
+# file_template = %%(rev)s_%%(slug)s
+
+# set to 'true' to run the environment during
+# the 'revision' command, regardless of autogenerate
+# revision_environment = false
+
+version_path_separator = os
+version_locations = %(here)s/versions
+
+# Logging configuration
+[loggers]
+keys = root,sqlalchemy,alembic,flask_migrate
+
+[handlers]
+keys = console
+
+[formatters]
+keys = generic
+
+[logger_root]
+level = WARN
+handlers = console
+qualname =
+
+[logger_sqlalchemy]
+level = WARN
+handlers =
+qualname = sqlalchemy.engine
+
+[logger_alembic]
+level = INFO
+handlers =
+qualname = alembic
+
+[logger_flask_migrate]
+level = INFO
+handlers =
+qualname = flask_migrate
+
+[handler_console]
+class = StreamHandler
+args = (sys.stderr,)
+level = NOTSET
+formatter = generic
+
+[formatter_generic]
+format = %(levelname)-5.5s [%(name)s] %(message)s
+datefmt = %H:%M:%S

migrations/env.py

@@ -0,0 +1,73 @@
+import logging
+from logging.config import fileConfig
+from flask import current_app
+from alembic import context
+
+# this is the Alembic Config object, which provides
+# access to the values within the .ini file in use.
+config = context.config
+
+# Interpret the config file for Python logging.
+# This line sets up loggers basically.
+fileConfig(config.config_file_name)
+logger = logging.getLogger("alembic.env")
+
+config.set_main_option("sqlalchemy.url", str(current_app.extensions["migrate"].db.get_engine().url).replace("%", "%%"))
+target_metadata = current_app.extensions["migrate"].db.metadata
+
+
+def run_migrations_offline():
+    """Run migrations in 'offline' mode.
+
+    This configures the context with just a URL
+    and not an Engine, though an Engine is acceptable
+    here as well.  By skipping the Engine creation
+    we don't even need a DBAPI to be available.
+
+    Calls to context.execute() here emit the given string to the
+    script output.
+
+    """
+    url = config.get_main_option("sqlalchemy.url")
+    context.configure(url=url, target_metadata=target_metadata, literal_binds=True)
+
+    with context.begin_transaction():
+        context.run_migrations()
+
+
+def run_migrations_online():
+    """Run migrations in 'online' mode.
+
+    In this scenario we need to create an Engine
+    and associate a connection with the context.
+
+    """
+
+    # this callback is used to prevent an auto-migration from being generated
+    # when there are no changes to the schema
+    # reference: http://alembic.zzzcomputing.com/en/latest/cookbook.html
+    def process_revision_directives(context, revision, directives):
+        if getattr(config.cmd_opts, "autogenerate", False):
+            script = directives[0]
+            if script.upgrade_ops.is_empty():
+                directives[:] = []
+                logger.info("No changes in schema detected.")
+
+    connectable = current_app.extensions["migrate"].db.get_engine()
+
+    with connectable.connect() as connection:
+        context.configure(
+            connection=connection,
+            target_metadata=target_metadata,
+            process_revision_directives=process_revision_directives,
+            **current_app.extensions["migrate"].configure_args
+        )
+
+        with context.begin_transaction():
+            context.run_migrations()
+
+
+if context.is_offline_mode():
+    run_migrations_offline()
+else:
+    run_migrations_online()

migrations/script.py.mako

@@ -0,0 +1,25 @@
+"""${message}
+
+Revision ID: ${up_revision}
+Revises: ${down_revision | comma,n}
+Create Date: ${create_date}
+
+"""
+from alembic import op
+import sqlalchemy as sa
+import flaschengeist
+${imports if imports else ""}
+
+# revision identifiers, used by Alembic.
+revision = ${repr(up_revision)}
+down_revision = ${repr(down_revision)}
+branch_labels = ${repr(branch_labels)}
+depends_on = ${repr(depends_on)}
+
+
+def upgrade():
+    ${upgrades if upgrades else "pass"}
+
+
+def downgrade():
+    ${downgrades if downgrades else "pass"}

migrations/versions/d3026757c7cb_initial_migration.py

@@ -0,0 +1,141 @@
+"""Initial migration.
+
+Revision ID: d3026757c7cb
+Revises: 
+Create Date: 2021-12-19 20:34:34.122576
+
+"""
+from alembic import op
+import sqlalchemy as sa
+import flaschengeist
+
+
+# revision identifiers, used by Alembic.
+revision = "d3026757c7cb"
+down_revision = None
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.create_table(
+        "image",
+        sa.Column("id", flaschengeist.models.Serial(), nullable=False),
+        sa.Column("filename_", sa.String(length=127), nullable=False),
+        sa.Column("mimetype_", sa.String(length=30), nullable=False),
+        sa.Column("thumbnail_", sa.String(length=127), nullable=True),
+        sa.Column("path_", sa.String(length=127), nullable=True),
+        sa.PrimaryKeyConstraint("id", name=op.f("pk_image")),
+    )
+    op.create_table(
+        "permission",
+        sa.Column("name", sa.String(length=30), nullable=True),
+        sa.Column("id", flaschengeist.models.Serial(), nullable=False),
+        sa.PrimaryKeyConstraint("id", name=op.f("pk_permission")),
+        sa.UniqueConstraint("name", name=op.f("uq_permission_name")),
+    )
+    op.create_table(
+        "plugin_setting",
+        sa.Column("id", flaschengeist.models.Serial(), nullable=False),
+        sa.Column("plugin", sa.String(length=30), nullable=True),
+        sa.Column("name", sa.String(length=30), nullable=False),
+        sa.Column("value", sa.PickleType(), nullable=True),
+        sa.PrimaryKeyConstraint("id", name=op.f("pk_plugin_setting")),
+    )
+    op.create_table(
+        "role",
+        sa.Column("id", flaschengeist.models.Serial(), nullable=False),
+        sa.Column("name", sa.String(length=30), nullable=True),
+        sa.PrimaryKeyConstraint("id", name=op.f("pk_role")),
+        sa.UniqueConstraint("name", name=op.f("uq_role_name")),
+    )
+    op.create_table(
+        "role_x_permission",
+        sa.Column("role_id", flaschengeist.models.Serial(), nullable=True),
+        sa.Column("permission_id", flaschengeist.models.Serial(), nullable=True),
+        sa.ForeignKeyConstraint(
+            ["permission_id"], ["permission.id"], name=op.f("fk_role_x_permission_permission_id_permission")
+        ),
+        sa.ForeignKeyConstraint(["role_id"], ["role.id"], name=op.f("fk_role_x_permission_role_id_role")),
+    )
+    op.create_table(
+        "user",
+        sa.Column("userid", sa.String(length=30), nullable=False),
+        sa.Column("display_name", sa.String(length=30), nullable=True),
+        sa.Column("firstname", sa.String(length=50), nullable=False),
+        sa.Column("lastname", sa.String(length=50), nullable=False),
+        sa.Column("deleted", sa.Boolean(), nullable=True),
+        sa.Column("birthday", sa.Date(), nullable=True),
+        sa.Column("mail", sa.String(length=60), nullable=True),
+        sa.Column("id", flaschengeist.models.Serial(), nullable=False),
+        sa.Column("avatar", flaschengeist.models.Serial(), nullable=True),
+        sa.ForeignKeyConstraint(["avatar"], ["image.id"], name=op.f("fk_user_avatar_image")),
+        sa.PrimaryKeyConstraint("id", name=op.f("pk_user")),
+        sa.UniqueConstraint("userid", name=op.f("uq_user_userid")),
+    )
+    op.create_table(
+        "notification",
+        sa.Column("id", flaschengeist.models.Serial(), nullable=False),
+        sa.Column("plugin", sa.String(length=127), nullable=False),
+        sa.Column("text", sa.Text(), nullable=True),
+        sa.Column("data", sa.PickleType(), nullable=True),
+        sa.Column("time", flaschengeist.models.UtcDateTime(), nullable=False),
+        sa.Column("user_id", flaschengeist.models.Serial(), nullable=False),
+        sa.ForeignKeyConstraint(["user_id"], ["user.id"], name=op.f("fk_notification_user_id_user")),
+        sa.PrimaryKeyConstraint("id", name=op.f("pk_notification")),
+    )
+    op.create_table(
+        "password_reset",
+        sa.Column("user", flaschengeist.models.Serial(), nullable=False),
+        sa.Column("token", sa.String(length=32), nullable=True),
+        sa.Column("expires", flaschengeist.models.UtcDateTime(), nullable=True),
+        sa.ForeignKeyConstraint(["user"], ["user.id"], name=op.f("fk_password_reset_user_user")),
+        sa.PrimaryKeyConstraint("user", name=op.f("pk_password_reset")),
+    )
+    op.create_table(
+        "session",
+        sa.Column("expires", flaschengeist.models.UtcDateTime(), nullable=True),
+        sa.Column("token", sa.String(length=32), nullable=True),
+        sa.Column("lifetime", sa.Integer(), nullable=True),
+        sa.Column("browser", sa.String(length=30), nullable=True),
+        sa.Column("platform", sa.String(length=30), nullable=True),
+        sa.Column("id", flaschengeist.models.Serial(), nullable=False),
+        sa.Column("user_id", flaschengeist.models.Serial(), nullable=True),
+        sa.ForeignKeyConstraint(["user_id"], ["user.id"], name=op.f("fk_session_user_id_user")),
+        sa.PrimaryKeyConstraint("id", name=op.f("pk_session")),
+        sa.UniqueConstraint("token", name=op.f("uq_session_token")),
+    )
+    op.create_table(
+        "user_attribute",
+        sa.Column("id", flaschengeist.models.Serial(), nullable=False),
+        sa.Column("user", flaschengeist.models.Serial(), nullable=False),
+        sa.Column("name", sa.String(length=30), nullable=True),
+        sa.Column("value", sa.PickleType(), nullable=True),
+        sa.ForeignKeyConstraint(["user"], ["user.id"], name=op.f("fk_user_attribute_user_user")),
+        sa.PrimaryKeyConstraint("id", name=op.f("pk_user_attribute")),
+    )
+    op.create_table(
+        "user_x_role",
+        sa.Column("user_id", flaschengeist.models.Serial(), nullable=True),
+        sa.Column("role_id", flaschengeist.models.Serial(), nullable=True),
+        sa.ForeignKeyConstraint(["role_id"], ["role.id"], name=op.f("fk_user_x_role_role_id_role")),
+        sa.ForeignKeyConstraint(["user_id"], ["user.id"], name=op.f("fk_user_x_role_user_id_user")),
+    )
+    # ### end Alembic commands ###
+
+
+def downgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_table("user_x_role")
+    op.drop_table("user_attribute")
+    op.drop_table("session")
+    op.drop_table("password_reset")
+    op.drop_table("notification")
+    op.drop_table("user")
+    op.drop_table("role_x_permission")
+    op.drop_table("role")
+    op.drop_table("plugin_setting")
+    op.drop_table("permission")
+    op.drop_table("image")
+    # ### end Alembic commands ###

setup.cfg

@@ -23,19 +23,21 @@ python_requires = >=3.7
 packages = find:
 install_requires =
     Flask >= 2.0
-    Pillow>=8.4.0
-    flask_cors
-    flask_sqlalchemy>=2.5
-    sqlalchemy>=1.4.26
+    Flask-Cors >= 3.0
+    Flask-Migrate >= 3.1.0 
+    Flask-SQLAlchemy >= 2.5
+    Pillow >= 8.4.0
+    SQLAlchemy >= 1.4.28
     toml
-    werkzeug
-    PyMySQL;platform_system=='Windows'
-    mysqlclient;platform_system!='Windows'
+    werkzeug >= 2.0
 
 [options.extras_require]
 argon = argon2-cffi
 ldap = flask_ldapconn; ldap3
-test = pytest; coverage
+tests = pytest; pytest-depends; coverage
+mysql =
+    PyMySQL;platform_system=='Windows'
+    mysqlclient;platform_system!='Windows'
 
 [options.package_data]
 * = *.toml

tests/conftest.py

@@ -3,8 +3,7 @@ import tempfile
 import pytest
 
 from flaschengeist import database
-from flaschengeist.app import create_app, install_all
-
+from flaschengeist.app import create_app
 
 # read in SQL for populating test data
 with open(os.path.join(os.path.dirname(__file__), "data.sql"), "r") as f:
@@ -25,12 +24,14 @@ def app():
     app = create_app(
         {
             "TESTING": True,
-            "DATABASE": {"file_path": f"/{db_path}"},
+            "DATABASE": {"engine": "sqlite", "database": f"/{db_path}"},
             "LOGGING": {"level": "DEBUG"},
         }
     )
     with app.app_context():
-        install_all()
+        database.db.create_all()
+        database.db.session.commit()
+
         engine = database.db.engine
         with engine.connect() as connection:
             for statement in _data_sql:

tests/data.sql

@@ -1,4 +1,8 @@
-INSERT INTO user ('userid', 'firstname', 'lastname', 'mail', 'id') VALUES ('user', 'Max', 'Mustermann', 'abc@def.gh', 1);
+INSERT INTO "user" ('userid', 'firstname', 'lastname', 'mail', 'deleted', 'id') VALUES ('user', 'Max', 'Mustermann', 'abc@def.gh', 0, 1);
+INSERT INTO "user" ('userid', 'firstname', 'lastname', 'mail', 'deleted', 'id') VALUES ('deleted_user', 'John', 'Doe', 'doe@example.com', 1, 2);
 -- Password = 1234
 INSERT INTO user_attribute VALUES(1,1,'password',X'800495c4000000000000008cc0373731346161336536623932613830366664353038656631323932623134393936393561386463353536623037363761323037623238346264623833313265323333373066376233663462643332666332653766303537333564366335393133366463366234356539633865613835643661643435343931376636626663343163653333643635646530386634396231323061316236386162613164373663663333306564306463303737303733336136353363393538396536343266393865942e');
 INSERT INTO session ('expires', 'token', 'lifetime', 'id', 'user_id') VALUES ('2999-01-01 00:00:00', 'f4ecbe14be3527ca998143a49200e294', 600, 1, 1);
+-- ROLES
+INSERT INTO role ('name', 'id') VALUES ('role_1', 1);
+INSERT INTO permission ('name', 'id') VALUES ('permission_1', 1);

tests/test_auth.py

@@ -15,9 +15,9 @@ def test_login(client):
     # Login successful
     assert result.status_code == 201
     # User set correctly
-    assert json["user"]["userid"] == USERID
+    assert json["userid"] == USERID
     # Token works
-    assert client.get("/auth", headers={"Authorization": f"Bearer {json['session']['token']}"}).status_code == 200
+    assert client.get("/auth", headers={"Authorization": f"Bearer {json['token']}"}).status_code == 200
 
 
 def test_login_decorator(client):

tests/test_events.py

@@ -1,17 +0,0 @@
-import pytest
-from werkzeug.exceptions import BadRequest
-
-import flaschengeist.plugins.events.event_controller as event_controller
-from flaschengeist.plugins.events.models import EventType
-
-VALID_TOKEN = "f4ecbe14be3527ca998143a49200e294"
-EVENT_TYPE_NAME = "Test Type"
-
-
-def test_create_event_type(app):
-    with app.app_context():
-        type = event_controller.create_event_type(EVENT_TYPE_NAME)
-        assert isinstance(type, EventType)
-
-        with pytest.raises(BadRequest):
-            event_controller.create_event_type(EVENT_TYPE_NAME)

tests/test_users.py

@@ -0,0 +1,52 @@
+import pytest
+from werkzeug.exceptions import BadRequest, NotFound
+from flaschengeist.controller import roleController, userController
+from flaschengeist.models.user import User
+
+VALID_TOKEN = "f4ecbe14be3527ca998143a49200e294"
+
+
+def test_get_user(app):
+    with app.app_context():
+        user = userController.get_user("user")
+        assert user is not None and isinstance(user, User)
+        assert user.userid == "user"
+
+        user = userController.get_user("deleted_user", deleted=True)
+        assert user is not None and isinstance(user, User)
+        assert user.userid == "deleted_user"
+
+        with pytest.raises(NotFound):
+            user = userController.get_user("__does_not_exist__")
+        with pytest.raises(NotFound):
+            user = userController.get_user("__does_not_exist__", deleted=True)
+        with pytest.raises(NotFound):
+            user = userController.get_user("deleted_user")
+
+
+def test_set_roles(app):
+    with app.app_context():
+        user = userController.get_user("user")
+        userController.set_roles(user, [])
+        assert user.roles_ == []
+
+        userController.set_roles(user, ["role_1"])
+        assert len(user.roles_) == 1 and user.roles_[0].id == 1
+
+        # Test unknown role + no create flag -> raise no changes
+        with pytest.raises(BadRequest):
+            userController.set_roles(user, ["__custom__"])
+        assert len(user.roles_) == 1
+
+        userController.set_roles(user, ["__custom__"], create=True)
+        assert len(user.roles_) == 1 and user.roles_[0].name == "__custom__"
+        assert roleController.get("__custom__").id == user.roles_[0].id
+
+        userController.set_roles(user, ["__custom__"], create=True)
+        assert len(user.roles_) == 1
+
+        userController.set_roles(user, ["__custom__", "role_1"])
+        assert len(user.roles_) == 2
+
+        userController.set_roles(user, [])
+        assert len(user.roles_) == 0