Configuration manual

Introduction

Though default configuration is suitable for many common cases, vstutils-based applications is highly configurable system. For advanced settings (scalability, dedicated DB, custom cache, logging or directories) you can configure vstutils-based application deeply by tweaking /etc/{{app_name or app_lib_name}}/settings.ini.

The most important thing to keep in mind when planning your application architecture is that vstutils-based applications have a service-oriented structure. To build a distributed scalable system you only need to connect to a shared database, shared cache, locks and a shared rpc service (MQ such as RabbitMQ, Redis, etc.). A shared file storage may be required in some cases, a but vstutils does not require it.

Let’s cover the main sections of the config and its parameters:

Main settings

Section [main].

This section is intended for settings related to whole vstutils-based application (both worker and web). Here you can specify verbosity level of vstutils-based application during work, which can be useful for troubleshooting (logging level etc). Also there are settings for changing timezone for whole app and allowed domains.

To use LDAP protocol, create following settings in section [main].

ldap-server = ldap://server-ip-or-host:port
ldap-default-domain = domain.name
ldap-auth_format = cn=<username>,ou=your-group-name,<domain>

ldap-default-domain is an optional argument, that is aimed to make user authorization easier (without input of domain name).

ldap-auth_format is an optional argument, that is aimed to customize LDAP authorization. Default value: cn=<username>,<domain>

In the example above authorization logic will be the following:

  1. System checks combination of login:password in database;

  2. System checks combination of login:password in LDAP:

    • if domain was mentioned, it will be set during authorization (if user enter login without user@domain.name or without DOMAIN\user );

    • if authorization was successful and there is user with entered credentials in database, server creates session that user.

  • debug - Enable debug mode. Default: false.

  • allowed_hosts - Comma separated list of domains, which allowed to serve. Default: *.

  • first_day_of_week - Integer value with first day of week. Default: 0.

  • ldap-server - LDAP server connection.

  • ldap-default-domain - Default domain for auth.

  • ldap-auth_format - Default search request format for auth. Default: cn=<username>,<domain>.

  • timezone - Timezone for web-application. Default: UTC.

  • log_level - Logging level. Default: WARNING.

  • enable_django_logs - Enable or disable Django logger to output. Useful for debugging. Default: false.

  • enable_admin_panel - Enable or disable Django Admin panel. Default: false.

  • enable_registration - Enable or disable user self-registration. Default: false.

  • enable_user_self_remove - Enable or disable user self-removing. Default: false.

  • auth-plugins - Comma separated list of django authentication backends. Authorization attempt is made until the first successful one in order specified in the list.

  • auth-cache-user - Enable or disable user instance caching. It increases session performance on each request but saves model instance in unsafe storage (default django cache). The instance is serialized to a string using the standard python module pickle and then encrypted with Vigenère cipher. Read more in the vstutils.utils.SecurePickling documentation. Default: false.

Databases settings

Section [databases].

The main section that is designed to manage multiple databases connected to the project.

These settings are for all databases and are vendor-independent, with the exception of tablespace management.

  • default_tablespace - Default tablespace to use for models that don’t specify one, if the backend supports it.

    Read more at Declaring tablespaces for tables.

  • default_index_tablespace - Default tablespace to use for indexes on fields that don’t specify one, if the backend supports it.

    Read more at Declaring tablespaces for indexes.

  • databases_without_cte_support - A comma-separated list of database section names that do not support CTEs (Common Table Experssions).

Warning

Although MariaDB supports Common Table Expressions, but database connected to MariaDB still needs to be added to databases_without_cte_support list. The problem is that the implementation of recursive queries in the MariaDB does not allow using it in a standard form. MySQL (since 8.0) works as expected.

Also, all subsections of this section are available connections to the DBMS. So the databases.default section will be used by django as the default connection.

Here you can change settings related to database, which vstutils-based application will use. vstutils-based application supports all databases supported by django. List of supported out of the box: SQLite (default choice), MySQL, Oracle, or PostgreSQL. Configuration details available at Django database documentation. To run vstutils-based application at multiple nodes (cluster), use client-server database (SQLite not suitable) shared for all nodes.

You can also set the base template for connecting to the database in the database section.

Section [database].

This section is designed to define the basic template for connections to various databases. This can be useful to reduce the list of settings in the databases.* subsections by setting the same connection for a different set of databases in the project. For more details read the django docs about Multiple databases

There is a list of settings, required for MySQL/MariaDB database.

Firstly, if you use MySQL/MariaDB and you have set timezone different from “UTC” you should run command below:

mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root -p mysql

Secondly, to use MySQL/MariaDB set following options in settings.ini file:

[database.options]
connect_timeout = 10
init_command = SET sql_mode='STRICT_TRANS_TABLES', default_storage_engine=INNODB, NAMES 'utf8', CHARACTER SET 'utf8', SESSION collation_connection = 'utf8_unicode_ci'

Finally, add some options to MySQL/MariaDB configuration:

[client]
default-character-set=utf8
init_command = SET collation_connection = @@collation_database

[mysqld]
character-set-server=utf8
collation-server=utf8_unicode_ci

Cache settings

Section [cache].

This section is cache backend related settings used by vstutils-based application. vstutils supports all cache backends that Django does. Filesystem, in-memory, memcached are supported out of the box and many others are supported with additional plugins. You can find details about cache configusupported Django caches documentation. In clusters we advice to share cache between nodes to improve performance using client-server cache realizations. We recommend to use Redis in production environments.

Locks settings

Section [locks].

Locks is a system that vstutils-based application uses to avoid damage from parallel actions working on the same entity simultaneously. It is based on Django cache, so there is another bunch of same settings as cache. And why there is another section for them, you may ask. Because cache backend is used for locking must provide some guarantees, which do not required to usual cache: it MUST be shared for all vstutils-based application threads and nodes. So, for example, in-memory backend is not suitable. In case of clusterization we strongly recommend to use Redis or Memcached as backend for that purpose. Cache and locks backend can be the same, but don’t forget about requirement we said above.

Session cache settings

Section [session].

vstutils-based application store sessions in database, but for better performance, we use a cache-based session backend. It is based on Django cache, so there is another bunch of same settings as cache. By default, settings are got from cache.

Rpc settings

Section [rpc].

vstutils-based application uses Celery for long-running async tasks. Celery is based on message queue concept, so between web-service and workers running under Celery bust be some kind of message broker (RabbitMQ or something). Those settings relate to this broker and Celery itself. Those kinds of settings: broker backend, number of worker-processes per node and some settings used for troubleshoot server-broker-worker interaction problems.

This section require vstutils with rpc extra dependency.

  • connection - Celery broker connection. Default: filesystem:///var/tmp.

  • concurrency - Count of celery worker threads. Default: 4.

  • heartbeat - Interval between sending heartbeat packages, which says that connection still alive. Default: 10.

  • enable_worker - Enable or disable worker with webserver. Default: true.

The following variables from Django settings are also supported (with the corresponding types):

Worker settings

Section [worker].

Warning

These settings are needed only for rpc-enabled applications.

Celery worker options:

  • loglevel - Celery worker log level. Default: from main section log_level.

  • pidfile - Celery worker pidfile. Default: /run/{app_name}_worker.pid

  • autoscale - Options for autoscaling. Two comma separated numbers: max,min.

  • beat - Enable or disable celery beat scheduler. Default: true.

See other settings via celery worker --help command.

SMTP settings

Section [mail].

Django comes with several email sending backends. With the exception of the SMTP backend (default when host is set), these backends are useful only in testing and development.

Applications based on vstutils uses only smtp and console backends.

  • host - IP or domain for smtp-server. If it not set vstutils uses console backends. Default: None.

  • port - Port for smtp-server connection. Default: 25.

  • user - Username for smtp-server connection. Default: "".

  • password - Auth password for smtp-server connection. Default: "".

  • tls - Enable/disable tls for smtp-server connection. Default: False.

  • send_confirmation - Enable/disable confirmation message after registration. Default: False.

  • authenticate_after_registration - Enable/disable autologin after registration confirmation. Default: False.

Web settings

Section [web].

These settings are related to web-server. Those settings includes: session_timeout, static_files_url and pagination limit.

  • allow_cors - enable cross-origin resource sharing. Default: False.

  • cors_allowed_origins, cors_allowed_origins_regexes, cors_expose_headers, cors_allow_methods, cors_allow_headers, cors_preflight_max_age - Settings from django-cors-headers lib with their defaults.

  • enable_gravatar - Enable/disable gravatar service using for users. Default: True.

  • rest_swagger_description - Help string in Swagger schema. Useful for dev-integrations.

  • openapi_cache_timeout - Cache timeout for storing schema data. Default: 120.

  • health_throttle_rate - Count of requests to /api/health/ endpoint. Default: 60.

  • bulk_threads - Threads count for PATCH /api/endpoint/ endpoint. Default: 3.

  • session_timeout - Session lifetime. Default: 2w (two weeks).

  • etag_default_timeout - Cache timeout for Etag headers to control models caching. Default: 1d (one day).

  • rest_page_limit and page_limit - Default limit of objects in API list. Default: 1000.

  • session_cookie_domain - The domain to use for session cookies. Read more. Default: None.

  • csrf_trusted_origins - A list of hosts which are trusted origins for unsafe requests. Read more. Default: from session_cookie_domain.

  • case_sensitive_api_filter - Enables/disables case sensitive search for name filtering. Default: True.

  • secure_proxy_ssl_header_name - Header name which activates SSL urls in responses. Read more. Default: HTTP_X_FORWARDED_PROTOCOL.

  • secure_proxy_ssl_header_value - Header value which activates SSL urls in responses. Read more. Default: https.

The following variables from Django settings are also supported (with the corresponding types):

The following settings affects prometheus metrics endpoint (which can be used for monitoring application):

  • metrics_throttle_rate - Count of requests to /api/metrics/ endpoint. Default: 120.

  • enable_metrics - Enable/disable /api/metrics/ endpoint for app. Default: true

  • metrics_backend - Python class path with metrics collector backend. Default: vstutils.api.metrics.DefaultBackend Default backend collects metrics from uwsgi workers and python version info.

Section [uvicorn].

You can configure the necessary settings to run the uvicorn server. vstutils supports almost all options from the cli, except for those that configure the application and connection.

See all available uvicorn settings via uvicorn --help command.

Centrifugo client settings

Section [centrifugo].

To install app with centrifugo client, [centrifugo] section must be set. Centrifugo is used by application to auto-update page data. When user change some data, other clients get notification on subscriptions_update channel with model label and primary key. Without the service all GUI-clients get page data every 5 seconds (by default).

  • address - Centrifugo server address.

  • api_key - API key for clients.

  • token_hmac_secret_key - API key for jwt-token generation.

  • timeout - Connection timeout.

  • verify - Connection verification.

  • subscriptions_prefix - Prefix used for generating update channels, by default “{VST_PROJECT}.update”.

Note

These settings also add parameters to the OpenApi schema and change how the auto-update system works in the GUI. token_hmac_secret_key is used for jwt-token generation (based on session expiration time). Token will be used for Centrifugo-JS client.

Storage settings

Section [storages].

Applications based on vstutils supports filesystem storage out of box. Setup media_root and media_url in [storages.filesystem] section to configure custom media dir and relative url. By default it would be {/path/to/project/module}/media and /media/.

Applications based on vstutils supports store files in external services with Apache Libcloud and Boto3.

Apache Libcloud settings grouped by sections named [storages.libcloud.provider], where provider is name of storage. Each section has four keys: type, user, key and bucket. Read more about the settings in django-storages libcloud docs

This setting is required to configure connections to cloud storage providers. Each entry corresponds to a single ‘bucket’ of storage. You can have multiple buckets for a single service provider (e.g., multiple S3 buckets), and you can define buckets at multiple providers.

For Boto3 all settings grouped by section named [storages.boto3]. Section must contain following keys: access_key_id, secret_access_key, storage_bucket_name. Read more about the settings in django-storages amazon-S3 docs

Storage has following priority to choose storage engine if multiple was provided:

  1. Libcloud store when config contains this section.

  2. Boto3 store, when you have section and has all required keys.

  3. FileSystem store otherwise.

Once you have defined your Libcloud providers, you have an option of setting one provider as the default provider of Libcloud storage. You can do it by setup [storages.libcloud.default] section or vstutils will set the first storage as default.

If you configure default libcloud provider, vstutils will use it as global file storage. To override it set default=django.core.files.storage.FileSystemStorage in [storages] section. When [storages.libcloud.default] is empty django.core.files.storage.FileSystemStorage is used as default. To override it set default=storages.backends.apache_libcloud.LibCloudStorage in [storages] section and use Libcloud provider as default.

Here is example for boto3 connection to minio cluster with public read permissions, external proxy domain and internal connection support:

[storages.boto3]
access_key_id = EXAMPLE_KEY
secret_access_key = EXAMPLEKEY_SECRET
# connection to internal service behind proxy
s3_endpoint_url = http://127.0.0.1:9000/
# external domain to bucket 'media'
storage_bucket_name = media
s3_custom_domain = media-api.example.com/media
# external domain works behind tls
s3_url_protocol = https:
s3_secure_urls = true
# settings to connect as plain http for uploading
s3_verify = false
s3_use_ssl = false
# allow to save files with similar names by adding prefix
s3_file_overwrite = false
# disables query string auth and setup default acl as RO for public users
querystring_auth = false
default_acl = public-read

Throttle settings

Section [throttle].

By including this section to your config, you can setup global and per-view throttle rates. Global throttle rates are specified under root [throttle] section.To specify per-view throttle rate, you need to include child section.

For example, if you want to apply throttle to api/v1/author:

[throttle.views.author]
rate=50/day
actions=create,update
  • rate - Throttle rate in format number_of_requests/time_period. Expected time_periods: second/minute/hour/day.

  • actions - Comma separated list of drf actions. Throttle will be applied only on specified here actions. Default: update, partial_update.

More on throttling at DRF Throttle docs.

Production web settings

Section [uwsgi].

Settings related to web-server used by vstutils-based application in production (for deb and rpm packages by default). Most of them related to system paths (logging, PID-file and so on). More settings in uWSGI docs.

But keep in mind that uWSGI is deprecated and may be removed in future releases. Use the uvicorn settings to manage your app server.

Configuration options

This section contains additional information for configure additional elements.

  1. If you need set https for your web settings, you can do it using HAProxy, Nginx, Traefik or configure it in settings.ini.

[uwsgi]
addrport = 0.0.0.0:8443

[uvicorn]
ssl_keyfile = /path/to/key.pem
ssl_certfile = /path/to/cert.crt
  1. We strictly do not recommend running the web server from root. Use HTTP proxy to run on privileged ports.

  2. You can use {ENV[HOME:-value]} (where HOME is environment variable, value is default value) in configuration values.

  3. You can use environment variables for setup important settings. But config variables has more priority then env. Available settings are: DEBUG, DJANGO_LOG_LEVEL, TIMEZONE and some settings with [ENV_NAME] prefix.

    For project without special settings and project levels named project this variables will stars with PROJECT_ prefix. There list of this variables: {ENV_NAME}_ENABLE_ADMIN_PANEL, {ENV_NAME}_ENABLE_REGISTRATION, {ENV_NAME}_MAX_TFA_ATTEMPTS, {ENV_NAME}_ETAG_TIMEOUT, {ENV_NAME}_SEND_CONFIRMATION_EMAIL, {ENV_NAME}_SEND_EMAIL_RETRIES, {ENV_NAME}_SEND_EMAIL_RETRY_DELAY, {ENV_NAME}_AUTHENTICATE_AFTER_REGISTRATION, {ENV_NAME}_MEDIA_ROOT (dir with uploads), {ENV_NAME}_GLOBAL_THROTTLE_RATE, and {ENV_NAME}_GLOBAL_THROTTLE_ACTIONS.

    There are also URI-specific variables for connecting to various services such as databases and caches. There are DATABASE_URL, CACHE_URL, LOCKS_CACHE_URL, SESSIONS_CACHE_URL and ETAG_CACHE_URL. As you can see from the names, they are closely related to the keys and names of the corresponding config sections.

  4. We recommend to install uvloop to your environment and setup loop = uvloop in [uvicorn] section for performance reasons.