continuwuity/docs/maintenance.mdx

# Maintaining your Continuwuity setup

## Moderation

Continuwuity has moderation through admin room commands. "binary commands" (medium
priority) and an admin API (low priority) is planned. Some moderation-related
config options are available in the example config such as "global ACLs" and
blocking media requests to certain servers. See the example config for the
moderation config options under the "Moderation / Privacy / Security" section.

Continuwuity has moderation admin commands for:

- managing room aliases (`!admin rooms alias`)
- managing room directory (`!admin rooms directory`)
- managing room banning/blocking and user removal (`!admin rooms moderation`)
- managing user accounts (`!admin users`)
- fetching `/.well-known/matrix/support` from servers (`!admin federation`)
- blocking incoming federation for certain rooms (not the same as room banning)
(`!admin federation`)
- deleting media

Any commands with `-list` in them will require a codeblock in the message with
each object being newline delimited. An example of doing this is:

````
!admin rooms moderation ban-list-of-rooms
```
!roomid1:server.name
#badroomalias1:server.name
!roomid2:server.name
!roomid3:server.name
#badroomalias2:server.name
```
````

## Database (RocksDB)

Generally there is very little you need to do. [Compaction][rocksdb-compaction]
is ran automatically based on various defined thresholds tuned for Continuwuity to
be high performance with the least I/O amplifcation or overhead. Manually
running compaction is not recommended, or compaction via a timer, due to
creating unnecessary I/O amplification. RocksDB is built with io_uring support
via liburing for improved read performance.

RocksDB troubleshooting can be found [in the RocksDB section of troubleshooting](./troubleshooting.mdx#rocksdb--database-issues).

### Compression

Some RocksDB settings can be adjusted, such as the chosen compression method and level.
See the RocksDB section in the [example config](./reference/config.mdx), and the
[database compression section](./guides/performance.mdx#tuning-database-compression)
in the performance tuning documentation for more.

#### Caveats for btrfs users

btrfs users have reported that database compression does not need to be disabled
on Continuwuity as the filesystem already does not attempt to compress. This can be
validated by using `filefrag -v` on a `.SST` file in your database, and ensure
the `physical_offset` matches (no filesystem compression). It is very important
to ensure no additional filesystem compression takes place as this can render
unbuffered Direct IO inoperable, significantly slowing down read and write
performance. See [the Btrfs docs](https://btrfs.readthedocs.io/en/latest/Compression.html#compatibility).

:::important
Compression is done using the COW mechanism so it’s incompatible with
`nodatacow`. Direct IO read works on compressed files but will fall back to
buffered writes and leads to no compression even if force compression is set.
Currently `nodatasum` and compression don’t work together.
:::

### Files in database

Do not touch any of the files in the database directory. This must be said due
to users being mislead by the `.log` files in the RocksDB directory, thinking
they're server logs or database logs, however they are critical RocksDB files
related to WAL tracking.

The only safe files that can be deleted are the `LOG` files (all caps). These
are the real RocksDB telemetry/log files, however Continuwuity has already
configured to only store up to 3 RocksDB `LOG` files due to generally being
useless for average users unless troubleshooting something low-level. If you
would like to store nearly none at all, see the `rocksdb_max_log_files`
config option.

## Backups

### Database online backup

f you'd like to run an online backup of your database - that is, a backup with
no downtime - check the [`!admin server` command](./reference/admin/server.md)
for the required commands and the `database_backup_path` config options in
the example config.

Please note that the format of the database backup is not the exact same as the
format of offline backups. This is unfortunately a bad design choice by Facebook
as we are using the database backup engine API from RocksDB, however the data
is still there and can still be joined together.

To restore a backup from an online RocksDB backup:

- Shutdown Continuwuity
- Create a new directory for merging together the data
- In the online backup created, copy all `.sst` files in
`$DATABASE_BACKUP_PATH/shared_checksum` to your new directory
- trim all the strings so instead of `######_sxxxxxxxxx.sst`, it reads
`######.sst`. A way of doing this with sed and bash is `for file in *.sst; do mv
"$file" "$(echo "$file" | sed 's/_s.*/.sst/')"; done`
- Copy all the files in `$DATABASE_BACKUP_PATH/private/1` (or the latest backup number
if you have multiple) to your new directory
- Set your `database_path` config option to your new directory, or replace your
old one with the new one you crafted
- Start up Continuwuity again and it should open as normal

Note: You can verify the necessary files to copy by comparing them with contents in the `$DATABASE_BACKUP_PATH/meta/1` (or latest backup number) file.

### Database offline backup

If you'd like to do an offline backup, shutdown Continuwuity and copy your
`database_path` directory elsewhere. This can be restored with no modifications
needed.

### Media backup

Media is stored in the `media/` subdirectory from your database directory.
Backing up media is also just copying that subdirectory.

## Media management

Media still needs various work, however Continuwuity implements media deletion via:

- MXC URI or Event ID (unencrypted and attempts to find the MXC URI in the
event)
- Delete list of MXC URIs
- Delete remote media in the past `N` seconds/minutes via filesystem metadata on
the file created time (`btime`) or file modified time (`mtime`)
- Delete all media from a local user
- Delete all media from a remote server
- Delete cached URL previews

See the [`!admin media` commands](./reference/admin/media.md) for further information.
All media in Continuwuity is stored at `$DATABASE_DIR/media`.

While Continuwuity does not implement built-in S3 support, using an S3 filesystem
mount on the `media/` path will work. Continuwuity also sends a `Cache-Control`
header of 1 year with the `immutable` directive for all media requests
(download and thumbnail) to reduce unnecessary bandwidth and load.

[rocksdb-compaction]: https://github.com/facebook/rocksdb/wiki/Compaction