this is dual-stack by default on linux, resolves
issues with nginx using `localhost` and randomly
choosing between 127.0.0.1 and [::1], causing
intermittent upstream issues
Signed-off-by: strawberry <strawberry@puppygock.gay>
The original implementation of this was really weird, so I restructed it
a lot while debugging, and am just gonna leave the restructured version.
Root cause of the segfault seems to be that upstream nixpkgs liburing
derivation is generating both static and dynamic libraries, causing
rocksdb to statically link liburing in a dynamic build, pulling in some
allocator stuff at the same time. I created a PR[1] to fix this upstream,
but it probably won't be available on nixos-unstable for quite a while,
so we can also patch it locally.
[1]: https://github.com/NixOS/nixpkgs/pull/314945
I talked to somebody yesterday in #conduwuit:puppygock.gay that was using
this output in their system config. The dynamically-linked jemalloc build
is quite fragile, and is not tested by anything else in CI. We want to
make sure we don't break it again in the future.
This failed to inherit the fix from bec507d739
because the crane package's buildInputs become propagatedBuildInputs in
a static stdenv, but become normal buildInputs in a dynamic stdenv. Since
we were only pulling propagatedBuildInputs into the devshell, dynamically
linked devshells did not include the rust-jemalloc-sys package. This
causes tikv-jemalloc-sys to build it's own static jemalloc package, and
we end up loading libc before jemalloc at runtime.
It turns out that this was actually fixed by
bec507d739 and
857ac42aac, but we didn't identify it at the
time. Notably, the `dynamic` devshell is still broken.
needed for hot reloading but rpath being true by default
causes linker errors on lld because of the sad rpath bug
Signed-off-by: strawberry <strawberry@puppygock.gay>
for a very long time, if a remote server responded to us with
a valid but unsuccessful (HTTP 4xx) response and the caller was the
`send_federation_request` function, we may find ourselves
with a warning message only containing the destination's
server name which was very unhelpful. the true error was
buried away in trace logs. this would primarily be noticed
with server key fetch requests from us.
conduit has been throwing away the ruma request error: https://gitlab.com/famedly/conduit/-/blame/next/src/utils/error.rs#L62
before: 2024-05-23T04:45:02.930224Z WARN router:{path=/_matrix/client/v3/publicRooms}:handle: conduit_api::client_server::directory: Failed to return our /publicRooms: matrix.org
after: 2024-05-23T05:05:02.435272Z WARN router:{path=/_matrix/client/v3/publicRooms}:handle: conduit_api::client_server::directory: Failed to return our /publicRooms: matrix.org: [401 / M_UNAUTHORIZED] Failed to find any key to satisfy: _FetchKeyRequest(server_name='your.server.name', minimum_valid_until_ts=1716440702337, key_ids=['ed25519:RQB3XPQX'])
Signed-off-by: strawberry <strawberry@puppygock.gay>
Previously we were relying on NIX_CFLAGS_COMPILE, but this is not being
set in static devshells. A cleaner solution for complement would likely
be to build the tests in their own nix derivation instead of building
them in the devshell, but this change unblocks CI for now.
This is causing build failures on Mac:
> In file included from /tmp/nix-build-rocksdb-static-aarch64-apple-darwin-9.1.1.drv-0/source/memory/memory_allocator.cc:8:
> In file included from /tmp/nix-build-rocksdb-static-aarch64-apple-darwin-9.1.1.drv-0/source/memory/jemalloc_nodump_allocator.h:11:
> /tmp/nix-build-rocksdb-static-aarch64-apple-darwin-9.1.1.drv-0/source/port/jemalloc_helper.h:63:36: warning: unknown attribute '_rjem_malloc' ignored [-Wunknown-attributes]
> mallocx(size_t, int) JEMALLOC_ATTR(malloc) JEMALLOC_ALLOC_SIZE(1)
> ^~~~~~
> /nix/store/3bix0kzy670dyhhizri3dwb1qfj3sdpa-jemalloc-static-aarch64-apple-darwin-5.3.0/include/jemalloc/jemalloc.h:412:18: note: expanded from macro 'malloc'
> # define malloc je_malloc
> ^~~~~~~~~
> /nix/store/3bix0kzy670dyhhizri3dwb1qfj3sdpa-jemalloc-static-aarch64-apple-darwin-5.3.0/include/jemalloc/jemalloc.h:75:21: note: expanded from macro 'je_malloc'
> # define je_malloc _rjem_malloc
> ^~~~~~~~~~~~
> /nix/store/3bix0kzy670dyhhizri3dwb1qfj3sdpa-jemalloc-static-aarch64-apple-darwin-5.3.0/include/jemalloc/jemalloc.h:183:43: note: expanded from macro 'JEMALLOC_ATTR'
> # define JEMALLOC_ATTR(s) __attribute__((s))
Full build log at <https://girlboss.ceo/~strawberry/pb/ygJ3>. This is
likely fixable with patches to rocksdb, but not worth it since darwin is
only a dev platform.
CI is running `cargo build --all-features`, so we should be passing all
the features to nix as well.
The only thing this currently affects is the jemalloc_prof feature, but if
we add any non-default features that affect nix in the future they should
also be handled correctly now.
Dynamically-linked jemalloc doesn't work due to link-order issues, and we
want CI to be testing a static binary anyway since that's what we're
publishing in releases.
Some of the features affect nix dependencies, so we need to have a
full feature list available when constructing the nix derivation. This
incidentally fixes the bug where we weren't enabling jemalloc on rocksdb
in CI/devshells, because jemalloc is now a default feature. It does not
fix the more general class of that issue, where CI is performing an
`--all-features` build in a nix devshell built for default-features.
I am now passing `--no-default-features` to cargo, and having it use our
unified feature list rather than duplicating the unification inside cargo.
Without setting JEMALLOC_OVERRIDE, we end up linking to two different
jemalloc builds. Once dynamically, as a transitive dependency through
rocksdb, and a second time to the static jemalloc that tikv-jemalloc-sys
builds.
Previously, we were returning redundant member count updates or encrypted
device updates from the /sync endpoint in some cases. The extra member
count updates are spec-compliant, but unnecessary, while the extra
encrypted device updates violate the spec.
The refactor necessary to fix this bug is also necessary to support
filtering on state events in sync.
Details:
Joined room incremental sync needs to examine state events for four
purposes:
1. determining whether we need to return an update to room member counts
2. determining the set of left/joined devices for encrypted rooms
(returned in `device_lists`)
3. returning state events to the client (in `rooms.joined.*.state`)
4. tracking which member events we have sent to the client, so they can
be omitted on future requests when lazy-loading is enabled.
The state events that we need to examine for the first two cases is member
events in the delta between `since` and the end of `timeline`. For the
second two cases, we need the delta between `since` and the start of
`timeline`, plus contextual member events for any senders that occur in
`timeline`. The second list is subject to filtering, while the first is
not.
Before this change, we were using the same set of state events that we are
returning to the client (cases 3/4) to do the analysis for cases 1/2.
In a compliant implementation, this would result in us missing some
relevant member events in 1/2 in addition to seeing redundant member
events. In current conduwuit this is not the case because the set of
events that we return to the client is always a superset of the set that
is needed for cases 1/2. This is because we don't support filtering, and
we have an existing bug[1] where we are returning the delta between
`since` and the end of `timeline` rather than the start.
[1]: https://github.com/girlbossceo/conduwuit/issues/361
Fixing this is necessary to implement filtering because otherwise
we would start missing some member events for member count or encrypted
device updates if the relevant member events are rejected by the filter.
This would be much worse than our current behavior.
This cache can serve invalid responses, and has an extremely low hit
rate.
It serves invalid responses because because it's only keyed off
the `since` parameter, but many of the other request parameters also
affect the response or it's side effects. This will become worse once we
implement filtering, because there will be a wider space of parameters
with different responses. This problem is fixable, but not worth it
because of the low hit rate.
The low hit rate is because normal clients will always issue the next
sync request with `since` set to the `prev_batch` value of the previous
response. The only time we expect to see multiple requests with the same
`since` is when the response is empty, but we don't cache empty
responses.
This was confirmed experimentally by logging cache hits and misses over
15 minutes with a wide variety of clients. This test was run on
matrix.computer.surgery, which has only a few active users, but a
large volume of sync traffic from many rooms. Over the test period, we
had 3 hits and 5309 misses. All hits occurred in the first minute, so I
suspect that they had something to do with client recovery from an
offline state. The clients that were connected during the test are:
- element web
- schildichat web
- iamb
- gomuks
- nheko
- fractal
- fluffychat web
- fluffychat android
- cinny web
- element android
- element X android
Fixes: #336
the namespace check on username login is unnecessary, hashes aren't ever
going to match, and axum auth handles this kind of stuff already
Signed-off-by: strawberry <strawberry@puppygock.gay>
debconf support needs to be done in a way that does not duplicate
the config file like upstream does.
Signed-off-by: strawberry <strawberry@puppygock.gay>
This reverts commit 321a6ca0fe.
These checks were not working as intended, resulting in the unban button not working
The join check gets kept since it slightly reduces the amount of sent joins in some cases
This check will probably be replaced soon for a more universal solution to the "made no change" issue
Signed-off-by: morguldir <morguldir@protonmail.com>
protects against a maintainer creating a downgrading version tag, and
uploading artifacts with that version
this check is only ran via workflow dispatch on the tag
Signed-off-by: strawberry <strawberry@puppygock.gay>
Only normal users should be prevented from creating an alias within an
exclusive namespace, not the appservice itself. This mirrors the
behaviour in api/client_server/room.rs on room creation.
this was already enabled by default by rocksdb technically, but
it wasn't building with it properly.
Signed-off-by: strawberry <strawberry@puppygock.gay>
this makes `CONDUWUIT_WELL_KNOWN__CLIENT` a valid env variable config
option as it would normally exist under `[well_known.client]` in toml
Signed-off-by: strawberry <strawberry@puppygock.gay>
after getting the shared rooms with the target user, we actually only
get the presence of ourselves instead of the requested user
Signed-off-by: strawberry <strawberry@puppygock.gay>
this is purely DNSSEC related which we don't use, and DNSSEC on matrix
is unbearable for federation (no one sets it up properly, it's extremely taxing, etc)
Signed-off-by: strawberry <strawberry@puppygock.gay>
there's really no point in trying to stay as low as possible for us,
and this makes development easier. Debian users should just use rustup,
Nix users already get the proper toolchains.
Signed-off-by: strawberry <strawberry@puppygock.gay>
Hardcoding the output path to something in CWD is a pain if you're running
conduwuit through systemd or similar. Also made the error message when
it's unable to create the output file a little more friendly.
Previously, enabling the `tracing_flame` or `allow_jaeger` options would
prevent any logs from being written to stdout. In addition, enabling the
`allow_jaeger` option would inhibit the `tracing_flame` option.
Now that we have a way to use separate tracing filters with different
layers, we can enable all three at the same time without issues.
This commit also prevents the `debug log_level` command from modifying
the `tracing-flame` filter. This was supported previously, but I don't
think it's something that you would ever want to do intentionally. Now
that we have both the normal log filter and the `tracing-flame` filter
enabled at the same time, we want to `debug log_level` to only modify the
normal filter.
imagine this SQL query but in conduwuit:
select * from users_in_public_rooms where user_id like '%user_id%';
Signed-off-by: strawberry <strawberry@puppygock.gay>
This turned out to be quite hairy, mostly because we need to apply the
config's log level filter to the actual logs (stdout and, optionally
sentry), but do not want to filter out the tokio tracing events needed by
the console_subscriber. I hit several edge cases in tracing getting
this to work, and we now depend on a git version of tracing with a
backported patch :(
All images should be generating correctly with parallelism and Docker manifests, and should output the end of the CI testing errors in a job summary box when the test fails.
When the test succeeds you get a big ✅ then at the end of the Docker publish it should include the `docker pull` commands for both Docker Hub and GHCR registries to make those pesky Docker users lives easier!
Should've been done in fe606f4fad but the
author didn't realize it.
Flake lock file updates:
• Updated input 'rocksdb':
'github:facebook/rocksdb/bcf88d48ce8aa8b536aee4dd305533b3b83cf435' (2024-04-16)
→ 'github:facebook/rocksdb/6f7cabeac80a3a6150be2c8a8369fcecb107bf43' (2024-04-22)
Apparently github actions VMs ship with it and that's how it was working
before? Cursed. We should control our own supply chain and also ensure
that local development uses the same version as CI.
Some of the improvements here include:
* rocksdb can actually use jemalloc now instead of just pulling in a
second rocksdb for no reason
* "complement-runtime" factored back out into shell file
* complement image no longer uses `mkDerivation` for `copyToRoot`
because that's what `buildEnv` is for
* complement image no longer sets `SERVER_NAME`, complement already does
that
* all packages were factored out into `callPackage`-able files for use
with a custom `lib.makeScope pkgs.newScope`
* new version of `mkPackage` has options that are easier to use and
override such as `features`
Apparently it uses `date -Iseconds` to parse, so we can use @ with a timestamp
Also it doesn't parse `created` in buildImage, only buildLayeredImage
Signed-off-by: morguldir <morguldir@protonmail.com>
Signed-off-by: strawberry <strawberry@puppygock.gay>
resolves X-Matrix signatures being invalid in some edge-cases,
and fixes Complement/Sytest federation tests
Signed-off-by: strawberry <strawberry@puppygock.gay>
this is hard to maintain now, and i think conduwuit is reaching a point that
it's way too different from upstream to simply "list all the differences" out
Signed-off-by: strawberry <strawberry@puppygock.gay>
this does deprecate the original `well_known_` prefixed config options
with a dedicated/proper config sub-block (`[config.well_known]`)
Signed-off-by: strawberry <strawberry@puppygock.gay>
there's nothing unstable about this, and per upstream only unstable
room versions are used if they are complex to support (versions <=5)
Signed-off-by: strawberry <strawberry@puppygock.gay>
this is not helpful informational logging
```
2024-04-02T04:40:52.590444Z INFO http_request{path=/_matrix/client/v1/rooms/:room_id/hierarchy}: conduit::service::rooms::spaces: Asking thomcat.rocks for /hierarchy
2024-04-02T04:40:52.628248Z INFO http_request{path=/_matrix/client/v1/rooms/:room_id/hierarchy}: conduit::service::rooms::spaces: Asking hackingfor.eu for /hierarchy
2024-04-02T04:40:52.808526Z INFO http_request{path=/_matrix/client/v1/rooms/:room_id/hierarchy}: conduit::service::rooms::spaces: Asking matrix.org for /hierarchy
2024-04-02T04:40:52.920936Z INFO http_request{path=/_matrix/client/v1/rooms/:room_id/hierarchy}: conduit::service::rooms::spaces: Asking matrix.org for /hierarchy
2024-04-02T04:40:52.959362Z INFO http_request{path=/_matrix/client/v1/rooms/:room_id/hierarchy}: conduit::service::rooms::spaces: Got response from matrix.org for /hierarchy
Response { children: [], inaccessible_children: [], room: SpaceHierarchyParentSummary { canonical_alias: Some("#cybersec-whonix:matrix.org"), name: Some("Whonix"), num_joined_members: 329, room_id: "!OJFkLJksWastbfdRuf:matrix.org", topic: Some("| Part of the Cybersec matrix.org community (#cyber-space:matrix.org) | RULES: https://cybersec-rules.thomcat.rocks | Whonix-focused room | Off-topic chat is fine. Keep it brief and/or move it to #cybersec-offtopic:matrix.org"), world_readable: true, guest_can_join: false, avatar_url: Some("mxc://matrix.org/DMJtrQdhQHKxeODrSibtrczX"), join_rule: "public", room_type: None, children_state: [], allowed_room_ids: [] } }
2024-04-02T04:40:52.959762Z INFO http_request{path=/_matrix/client/v1/rooms/:room_id/hierarchy}: conduit::service::rooms::spaces: Asking matrix.org for /hierarchy
2024-04-02T04:40:53.109611Z INFO http_request{path=/_matrix/client/v1/rooms/:room_id/hierarchy}: conduit::service::rooms::spaces: Got response from matrix.org for /hierarchy
Response { children: [], inaccessible_children: [], room: SpaceHierarchyParentSummary { canonical_alias: Some("#lockpicking:matrix.org"), name: Some("Lockpicking"), num_joined_members: 284, room_id: "!uEYjSxQOZnHEkiurTP:matrix.org", topic: Some("| Part of the Cybersec matrix.org community (#cyber-space:matrix.org) | RULES: https://cybersec-rules.thomcat.rocks | Off-topic chat is fine. Keep it brief and/or move it to #cybersec-offtopic:matrix.org"), world_readable: false, guest_can_join: false, avatar_url: Some("mxc://thomcat.rocks/cSeFfMLUdjymKdkUmXtIQTjf"), join_rule: "public", room_type: None, children_state: [], allowed_room_ids: [] } }
2024-04-02T04:40:53.109880Z INFO http_request{path=/_matrix/client/v1/rooms/:room_id/hierarchy}: conduit::service::rooms::spaces: Asking matrix.org for /hierarchy
2024-04-02T04:40:53.261581Z INFO http_request{path=/_matrix/client/v1/rooms/:room_id/hierarchy}: conduit::service::rooms::spaces: Got response from matrix.org for /hierarchy
Response { children: [], inaccessible_children: [], room: SpaceHierarchyParentSummary { canonical_alias: Some("#cybersec-rss:matrix.org"), name: Some("Cybersecurity-RSS"), num_joined_members: 347, room_id: "!IVHnAZkhJOhdZxlHRA:matrix.org", topic: Some("| Part of the Cybersec matrix.org community (#cyber-space:matrix.org) | Suggestions to our RSS feed list welcome at #cybersecurity:matrix.org"), world_readable: true, guest_can_join: false, avatar_url: Some("mxc://thomcat.rocks/ugdubBloOkVCYxqFzdVoPwcV"), join_rule: "public", room_type: None, children_state: [], allowed_room_ids: [] } }
2024-04-02T04:40:53.305166Z INFO http_request{path=/_matrix/client/v1/rooms/:room_id/hierarchy}: conduit::service::rooms::spaces: Got response from matrix.org for /hierarchy
Response { children: [], inaccessible_children: [], room: SpaceHierarchyParentSummary { canonical_alias: Some("#cybersec-whonix:matrix.org"), name: Some("Whonix"), num_joined_members: 329, room_id: "!OJFkLJksWastbfdRuf:matrix.org", topic: Some("| Part of the Cybersec matrix.org community (#cyber-space:matrix.org) | RULES: https://cybersec-rules.thomcat.rocks | Whonix-focused room | Off-topic chat is fine. Keep it brief and/or move it to #cybersec-offtopic:matrix.org"), world_readable: true, guest_can_join: false, avatar_url: Some("mxc://matrix.org/DMJtrQdhQHKxeODrSibtrczX"), join_rule: "public", room_type: None, children_state: [], allowed_room_ids: [] } }
2024-04-02T04:40:53.323257Z INFO http_request{path=/_matrix/client/v1/rooms/:room_id/hierarchy}: conduit::service::rooms::spaces: Asking thomcat.rocks for /hierarchy
2024-04-02T04:40:53.323672Z INFO http_request{path=/_matrix/client/v1/rooms/:room_id/hierarchy}: conduit::service::rooms::spaces: Asking matrix.org for /hierarchy
2024-04-02T04:40:53.369721Z INFO http_request{path=/_matrix/client/v1/rooms/:room_id/hierarchy}: conduit::service::rooms::spaces: Asking matrix.org for /hierarchy
2024-04-02T04:40:53.529250Z INFO http_request{path=/_matrix/client/v1/rooms/:room_id/hierarchy}: conduit::service::rooms::spaces: Got response from matrix.org for /hierarchy
Response { children: [], inaccessible_children: [], room: SpaceHierarchyParentSummary { canonical_alias: None, name: None, num_joined_members: 1463, room_id: "!aBXqGDWIxVYeYxVbRu:matrix.org", topic: Some("| Part of the Cybersec matrix.org community (#cyber-space:matrix.org) | The room needs urgent attention? @ all the mods! | RULES: https://cybersec-rules.thomcat.rocks | \"you seem to be completely in lack of basic knowledge of how a computer or any programming language operates, to the point that any attempt of discussing any security measures is a complete waste of time\""), world_readable: false, guest_can_join: false, avatar_url: Some("mxc://thomcat.rocks/DjRWzuYjLkScQYPySYNedlFr"), join_rule: "public", room_type: None, children_state: [], allowed_room_ids: [] } }
2024-04-02T04:40:53.546581Z INFO http_request{path=/_matrix/client/v1/rooms/:room_id/hierarchy}: conduit::service::rooms::spaces: Got response from matrix.org for /hierarchy
Response { children: [], inaccessible_children: [], room: SpaceHierarchyParentSummary { canonical_alias: None, name: None, num_joined_members: 1463, room_id: "!aBXqGDWIxVYeYxVbRu:matrix.org", topic: Some("| Part of the Cybersec matrix.org community (#cyber-space:matrix.org) | The room needs urgent attention? @ all the mods! | RULES: https://cybersec-rules.thomcat.rocks | \"you seem to be completely in lack of basic knowledge of how a computer or any programming language operates, to the point that any attempt of discussing any security measures is a complete waste of time\""), world_readable: false, guest_can_join: false, avatar_url: Some("mxc://thomcat.rocks/DjRWzuYjLkScQYPySYNedlFr"), join_rule: "public", room_type: None, children_state: [], allowed_room_ids: [] } }
```
Signed-off-by: strawberry <strawberry@puppygock.gay>
this is the url field in the appservice registration file,
this is almost always localhost and the admin should
be vetting the appservice registration yaml file before
registering it anyways.
Signed-off-by: strawberry <strawberry@puppygock.gay>
there seems to be a regression, likely from https://github.com/facebook/rocksdb/issues/12361 / https://github.com/facebook/rocksdb/pull/12309
```
[1/0/2 built] building rocksdb-9.0.0 (configurePhase): -- Detecting CXX compile features - donedirenv: ([/Users/strawberry/.nix-profile/bin/direnv export zsh]) is taking a while to execute. Use CTRL-C to give up.
error: builder for '/nix/store/9slwgpnardhn2vqzqhn361ic668n38wq-rocksdb-9.0.0.drv' failed with exit code 1;
last 10 log lines:
> -- Found lz4: /nix/store/cafwv4439qbm2ij04mpc7xz5m3f7mfix-lz4-1.9.4/lib/liblz4.dylib
> CMake Error at /nix/store/bin32lqag7lx38994xpf9jvhk1xbd64c-cmake-3.28.2/share/cmake-3.28/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
> Could NOT find zstd (missing: ZSTD_INCLUDE_DIRS)
> Call Stack (most recent call first):
> /nix/store/bin32lqag7lx38994xpf9jvhk1xbd64c-cmake-3.28.2/share/cmake-3.28/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
> cmake/modules/Findzstd.cmake:17 (find_package_handle_standard_args)
> CMakeLists.txt:167 (find_package)
>
>
> -- Configuring incomplete, errors occurred!
For full logs, run 'nix log /nix/store/9slwgpnardhn2vqzqhn361ic668n38wq-rocksdb-9.0.0.drv'.
error: 1 dependencies of derivation '/nix/store/ir8jf2wic98iymjlk7d2i1kjjsgv15v2-nix-shell-env.drv' failed to build
```
happens in both rust-rocksdb and our fork of rust-rocksdb
Signed-off-by: strawberry <strawberry@puppygock.gay>
There are so many ways to do this we realistically shouldn't bother
describing any of them, especially because people should be learning all
the options and choosing the one that suits them best anyway.
Co-authored-by: strawberry <strawberry@puppygock.gay>
Signed-off-by: strawberry <strawberry@puppygock.gay>
The main thing this section is really useful for is explaining how to
configure various reverse proxies, which applies to basically anything.
Also, remove all the language about this being "recommended", because
nothing in this documentation is actually tested in CI.
Signed-off-by: strawberry <strawberry@puppygock.gay>
It is very stale. Please just use Nix. Trying to do it outside of Nix
will be an exercise in frustration, I guarantee it.
Signed-off-by: strawberry <strawberry@puppygock.gay>
this should fix FluffyChat password resets, and
other possible client issues that expose features
based on server capabilities.
Signed-off-by: strawberry <strawberry@puppygock.gay>
save the few bytes of bandwidth for something else
also now that we send our User-Agent, it is
technically sending conduwuit versions now
Signed-off-by: strawberry <strawberry@puppygock.gay>
i build/develop on a mac, i have a windows machine,
my servers run linux and i do prod builds on linux,
and BSD support is generally inherent with linux
and mac.
i don't think it hurts or would be difficult for me
to say i support all of these.
upstream conduit only officially supports Linux
Signed-off-by: strawberry <strawberry@puppygock.gay>
yes windows technically supports unix sockets,
but its not as good or the same as actual
nix platform unix sockets
Signed-off-by: strawberry <strawberry@puppygock.gay>
reduces unnecessary crates being compiled. splits them
into features.
i have yet to see anyone use conduit's opentelemetry
stuff, and realistically those people who do
performance benchmarking and measurements will be
building stuff anyways so they can just enable this
feature.
Signed-off-by: strawberry <strawberry@puppygock.gay>
if https://github.com/ruma/ruma/issues/1240 is to
be trusted, there are apparently still servers
and clients that may call these endpoints.
i'm unable to read the history of that matrix.to
link so i don't know the full context, but this
is trivial to implement so..
Signed-off-by: strawberry <strawberry@puppygock.gay>
retains compatibility for container users
who set it to empty.
if the variable is unspecified, it will
use the CONDUIT_ variables as normal.
Signed-off-by: strawberry <strawberry@puppygock.gay>
this can be used as a way to deal with the thumbnail
and the media file at the same time without knowing
the thumbnail MXC URL.
Signed-off-by: strawberry <strawberry@puppygock.gay>
complement sends C-S requests over HTTP, and federation
over HTTPS.
complement without caddy *almost* works. unfortunately
i am now dealing with invalid X-Matrix signatures
due to non-percent encoded URIs and it does not
seem trivial to percent-encode URIs that a
reverse proxy would normally do for you.
Signed-off-by: strawberry <strawberry@puppygock.gay>
this provides not only some future compatibility with MSC4051,
but it just makes sense to not crash/error if we can't get a server_name
from the room ID and should just use the server_name from the sender
user's invite event. there is already code ahead that accounts for
an empty vector so this is safe.
Signed-off-by: strawberry <strawberry@puppygock.gay>
default of 90 seconds can be too low if hardware is not very fast
and rocksdb compaction or shutdown takes too long
Signed-off-by: strawberry <strawberry@puppygock.gay>
takes a full room ID, evicts all our users from that room,
adds room ID to banned room IDs metadata db table, and
forbids any new local users from attempting to join it.
Signed-off-by: strawberry <strawberry@puppygock.gay>
from upstream MR https://gitlab.com/famedly/conduit/-/merge_requests/347
with the following changes (so far):
- remove hardcoded list of allowed hosts (strongly disagree with this,
even if it is desired, it should not be harcoded)
- add more allow config options for granularity via URL contains,
host contains, and domain is (explicit match) for security
- warn if a user is allowing all URLs to be previewed for security reasons
- replace an expect with proper error handling
- bump webpage to 2.0
- improved code style a tad
Co-authored-by: rooot <hey@rooot.gay>
Signed-off-by: rooot <hey@rooot.gay>
Signed-off-by: strawberry <strawberry@puppygock.gay>
This uses flakes-compat to read the `flake.nix` and expose it
to non-flake users.
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
Signed-off-by: strawberry <strawberry@puppygock.gay>
`IndexMap::remove` was deprecated in favor of explicitly named methods.
I assume that we actually needed to be using `shift_remove`, otherwise
we probably wouldn't be bothering with `indexmap` here in the first
place. I wonder if this fixes any bugs lol
from: https://gitlab.com/famedly/conduit/-/merge_requests/580
Signed-off-by: strawberry <strawberry@puppygock.gay>
This fixes a bug where the aarch64 OCI image had metadata saying it was
an x86_64 OCI image. On top of that, I think the metadata was actually
right (aside from Conduit's binary): since all other packages were being
pulled from `pkgsHost`, an OCI image cross compiled for aarch64 from a
different architecture would result in unexecutable binaries (e.g. tini)
since they were compiled for the completely wrong architecture.
from: https://gitlab.com/famedly/conduit/-/merge_requests/579
Signed-off-by: strawberry <strawberry@puppygock.gay>
Well, kinda. It crashed on me after 10 minutes because the tests timed
out like in <https://github.com/matrix-org/complement/issues/394>.
Sounds like this means it's a them problem though.
I want to use Nix to build this image instead in the future but this
will at least make it work for now and give me a reference for while I'm
porting it. I also want to make Conduit natively understand Complement's
requirements instead of `sed`ing a bunch of stuff and needing a reverse
proxy in the container. Should be more reliable that way.
I'm not making this run in CI until the above stuff is addressed and
until I can decide on a way to pin the revision of Complement being
tested against.
from: https://gitlab.com/famedly/conduit/-/merge_requests/575
Signed-off-by: strawberry <strawberry@puppygock.gay>
Without this, checking the authority of TLS certificates fails, making
Conduit (rightly) refuse to connect to anything.
Signed-off-by: strawberry <strawberry@puppygock.gay>
Honestly not sure why it wasn't done like this before. This code is much
less awkward to follow and more compartmentalized.
These changes were mainly motivated by a clippy lint triggering on the
original code, which then made me wonder if I could get rid of some of
the `Box`ing. Turns out I could, and this is the result of that.
Co-authored-by: strawberry <strawberry@puppygock.gay>
Signed-off-by: strawberry <strawberry@puppygock.gay>
a 6+ year old deprecated field that isnt even spelled
right, and no clients use it must still be sent
according to spec
Signed-off-by: strawberry <strawberry@puppygock.gay>
this can most definitely be improved but this is a decent attempt.
the only annoying this is i couldn't just use a Vec<IPAddress> which
would have significantly simplified all of this, but serde can't
deserialise it on the config side i guess.
i may find a better way to do this in the future, but this should cover
most areas anyways.
Signed-off-by: strawberry <strawberry@puppygock.gay>
as far as i can tell, it will return a normal
error in the auth token handling code so this is fine.
we also shouldnt assume all errors from this are
access_token related.
Signed-off-by: strawberry <strawberry@puppygock.gay>
latest rocksdb now has WriteBufferManager support
i hope no one is using sqlite with conduwuit, but if they are let's
bump it to latest git too for the latest sqlite version available.
Signed-off-by: strawberry <strawberry@puppygock.gay>
im not sure what the TODO is trying to say here,
but since it's many years old and conduwuit is
fast, i dont see an issue with this.
Signed-off-by: strawberry <strawberry@puppygock.gay>
these are benign errors that are from things like
conduwuit fetching remote media from dead servers
Signed-off-by: strawberry <strawberry@puppygock.gay>
default for RocksDB is -1 and conduwuit already raises the
soft and hard nofile limits at startup.
Signed-off-by: strawberry <strawberry@puppygock.gay>
in short, the `/report` endpoint now checks if:
- the reporting room in the URI matches the PDU/event reported
- sender user is in the room reported
- raises report reasoning to 750 characters (spec doesn't say to limit
these, but thorough and informative reports for server admins are not
a bad thing)
- (hopefully) fixes some broken formatting
- add a random short delay before sending a successful response to the
client to make it more annoying to enumerate for events on our server
(security by obscurity but spec suggests it)
basically, secure reports better lol
see https://spec.matrix.org/v1.9/client-server-api/#post_matrixclientv3roomsroomidreporteventid
Signed-off-by: strawberry <strawberry@puppygock.gay>
this seems to require some more work to properly ignore
dead server errors without breaking the entire room join
Signed-off-by: strawberry <strawberry@puppygock.gay>
this will be a major pain to work through. for now, let's
just add them and overtime work through these.
Signed-off-by: strawberry <strawberry@puppygock.gay>
these database backends are either unmaintained, broken in conduit, or
incredibly niche for something like conduwuit.
also i want to bump the MSRV.
Signed-off-by: strawberry <strawberry@puppygock.gay>
amazing that this wasn't done in the first place. a lack of a UA is
suspicious and some providers may block our requests just for the sake
of no UA.
Signed-off-by: strawberry <strawberry@puppygock.gay>
the last endpoint is a non-standard health check endpoint used by at
least Element Web as a weird way to determine if syncv3 is available
there can also be some valid use-cases for serving well-knowns from the
application itself
Signed-off-by: strawberry <strawberry@puppygock.gay>
info is still logging useless database statistics and metrics that will
accumulate, less I/O activity.
Signed-off-by: strawberry <strawberry@puppygock.gay>
* feat: replaced flaky argon2 with better argon2 crate
* fix: applied cargo fmt nightly
* docs: added comment specifying what the settings for Argon2 mean
* fix: made hashing error a bit more descriptive
* fix: fixed incorrect value for Kib
this might be an unreachable state with the next commit, but let's be
extra sure here just in case a guest still managed to register before
a real admin was created.
Signed-off-by: strawberry <strawberry@puppygock.gay>
Also removed all instances of `#[command(verbatim_doc_comment)]` because
I'm pretty sure it's not necessary anymore. The `[commandbody]` things
were making rustdoc upset about broken link syntax. I also normalized
"code-block" to "code block" in that file since the latter appears more
often.
split out the spinning disk focused options into a configurable option, current
conduwuit users are NVMe/SSDs anyways so those options are just hindering performance.
rocksdb logging builds up overtime with no cleanup or anything, adds support for
configuring the amount of logging, size of files, log rotate, etc.
fixes https://gitlab.com/girlbossceo/conduwuit/-/issues/1
misc conduit logging improvements for help debugging issues and maybe a future feature
experimental Room V11 support from https://gitlab.com/famedly/conduit/-/merge_requests/562
Signed-off-by: strawberry <strawberry@puppygock.gay>
room IDs in conduit have a server name attached to them already
outputting the server name again just makes it look incorrect
also add codeblock to the internal room ID portion
Signed-off-by: strawberry <june@girlboss.ceo>
this project's codebase is so horrendous, im shocked that no one has ran
clippy at all. it had ~200 total lint warnings, some with performance
issues and unsoundness, and the rest just very ugly codebase. i have sat
down and fixed as many of these as possible and i am exhausted.
i haven't fixed some extremely complex ones, but i brought it down from
~200 to ~30.
i have also removed io_uring as a default feature due to it falling
under the same category as linux eBPF: major kernel attack surface for
minimal performance gains. this also makes it impossible to cross-compile
from macOS to Linux because io_uring does not exist in Darwin land.
there are far better ways to achieve better performance than io_uring on
the codebase level.
Signed-off-by: strawberry <june@girlboss.ceo>
iouring falls into the same category as eBPF and is a major source of kernel vulnerabilities. the benefits gained here are too minimal to bother keeping this enabled, and makes cross-compiling from macOS impossible
Signed-off-by: strawberry <june@girlboss.ceo>
can't update ruma to very latest commit because of the weird JsOption thing for syncv4 that i can't wrap my head around how to use, not important anyways
Signed-off-by: strawberry <strawberry@pupbrain.dev>
This speeds up handling of /_matrix/federation/v1/send/:transaction_id
when more than one event contains unknown keys.
In particular, when receiving multiple PDUs with dead servers in their
auth chain, timeouts of each server accumulate and can make handling of
incoming requests take several minutes, to the point the client closes
the connection (eg. matrix.org has a 2 minute timeout), causing new
events to be dropped eventually.
rust-rocksdb fork, add optional opt-in zstd response body compression
config, add webp support for images
cant upgrade things like axum, http, tower, hyper, etc due to the 1.0
release of http which are breaking all of these libs
Signed-off-by: strawberry <strawberry@pupbrain.dev>
the ones now used by default in rust-argon2 are extremely aggressive and
resource intensive. owasp's 2nd default is very reasonable with the same
security.
Signed-off-by: girlbossceo <june@girlboss.ceo>
causes tokio runtime worker to panic as we attempt to convert an HTTP
Request from ruma (try_into_http_request) into a reqwest Request so the
reqwest http client on the server can execute said request. error
message is not ideal ("invalid port number") but core issue is fixed.
Co-authored-by: infamous <ehuff007@gmail.com>
Signed-off-by: girlbossceo <june@girlboss.ceo>
even though these rooms would be quite broken as they typically would be
super super old rooms, people seem to still complain about conduit not
"supporting" room versions 1 and 2. to make them happy, just make them
as unstable so they can at least join it.
Signed-off-by: girlbossceo <june@girlboss.ceo>
this has been the new default since Matrix 1.6, and V10 has been flagged
as stable for a while in Conduit
Signed-off-by: girlbossceo <june@girlboss.ceo>
prevents errors like DB or I/O errors from leaking filesystem paths
Co-authored-by: infamous <ehuff007@gmail.com>
Signed-off-by: girlbossceo <june@girlboss.ceo>
These are room ACLs, not server ACLs. Causes confusion where people
think their Conduit homeserver was ACL'd. Print where these are coming from
in informational level.
Signed-off-by: girlbossceo <june@girlboss.ceo>
The `chown` command mentioned later in `DEPLOY.md` needs this group to
exist. Also make sure this account cannot be used to login with by
disabling its password and its shell.
This is similar to how the Debian `postinst` script does this.
It is common to have a markdown file per deployment subdirectory.
Still install it as `README.Debian` to `/usr/share/doc/matrix-conduit`
as per Debian policy.
Also update the link in the main `README.md` file.
1. Recommended GitLab's own registry over Docker Hub. (Reason: https://gitlab.com/famedly/conduit/-/merge_requests/492#note_1457220261)
2. Added the development image :next to the list of options.
3. Displayed text for Docker Hub now contains "docker.io" as part of the link for easier copy-paste for podman users. Clicking on the link still takes to the website.
`nixos-unstable` is the rolling release channel of NixOS. The default is
the master branch, which doesn't always have a populated binary cache
and so may result in compiling a bunch of stuff unnecessarily.
* Fix CI by explicitly setting hostname of docker in docker service
* Fix Docker build by bumping the Rust version to 1.69
* Fix cargo check in CI by bumping the Rust version to 1.69
add_header will not override the Content-Type header set by the server,
but will instead add another header below, which is obviously not ideal.
The proposed change will instead tell nginx to set the correct value for
this header straight away.
it convert :
- ".m.rules.call" to ".m.rule.call"
- ".m.rules.room_one_to_one" to ".m.rule.room_one_to_one"
- ".m.rules.encrypted_room_one_to_one" to ".m.rule.encrypted_room_one_to_one"
- ".m.rules.message" to ".m.rule.message"
- ".m.rules.encrypted" to ".m.rule.encrypted"
related to issue #264
Also add `.envrc` for direnv + Nix users. This makes developing locally
easier for us NixOS folks.
The flake itself will allow NixOS users to pull code directly from
Conduit's repository, making it completely trivial to stay up-to-date
with every commit.
I'd also like to add a NixOS module directly to this repository at some
point so that new configuration options will be available in the NixOS
module faster. But for now, NixOS users can simply override
`serivces.matrix-conduit.package` and get pretty much all the
functionality.
I've added myself to the `CODEOWNERS` file for the Nix files, since I am
willing to maintain this stuff. I use Conduit on NixOS so I'm personally
invested in having this work.
Lastly, `.gitignore` was updated to exclude symlinks created by `direnv`
and `nix build` and other such Nix commands.
This doesn't come without maintenance burden, however:
* The `sha256` in `flake.nix` will need to be updated whenever Conduit's
MSRV is updated, but that should be pretty infrequent.
* `nix flake update` should be run every so often to pull in updates to
`nixpkgs` and other flake inputs. I think downstream users can also
override this themselves with `inputs.<name>.inputs.<name>.follows`.
* `nix flake check` should be run in CI to ensure Nix builds keep
working.
* `nixpkgs-fmt --check $(fd '\.nix')` (or similar) should be run in CI
to ensure style uniformity.
Raise 404 "Room not found" when changing or accessing room visibility
settings (`GET` and `PUT
/_matrix/client/r0/directory/list/room/{roomId}`).
See issue #290
Use `leave_room` in `leave_all_rooms`
WIP: Add command to delete a list of users
also implements a flag to prevent the user from being removed from their joined rooms.
Report user deactivation failure reason
Don't send leave events by default when mass deactivating user accounts
Don't stop leaving rooms if an error was encountered
WIP: Rename command, make flags consistent, don't deactivate admin accounts.
Accounts should be deactivated as fast as possible and removing users from joined groups is completed afterwards.
Fix admin safety logic, improve command output
Continue leaving rooms if a room_id is invalid
Ignore errors from leave_room
Add notice to the list-local-users command
Output form list-local-users can be used directly without modification with the deactivate-all command
Only get mutex lock for admin room when sending message
This override was accidentally introduced by commit de6c331.
The Debian postinst script will ask for and generate a config with the
address set. This should not be overriden by what is set in the default
config and is thus a deviation from the standard docs.
Fix the scenario where a MembershipState change event was not sent to the server of a user kicked/banned from a room on a Conduit instance if there were not any other users from that server in the room.
The docs state that you need to make the config file _readable_
and then proceeds to make the file writable.
This changes it to make the file to be owned by root and readable by
anyone. This is the default for unix / linux and suggested practice
for files in /etc.
When a user mounts a volume into the default volume path,
it uses the permissions and ownership from the host volume.
In most cases, this is 1000:1000, which it also uses on the inside.
If you don't mount a volume though (e.g., for testing), conduit cries:
“The database couldn't be loaded or created.”
This fix chowns the default db dir to remedy this.
The text just sets the ownership and ignores that defaults on unix
are to have newly created dirs be readable by everyone.
This closes the database to unauthorized users on multi-user systems.
The nginx instance which is serving the .well-known endpoints can serve
the simple JSON replies directly from memory, instead of having them
as external files on disk.
some uiaa requests contain plaintext passwords which should never be
persisted to disk.
Currently there is no cleanup implemented (you have to restart conduit)
This moves compiler caching for incremental builds away from GitLab
caching the whole target/ folder to caching each code unit in S3.
This aleviates the need to zip and unzip and just caches on the fly.
This feature is optional and gated behind the SCCACHE_BIN_URL env
Remove device_one_time_keys_count from is_empty() sync checks, fixing sync issue as reported by Nekron
Closes#115
See merge request famedly/conduit!215
gcc-8-aarch64-linux-gnu is not available in debian 11 (which the rust image uses), so update to gcc-10
Signed-off-by: Jonas Zohren <git-pbkyr@jzohren.de>
- Build release builds for branches "master" and "next"
- Push docker images under different tags, depending on why the pipeline started
- branch master: push to `latest`
- branch next: push to `next`
- tag: push to `$TAG_NAME`
Signed-off-by: Jonas Zohren <git-pbkyr@jzohren.de>
Using `$CI_COMMIT_REF_NAME` means we get `master` for every image build,
which is not very useful/informative. Using `$CI_COMMIT_SHORT_SHA`,
on the other hand, makes it possible to see exactly from which commit an
image was built.
The average german man has a life expectancy of 78.7 years, or 689884.2 hours.
Assuming that Timo is 20 years old, he has rougly 514564.2 hours left on planet earth.
Also assuming that cross release builds took him 25 minutes before,
but 2-2.5x of that with the current release compilation config he wasted roughly an hour waiting for it to complete.
If he continued to work on Conduit for 20 more years (or 175320 hours),
and makes a release compilation about once per day, this means 7305 hours or 304 days wasted waiting for the rust compiler.
By cutting that back down to the original settings, he get's 182 days of his life back.
That's about 0.63% of his remaining life.
182 joyful days he can spend with family and loved ones.
1. The fallback text of the get_pdu admin room command response message
now contains the same text as the formatted_body content (namely, the
json instead of Debug-formatting of a serde type).
2. The formatted_body content of the get_pdu response is now
html-escaped.
The spec does not require servers to apply a room canonical alias event upon room creation (yet).
Still, synapse does that, since users can set their desired alias in Elements room creation dialog.
With this commit, conduit also sets that alias if it is a valid one.
This closes https://gitlab.com/famedly/conduit/-/issues/123
Testing needs compilation and is slow. Format checking is quick.
As format checking fails more often than tests,
switching them should result in faster failure and feedback
Conduit did not send the event in /sync because of a race condition.
There is a brief moment in time where Conduit accepted the event, but
did not store it yet. So when the client /syncs it updates the since
token without sending the event. I hope I fixed it by significantly
shortening the race-condition period.
Setting cargo to run incremental builds means partial build results should be cached. This is not enabled by default in release mode.
Incremental builds use 256 codegen units by default [1].
We set them to 16 (release default) again for somewhat faster code but slightly slower builds.
[1]: https://doc.rust-lang.org/cargo/reference/profiles.html#codegen-units
By default, jobs without tags only run on CI runners configured to do so [1].
Conduit can use famedly runners, which are more powerfull than gitlab's runners, but require a tag on the job to run it there.
This commit tags each job with the "docker" tag.
On the famedly/conduit repo this means faster CI.
On other gitlab.com forks the normal ci.
Selfhosted gitlab's might need to add a "docker" tag to their runner.
[1]: https://docs.gitlab.com/ee/ci/runners/configure_runners.html#use-tags-to-limit-the-number-of-jobs-using-the-runner
json_body is used in places that need authentication. In
case an unknown field is set, Ruma doesn't parse the field
and so doesn't give an error on invalid UTF-8. But Conduit
has parsed and on error makes json_body None. Return an
error to the client instead of generating an internal error.
- Dockerfile now tracks the gitlab repository and the master branch.
- docker-compose now points to conduit.toml instead of Rocket.toml and
its env vars were also renamed from ROCKET_ to CONDUIT_.
Furthermore vectorim/riot-web was changed to vectorim/element-web
I'm a bit torn on the "auth check based on the current state of the
room". It can mean multiple things:
1. The state of the room before the homeserver looked at the event at
all. But that means if a message event from a user arrives, but we
didn't see their join event before, we soft fail the message (even
though we would find the join event when going through the auth events
of the event and doing state res)
2. The state of the room after doing state-res with the event and our
previous room state. We need to do this state resolution to find the new
room state anyway, so we could just use the new room state for the auth
check. The problem is that if the incoming event is a membership leave
event, the new room state does not allow another leave event. This is
obviously the wrong option.
3. The state of the room after doing state-res with the state **before**
the event and our previous room state. This will mean a lot more
calculations because we have to run state-res again
We used 2. before and now use 1. again
- power level content override adds to the default event instead of
replacing it
- sending code refactored to make edus possible
- remove presence events when restarting conduit
- remove room_id field from read receipts over /sync
- handle incoming read receipts
- fix array bounds bug in server_server.rs
The final container tried to COPY from the default cargo target dir, but we only checked if cached_target is present.
We copy from cached_target to target when cached_target is present now in the builder
Ruma updated the event signing validation code and there was a dep
resolving failure with serde rocket and tokio so I updated rocket latest and
tokio 1.0 to fix.
As the docker ignore file includes the target dir, content in this dir
is no accessible to the docker daemon.
We circumvent this by providing the build artifact in a dir called
cached_dir
Ruma updated the event signing validation code and there was a dep
resolving failure with serde rocket and tokio so I updated rocket latest and
tokio 1.0 to fix.
The latest state-res crate uses ruma's PduRoomV3 PDU's which don't have
tuples of (EventId, EventHashs) like previous versions did (this was
left from rebasing onto master). The Media DB
now takes an optional content_type like the updated ruma structs.
* The Debian part will be generated and managed by Debconf and configure
homeserver name, address and port
* The local part will just be a config file that shows the other
configuration options
Added the address configuration and moved the config generation
from the config to the postinst script.
Most common filesystems limit paths to 255 bytes.
This change brings down the media ID length to be similar to
Synapse servers (25), and makes it possible for clients to
download media with the ID included in the filename.
The 404 error for /profile in the spec says "There is no profile
information for this user or this user does not exist.", but Element
assumes every 404 is a user that does not exist.
feat: first steps towards joining rooms over federation
Add state-res as a dependency of conduit
Add reverse_topological_power_sort before append_pdu
Implement statehashstatid_pduid tree for keeping track of state
Clean up implementation of state_hash as key for tracking state
Too many people fell into the trap and left conduit.rs in the deployment
configuration, wondering why users are all listed as @conduit.rs.
Make the default server name really easy to identify and make it obvious
that this needs to be changed.
Moved back to a fork of ruma with timo's key-backup and cross-signing
branch. Ephemeral events in sync responses are EphemeralRoomEventStub
(they also have no room_id like all of sync responses events)
When creating some responses (sync) an AnyRoomEventStub is needed for this
PduEvent will deserialize the JSON as a Stub event and a non stub event
when needed. Ephemeral and account events are checked to be the correct
type and filtered out if not. This requires an extra `deserialize` call
which could/should be removed.
TODO: Possibly get rid of EventJson in some places.
When I was investigating the DM issue, I found that join event contained
the `is_direct` flag. According to the spec, this flag should only be
set on "invite" events, but not on join.
Other homeservers do not have this flag on join.
On /sync, check if a room is a new join between `since` parameter and
now. If it's a newly joined room, set the limited flag to true, which will force
the client to load room messages via the `/messages` endpoint.
On `master`, I could not reproduce the messages not showing to others
when joining after being invited.
Fixes#39
small improvements
Cargo fmt
Simplify insert and update methods
Review feedback
Remove has_device method calls
Load all devices with a single db call
Remove device as in logout
Put all metadata on the same tree
Create userdevice key fucntion
Implement devices API
Implement all the devices endpoints. There's a couple of pending tasks:
- Integrate the "logout" logic once it lands to master (this should
remove the given device from the database).
- Track and store last seen timestamp and IP.
Co-authored-by: timokoesters <timo@koesters.xyz>
Co-authored-by: Guillem Nieto <gnieto.talo@gmail.com>
small improvements
Cargo fmt
Simplify insert and update methods
Review feedback
Remove has_device method calls
Load all devices with a single db call
Remove device as in logout
Put all metadata on the same tree
Create userdevice key fucntion
Implement devices API
Implement all the devices endpoints. There's a couple of pending tasks:
- Integrate the "logout" logic once it lands to master (this should
remove the given device from the database).
- Track and store last seen timestamp and IP.
Co-authored-by: timokoesters <timo@koesters.xyz>
Co-authored-by: Guillem Nieto <gnieto.talo@gmail.com>
Device_id was retrieved from the auth data instead of login's body and
this was causing that a new device was created on every login.
This is (I guess) provoking that some sytests are failing (for example,
"POST /login returns the same device_id as that in the request").
Use if let instead of unwrap
Default to invalid password if could not calculate
Move hash password methdo and return Result
Rename get_password method
Default to empty password when no pwd is received
Store hashed passwords
Store passwords hashed with Argon2 and verify password with that stored
hash.
Co-authored-by: Guillem Nieto <gnieto.talo@gmail.com>
Add a missing dot in the errormessage
Require mxc:// to be present at the start of an avatar_url
Update mxc check TODO
Show displayname or avatar_url if either is available when getting the profile Endpoint
Return the correct data in case of a empty displayname or an empty avatar_url
Took 50 minutes
Took 34 seconds
Add PUT and GET /_matrix/client/r0/profile/{userId}/displayname Endpoint
Add PUT and GET /_matrix/client/r0/profile/{userId}/avatar_url Endpoint
Add GET /_matrix/client/r0/profile/{userId} Endpoint
Took 2 hours 16 minutes
- if command -v nix > /dev/null; then echo "experimental-features = nix-command flakes" >> /etc/nix/nix.conf; fi
# Add conduwuit binary cache
- if command -v nix > /dev/null; then echo "extra-substituters = https://attic.kennel.juneis.dog/conduwuit" >> /etc/nix/nix.conf; fi
- if command -v nix > /dev/null; then echo "extra-trusted-public-keys = conduwuit:BbycGUgTISsltcmH0qNjFR9dbrQNYgdIAcmViSGoVTE=" >> /etc/nix/nix.conf; fi
- if command -v nix > /dev/null; then echo "extra-substituters = https://attic.kennel.juneis.dog/conduit" >> /etc/nix/nix.conf; fi
- if command -v nix > /dev/null; then echo "extra-trusted-public-keys = conduit:eEKoUwlQGDdYmAI/Q/0slVlegqh/QmAvQd7HBSm21Wk=" >> /etc/nix/nix.conf; fi
# Add alternate binary cache
- if command -v nix > /dev/null && [ -n "$ATTIC_ENDPOINT" ]; then echo "extra-substituters = $ATTIC_ENDPOINT" >> /etc/nix/nix.conf; fi
- if command -v nix > /dev/null && [ -n "$ATTIC_PUBLIC_KEY" ]; then echo "extra-trusted-public-keys = $ATTIC_PUBLIC_KEY" >> /etc/nix/nix.conf; fi
# Add Lix binary cache
- if command -v nix > /dev/null; then echo "extra-substituters = https://cache.lix.systems" >> /etc/nix/nix.conf; fi
- if command -v nix > /dev/null; then echo "extra-trusted-public-keys = cache.lix.systems:aBnZUw8zA7H35Cz2RyKFVs3H4PlGTLawyY5KRbvJR8o=" >> /etc/nix/nix.conf; fi
# Add crane binary cache
- if command -v nix > /dev/null; then echo "extra-substituters = https://crane.cachix.org" >> /etc/nix/nix.conf; fi
- if command -v nix > /dev/null; then echo "extra-trusted-public-keys = crane.cachix.org-1:8Scfpmn9w+hGdXH/Q9tTLiYAE/2dnJYRJP7kl80GuRk=" >> /etc/nix/nix.conf; fi
# Add nix-community binary cache
- if command -v nix > /dev/null; then echo "extra-substituters = https://nix-community.cachix.org" >> /etc/nix/nix.conf; fi
- if command -v nix > /dev/null; then echo "extra-trusted-public-keys = nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=" >> /etc/nix/nix.conf; fi
# Install direnv and nix-direnv
- if command -v nix > /dev/null; then nix-env -iA nixpkgs.direnv nixpkgs.nix-direnv; fi
# Allow .envrc
- if command -v nix > /dev/null; then direnv allow; fi
This page is for about contributing to conduwuit. The [development](docs/development.md) page may be of interest for you as well.
If you would like to work on an [issue][issues] that is not assigned, preferably ask in the Matrix room first at [#conduwuit:puppygock.gay][conduwuit-matrix], and comment on it.
### Linting and Formatting
It is mandatory all your changes satisfy the lints (clippy, rustc, rustdoc, etc) and your code is formatted via the **nightly**`cargo fmt`. A lot of the `rustfmt.toml` features depend on nightly toolchain. It would be ideal if they weren't nightly-exclusive features, but they currently still are. CI's rustfmt uses nightly.
If you need to allow a lint, please make sure it's either obvious as to why (e.g. clippy saying redundant clone but it's actually required) or it has a comment saying why. Do not write inefficient code for the sake of satisfying lints. If a lint is wrong and provides a more inefficient solution or suggestion, allow the lint and mention that in a comment.
### Running CI tests locally
conduwuit's CI for tests, linting, formatting, audit, etc use [`engage`][engage]. engage can be installed from nixpkgs or `cargo install engage`. conduwuit's Nix flake devshell has the nixpkgs engage with `direnv`. Use `engage --help` for more usage details.
To test, format, lint, etc that CI would do, install engage, allow the `.envrc` file using `direnv allow`, and run `engage`.
All of the tasks are defined at the [engage.toml][engage.toml] file. You can view all of them neatly by running `engage list`
If you would like to run only a specific engage task group, use `just`:
-`engage just <group>`
- Example: `engage just lints`
If you would like to run a specific engage task in a specific group, use `just <GROUP> [TASK]`: `engage just lints cargo-fmt`
The following binaries are used in [`engage.toml`][engage.toml]:
- [`engage`][engage]
-`nix`
- [`direnv`][direnv]
-`rustc`
-`cargo`
-`cargo-fmt`
-`rustdoc`
-`cargo-clippy`
- [`cargo-audit`][cargo-audit]
- [`cargo-deb`][cargo-deb]
- [`lychee`][lychee]
### Matrix tests
CI runs [Complement][complement], but currently does not fail if results from the checked-in results differ with the new results. If your changes are done to fix Matrix tests, note that in your pull request. If more Complement tests start failing from your changes, please review the logs (they are uploaded as artifacts) and determine if they're intended or not.
If you'd like to run Complement locally using Nix, see the [testing](docs/development/testing.md) page.
[Sytest][sytest] support will come soon.
### Writing documentation
conduwuit's website uses [`mdbook`][mdbook] and deployed via CI using GitHub Pages in the [`documentation.yml`][documentation.yml] workflow file with Nix's mdbook in the devshell. All documentation is in the `docs/` directory at the top level. The compiled mdbook website is also uploaded as an artifact.
To build the documentation using Nix, run: `bin/nix-build-and-cache just .#book`
The output of the mdbook generation is in `result/`. mdbooks can be opened in your browser from the individual HTML files without any web server needed.
### Inclusivity and Diversity
All **MUST** code and write with inclusivity and diversity in mind. See the [following page by Google on writing inclusive code and documentation](https://developers.google.com/style/inclusive-documentation).
This **EXPLICITLY** forbids usage of terms like "blacklist"/"whitelist" and "master"/"slave", [forbids gender-specific words and phrases](https://developers.google.com/style/pronouns#gender-neutral-pronouns), forbids ableist language like "sanity-check", "cripple", or "insane", and forbids culture-specific language (e.g. US-only holidays or cultures).
No exceptions are allowed. Dependencies that may use these terms are allowed but [do not replicate the name in your functions or variables](https://developers.google.com/style/inclusive-documentation#write-around).
In addition to language, write and code with the user experience in mind. This is software that intends to be used by everyone, so make it easy and comfortable for everyone to use. 🏳️⚧️
### Variable, comment, function, etc standards
Rust's default style and standards with regards to [function names, variable names, comments](https://rust-lang.github.io/api-guidelines/naming.html), etc applies here.
### Creating pull requests
Please try to keep contributions to the GitHub. While the mirrors of conduwuit allow for pull/merge requests, there is no guarantee I will see them in a timely manner. Additionally, please mark WIP or unfinished or incomplete PRs as drafts. This prevents me from having to ping once in a while to double check the status of it, especially when the CI completed successfully and everything so it *looks* done.
If you open a pull request on one of the mirrors, it is your responsibility to inform me about its existence. In the future I may try to solve this with more repo bots in the conduwuit Matrix room. There is no mailing list or email-patch support on the sr.ht mirror, but if you'd like to email me a git patch you can do so at `strawberry@puppygock.gay`.
Direct all PRs/MRs to the `main` branch.
By sending a pull request or patch, you are agreeing that your changes are allowed to be licenced under the Apache-2.0 licence and all of your conduct is in line with the Contributor's Covenant.
`main` / stable: [](https://github.com/girlbossceo/conduwuit/actions/workflows/ci.yml)
<!-- ANCHOR: catchphrase -->
### a very cool, featureful fork of [Conduit](https://conduit.rs/)
<!-- ANCHOR_END: catchphrase -->
Visit the [Conduwuit documentation](https://conduwuit.puppyirl.gay/) for more information.
<!-- ANCHOR: body -->
#### What is Matrix?
[Matrix](https://matrix.org) is an open network for secure and decentralized
communication. Users from every Matrix homeserver can chat with users from all
other Matrix servers. You can even use bridges (also called Matrix Appservices)
to communicate with users outside of Matrix, like a community on Discord.
#### What is the goal?
An efficient Matrix homeserver that's easy to set up and just works. You can install
it on a mini-computer like the Raspberry Pi to host Matrix for your family,
friends or company.
#### Can I try it out?
An official conduwuit server ran by me is available at transfem.dev ([element.transfem.dev](https://element.transfem.dev) / [cinny.transfem.dev](https://cinny.transfem.dev))
transfem.dev is a public homeserver that can be used, it is not a "test only homeserver". This means there are rules, so please read the rules: [https://transfem.dev/homeserver_rules.txt](https://transfem.dev/homeserver_rules.txt)
transfem.dev is also listed at [servers.joinmatrix.org](https://servers.joinmatrix.org/)
#### What is the current status?
conduwuit is a hard fork of Conduit which is in beta, meaning you can join and participate in most
Matrix rooms, but not all features are supported and you might run into bugs
from time to time.
<!-- ANCHOR_END: body -->
<!-- ANCHOR: footer -->
#### Contact
If you run into any question, feel free to
- Ask us in `#conduwuit:puppygock.gay` on Matrix
- [Open an issue on GitHub](https://github.com/girlbossceo/conduwuit/issues/new)
Original repo and Matrix room picture was from bran (<3). Current banner image and logo is directly from [this cohost post](https://cohost.org/RatBaby/post/1028290-finally-a-flag-for).
# let containers = podman ps -a --format json | from json | where Labels."com.docker.compose.project" == "jade-website"
# podman compose create; let containers = podman ps -a --format json | from json | where Labels."com.docker.compose.project" == "jade-website"; podman kube generate ($containers | get Id) | save deployment.yml
# Enables registration. If set to false, no users can register on this
# server.
# If set to true without a token configured, users can register with no form of 2nd-
# step only if you set
# `yes_i_am_very_very_sure_i_want_an_open_registration_server_prone_to_abuse` to
# true in your config. If you would like
# registration only via token reg, please configure the `registration_token` key.
allow_registration=false
# Please note that an open registration homeserver with no second-step verification
# is highly prone to abuse and potential defederation by homeservers, including
# matrix.org.
# A static registration token that new users will have to provide when creating
# an account. If unset and `allow_registration` is true, registration is open
# without any condition. YOU NEED TO EDIT THIS.
registration_token="change this token for something specific to your server"
# controls whether federation is allowed or not
# defaults to true
# allow_federation = true
# controls whether users are allowed to create rooms.
# appservices and admins are always allowed to create rooms
# defaults to true
# allow_room_creation = true
# controls whether non-admin local users are forbidden from sending room invites (local and remote),
# and if non-admin users can receive remote room invites. admins are always allowed to send and receive all room invites.
# defaults to false
# block_non_admin_invites = false
# List of forbidden username patterns/strings. Values in this list are matched as *contains*.
# This is checked upon username availability check, registration, and startup as warnings if any local users in your database
# have a forbidden username.
# No default.
# forbidden_usernames = []
# List of forbidden room aliases and room IDs as patterns/strings. Values in this list are matched as *contains*.
# This is checked upon room alias creation, custom room ID creation if used, and startup as warnings if any room aliases
# in your database have a forbidden room alias/ID.
# No default.
# forbidden_alias_names = []
# List of forbidden server names that we will block all client room joins, incoming federated room directory requests, incoming federated invites for, and incoming federated joins. This check is applied on the room ID, room alias, sender server name, and sender user's server name.
# Basically "global" ACLs. For our user (client) checks, admin users are allowed.
# No default.
# forbidden_remote_server_names = []
# List of forbidden server names that we will block all outgoing federated room directory requests for. Useful for preventing our users from wandering into bad servers or spaces.
# Set this to true to allow your server's public room directory to be federated.
# Set this to false to protect against /publicRooms spiders, but will forbid external users
# from viewing your server's public room directory. If federation is disabled entirely
# (`allow_federation`), this is inherently false.
allow_public_room_directory_over_federation=false
# Set this to true to allow your server's public room directory to be queried without client
# authentication (access token) through the Client APIs. Set this to false to protect against /publicRooms spiders.
allow_public_room_directory_without_auth=false
# Set this to true to lock down your server's public room directory and only allow admins to publish rooms to the room directory.
# Unpublishing is still allowed by all users with this enabled.
#
# Defaults to false
lockdown_public_room_directory=false
# Set this to true to allow federating device display names / allow external users to see your device display name.
# If federation is disabled entirely (`allow_federation`), this is inherently false. For privacy, this is best disabled.
allow_device_name_federation=false
# Vector list of domains allowed to send requests to for URL previews. Defaults to none.
# Note: this is a *contains* match, not an explicit match. Putting "google.com" will match "https://google.com" and "http://mymaliciousdomainexamplegoogle.com"
# Setting this to "*" will allow all URL previews. Please note that this opens up significant attack surface to your server, you are expected to be aware of the risks by doing so.
url_preview_domain_contains_allowlist=[]
# Vector list of explicit domains allowed to send requests to for URL previews. Defaults to none.
# Note: This is an *explicit* match, not a contains match. Putting "google.com" will match "https://google.com", "http://google.com", but not "https://mymaliciousdomainexamplegoogle.com"
# Setting this to "*" will allow all URL previews. Please note that this opens up significant attack surface to your server, you are expected to be aware of the risks by doing so.
url_preview_domain_explicit_allowlist=[]
# Vector list of URLs allowed to send requests to for URL previews. Defaults to none.
# Note that this is a *contains* match, not an explicit match. Putting "google.com" will match "https://google.com/", "https://google.com/url?q=https://mymaliciousdomainexample.com", and "https://mymaliciousdomainexample.com/hi/google.com"
# Setting this to "*" will allow all URL previews. Please note that this opens up significant attack surface to your server, you are expected to be aware of the risks by doing so.
url_preview_url_contains_allowlist=[]
# Vector list of explicit domains not allowed to send requests to for URL previews. Defaults to none.
# Note: This is an *explicit* match, not a contains match. Putting "google.com" will match "https://google.com", "http://google.com", but not "https://mymaliciousdomainexamplegoogle.com"
# The denylist is checked first before allowlist. Setting this to "*" will not do anything.
url_preview_domain_explicit_denylist=[]
# Maximum amount of bytes allowed in a URL preview body size when spidering. Defaults to 384KB (384_000 bytes)
url_preview_max_spider_size=384_000
# Option to decide whether you would like to run the domain allowlist checks (contains and explicit) on the root domain or not. Does not apply to URL contains allowlist. Defaults to false.
# Example: If this is enabled and you have "wikipedia.org" allowed in the explicit and/or contains domain allowlist, it will allow all subdomains under "wikipedia.org" such as "en.m.wikipedia.org" as the root domain is checked and matched.
# Useful if the domain contains allowlist is still too broad for you but you still want to allow all the subdomains under a root domain.
url_preview_check_root_domain=false
# Config option to allow or disallow incoming federation requests that obtain the profiles
# of our local users from `/_matrix/federation/v1/query/profile`
#
# This is inherently false if `allow_federation` is disabled
#
# Defaults to true
allow_profile_lookup_federation_requests=true
# Config option to automatically deactivate the account of any user who attempts to join a:
# - banned room
# - forbidden room alias
# - room alias or ID with a forbidden server name
#
# This may be useful if all your banned lists consist of toxic rooms or servers that no good faith user would ever attempt to join, and
# to automatically remediate the problem without any admin user intervention.
#
# This will also make the user leave all rooms. Federation (e.g. remote room invites) are ignored here.
#
# Defaults to false as rooms can be banned for non-moderation-related reasons
#auto_deactivate_banned_room_attempts = false
### Misc
# max log level for conduwuit. allows debug, info, warn, or error
# see also: https://docs.rs/tracing-subscriber/latest/tracing_subscriber/filter/struct.EnvFilter.html#directives
# **Caveat**:
# For release builds, the tracing crate is configured to only implement levels higher than error to avoid unnecessary overhead in the compiled binary from trace macros.
# For debug builds, this restriction is not applied.
#
# Defaults to "info"
#log = "info"
# controls whether encrypted rooms and events are allowed (default true)
#allow_encryption = false
# if enabled, conduwuit will send a simple GET request periodically to `https://pupbrain.dev/check-for-updates/stable`
# for any new announcements made. Despite the name, this is not an update check
# endpoint, it is simply an announcement check endpoint.
# Defaults to false.
#allow_check_for_updates = false
# Set to false to disable users from joining or creating room versions that aren't 100% officially supported by conduwuit.
# conduwuit officially supports room versions 6 - 10. conduwuit has experimental/unstable support for 3 - 5, and 11.
# Defaults to true.
#allow_unstable_room_versions = true
# Option to control adding arbitrary text to the end of the user's displayname upon registration with a space before the text.
# This was the lightning bolt emoji option, just replaced with support for adding your own custom text or emojis.
# To disable, set this to "" (an empty string)
# Defaults to "🏳️⚧️" (trans pride flag)
#new_user_displayname_suffix = "🏳️⚧️"
# Option to control whether conduwuit will query your list of trusted notary key servers (`trusted_servers`) for
# remote homeserver signing keys it doesn't know *first*, or query the individual servers first before falling back to the trusted
# key servers.
#
# The former/default behaviour makes federated/remote rooms joins generally faster because we're querying a single (or list of) server
# that we know works, is reasonably fast, and is reliable for just about all the homeserver signing keys in the room. Querying individual
# servers may take longer depending on the general infrastructure of everyone in there, how many dead servers there are, etc.
#
# However, this does create an increased reliance on one single or multiple large entities as `trusted_servers` should generally
# contain long-term and large servers who know a very large number of homeservers.
#
# If you don't know what any of this means, leave this and `trusted_servers` alone to their defaults.
#
# Defaults to true as this is the fastest option for federation.
#query_trusted_key_servers_first = true
# List/vector of room **IDs** that conduwuit will make newly registered users join.
# The room IDs specified must be rooms that you have joined at least once on the server, and must be public.
#
# No default.
#auto_join_rooms = []
# Retry failed and incomplete messages to remote servers immediately upon startup. This is called bursting.
# If this is disabled, said messages may not be delivered until more messages are queued for that server.
# Do not change this option unless server resources are extremely limited or the scale of the server's
# deployment is huge. Do not disable this unless you know what you are doing.
#startup_netburst = true
# Limit the startup netburst to the most recent (default: 50) messages queued for each remote server. All older
# messages are dropped and not reattempted. The `startup_netburst` option must be enabled for this value to have
# any effect. Do not change this value unless you know what you are doing. Set this value to -1 to reattempt
# every message without trimming the queues; this may consume significant disk. Set this value to 0 to drop all
# messages without any attempt at redelivery.
#startup_netburst_keep = 50
# If the 'perf_measurements' feature is enabled, enables collecting folded stack trace profile of tracing spans using
# tracing_flame. The resulting profile can be visualized with inferno[1], speedscope[2], or a number of other tools.
# [1]: https://github.com/jonhoo/inferno
# [2]: www.speedscope.app
# tracing_flame = false
# If 'tracing_flame' is enabled, sets a filter for which events will be included in the profile.
# Supported syntax is documented at https://docs.rs/tracing-subscriber/latest/tracing_subscriber/filter/struct.EnvFilter.html#directives
# tracing_flame_filter = "trace,h2=off"
# If 'tracing_flame' is enabled, set the path to write the generated profile.
# tracing_flame_output_path = "./tracing.folded"
### Generic database options
# Set this to any float value to multiply conduwuit's in-memory LRU caches with.
# May be useful if you have significant memory to spare to increase performance.
# Defaults to 1.0.
#conduit_cache_capacity_modifier = 1.0
# Set this to any float value in megabytes for conduwuit to tell the database engine that this much memory is available for database-related caches.
# May be useful if you have significant memory to spare to increase performance.
# Defaults to 256.0
#db_cache_capacity_mb = 256.0
# Interval in seconds when conduwuit will run database cleanup operations.
#
# For SQLite: this will flush the WAL by executing `PRAGMA wal_checkpoint(RESTART)` (https://www.sqlite.org/pragma.html#pragma_wal_checkpoint)
# For RocksDB: this will run `flush_opt` to flush database memtables to SST files on disk (https://docs.rs/rocksdb/latest/rocksdb/struct.DBCommon.html#method.flush_opt)
# These operations always run on shutdown.
#
# Defaults to 30 minutes (1800 seconds) to avoid IO amplification from too frequent cleanups
#cleanup_second_interval = 1800
### RocksDB options
# Set this to true to use RocksDB config options that are tailored to HDDs (slower device storage)
#
# It is worth noting that by default, conduwuit will use RocksDB with Direct IO enabled. *Generally* speaking this improves performance as it bypasses buffered I/O (system page cache).
# However there is a potential chance that Direct IO may cause issues with database operations if your setup is uncommon. This has been observed with FUSE filesystems, and possibly ZFS filesystem.
# RocksDB generally deals/corrects these issues but it cannot account for all setups.
# If you experience any weird RocksDB issues, try enabling this option as it turns off Direct IO and feel free to report in the conduwuit Matrix room if this option fixes your DB issues.
# See https://github.com/facebook/rocksdb/wiki/Direct-IO for more information.
#
# Defaults to false
#rocksdb_optimize_for_spinning_disks = false
# Enables direct-io to increase database performance. This is enabled by default. Set this option to false if the
# database resides on a filesystem which does not support direct-io.
#rocksdb_direct_io = true
# RocksDB log level. This is not the same as conduwuit's log level. This is the log level for the RocksDB engine/library
# which show up in your database folder/path as `LOG` files. Defaults to error. conduwuit will typically log RocksDB errors as normal.
#rocksdb_log_level = "error"
# Max RocksDB `LOG` file size before rotating in bytes. Defaults to 4MB.
#rocksdb_max_log_file_size = 4194304
# Time in seconds before RocksDB will forcibly rotate logs. Defaults to 0.
#rocksdb_log_time_to_roll = 0
# Amount of threads that RocksDB will use for parallelism on database operatons such as cleanup, sync, flush, compaction, etc. Set to 0 to use all your logical threads.
#
# Defaults to your CPU logical thread count.
#rocksdb_parallelism_threads = 0
# Enables idle IO priority for compaction thread. This prevents any unexpected lag in the server's operation and
# is usually a good idea. Enabled by default.
#rocksdb_compaction_ioprio_idle = true
# Enables idle CPU priority for compaction thread. This is not enabled by default to prevent compaction from
# falling too far behind on busy systems.
#rocksdb_compaction_prio_idle = false
# Maximum number of LOG files RocksDB will keep. This must *not* be set to 0. It must be at least 1.
# Defaults to 3 as these are not very useful.
#rocksdb_max_log_files = 3
# Type of RocksDB database compression to use.
# Available options are "zstd", "zlib", "bz2", "lz4", or "none"
# It is best to use ZSTD as an overall good balance between speed/performance, storage, IO amplification, and CPU usage.
# For more performance but less compression (more storage used) and less CPU usage, use LZ4.
# See https://github.com/facebook/rocksdb/wiki/Compression for more details.
#
# "none" will disable compression.
#
# Defaults to "zstd"
#rocksdb_compression_algo = "zstd"
# Level of compression the specified compression algorithm for RocksDB to use.
# Default is 32767, which is internally read by RocksDB as the default magic number and
# translated to the library's default compression level as they all differ.
# See their `kDefaultCompressionLevel`.
#
#rocksdb_compression_level = 32767
# Level of compression the specified compression algorithm for the bottommost level/data for RocksDB to use.
# Default is 32767, which is internally read by RocksDB as the default magic number and
# translated to the library's default compression level as they all differ.
# See their `kDefaultCompressionLevel`.
#
# Since this is the bottommost level (generally old and least used data), it may be desirable to have a very
# high compression level here as it's lesss likely for this data to be used. Research your chosen compression algorithm.
#
#rocksdb_bottommost_compression_level = 32767
# Whether to enable RocksDB "bottommost_compression".
# At the expense of more CPU usage, this will further compress the database to reduce more storage.
# It is recommended to use ZSTD compression with this for best compression results.
# See https://github.com/facebook/rocksdb/wiki/Compression for more details.
#
# Defaults to false as this uses more CPU when compressing.
# Defaults to 35 seconds as generally appservices are hosted within the same network
#appservice_timeout = 35
# Appservice URL idle connection pool timeout
#
# Defaults to 300 seconds
#appservice_idle_timeout = 300
# Notification gateway pusher idle connection pool timeout
#
# Defaults to 15 seconds
#pusher_idle_timeout = 15
### Presence / Typing Indicators / Read Receipts
# Config option to control local (your server only) presence updates/requests. Defaults to true.
# Note that presence on conduwuit is very fast unlike Synapse's.
# If using outgoing presence, this MUST be enabled.
#
#allow_local_presence = true
# Config option to control incoming federated presence updates/requests. Defaults to true.
# This option receives presence updates from other servers, but does not send any unless `allow_outgoing_presence` is true.
# Note that presence on conduwuit is very fast unlike Synapse's.
#
#allow_incoming_presence = true
# Config option to control outgoing presence updates/requests. Defaults to true.
# This option sends presence updates to other servers, but does not receive any unless `allow_incoming_presence` is true.
# Note that presence on conduwuit is very fast unlike Synapse's.
# If using outgoing presence, you MUST enable `allow_local_presence` as well.
#
#allow_outgoing_presence = true
# Config option to enable the presence idle timer for remote users. Disabling is offered as an optimization for
# servers participating in many large rooms or when resources are limited. Disabling it may cause incorrect
# presence states (i.e. stuck online) to be seen for some remote users. Defaults to true.
#presence_timeout_remote_users = true
# Config option to control how many seconds before presence updates that you are idle. Defaults to 5 minutes.
#presence_idle_timeout_s = 300
# Config option to control how many seconds before presence updates that you are offline. Defaults to 30 minutes.
#presence_offline_timeout_s = 1800
# Config option to control whether we should receive remote incoming read receipts.
# Defaults to true.
#allow_incoming_read_receipts = true
# Config option to control whether we should send read receipts to remote servers.
# Defaults to true.
#allow_outgoing_read_receipts = true
# Config option to control outgoing typing updates to federation. Defaults to true.
#allow_outgoing_typing = true
# Config option to control incoming typing updates from federation. Defaults to true.
#allow_incoming_typing = true
# Config option to control maximum time federation user can indicate typing.
#typing_federation_timeout_s = 30
# Config option to control minimum time local client can indicate typing. This does not override
# a client's request to stop typing. It only enforces a minimum value in case of no stop request.
#typing_client_timeout_min_s = 15
# Config option to control maximum time local client can indicate typing.
#typing_client_timeout_max_s = 45
# Other options not in [global]:
#
#
# Enables running conduwuit with direct TLS support
# It is strongly recommended you use a reverse proxy instead. This is primarily relevant for test suites like complement that require a private CA setup.
# [global.tls]
# certs = "/path/to/my/certificate.crt"
# key = "/path/to/my/private_key.key"
#
# Whether to listen and allow for HTTP and HTTPS connections (insecure!)
# This config option is only available if conduwuit was built with `axum_dual_protocol` feature (not default feature)
# Defaults to false
#dual_protocol = false
# If you are using delegation via well-known files and you cannot serve them from your reverse proxy, you can
# uncomment these to serve them directly from conduwuit. This requires proxying all requests to conduwuit, not just `/_matrix` to work.
#
#[global.well_known]
#server = "matrix.example.com:443"
#client = "https://matrix.example.com"
#
# A single contact and/or support page for /.well-known/matrix/support
# All options here are strings. Currently only supports 1 single contact.
Information about downloading and deploying the Debian package. This may also be referenced for other `apt`-based distros such as Ubuntu.
### Installation
It is recommended to see the [generic deployment guide](../deploying/generic.md) for further information if needed as usage of the Debian package is generally related.
### Configuration
When installed, the example config is placed at `/etc/conduwuit/conduwuit.toml` as the default config. At the minimum, you will need to change your `server_name` here.
You can tweak more detailed settings by uncommenting and setting the config options
in `/etc/conduwuit/conduwuit.toml`.
### Running
The package uses the [`conduwuit.service`](../configuration.md#example-systemd-unit-file) systemd unit file to start and stop conduwuit. The binary is installed at `/usr/sbin/conduwuit`.
This package assumes by default that conduwuit will be placed behind a reverse proxy. The default config options apply (listening on `localhost` and TCP port `6167`). Matrix federation requires a valid domain name and TLS, so you will need to set up TLS certificates and renewal for it to work properly if you intend to federate.
Consult various online documentation and guides on setting up a reverse proxy and TLS. Caddy is documented at the [generic deployment guide](../deploying/generic.md#setting-up-the-reverse-proxy) as it's the easiest and most user friendly.
If you run into any problems while setting up an Appservice: ask us in [#conduwuit:puppygock.gay](https://matrix.to/#/#conduwuit:puppygock.gay) or [open an issue on GitHub](https://github.com/girlbossceo/conduwuit/issues/new).
## Set up the appservice - general instructions
Follow whatever instructions are given by the appservice. This usually includes
downloading, changing its config (setting domain, homeserver url, port etc.)
and later starting it.
At some point the appservice guide should ask you to add a registration yaml
file to the homeserver. In Synapse you would do this by adding the path to the
homeserver.yaml, but in Conduit you can do this from within Matrix:
First, go into the #admins room of your homeserver. The first person that
registered on the homeserver automatically joins it. Then send a message into
the room like this:
@conduit:your.server.name: register-appservice
```
paste
the
contents
of
the
yaml
registration
here
```
You can confirm it worked by sending a message like this:
`@conduit:your.server.name: appservices list`
The `@conduit` bot should answer with `Appservices (1): your-bridge`
Then you are done. Conduit will send messages to the appservices and the
appservice can send requests to the homeserver. You don't need to restart
Conduit, but if it doesn't work, restarting while the appservice is running
could help.
## Appservice-specific instructions
### Remove an appservice
To remove an appservice go to your admin room and execute
Currently conduwuit is only on the Arch User Repository (AUR).
The conduwuit AUR packages are community maintained and are not maintained by conduwuit development team, but the AUR package maintainers are in the Matrix room. Please attempt to verify your AUR package's PKGBUILD file looks fine before asking for support.
When you have the image you can simply run it with
```bash
docker run -d -p 8448:6167 \
-v db:/var/lib/conduwuit/ \
-e CONDUIT_SERVER_NAME="your.server.name"\
-e CONDUIT_DATABASE_BACKEND="rocksdb"\
-e CONDUIT_ALLOW_REGISTRATION=false\
-e CONDUIT_ALLOW_FEDERATION=true\
-e CONDUIT_MAX_REQUEST_SIZE="40000000"\
-e CONDUIT_TRUSTED_SERVERS="[\"matrix.org\"]"\
-e CONDUIT_LOG="warn,ruma_state_res=warn"\
--name conduit <link>
```
or you can use [docker compose](#docker-compose).
The `-d` flag lets the container run in detached mode. You may supply an optional `conduwuit.toml` config file, the example config can be found [here](../configuration.md).
You can pass in different env vars to change config values on the fly. You can even configure conduwuit completely by using env vars. For an overview of possible
values, please take a look at the [`docker-compose.yml`](docker-compose.yml) file.
If you just want to test conduwuit for a short time, you can use the `--rm` flag, which will clean up everything related to your container after you stop it.
### Docker-compose
If the `docker run` command is not for you or your setup, you can also use one of the provided `docker-compose` files.
Depending on your proxy setup, you can use one of the following files;
- If you already have a `traefik` instance set up, use [`docker-compose.for-traefik.yml`](docker-compose.for-traefik.yml)
- If you don't have a `traefik` instance set up (or any other reverse proxy), use [`docker-compose.with-traefik.yml`](docker-compose.with-traefik.yml)
- For any other reverse proxy, use [`docker-compose.yml`](docker-compose.yml)
When picking the traefik-related compose file, rename it so it matches `docker-compose.yml`, and
rename the override file to `docker-compose.override.yml`. Edit the latter with the values you want
for your server.
Additional info about deploying conduwuit can be found [here](generic.md).
### Build
To build the conduwuit image with docker-compose, you first need to open and modify the `docker-compose.yml` file. There you need to comment the `image:` option and uncomment the `build:` option. Then call docker compose with:
```bash
docker compose up
```
This will also start the container right afterwards, so if want it to run in detached mode, you also should use the `-d` flag.
### Run
If you already have built the image or want to use one from the registries, you can just start the container and everything else in the compose file in detached mode with:
```bash
docker compose up -d
```
> **Note:** Don't forget to modify and adjust the compose file to your needs.
### Use Traefik as Proxy
As a container user, you probably know about Traefik. It is a easy to use reverse proxy for making
containerized app and services available through the web. With the two provided files,
> If you run into any problems while setting up conduwuit, ask us
> in `#conduwuit:puppygock.gay` or [open an issue on GitHub](https://github.com/girlbossceo/conduwuit/issues/new).
## Installing conduwuit
You may simply download the binary that fits your machine. Run `uname -m` to see what you need.
Prebuilt binaries can be downloaded from the latest tagged release [here](https://github.com/girlbossceo/conduwuit/releases/latest).
The latest tagged release also includes the Debian packages.
Alternatively, you may compile the binary yourself. We recommend using [Lix](https://lix.systems) to build conduwuit as this has the most guaranteed
reproducibiltiy and easiest to get a build environment and output going.
Otherwise, follow standard Rust project build guides (installing git and cloning the repo, getting the Rust toolchain via rustup, installing LLVM toolchain + libclang, installing liburing for io_uring and RocksDB, etc).
## Adding a conduwuit user
While conduwuit can run as any user it is better to use dedicated users for different services. This also allows
you to make sure that the file permissions are correctly set up.
In Debian or RHEL, you can use this command to create a conduwuit user:
conduwuit uses the ports 443 and 8448 both of which need to be open in the firewall.
If conduwuit runs behind a router or in a container and has a different public IP address than the host system these public ports need to be forwarded directly or indirectly to the port mentioned in the config.
## Setting up a systemd service
The systemd unit for conduwuit can be found [here](../configuration.md#example-systemd-unit-file). You may need to change the `ExecStart=` path to where you placed the conduwuit binary.
## Creating the conduwuit configuration file
Now we need to create the conduwuit's config file in `/etc/conduwuit/conduwuit.toml`. The example config can be found at [conduwuit-example.toml](../configuration.md).**Please take a moment to read it. You need to change at least the server name.**
RocksDB (`rocksdb`) is the only supported database backend. SQLite only exists for historical reasons and is not recommended. Any performance issues, storage issues, database issues, etc will not be assisted if using SQLite and you will be asked to migrate to RocksDB first.
## Setting the correct file permissions
If you are using a dedicated user for conduwuit, you will need to allow it to read the config. To do that you can run this command on
Debian or RHEL:
```bash
sudo chown -R root:root /etc/conduwuit
sudo chmod 755 /etc/conduwuit
```
If you use the default database path you also need to run this:
Refer to the documentation or various guides online of your chosen reverse proxy software. A [Caddy](https://caddyserver.com/) example will be provided as this is the recommended reverse proxy for new users and is very trivial to use (handles TLS, reverse proxy headers, etc transparently with proper defaults).
### Caddy
Create `/etc/caddy/conf.d/conduwuit_caddyfile` and enter this (substitute for your server name).
```caddy
your.server.name, your.server.name:8448{
# TCP
reverse_proxy127.0.0.1:6167
# UNIX socket
#reverse_proxy unix//run/conduwuit/conduwuit.sock
}
```
That's it! Just start and enable the service and you're set.
```bash
$ sudo systemctl enable --now caddy
```
## You're done!
Now you can start conduwuit with:
```bash
$ sudo systemctl start conduwuit
```
Set it to start automatically when your system boots with:
```bash
$ sudo systemctl enable conduwuit
```
## How do I know it works?
You can open [a Matrix client](https://matrix.org/ecosystem/clients), enter your homeserver and try to register.
You can also use these commands as a quick health check.
When developing in debug-builds with the nightly toolchain, conduwuit is modular using dynamic libraries and various parts of the application are hot-reloadable while the server is running: http api handlers, admin commands, services, database, etc. These are all split up into individual workspace crates as seen in the `src/` directory. Changes to sourcecode in a crate rebuild that crate and subsequent crates depending on it. Reloading then occurs for the changed crates.
Release builds still produce static binaries which are unaffected. Rust's soundness guarantees are in full force. Thus you cannot hot-reload release binaries.
### Requirements
Currently, this development setup only works on x86_64 and aarch64 Linux glibc. [musl explicitly does not support hot reloadable libraries, and does not implement `dlclose`][2]. macOS does not fully support our usage of `RTLD_GLOBAL` possibly due to some thread-local issues. [This Rust issue][3] may be of relevance, specifically [this comment][4]. It may be possible to get it working on only very modern macOS versions such as at least Sonoma, as currently loading dylibs is supported, but not unloading them in our setup, and the cited comment mentions an Apple WWDC confirming there have been TLS changes to somewhat make this possible.
As mentioned above this requires the nightly toolchain. This is due to reliance on various Cargo.toml features that are only available on nightly, most specifically `RUSTFLAGS` in Cargo.toml. Some of the implementation could also be simpler based on other various nightly features. We hope lots of nightly features start making it out of nightly sooner as there have been dozens of very helpful features that have been stuck in nightly ("unstable") for at least 5+ years that would make this simpler. We encourage greater community consensus to move these features into stability.
This currently only works on x86_64/aarch64 Linux with a glibc C library. musl C library, macOS, and likely other host architectures are not supported (if other architectures work, feel free to let us know and/or make a PR updating this). This should work on GNU ld and lld (rust-lld) and gcc/clang, however if you happen to have linker issues it's recommended to try using `mold` or `gold` linkers, and please let us know in the [conduwuit Matrix room][7] the linker error and what linker solved this issue so we can figure out a solution. Ideally there should be minimal friction to using this, and in the future a build script (`build.rs`) may be suitable to making this easier to use if the capabilities allow us.
### Usage
As of 19 May 2024, the instructions for using this are:
0. Have patience. Don't hesitate to join the [conduwuit Matrix room][7] to receive help using this. As indicated by the various rustflags used and some of the interesting issues linked at the bottom, this is definitely not something the Rust ecosystem or toolchain is used to doing.
1. Install the nightly toolchain using rustup. You may need to use `rustup override set nightly` in your local conduwuit directory, or use `cargo +nightly` for all actions.
2. Uncomment `cargo-features` at the top level / root Cargo.toml
3. Scroll down to the `# Developer profile` section and uncomment ALL the rustflags for each dev profile and their respective packages.
4. In each workspace crate's Cargo.toml (everything under `src/*` AND `deps/rust-rocksdb/Cargo.toml`), uncomment the `dylib` crate type under `[lib]`.
5. Due to [this rpath issue][5], you must export the `LD_LIBRARY_PATH` environment variable to your nightly Rust toolchain library directory. If using rustup (hopefully), use this: `export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/`
6. Start the server. You can use `cargo +nightly run` for this along with the standard.
7. Make some changes where you need to.
8. In a separate terminal window in the same directory (or using a terminal multiplexer like tmux), run the *build* Cargo command `cargo +nightly build`. Cargo should only rebuild what was changed / what's necessary, so it should not be rebuilding all the crates.
9. In your conduwuit server terminal, hit/send `CTRL+C` signal. This will tell conduwuit to find which libraries need to be reloaded, and reloads them as necessary.
10. If there were no errors, it will tell you it successfully reloaded `#` modules, and your changes should now be visible. Repeat 7 - 9 as needed.
To shutdown conduwuit in this setup, hit/send `CTRL+\`. Normal builds still shutdown with `CTRL+C` as usual.
Steps 1 - 5 are the initial first-time steps for using this. To remove the hot reload setup, revert/comment all the Cargo.toml changes.
As mentioned in the requirements section, if you happen to have some linker issues, try using the `-fuse-ld=` rustflag and specify mold or gold in all the `rustflags` definitions in the top level Cargo.toml, and please let us know in the [conduwuit Matrix room][7] the problem. mold can be installed typically through your distro, and gold is provided by the binutils package.
It's possible a helper script can be made to do all of this, or most preferably a specially made build script (build.rs). `cargo watch` support will be implemented soon which will eliminate the need to manually run `cargo build` all together.
### Addendum
Conduit was inherited as a single crate without modularity or reloading in its design. Reasonable partitioning and abstraction allowed a split into several crates, though many circular dependencies had to be corrected. The resulting crates now form a directed graph as depicted in figures below. The interfacing between these crates is still extremely broad which is not mitigable.
Initially [hot_lib_reload][6] was investigated but found appropriate for a project designed with modularity through limited interfaces, not a large and complex existing codebase. Instead a bespoke solution built directly on [libloading][8] satisfied our constraints. This required relatively minimal modifications and zero maintenance burden compared to what would be required otherwise. The technical difference lies with relocation processing: we leverage global bindings (`RTLD_GLOBAL`) in a very intentional way. Most libraries and off-the-shelf module systems (such as [hot_lib_reload][6]) restrict themselves to local bindings (`RTLD_LOCAL`). This allows them to release software to multiple platforms with much greater consistency, but at the cost of burdening applications to explicitly manage these bindings. In our case with an optional feature for developers, we shrug any such requirement to enjoy the cost/benefit on platforms where global relocations are properly cooperative.
To make use of `RTLD_GLOBAL` the application has to be oriented as a directed acyclic graph. The primary rule is simple and illustrated in the figure below: **no crate is allowed to call a function or use a variable from a crate below it.**
When a symbol is referenced between crates they become bound: **crates cannot be unloaded until their calling crates are first unloaded.** Thus we start the reloading process from the crate which has no callers. There is a small problem though: the first crate is called by the base executable itself! This is solved by using an `RTLD_LOCAL` binding for just one link between the main executable and the first crate, freeing the executable from all modules as no global binding ever occurs between them.
Proper resource management is essential for reliable reloading to occur. This is a very basic ask in RAII-idiomatic Rust and the exposure to reloading hazards is remarkably low, generally stemming from poor patterns and practices. Unfortunately static analysis doesn't enforce reload-safety programmatically (though it could one day), for now hazards can be avoided by knowing a few basic do's and dont's:
1. Understand that code is memory. Just like one is forbidden from referencing free'd memory, one must not transfer control to free'd code. Exposure to this is primarily from two things:
- Callbacks, which this project makes very little use of.
- Async tasks, which are addressed below.
2. Tie all resources to a scope or object lifetime with greatest possible symmetry (locality). For our purposes this applies to code resources, which means async blocks and tokio tasks.
- **Never spawn a task without receiving and storing its JoinHandle**.
- **Always wait on join handles** before leaving a scope or in another cleanup function called by an owning scope.
3. Know any minor specific quirks documented in code or here:
- Don't use `tokio::spawn`, instead use our `Handle` in `core/server.rs`, which is reachable in most of the codebase via `services()` or other state. This is due to some bugs or assumptions made in tokio, as it happens in `unsafe {}` blocks, which are mitigated by circumventing some thread-local variables. Using runtime handles is good practice in any case.
The initial implementation PR is available [here][1].
### Interesting related issues/bugs
- [DT_RUNPATH produced in binary with rpath = true is wrong (cargo)][5]
- [Disabling MIR Optimization in Rust Compilation (cargo)](https://internals.rust-lang.org/t/disabling-mir-optimization-in-rust-compilation/19066/5)
#### **Note: This list may not up to date. There are rapidly more and more improvements, fixes, changes, etc being made that it is becoming more difficult to maintain this list. I recommend that you give conduwuit a try and see the differences for yourself. If you have any concerns, feel free to join the conduwuit Matrix room and ask any pre-usage questions.**
### list of features, bug fixes, etc that conduwuit does that Conduit does not:
- Concurrency support for key fetching for faster remote room joins and room joins that will error less frequently
- Send `Cache-Control` response header with `immutable` and 1 year cache length for all media requests (download and thumbnail) to instruct clients to cache media, and reduce server load from media requests that could be otherwise cached
- Add feature flags and config options to enable/build with zstd, brotli, and/or gzip HTTP body compression (response and request)
- Eliminate all usage of the thread-blocking `getaddrinfo(3)` call upon DNS queries, significantly improving federation latency/ping and cache DNS results (NXDOMAINs, successful queries, etc) using hickory-dns / hickory-resolver
- Vastly improve RocksDB default settings to use new features that help with performance significantly, uses settings tailored to SSDs, various ways to tweak RocksDB, and a conduwuit setting to tell RocksDB to use settings that are tailored to HDDs or slow spinning rust storage or buggy filesystems.
- Add a Cargo build profile for aggressive build-time performance optimisations for release builds (1 codegen unit, no debug, fat LTO, etc, and optimise all crates with same)
- Implement database flush and cleanup conduwuit operations when using RocksDB
- Implement RocksDB write buffer corking and coalescing in database write-heavy areas
- Perform connection pooling and keepalives where necessary to significantly improve federation performance and latency
- Various config options to tweak connection pooling, request timeouts, connection timeouts, DNS timeouts and settings, etc with good defaults which also help huge with performance via reusing connections and retrying where needed
- Implement building conduwuit with jemalloc (which extends to the RocksDB jemalloc feature for maximum gains) or hardened_malloc light variant, and produce CI builds with jemalloc for performance (Nix doesn't seem to build [hardened_malloc-rs](https://github.com/girlbossceo/hardened_malloc-rs) properly)
- Add support for caching DNS results with hickory-dns / hickory-resolver in conduwuit (not a replacement for a proper resolver cache, but still far better than nothing)
- Add config option for using DNS over TCP, and config option for controlling A/AAAA record lookup strategy (e.g. don't query AAAA records if you don't have IPv6 connectivity)
- Overall significant database, Client-Server, and federation performance and latency improvements (check out the ping room leaderboards if you don't believe me :>)
- Add config options for RocksDB compression and bottommost compression, including choosing the algorithm and compression level
- Use [loole](https://github.com/mahdi-shojaee/loole) MPSC channels instead of tokio MPSC channels for huge performance boosts in sending channels (mainly relevant for federation) and presence channels
- Use `tracing`/`log`'s `release_max_level_info` feature to improve performance, build speeds, binary size, and CPU usage in release builds by avoid compiling debug/trace log level macros that users will generally never use (can be disabled with a build-time feature flag)
- Enable RocksDB async read I/O via `io_uring` by default
## General Fixes:
- Raise and improve all the various request timeouts making some things like room joins and client bugs error less or none at all than they should, and make them all user configurable
- Add missing `reason` field to user ban events (`/ban`)
- Fixed spec compliance issue with room version 8 - 11 joins (https://github.com/matrix-org/synapse/issues/16717 / https://github.com/matrix-org/matrix-spec/issues/1708)
- Safer and cleaner shutdowns on both database side as we run cleanup on shutdown and exits database loop better (no potential hanging issues in database loop), overall cleaner shutdown logic
- Stop sending `make_join` requests on room joins if 15 servers respond with `M_UNSUPPORTED_ROOM_VERSION` or `M_INVALID_ROOM_VERSION`
- Stop sending `make_join` requests if 50 servers cannot provide `make_join` for us
- Respect *most* client parameters for `/media/` requests (`allow_redirect` still needs work)
- Increased graceful shutdown timeout from a low 60 seconds to 180 seconds to avoid killing connections and let the remaining ones finish processing
- Return joined member count of rooms for push rules/conditions instead of a hardcoded value of 10
- Make `CONDUIT_CONFIG` optional, relevant for container users that configure only by environment variables and no longer need to set `CONDUIT_CONFIG` to an empty string.
- Allow HEAD and PATCH (MSC4138) HTTP requests in CORS for clients (despite not being explicity mentioned in Matrix spec, HTTP spec says all HEAD requests need to behave the same as GET requests, Synapse supports HEAD requests)
- Resolve and remove some "features" from upstream that result in concurrency hazards, exponential backoff issues, or arbitrary performance limiters
- Find more servers for outbound federation `/hierarchy` requests instead of just the room ID server name
- Support for suggesting servers to join through at `/_matrix/client/v3/directory/room/{roomAlias}`
- Support for suggesting servers to join through us at `/_matrix/federation/v1/query/directory`
- Add workaround for [Out Of Your Element](https://gitdab.com/cadence/out-of-your-element) appservice bridge to make it functional on conduwuit (bug has already been reported)
## Moderation:
- (Also see [Admin Room](#admin-room) for all the admin commands pertaining to moderation, there's a lot!)
- Add support for room banning/blocking by ID using admin command
- Add support for serving `support` well-known from `[well_known.support]` (MSC1929)
- Config option to forbid publishing rooms to the room directory (`lockdown_public_room_directory`) except for admins
- Admin commands to delete room aliases and unpublish rooms from our room directory
- For all [`/report`](https://spec.matrix.org/v1.9/client-server-api/#post_matrixclientv3roomsroomidreporteventid) requests: check if the reported event ID belongs to the reported room ID, raise report reasoning character limit to 750, fix broken formatting, make a small delayed random response per spec suggestion on privacy, and check if the sender user is in the reported room.
- Support blocking servers from downloading remote media from, returning a 404
- Don't allow `m.call.invite` events to be sent in public rooms (prevents calling the entire room)
- On new public room creations, only allow moderators to send `m.call.invite`, `org.matrix.msc3401.call`, and `org.matrix.msc3401.call.member` events
- Add support for a "global ACLs" feature (`forbidden_remote_server_names`) that blocks inbound remote room invites, room joins by room ID on server name, room joins by room alias on server name, incoming federated joins, and incoming federated room directory requests. This is very helpful for blocking servers that are purely toxic/bad and serve no value in allowing our users to suffer from things like room invite spam or such. Please note that this is not a substitute for room ACLs.
- Add support for a config option to forbid our local users from sending federated room directory requests for (`forbidden_remote_room_directory_server_names`). Similar to above, useful for blocking servers that help prevent our users from wandering into bad areas of Matrix via room directories of those malicious servers.
- Add config option for auto remediating/deactivating local non-admin users who attempt to join bad/forbidden rooms (`auto_deactivate_banned_room_attempts`)
## Privacy/Security:
- Add config option for device name federation with a privacy-friendly default (disabled)
- Add config option for requiring authentication to the `/publicRooms` endpoint (room directory) with a default enabled for privacy
- Add config option for federating `/publicRooms` endpoint (room directory) to other servers with a default disabled for privacy
- Uses proper `argon2` crate by RustCrypto instead of questionable `rust-argon2` crate
- Generate passwords with 25 characters instead of 15
- Config option `ip_range_denylist` to support refusing to send requests (typically federation) to specific IP ranges, typically RFC 1918, non-routable, testnet, etc addresses like Synapse for security (note: this is not a guaranteed protection, and you should be using a firewall with zones if you want guaranteed protection as doing this on the application level is prone to bypasses).
- Config option to block non-admin users from sending room invites or receiving remote room invites. Admin users are still allowed.
- Config option to disable incoming, outgoing, and/or local presence
- Sanitise file names for the `Content-Disposition` header for all media requests (thumbnails, downloads, uploads)
- Return `inline` or `attachment` based on the detected file MIME type for the `Content-Disposition` and only allow images/videos/text/audio to be `inline`
- Send secure default HTTP headers such as a strong restrictive CSP, deny iframes, disable `X-XSS-Protection`, disable interest cohort in `Permission-Policy`, etc to mitigate any potential attack surface such as from untrusted media
## Administration/Logging:
- Commandline argument to specify the path to a config file instead of relying on `CONDUIT_CONFIG`
- Revamped admin room infrastructure and commands
- Substantially clean up, improve, and fix logging (less noisy dead server logging, registration attempts, more useful troubleshooting logging, proper error propagation, etc)
- Configurable RocksDB logging (`LOG` files) with proper defaults (rotate, max size, verbosity, etc) to stop LOG files from accumulating so much
- Explicit startup error if your configuration allows open registration without a token or such like Synapse with a way to bypass it if needed
- Replace the lightning bolt emoji option with support for setting any arbitrary text (e.g. another emoji) to suffix to all new user registrations, with a conduwuit default of 🏳️⚧️
- Implement config option to auto join rooms upon registration
- Warn on unknown config options specified
- Add `/_conduwuit/server_version` route to return the version of conduwuit without relying on the federation API `/_matrix/federation/v1/version`
- Add configurable RocksDB recovery modes to aid in recovering corrupted RocksDB databases
- Support config options via `CONDUWUIT_` prefix and accessing non-global struct config options with the `__` split (e.g. `CONDUWUIT_WELL_KNOWN__SERVER`)
- Add support for listening on multiple TCP ports
- Disable update check by default as it's not useful for conduwuit
- **Opt-in** Sentry.io telemetry and metrics, mainly used for crash reporting
## Maintenance/Stability:
- GitLab CI ported to GitHub Actions
- Repo is mirrored to GitHub, GitLab, git.gay, git.girlcock.ceo, sourcehut, and Codeberg (see README.md for their links)
- Docker container images published to GitLab Container Registry, GitHub Container Registry, and Dockerhub
- Extensively revamp the example config to be extremely helpful and useful to both new users and power users
- Fixed every single clippy (default lints) and rustc warnings, including some that were performance related or potential safety issues / unsoundness
- Add a **lot** of other clippy and rustc lints and a rustfmt.toml file
- Has [Renovate](https://docs.renovatebot.com/), [Trivy](https://github.com/aquasecurity/trivy-action), and keeps ALL dependencies as up to date as possible
- Attempts and interest in removing extreme and unnecessary panics/unwraps/expects that can lead to denial of service or such (upstream and upstream contributors want this unusual behaviour for some reason)
- Purge unmaintained/irrelevant/broken database backends (heed, sled, persy) and other unnecessary code or overhead
- webp support for images
- Add cargo audit support to CI
- CI tests with all features
- Add timestamp by commit date support to building OCI images for keeping image build reproducibility and still have a meaningful "last modified date" for OCI image metadata
- Update rusqlite/sqlite (not that you should be using it)
- Startup check if conduwuit running in a container and is listening on 127.0.0.1 (generally containers are using NAT networking and 0.0.0.0 is the intended listening address)
## Admin Room:
- Fix admin room handler to not panic/crash if the admin room command response fails (e.g. too large message)
- Add command to dynamically change conduwuit's tracing log level filter on the fly
- Add admin command to fetch a server's `/.well-known/matrix/support` file
- Add debug admin command to force update user device lists (could potentially resolve some E2EE flukes)
- Implement **RocksDB online backups**, listing RocksDB backups, and listing database file counts all via admin commands
- Add various database visibility commands such as being able to query the getters and iterators used in conduwuit, a very helpful online debugging utility
- Forbid the admin room from being made public or world readable history
- Add `!admin` as a way to call the admin bot
- Extend clear cache admin command to support clearing more caches such as DNS and TLS name overrides
- Admin debug command to send a federation request/ping to a server's `/_matrix/federation/v1/version` endpoint and measures the latency it took
- Add admin command to bulk delete media via a codeblock list of MXC URLs.
- Add admin command to delete both the thumbnail and media MXC URLs from an event ID (e.g. from an abuse report)
- Add admin command to list all the rooms a local user is joined in
- Add admin command to delete all remote media in the past X minutes as a form of deleting media that you don't want on your server that a remote user posted in a room
- Add admin command to return a room's state
- Admin debug command to fetch a PDU from a remote server and inserts it into our database/timeline as backfill
- Add admin command to delete media via a specific MXC. This deletes the MXC from our database, and the file locally.
- Add admin commands for banning (blocking) room IDs from our local users joining (admins are always allowed) and evicts all our local users from that room, in addition to bulk room banning support, and blocks room invites (remote and local) to the banned room, as a moderation feature
- Add admin commands to output jemalloc memory stats and memory usage
- Add admin command to get conduwuit's uptime
- Add admin command to get rooms a *remote* user shares with us
## Misc:
- Support for creating rooms with custom room IDs like Maunium Synapse (`room_id` request body field to `/createRoom`)
- Query parameter `?format=event|content` for returning either the room state event's content (default) for the full room state event on ` /_matrix/client/v3/rooms/{roomId}/state/{eventType}[/{stateKey}]` requests (see https://github.com/matrix-org/matrix-spec/issues/1047)
- Add **optional** feature flag to use SHA256 key names for media instead of base64 to overcome filesystem file name length limitations (OS error file name too long)
- Send a User-Agent on all of our requests
- Send `avatar_url` on invite room membership events/changes
- Support sending [`well_known` response to client login responses](https://spec.matrix.org/v1.10/client-server-api/#post_matrixclientv3login) if using config option `[well_known.client]`
- Implement `include_state` search criteria support for `/search` requests (response now can include room states)
- Declare various missing Matrix versions and features at `/_matrix/client/versions`
- Implement legacy Matrix `/v1/` media endpoints that some clients and servers may still call
- Config option to change Conduit's behaviour of homeserver key fetching (`query_trusted_key_servers_first`). This option sets whether conduwuit will query trusted notary key servers first before the individual homeserver(s), or vice versa which may help in joining certain rooms.
- Implement unstable MSC2666 support for querying mutual rooms with a user
- Assume well-knowns are broken if they exceed past 10000 characters.
- Add support for the Matrix spec compliance test suite [Complement](https://github.com/matrix-org/complement/) via the Nix flake and various other fixes for it
- Add support for listening on both HTTP and HTTPS if using direct TLS with conduwuit for usecases such as Complement
- Implement running and diff'ing Complement results in CI
- Interest in supporting other operating systems such as macOS, BSDs, and Windows, and getting them added into CI and doing builds for them
- Add config option for disabling RocksDB Direct IO if needed
- Add various documentation on maintaining conduwuit, using RocksDB online backups, some troubleshooting, using admin commands, etc
- (Developers): Add support for [hot reloadable/"live" modular development](development/hot_reload.md)
- (Developers): Add support for tokio-console
- (Developers): Add support for tracing flame graphs
- Add `release-debuginfo` Cargo build profile
- No cryptocurrency donations allowed, conduwuit is fully maintained by independent queer maintainers, and with a strong priority on inclusitivity and comfort for protected groups 🏳️⚧️
conduwuit has moderation through admin room commands. "binary commands" (medium priority) and an admin API (low priority) is planned. Some moderation-related config options are available in the example config such as "global ACLs" and blocking media requests to certain servers. See the example config for the moderation config options under the "Moderation / Privacy / Security" section.
- managing room banning/blocking and user removal (`!admin rooms moderation`)
- managing user accounts (`!admin users`)
- fetching `/.well-known/matrix/support` from servers (`!admin federation`)
- blocking incoming federation for certain rooms (not the same as room banning) (`!admin federation`)
- deleting media (see [the media section](#media))
Any commands with `-list` in them will require a codeblock in the message with each object being newline delimited. An example of doing this is:
````
!admin rooms moderation ban-list-of-rooms
```
!roomid1:server.name
!roomid2:server.name
!roomid3:server.name
```
````
## Database
If using RocksDB, there's very little you need to do. Compaction is ran automatically based on various defined thresholds tuned for conduwuit to be high performance with the least I/O amplifcation or overhead. Manually running compaction is not recommended, or compaction via a timer. RocksDB is built with io_uring support via liburing for async read I/O.
Some RocksDB settings can be adjusted such as the compression method chosen. See the RocksDB section in the [example config](configuration.md). btrfs users may benefit from disabling compression on RocksDB if CoW is in use.
RocksDB troubleshooting can be found [in the RocksDB section of troubleshooting](troubleshooting.md).
## Backups
Currently only RocksDB supports online backups. If you'd like to backup your database online without any downtime, see the `!admin server` command for the backup commands and the `database_backup_path` config options in the example config. Please note that the format of the database backup is not the exact same. This is unfortunately a bad design choice by Facebook as we are using the database backup engine API from RocksDB, however the data is still there and can still be joined together.
To restore a backup from an online RocksDB backup:
- shutdown conduwuit
- create a new directory for merging together the data
- in the online backup created, copy all `.sst` files in `$DATABASE_BACKUP_PATH/shared_checksum` to your new directory
- trim all the strings so instead of `######_sxxxxxxxxx.sst`, it reads `######.sst`. A way of doing this with sed and bash is `for file in *.sst; do mv "$file" "$(echo "$file" | sed 's/_s.*/.sst/')"; done`
- copy all the files in `$DATABASE_BACKUP_PATH/1` to your new directory
- set your `database_path` config option to your new directory, or replace your old one with the new one you crafted
- start up conduwuit again and it should open as normal
If you'd like to do an offline backup, shutdown conduwuit and copy your `database_path` directory elsewhere. This can be restored with no modifications needed.
Backing up media is also just copying the `media/` directory from your database directory.
## Media
Media still needs various work, however conduwuit implements media deletion via:
- MXC URI
- Delete list of MXC URIs
- Delete remote media in the past `N` seconds/minutes
See the `!admin media` command for further information. All media in conduwuit is stored at `$DATABASE_DIR/media`. This will be configurable soon.
If you are finding yourself needing extensive granular control over media, we recommend looking into [Matrix Media Repo](https://github.com/t2bot/matrix-media-repo). conduwuit intends to implement various utilities for media, but MMR is dedicated to extensive media management.
Built-in S3 support is also planned, but for now using a "S3 filesystem" on `media/` works. conduwuit also sends a `Cache-Control` header of 1 year and immutable for all media requests (download and thumbnail) to reduce unnecessary media requests from browsers.
> Docker is extremely UX unfriendly. Because of this, a ton of issues or support is actually Docker support, not conduwuit support. We also cannot document the ever-growing list of Docker issues here.
>
> If you intend on asking for support and you are using Docker, **PLEASE** triple validate your issues are **NOT** because you have a misconfiguration in your Docker setup.
>
> If there are things like Compose file issues or Dockerhub image issues, those can still be mentioned as long as they're something we can fix.
## Rocksdb / database issues
#### Direct IO
Some filesystems may not like RocksDB using [Direct IO](https://github.com/facebook/rocksdb/wiki/Direct-IO). Direct IO is for non-buffered I/O which improves conduwuit performance, but at least FUSE is a filesystem potentially known to not like this. See the [example config](configuration.md) for disabling it if needed. Issues from Direct IO on unsupported filesystems are usually shown as startup errors.
#### Database corruption
If your database is corrupted and is failing to start (e.g. checksum mismatch), it may be recoverable but careful steps must be taken, and there is no guarantee it may be recoverable.
RocksDB has the following recovery modes:
-`TolerateCorruptedTailRecords`
-`AbsoluteConsistency`
-`PointInTime`
-`SkipAnyCorruptedRecord`
By default, conduwuit uses `TolerateCorruptedTailRecords` as generally these may be due to bad federation and we can re-fetch the correct data over federation. The RocksDB default is `PointInTime` which will attempt to restore a "snapshot" of the data when it was last known to be good. This data can be either a few seconds old, or multiple minutes prior. `PointInTime` may not be suitable for default usage due to clients and servers possibly not being able to handle sudden "backwards time travels", and `AbsoluteConsistency` may be too strict.
`AbsoluteConsistency` will fail to start the database if any sign of corruption is detected. `SkipAnyCorruptedRecord` will skip all forms of corruption unless it forbids the database from opening (e.g. too severe). Usage of `SkipAnyCorruptedRecord` voids any support as this may cause more damage and/or leave your database in a permanently inconsistent state, but it may do something if `PointInTime` does not work as a last ditch effort.
With this in mind:
- First start conduwuit with the `PointInTime` recovery method. See the [example config](configuration.md) for how to do this using `rocksdb_recovery_mode`
- If your database successfully opens, clients are recommended to clear their client cache to account for the rollback
- Leave your conduwuit running in `PointInTime` for at least 30-60 minutes so as much possible corruption is restored
- If all goes will, you should be able to restore back to using `TolerateCorruptedTailRecords` and you have successfully recovered your database
## Media
#### "File name too long"
If you are running into the "file name is too long" OS error for media requests, your filesystem cannot handle file name lengths >=255 characters. This is unfortuntely due to Conduit (upstream) using base64 for file name keys which is very problematic for some filesystems as the base64 input is untrusted and long file names or specific inputs can cause this. If you would like to avoid this, you may build conduwuit yourself with the `sha256_media` feature. **This will lose database compatibility with upstream**.
## Debugging
Note that users should not really be debugging things. If you find yourself debugging and find the issue, please let us know and/or how we can fix it. Various debug commands can be found in `!admin debug`.
#### Debug/Trace log level
conduwuit builds without debug or trace log levels by default for at least performance reasons. This may change in the future and/or binaries providing such configurations may be provided. If you need to access debug/trace log levels, you will need to build without the `release_max_log_level` feature.
#### Changing log level dynamically
conduwuit supports changing the tracing log environment filter on-the-fly using the admin command `!admin debug change-log-level`. This accepts a string **without quotes** the same format as the `log` config option.
#### Pinging servers
conduwuit can ping other servers using `!admin debug ping`. This takes a server name and goes through the server discovery process and queries `/_matrix/federation/v1/version`. Errors are outputted.
#### Allocator memory stats
When using jemalloc with jemallocator's `stats` feature, you can see conduwuit's jemalloc memory stats by using `!admin debug memory-stats`
In order to make or receive calls, a TURN server is required. conduwuit suggests using [Coturn](https://github.com/coturn/coturn) for this purpose, which is also available as a Docker image.
### Configuration
Create a configuration file called `coturn.conf` containing:
```conf
use-auth-secret
static-auth-secret=<a secret key>
realm=<your server domain>
```
A common way to generate a suitable alphanumeric secret key is by using `pwgen -s 64 1`.
These same values need to be set in conduwuit. You can either modify conduwuit.toml to include these lines:
```
turn_uris = ["turn:<your server domain>?transport=udp", "turn:<your server domain>?transport=tcp"]
turn_secret = "<secret key from coturn configuration>"
```
or append the following to the docker environment variables dependig on which configuration method you used earlier:
```yml
CONDUIT_TURN_URIS:'["turn:<your server domain>?transport=udp", "turn:<your server domain>?transport=tcp"]'
CONDUIT_TURN_SECRET:"<secret key from coturn configuration>"
```
Restart conduwuit to apply these changes.
### Run
Run the [Coturn](https://hub.docker.com/r/coturn/coturn) image using
```bash
docker run -d --network=host -v $(pwd)/coturn.conf:/etc/coturn/turnserver.conf coturn/coturn
```
or docker-compose. For the latter, paste the following section into a file called `docker-compose.yml`
and run `docker compose up -d` in the same directory.
```yml
version:3
services:
turn:
container_name:coturn-server
image:docker.io/coturn/coturn
restart:unless-stopped
network_mode:"host"
volumes:
- ./coturn.conf:/etc/coturn/turnserver.conf
```
To understand why the host networking mode is used and explore alternative configuration options, please visit [Coturn's Docker documentation](https://github.com/coturn/coturn/blob/master/docker/coturn/README.md).
For security recommendations see Synapse's [Coturn documentation](https://element-hq.github.io/synapse/latest/turn-howto.html).
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Jason Volk
strawberry
Benjamin Lee
morguldir
renovate[bot]
slonkazoid
Charles Hall
Xiretza
Matt Moriarity
Tom Foster
AwesomeQubic
Matthias Ahouansou
strawberry
raizo
K900
Timo Kösters
Samuel Meenzen
sininenkissa
olly1240
EPS-DEV
dependabot[bot]
Reiner Herrmann
Raito Bezarius
kubo6472
Val Lorentz
Nineko
Jakub Kubík
nat
strawberry
tezlm
AndSDev
Rohan Kumar
Tobias Bucher
Luke McCarthy
Jonas Zohren
purplemeteorite
Maarten Steenhagen
Tobias Tom
uak
Paul van Tilburg
x4u
Jacob Taylor
Jonas Platte
digital
Jade
x4u
Jonathan Flueren
Kévin Commaille
Moritz Heiber
Yusuf Bera Ertan
Jonathan de Jong
The one with the braid
Nyaaori
Vladan Popovic
Orhideous
Ticho 34782694
Paul Beziau
Nabulator
James Blachly
Max Cohen
Tglman
exin
majso
Jim
Zeyphros
Dietrich
Radek Stępień
Miepee
Jakub Doboš
=
rmsthebest
Jan Christian Grünhage
chenyuqide
Andrej Kacian
LordMZTE
kk
TomZ
Jonas Platte
Andrei Vasiliu
Torsten Flammiger
user
Aode (lion)
Maxim De Clercq
Jonas Zohren
Julius de Bruijn
Moritz Bitsch
Andrew Morgan
Luc-pascal Ceccaldi
Greg Sutcliffe
Daniel Wiesenberg
Rasmus Thomsen
charludo
Kurt Roeckx
Devin Ragotzy
Tom Smeding
Jonathan de Jong
Ahmed Charles
phesch
Aiden McClelland
hamidreza kalbasi
Damian Poddebniak
Jonas Fowl
Vadim Zeitlin
Gabriel Souza Franco
Tobias Fella
Marcel
Mariusz Kogen
Rudi Floren
Rudi Floren
Niklas Zender
Devin Ragotzy
Devon Hudson
Leonhard Kuboschek
miruka
Faelar
Devin Ragotzy
Sebastian Spaeth
Daniel Wiesenberg
Guillem Nieto
CapsizeGlimmer
aura
the0
PublicByte
Jean-Baptiste Petit
josias
TheCycoONE
