Compare commits

...
32 Commits
Author SHA1 Message Date
Jason Volk 9af15ecbba Bump 1.4.5
Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-25 13:33:32 +00:00
Jason Volk f66a83763e Persist sliding-sync state; mitigate initial-sync.
Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-25 13:33:32 +00:00
Jason Volk 718c3adcb2 ci: Expand the rust-sdk integration matrix.
Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-25 08:44:16 +00:00
Jason Volk af80482c04 Add branch in room handler for left rooms.
Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-25 08:34:49 +00:00
Jason Volk a20ddcd586 Recency bump on own membership event.
Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-25 08:08:26 +00:00
Jason Volk b483306367 Fix sliding-sync invite regression.
Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-25 07:52:40 +00:00
Jason Volk f5e98467be Improve sliding-sync robustness to deeper replays.
Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-24 18:31:48 +00:00
Jason Volk d44db45f83 Adjust the order of operations during read_marker handling.
Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-24 18:31:48 +00:00
Jason Volk 0397bb8237 Implement specified non-blocking semantic for sliding-sync.
Simplify sliding-sync watch loop and bounds.

Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-23 11:00:55 +00:00
Jason Volk 4010fc62bc Use take_while rather than filter for scanning last_receipt_count.
Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-23 03:37:52 +00:00
Jason Volk 0d823a2822 Revert "Involve m.fully_read for resetting notification counts."
This reverts commit 0df5e5e7ac.
2025-10-23 03:37:52 +00:00
Jason Volk 9cd175b125 Additional logging/tracing tweaks around sliding-sync.
Additional spans around receipt service interface.

Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-23 03:37:52 +00:00
Jason Volk 050a1a350a Add dbg profile to Cargo; default to faster builds accomodating new users. 2025-10-23 03:37:52 +00:00
Jason Volk ec0f872f8f Add tracing around db watcher notify including cf name.
Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-22 20:23:30 +00:00
Jason Volk 024e8eae62 Implement num_live for sliding-sync room response.
Avoid using origin_server_ts for recency stamp.

Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-22 09:14:41 +00:00
Jason Volk 4fd60b2605 Improve sliding-sync selector and windowing logic.
Bump Ruma; eliminate premature release compat.

Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-22 08:15:25 +00:00
Jason Volk e53968d9eb Improve tracing spans for notif counts and sliding-sync exts.
Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-22 07:05:03 +00:00
Jason Volk 7cbc2ee385 Fix the accidentally dual-keyspaced cf from conduit.
Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-22 07:03:10 +00:00
Jason Volk 0df5e5e7ac Involve m.fully_read for resetting notification counts.
EXA is not reliably removing its unread marker any other way. Clients
wishing to move the fully_read marker in the other direction should
file an issue to make themselves known.

Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-22 06:52:30 +00:00
Jason Volk 312eb69450 Add tool to get sibling connections for a device.
Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-21 17:48:08 +00:00
Jason Volk 5b620a2c37 Add .and_is() / .and_if() to the BoolExt smalltalk.
Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-21 17:41:42 +00:00
Jason Volk 3b4fbb8c1a Fix inverted debug assert message.
Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-21 15:48:05 +00:00
Jason Volk 158d44e1a9 docs: Remove line about other reverse proxies. (#197)
Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-21 15:02:36 +00:00
Jason Volk d2aab468cf Fixes for sliding-sync request replays.
Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-21 06:46:02 +00:00
Jason Volk aa4486dfdf Simplify types, mitigate expansion; eliminate unnecessary move.
Further simplify future::BoolExt toward type expansion mitigation.

Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-21 05:25:11 +00:00
Jason Volk 2a662445b6 Make account_data.last_count() upper bound argument optional.
Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-20 23:58:38 +00:00
Jason Volk 5b3f0fde23 Add invite rooms to sync v3 watch rooms list.
Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-20 23:57:52 +00:00
Jason Volk 9640afebff Join the async prologue operations in sync v3.
Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-20 23:55:42 +00:00
Jason Volk dd5c5c7a4a Notify watchers on delete operations in addition to inserts. 2025-10-20 20:45:38 +00:00
Jason Volk fc0f04defa Optimize watcher range collection on notify.
Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-20 20:44:45 +00:00
Jason Volk 7d8f7cbe5d Bump 1.4.4
Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-16 01:55:43 +00:00
Jason Volk 4e0249cd2f Fix sliding-sync again by resetting after interrupted requests. (fixes #190)
This solution is not elegant but appears to be correct for fixing the issue. A
a more efficient solution might be possible without resetting the connection
which should be further investigated.

Signed-off-by: Jason Volk <jason@zemos.net>
2025-10-16 01:05:16 +00:00
35 changed files with 718 additions and 497 deletions

No files matched your search

+15 -8
View File
@@ -226,7 +226,7 @@ jobs:
&& !contains(inputs.head_msg, '[ci only it]')
&& !contains(inputs.head_msg, '[ci no build]')
&& contains(fromJSON(inputs.cargo_profiles), fromJSON('["release"]')[0])
&& contains(fromJSON(inputs.complement_feat_sets), fromJSON('["all"]')[0])
&& contains(fromJSON(inputs.feat_sets), fromJSON('["all"]')[0])
&& contains(fromJSON(inputs.rust_toolchains), fromJSON('["stable"]')[0])
&& contains(fromJSON(inputs.sys_targets), fromJSON('["x86_64-v1-linux-gnu"]')[0])
&& fromJSON(inputs.build_nix)
@@ -251,9 +251,9 @@ jobs:
!failure() && !cancelled()
&& !contains(inputs.head_msg, '[ci no build]')
&& inputs.rust_sdk_integration
&& contains(fromJSON(inputs.cargo_profiles), fromJSON('["test"]')[0])
&& contains(fromJSON(inputs.complement_feat_sets), fromJSON('["all"]')[0])
&& contains(fromJSON(inputs.feat_sets), fromJSON('["all"]')[0])
&& contains(fromJSON(inputs.rust_toolchains), fromJSON('["nightly"]')[0])
&& contains(fromJSON(inputs.rust_targets), fromJSON('["x86_64-unknown-linux-gnu"]')[0])
&& contains(fromJSON(inputs.sys_targets), fromJSON('["x86_64-v1-linux-gnu"]')[0])
name: Matrix SDK Integration
@@ -261,16 +261,15 @@ jobs:
uses: ./.github/workflows/bake.yml
with:
bake_targets: '["matrix-rust-sdk-integration"]'
cargo_profiles: '["test"]'
feat_sets: ${{inputs.complement_feat_sets}}
cargo_profiles: ${{inputs.cargo_profiles}}
feat_sets: '["all"]'
rust_toolchains: '["nightly"]'
sys_names: ${{inputs.sys_names}}
sys_versions: ${{inputs.sys_versions}}
rust_targets: ${{inputs.rust_targets}}
rust_targets: '["x86_64-unknown-linux-gnu"]'
sys_targets: '["x86_64-v1-linux-gnu"]'
machines: ${{inputs.machines}}
machines: '["X64"]'
runner: ${{inputs.complement_runner}}
excludes: ${{inputs.excludes}}
includes: ${{inputs.includes}}
artifact: >
{
@@ -279,6 +278,14 @@ jobs:
"dst": "matrix-rust-sdk-integration.tuwunel.log",
}
}
excludes: >
[
{"feat_set": "none"},
{"feat_set": "logging"},
{"cargo_profile": "release"},
{"cargo_profile": "release-debuginfo"},
{"cargo_profile": "release-native"},
]
complement:
if: >
Generated
+19 -18
View File
@@ -3509,7 +3509,7 @@ dependencies = [
[[package]]
name = "ruma"
version = "0.13.0"
source = "git+https://github.com/matrix-construct/ruma?rev=3c26be1e4d670c2c71e2bc84b3f09c67c4188612#3c26be1e4d670c2c71e2bc84b3f09c67c4188612"
source = "git+https://github.com/matrix-construct/ruma?rev=b858d3478f2792a381463ef9b63f6c1469a5d81c#b858d3478f2792a381463ef9b63f6c1469a5d81c"
dependencies = [
"assign",
"js_int",
@@ -3528,7 +3528,7 @@ dependencies = [
[[package]]
name = "ruma-appservice-api"
version = "0.13.0"
source = "git+https://github.com/matrix-construct/ruma?rev=3c26be1e4d670c2c71e2bc84b3f09c67c4188612#3c26be1e4d670c2c71e2bc84b3f09c67c4188612"
source = "git+https://github.com/matrix-construct/ruma?rev=b858d3478f2792a381463ef9b63f6c1469a5d81c#b858d3478f2792a381463ef9b63f6c1469a5d81c"
dependencies = [
"js_int",
"ruma-common",
@@ -3540,7 +3540,7 @@ dependencies = [
[[package]]
name = "ruma-client-api"
version = "0.21.0"
source = "git+https://github.com/matrix-construct/ruma?rev=3c26be1e4d670c2c71e2bc84b3f09c67c4188612#3c26be1e4d670c2c71e2bc84b3f09c67c4188612"
source = "git+https://github.com/matrix-construct/ruma?rev=b858d3478f2792a381463ef9b63f6c1469a5d81c#b858d3478f2792a381463ef9b63f6c1469a5d81c"
dependencies = [
"as_variant",
"assign",
@@ -3565,7 +3565,7 @@ dependencies = [
[[package]]
name = "ruma-common"
version = "0.16.0"
source = "git+https://github.com/matrix-construct/ruma?rev=3c26be1e4d670c2c71e2bc84b3f09c67c4188612#3c26be1e4d670c2c71e2bc84b3f09c67c4188612"
source = "git+https://github.com/matrix-construct/ruma?rev=b858d3478f2792a381463ef9b63f6c1469a5d81c#b858d3478f2792a381463ef9b63f6c1469a5d81c"
dependencies = [
"as_variant",
"base64",
@@ -3598,7 +3598,7 @@ dependencies = [
[[package]]
name = "ruma-events"
version = "0.31.0"
source = "git+https://github.com/matrix-construct/ruma?rev=3c26be1e4d670c2c71e2bc84b3f09c67c4188612#3c26be1e4d670c2c71e2bc84b3f09c67c4188612"
source = "git+https://github.com/matrix-construct/ruma?rev=b858d3478f2792a381463ef9b63f6c1469a5d81c#b858d3478f2792a381463ef9b63f6c1469a5d81c"
dependencies = [
"as_variant",
"indexmap",
@@ -3612,6 +3612,7 @@ dependencies = [
"ruma-macros",
"serde",
"serde_json",
"smallstr",
"smallvec",
"thiserror 2.0.16",
"tracing",
@@ -3624,7 +3625,7 @@ dependencies = [
[[package]]
name = "ruma-federation-api"
version = "0.12.0"
source = "git+https://github.com/matrix-construct/ruma?rev=3c26be1e4d670c2c71e2bc84b3f09c67c4188612#3c26be1e4d670c2c71e2bc84b3f09c67c4188612"
source = "git+https://github.com/matrix-construct/ruma?rev=b858d3478f2792a381463ef9b63f6c1469a5d81c#b858d3478f2792a381463ef9b63f6c1469a5d81c"
dependencies = [
"bytes",
"headers",
@@ -3646,7 +3647,7 @@ dependencies = [
[[package]]
name = "ruma-identifiers-validation"
version = "0.11.0"
source = "git+https://github.com/matrix-construct/ruma?rev=3c26be1e4d670c2c71e2bc84b3f09c67c4188612#3c26be1e4d670c2c71e2bc84b3f09c67c4188612"
source = "git+https://github.com/matrix-construct/ruma?rev=b858d3478f2792a381463ef9b63f6c1469a5d81c#b858d3478f2792a381463ef9b63f6c1469a5d81c"
dependencies = [
"js_int",
"thiserror 2.0.16",
@@ -3655,7 +3656,7 @@ dependencies = [
[[package]]
name = "ruma-macros"
version = "0.16.0"
source = "git+https://github.com/matrix-construct/ruma?rev=3c26be1e4d670c2c71e2bc84b3f09c67c4188612#3c26be1e4d670c2c71e2bc84b3f09c67c4188612"
source = "git+https://github.com/matrix-construct/ruma?rev=b858d3478f2792a381463ef9b63f6c1469a5d81c#b858d3478f2792a381463ef9b63f6c1469a5d81c"
dependencies = [
"cfg-if",
"proc-macro-crate",
@@ -3670,7 +3671,7 @@ dependencies = [
[[package]]
name = "ruma-push-gateway-api"
version = "0.12.0"
source = "git+https://github.com/matrix-construct/ruma?rev=3c26be1e4d670c2c71e2bc84b3f09c67c4188612#3c26be1e4d670c2c71e2bc84b3f09c67c4188612"
source = "git+https://github.com/matrix-construct/ruma?rev=b858d3478f2792a381463ef9b63f6c1469a5d81c#b858d3478f2792a381463ef9b63f6c1469a5d81c"
dependencies = [
"js_int",
"ruma-common",
@@ -3682,7 +3683,7 @@ dependencies = [
[[package]]
name = "ruma-signatures"
version = "0.18.0"
source = "git+https://github.com/matrix-construct/ruma?rev=3c26be1e4d670c2c71e2bc84b3f09c67c4188612#3c26be1e4d670c2c71e2bc84b3f09c67c4188612"
source = "git+https://github.com/matrix-construct/ruma?rev=b858d3478f2792a381463ef9b63f6c1469a5d81c#b858d3478f2792a381463ef9b63f6c1469a5d81c"
dependencies = [
"base64",
"ed25519-dalek",
@@ -4960,7 +4961,7 @@ checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b"
[[package]]
name = "tuwunel"
version = "1.4.3"
version = "1.4.5"
dependencies = [
"clap",
"const-str",
@@ -4992,7 +4993,7 @@ dependencies = [
[[package]]
name = "tuwunel_admin"
version = "1.4.3"
version = "1.4.5"
dependencies = [
"base64",
"clap",
@@ -5015,7 +5016,7 @@ dependencies = [
[[package]]
name = "tuwunel_api"
version = "1.4.3"
version = "1.4.5"
dependencies = [
"async-trait",
"axum",
@@ -5048,7 +5049,7 @@ dependencies = [
[[package]]
name = "tuwunel_core"
version = "1.4.3"
version = "1.4.5"
dependencies = [
"argon2",
"arrayvec",
@@ -5109,7 +5110,7 @@ dependencies = [
[[package]]
name = "tuwunel_database"
version = "1.4.3"
version = "1.4.5"
dependencies = [
"async-channel",
"const-str",
@@ -5129,7 +5130,7 @@ dependencies = [
[[package]]
name = "tuwunel_macros"
version = "1.4.3"
version = "1.4.5"
dependencies = [
"itertools 0.14.0",
"proc-macro2",
@@ -5139,7 +5140,7 @@ dependencies = [
[[package]]
name = "tuwunel_router"
version = "1.4.3"
version = "1.4.5"
dependencies = [
"axum",
"axum-client-ip",
@@ -5173,7 +5174,7 @@ dependencies = [
[[package]]
name = "tuwunel_service"
version = "1.4.3"
version = "1.4.5"
dependencies = [
"async-trait",
"base64",
+9 -4
View File
@@ -29,7 +29,7 @@ license = "Apache-2.0"
readme = "README.md"
repository = "https://github.com/matrix-construct/tuwunel"
rust-version = "1.88.0"
version = "1.4.3"
version = "1.4.5"
[workspace.metadata.crane]
name = "tuwunel"
@@ -320,7 +320,7 @@ default-features = false
[workspace.dependencies.ruma]
git = "https://github.com/matrix-construct/ruma"
rev = "3c26be1e4d670c2c71e2bc84b3f09c67c4188612"
rev = "b858d3478f2792a381463ef9b63f6c1469a5d81c"
features = [
"__compat",
"appservice-api-c",
@@ -723,7 +723,7 @@ strip = false
# and can be raised if build times are tolerable.
[profile.dev]
debug = "full"
debug = 0
#rustflags = [
# '--cfg', 'tuwunel_mods',
# '-Ztime-passes',
@@ -783,7 +783,7 @@ inherits = "dev"
[profile.dev.package.'*']
inherits = "dev"
debug = "limited"
debug = 0
incremental = false
codegen-units = 1
opt-level = 'z'
@@ -800,6 +800,11 @@ opt-level = 'z'
# '-Clink-arg=-Wl,-z,nodelete',
#]
# same as dev but slower.
[profile.dbg]
inherits = "dev"
debug = "full"
# primarily used for CI
[profile.test]
debug = "limited"
+4 -45
View File
@@ -1,50 +1,9 @@
# Tuwunel 1.4.3
# Tuwunel 1.4.5
October 10, 2025
October 24, 2025
### Featured
**Sync Tokens have been eliminated** now with stateless sync. Users should notice a reduction in database size after running this version. Long-time users, including from conduwuit and even Conduit will benefit the most. New users may not notice reductions, but nevertheless will be preventing database growth going forward. With the prior addition of room deletions courtesy of @dasha-uwu, only enhanced media retention remains between Tuwunel and sustainable cost-of-ownership.
**Sliding Sync has been fixed (Element X)** after a third pass was made to install an entirely new architecture based on the latest evolution of the highly active MSC4186. For background, the first work done earlier this year was for optimization without logical changes to what was inherited from conduwuit. The existing implementation worked by all appearances, but didn't meet specific production quality demands. The second pass made last month to rapidly prepare this passé implementation for production use against highly developed modern clients did not turn out well. More tests passed; fewer things worked. This time the core logic had been rewritten. These three iterations have now modernized the entire module to keep up with the final stages of the specification's development. It is still not perfect, so your input and issue reports are greatly appreciated as always.
### Enhancements
- Configuration options passed after arguments `--option` or `-O` now accept dots to address the TOML tables found in the config file. Thanks to the suggestion by @lucat1 while trying to configure `ldap.bind_password_file` from the command line (#162). This was separately uncovered by @andrewerf trying to configure the `tls` section (#167).
- Configuring `log_to_stderr` now provides an explicit way to redirect logging and tracing to stderr. This is often requested for use with systemd.
- The `!admin query raw` commands no longer require the redundant `raw-` prefix on every command name.
- Thanks to @SophiaH67 for pointing out that our new verbose-logging docker images aren't very useful without being pushed to registries, then taking the initiative to host it themselves until we corrected that.
- After a report by @munkinasack in (#186), @dasha-uwu determined we could solve a lot of recurring network issues by clearing the well-known cache entry for a destination that becomes unreachable.
- Thanks to @dasha-uwu for general improvements, refactoring and fixes for the room alias and presence subsystems.
- Thanks to a report by @ohitsdylan a cryptic error message from the DNS resolver has now been caught and reworded to indicate a missing or empty `/etc/resolv.conf` (#179).
- Inspired by @boarfish offering due confusion about our many build variants in (#175), some documentation about `x86_64-v1` `-v2` and `-v3` has been improved. Special thanks to @Hashbrown777 for providing a quick one-liner shell script which made its way to the documentation.
- Various performance improvements took place in s2s request handlers, and the ubiquitous matrix event `Pdu` structure.
- Nix builds have been added to CI.
All deployments serving ElementX, Element Pro, SchildiChat Next, or any client using sliding-sync should upgrade to this patch for an improved experience.
### Bug Fixes
- Thanks to @harharlinks for reporting the Code of Conduct link on our github-pages was 404. Also thanks to @Tronde for reporting additional broken links in (#165). This helped discover pages had stopped deploying at some point and needed fixing.
- Thanks to @agx for contributing a fix for the systemd units on multiple platforms which contained unknown or deprecated keys (#168). And thanks again for adding missing documentation to the well-known sections of the example configuration which many users will greatly appreciate (#173).
- @mitch9911:matrix.org reported the `device_signing/upload` endpoint was omitted when adding JWT as a UIAA flow to other API's. This was subsequently patched (#169).
- The report by @orhtej2 of an invalid `?via` parameter sent by FluffyChat when joining a version 12 room was greatly appreciated, even though there was nothing more we could do on our end (#171). Thanks to @dasha-uwu for triaging and investigating this issue to conclusion.
- Thanks to a follow-up by @alaviss in (#176) the DNS-passthru feature was found to still be enforcing a large minimum-TTL for its DNS cache. This was subsequently corrected.
- Thank you @canarysnort01 for the apropos and rather surgical off-by-one fix to sliding-sync range selection in (#188). Unfortunately this entire unit had to be rewritten for the latest iteration of sliding-sync, but the fix carried value in any case to improve the rewrite.
### Notices
It has come to our attention courtesy of @andrewerf that the Arch packages are built with `--no-default-features`. This may be a problem for an ideal experience. The backstory is that conduwuit underwent a "feature skew" over its lifetime which still remains today: our default-features are basically minimal requirements, while `--all-features` should be default features. Let us first take a moment to reiterate our gratitude to AUR package maintainers @drrossum and @kimiblock who have supported this project from the first hours of its existence. No action is required on their part as the plan now is to remove several optional features by the next release to make `--no-default-features` viable. We still recommend default features in general unless this conflicts with AUR policies or philosophies.
- Progress made for sliding-sync in 1.4.3 was effectively reverted by 1.4.4. This release puts sliding-sync back on track with further progress exceeding 1.4.3. This release is now shipping without any known issues. Please report all issues. **IMPORTANT:** Issues from prior releases (such as stuck or incorrect unread counts) benefit from clearing the app's cache to immediately resolve. Go to settings, tap the version number 7 times, then enter the menu to select `clear cache`.
+2 -3
View File
@@ -13,14 +13,13 @@ ARG mrsdk_profile="release"
ARG mrsdk_target_share="${MRSDK_TARGET_DIR}/${sys_name}/${sys_version}/${rust_target}/${rust_toolchain}/${mrsdk_profile}/_shared_cache"
ARG mrsdk_test_args=""
ARG mrsdk_test_opts=""
#TODO!!!
ARG mrsdk_skip_list="--skip test_history_share_on_invite_pin_violation --skip test_room_notification_count"
ARG mrsdk_skip_list=""
WORKDIR /
COPY --link --from=input . .
WORKDIR /usr/src
ADD --link https://github.com/matrix-construct/matrix-rust-sdk.git matrix-rust-sdk
ADD --link https://github.com/matrix-construct/matrix-rust-sdk.git#integration matrix-rust-sdk
WORKDIR /etc
COPY <<EOF tuwunel.toml
+1 -1
View File
@@ -166,7 +166,7 @@ sudo systemctl enable --now caddy
### Other Reverse Proxies
As we would prefer our users to use Caddy, we will not provide configuration files for other proxies.
_Specific contributions for other proxies are welcome!_
You will need to reverse proxy everything under following routes:
- `/_matrix/` - core Matrix C-S and S-S APIs
+14 -7
View File
@@ -29,7 +29,7 @@ pub(crate) enum SyncCommand {
#[admin_command]
pub(super) async fn list_connections(&self) -> Result {
let connections = self.services.sync.list_connections();
let connections = self.services.sync.list_loaded_connections().await;
for connection_key in connections {
self.write_str(&format!("{connection_key:?}\n"))
@@ -47,7 +47,11 @@ pub(super) async fn show_connection(
conn_id: Option<String>,
) -> Result {
let key = into_connection_key(user_id, device_id, conn_id);
let cache = self.services.sync.find_connection(&key)?;
let cache = self
.services
.sync
.get_loaded_connection(&key)
.await?;
let out;
{
@@ -65,11 +69,14 @@ pub(super) async fn drop_connections(
device_id: Option<OwnedDeviceId>,
conn_id: Option<String>,
) -> Result {
self.services.sync.clear_connections(
user_id.as_deref(),
device_id.as_deref(),
conn_id.map(Into::into).as_ref(),
);
self.services
.sync
.clear_connections(
user_id.as_deref(),
device_id.as_deref(),
conn_id.map(Into::into).as_ref(),
)
.await;
Ok(())
}
+33 -30
View File
@@ -26,6 +26,12 @@ pub(crate) async fn set_read_marker_route(
) -> Result<set_read_marker::v3::Response> {
let sender_user = body.sender_user();
if body.private_read_receipt.is_some() || body.read_receipt.is_some() {
services
.user
.reset_notification_counts(sender_user, &body.room_id);
}
if let Some(event) = &body.fully_read {
let fully_read_event = ruma::events::fully_read::FullyReadEvent {
content: ruma::events::fully_read::FullyReadEventContent { event_id: event.clone() },
@@ -39,21 +45,29 @@ pub(crate) async fn set_read_marker_route(
RoomAccountDataEventType::FullyRead,
&serde_json::to_value(fully_read_event)?,
)
.await?;
.await
.ok();
}
if body.private_read_receipt.is_some() || body.read_receipt.is_some() {
if let Some(event) = &body.private_read_receipt {
let count = services
.timeline
.get_pdu_count(event)
.await
.map_err(|_| err!(Request(NotFound("Event not found."))))?;
let PduCount::Normal(count) = count else {
return Err!(Request(InvalidParam(
"Event is a backfilled PDU and cannot be marked as read."
)));
};
services
.user
.reset_notification_counts(sender_user, &body.room_id);
.read_receipt
.private_read_set(&body.room_id, sender_user, count);
}
if let Some(event) = &body.read_receipt {
services
.presence
.maybe_ping_presence(sender_user, &ruma::presence::PresenceState::Online)
.await?;
let receipt_content = BTreeMap::from_iter([(
event.to_owned(),
BTreeMap::from_iter([(
@@ -76,24 +90,12 @@ pub(crate) async fn set_read_marker_route(
},
)
.await;
}
if let Some(event) = &body.private_read_receipt {
let count = services
.timeline
.get_pdu_count(event)
.await
.map_err(|_| err!(Request(NotFound("Event not found."))))?;
let PduCount::Normal(count) = count else {
return Err!(Request(InvalidParam(
"Event is a backfilled PDU and cannot be marked as read."
)));
};
services
.read_receipt
.private_read_set(&body.room_id, sender_user, count);
.presence
.maybe_ping_presence(sender_user, &ruma::presence::PresenceState::Online)
.await
.ok();
}
Ok(set_read_marker::v3::Response {})
@@ -135,11 +137,6 @@ pub(crate) async fn create_receipt_route(
.await?;
},
| create_receipt::v3::ReceiptType::Read => {
services
.presence
.maybe_ping_presence(sender_user, &ruma::presence::PresenceState::Online)
.await?;
let receipt_content = BTreeMap::from_iter([(
body.event_id.clone(),
BTreeMap::from_iter([(
@@ -165,6 +162,12 @@ pub(crate) async fn create_receipt_route(
},
)
.await;
services
.presence
.maybe_ping_presence(sender_user, &ruma::presence::PresenceState::Online)
.await
.ok();
},
| create_receipt::v3::ReceiptType::ReadPrivate => {
let count = services
+5 -3
View File
@@ -1,7 +1,7 @@
mod v3;
mod v5;
use futures::{StreamExt, pin_mut};
use futures::{FutureExt, StreamExt, pin_mut};
use ruma::{RoomId, UserId};
use tuwunel_core::{
Error, PduCount, Result,
@@ -42,10 +42,12 @@ async fn load_timeline(
.by_ref()
.take(limit)
.collect()
.map(|mut pdus: Vec<_>| {
pdus.reverse();
pdus
})
.await;
let timeline_pdus: Vec<_> = timeline_pdus.into_iter().rev().collect();
// They /sync response doesn't always return all messages, so we say the output
// is limited unless there are events in non_timeline_pdus
let limited = non_timeline_pdus.next().await.is_some();
+11 -7
View File
@@ -39,7 +39,7 @@ use tuwunel_core::{
Error, Result, at,
debug::INFO_SPAN_LEVEL,
err, error,
error::inspect_debug_log,
error::{inspect_debug_log, inspect_log},
extract_variant, is_equal_to,
matrix::{
Event,
@@ -53,7 +53,6 @@ use tuwunel_core::{
self, BoolExt, FutureBoolExt, IterStream, ReadyExt, TryFutureExtExt,
future::{OptionStream, ReadyEqExt},
math::ruma_from_u64,
result::LogErr,
stream::{BroadbandExt, Tools, TryExpect, WidebandExt},
},
warn,
@@ -129,15 +128,16 @@ pub(crate) async fn sync_events_route(
) -> Result<sync_events::v3::Response, RumaResponse<UiaaResponse>> {
let (sender_user, sender_device) = body.sender();
services
let ping_presence = services
.presence
.maybe_ping_presence(sender_user, &body.body.set_presence)
.await
.log_err()
.inspect_err(inspect_log)
.ok();
// Record user as actively syncing for push suppression heuristic.
services.presence.note_sync(sender_user).await;
let note_sync = services.presence.note_sync(sender_user);
join(ping_presence, note_sync).await;
let mut since = body
.body
@@ -163,7 +163,11 @@ pub(crate) async fn sync_events_route(
.expect("configuration must limit maximum timeout");
loop {
let watch_rooms = services.state_cache.rooms_joined(sender_user);
let watch_rooms = services
.state_cache
.rooms_joined(sender_user)
.chain(services.state_cache.rooms_invited(sender_user));
let watchers = services
.sync
.watch(sender_user, sender_device, watch_rooms);
+83 -65
View File
@@ -23,9 +23,9 @@ use ruma::{
};
use tokio::time::{Instant, timeout_at};
use tuwunel_core::{
Result, apply, at,
Err, Result, apply, at, debug,
debug::INFO_SPAN_LEVEL,
err,
debug_warn,
error::inspect_log,
extract_variant,
smallvec::SmallVec,
@@ -53,7 +53,6 @@ struct SyncInfo<'a> {
services: &'a Services,
sender_user: &'a UserId,
sender_device: &'a DeviceId,
request: &'a Request,
}
#[derive(Clone, Debug)]
@@ -87,28 +86,39 @@ type ListIds = SmallVec<[ListId; 1]>;
user_id = %body.sender_user().localpart(),
device_id = %body.sender_device(),
conn_id = ?body.body.conn_id.clone().unwrap_or_default(),
since = ?body.body.pos.clone().or_else(|| body.body.pos_qrs_.clone()).unwrap_or_default(),
since = ?body.body.pos.clone().unwrap_or_default(),
)
)]
pub(crate) async fn sync_events_v5_route(
State(ref services): State<crate::State>,
body: Ruma<Request>,
) -> Result<Response> {
let (sender_user, sender_device) = body.sender();
let request = &body.body;
let since = request
.pos
.as_ref()
.or(request.pos_qrs_.as_ref())
.and_then(|string| string.parse().ok())
.unwrap_or(0);
let (sender_user, sender_device) = body.sender();
let timeout = request
.timeout
.as_ref()
.map(Duration::as_millis)
.map(TryInto::try_into)
.flat_ok()
.map(|timeout: u64| {
timeout
.max(services.config.client_sync_timeout_min)
.min(services.config.client_sync_timeout_max)
})
.unwrap_or(0);
let conn_key = into_connection_key(sender_user, sender_device, request.conn_id.as_deref());
let conn_val = since
.ne(&0)
.then(|| services.sync.find_connection(&conn_key))
.unwrap_or_else(|| Ok(services.sync.init_connection(&conn_key)))
.map_err(|_| err!(Request(UnknownPos("Connection lost; restarting sync stream."))))?;
let conn_val = services
.sync
.load_or_init_connection(&conn_key)
.await;
let conn = conn_val.lock();
let ping_presence = services
@@ -119,31 +129,34 @@ pub(crate) async fn sync_events_v5_route(
let (mut conn, _) = join(conn, ping_presence).await;
if since != 0 && conn.next_batch == 0 {
return Err!(Request(UnknownPos(warn!("Connection lost; restarting sync stream."))));
}
if since == 0 {
*conn = Connection::default();
conn.store(&services.sync, &conn_key);
debug_warn!(?conn_key, "Client cleared cache and reloaded.");
}
let advancing = since == conn.next_batch;
let retarding = since != 0 && since <= conn.globalsince;
if !advancing && !retarding {
return Err!(Request(UnknownPos(warn!(
"Requesting unknown or invalid stream position."
))));
}
debug_assert!(
advancing || retarding,
"Request should either be advancing or replaying the since token."
);
// Update parameters regardless of replay or advance
conn.next_batch = services.globals.wait_pending().await?;
conn.globalsince = since.min(conn.next_batch);
conn.update_cache(request);
conn.globalsince = since;
conn.next_batch = since;
let sync_info = SyncInfo {
services,
sender_user,
sender_device,
request,
};
let timeout = request
.timeout
.as_ref()
.or(request.timeout_qrs_.as_ref())
.map(Duration::as_millis)
.map(TryInto::try_into)
.flat_ok()
.unwrap_or(services.config.client_sync_timeout_default)
.max(services.config.client_sync_timeout_min)
.min(services.config.client_sync_timeout_max);
let stop_at = Instant::now()
.checked_add(Duration::from_millis(timeout))
.expect("configuration must limit maximum timeout");
conn.update_rooms_prologue(retarding.then_some(since));
let mut response = Response {
txn_id: request.txn_id.clone(),
@@ -153,54 +166,63 @@ pub(crate) async fn sync_events_v5_route(
extensions: Default::default(),
};
loop {
let window;
(window, response.lists) = selector(&mut conn, sync_info).boxed().await;
let watch_rooms = window.keys().map(AsRef::as_ref).stream();
let watchers = services
.sync
.watch(sender_user, sender_device, watch_rooms);
let stop_at = Instant::now()
.checked_add(Duration::from_millis(timeout))
.expect("configuration must limit maximum timeout");
conn.next_batch = services.globals.wait_pending().await?;
let sync_info = SyncInfo { services, sender_user, sender_device };
loop {
debug_assert!(
conn.globalsince <= conn.next_batch,
"next_batch should not be greater than since."
"since should not be greater than next_batch."
);
let window;
let watchers = services.sync.watch(
sender_user,
sender_device,
services.state_cache.rooms_joined(sender_user),
);
conn.next_batch = services.globals.wait_pending().await?;
(window, response.lists) = selector(&mut conn, sync_info).boxed().await;
if conn.globalsince < conn.next_batch {
let rooms =
handle_rooms(sync_info, &conn, &window).map_ok(|rooms| response.rooms = rooms);
let rooms = handle_rooms(sync_info, &conn, &window)
.map_ok(|response_rooms| response.rooms = response_rooms);
let extensions = handle_extensions(sync_info, &conn, &window)
.map_ok(|extensions| response.extensions = extensions);
.map_ok(|response_extensions| response.extensions = response_extensions);
try_join(rooms, extensions).boxed().await?;
for room_id in window.keys() {
conn.rooms
.entry(room_id.into())
.or_default()
.roomsince = conn.next_batch;
}
conn.update_rooms_epilogue(window.keys().map(AsRef::as_ref));
if !is_empty_response(&response) {
response.pos = conn.next_batch.to_string().into();
trace!(conn.globalsince, conn.next_batch, "response {response:?}");
conn.store(&services.sync, &conn_key);
return Ok(response);
}
}
if timeout_at(stop_at, watchers).await.is_err() || services.server.is_stopping() {
if timeout == 0
|| services.server.is_stopping()
|| timeout_at(stop_at, watchers)
.boxed()
.await
.is_err()
{
response.pos = conn.next_batch.to_string().into();
trace!(conn.globalsince, conn.next_batch, "timeout; empty response");
trace!(conn.globalsince, conn.next_batch, "timeout; empty response {response:?}");
conn.store(&services.sync, &conn_key);
return Ok(response);
}
trace!(
conn.globalsince,
last_batch = ?conn.next_batch,
count = ?services.globals.pending_count(),
stop_at = ?stop_at,
debug!(
?timeout,
last_since = conn.globalsince,
last_batch = conn.next_batch,
pend_count = ?services.globals.pending_count(),
"notified by watcher"
);
@@ -209,11 +231,7 @@ pub(crate) async fn sync_events_v5_route(
}
fn is_empty_response(response: &Response) -> bool {
response.extensions.is_empty()
&& response
.rooms
.iter()
.all(|(_, room)| room.timeline.is_empty() && room.invite_state.is_none())
response.extensions.is_empty() && response.rooms.is_empty()
}
#[tracing::instrument(
+1 -1
View File
@@ -8,7 +8,7 @@ use tuwunel_service::sync::Room;
use super::{Connection, SyncInfo, Window, extension_rooms_selector};
#[tracing::instrument(level = "trace", skip_all)]
#[tracing::instrument(name = "account_data", level = "trace", skip_all)]
pub(super) async fn collect(
sync_info: SyncInfo<'_>,
conn: &Connection,
+2 -2
View File
@@ -26,7 +26,7 @@ use tuwunel_service::sync::Connection;
use super::{SyncInfo, share_encrypted_room};
#[tracing::instrument(level = "trace", skip_all)]
#[tracing::instrument(name = "e2ee", level = "trace", skip_all)]
pub(super) async fn collect(
sync_info: SyncInfo<'_>,
conn: &Connection,
@@ -94,7 +94,7 @@ pub(super) async fn collect(
})
}
#[tracing::instrument(level = "trace", skip_all, fields(room_id))]
#[tracing::instrument(level = "trace", skip_all, fields(room_id), ret)]
async fn collect_room(
SyncInfo { services, sender_user, .. }: SyncInfo<'_>,
conn: &Connection,
+11 -14
View File
@@ -7,7 +7,7 @@ use tuwunel_core::{
is_equal_to, is_true,
utils::{
BoolExt, FutureBoolExt, IterStream, ReadyExt,
future::{OptionExt, ReadyEqExt},
future::{self, OptionExt, ReadyEqExt},
},
};
@@ -115,19 +115,16 @@ pub(super) async fn filter_room(
})
.into();
match_encrypted
.is_none_or(is_true!())
.and3(
match_invite.is_none_or(is_true!()),
match_direct.is_none_or(is_true!()),
match_direct_member.is_none_or(is_true!()),
)
.and3(
match_space_child.is_none_or(is_true!()),
match_room_type.is_none_or(is_true!()),
match_room_tag.is_none_or(is_true!()),
)
.await
future::and7(
match_invite.is_none_or(is_true!()),
match_encrypted.is_none_or(is_true!()),
match_direct.is_none_or(is_true!()),
match_direct_member.is_none_or(is_true!()),
match_space_child.is_none_or(is_true!()),
match_room_type.is_none_or(is_true!()),
match_room_tag.is_none_or(is_true!()),
)
.await
}
#[tracing::instrument(name = "filter_meta", level = "trace", skip_all)]
+2 -2
View File
@@ -13,7 +13,7 @@ use tuwunel_service::{rooms::read_receipt::pack_receipts, sync::Room};
use super::{Connection, SyncInfo, Window, extension_rooms_selector};
#[tracing::instrument(level = "trace", skip_all)]
#[tracing::instrument(name = "receipts", level = "trace", skip_all)]
pub(super) async fn collect(
sync_info: SyncInfo<'_>,
conn: &Connection,
@@ -44,7 +44,7 @@ pub(super) async fn collect(
Ok(response::Receipts { rooms })
}
#[tracing::instrument(level = "trace", skip_all, fields(room_id))]
#[tracing::instrument(level = "trace", skip_all, fields(room_id), ret)]
async fn collect_room(
SyncInfo { services, sender_user, .. }: SyncInfo<'_>,
conn: &Connection,
+88 -43
View File
@@ -5,10 +5,10 @@ use futures::{
future::{OptionFuture, join, join3, join4},
};
use ruma::{
JsOption, MxcUri, OwnedMxcUri, RoomId, UInt, UserId,
JsOption, MxcUri, OwnedMxcUri, RoomId, UserId,
api::client::sync::sync_events::{
UnreadNotificationsCount,
v5::{DisplayName, response},
v5::{DisplayName, response, response::Heroes},
},
events::{
StateEventType,
@@ -19,7 +19,7 @@ use ruma::{
},
};
use tuwunel_core::{
Result, at, debug_error, err, is_equal_to,
Result, at, err, is_equal_to,
matrix::{Event, StateKey, pdu::PduCount},
ref_at,
utils::{
@@ -45,15 +45,44 @@ static DEFAULT_BUMP_TYPES: [TimelineEventType; 6] =
pub(super) async fn handle(
SyncInfo { services, sender_user, .. }: SyncInfo<'_>,
conn: &Connection,
WindowRoom { lists, membership, room_id, .. }: &WindowRoom,
WindowRoom {
lists, membership, room_id, last_count, ..
}: &WindowRoom,
) -> Result<Option<response::Room>> {
debug_assert!(DEFAULT_BUMP_TYPES.is_sorted(), "DEFAULT_BUMP_TYPES is not sorted");
debug_assert!(
DEFAULT_BUMP_TYPES.is_sorted(),
"DEFAULT_BUMP_TYPES must be sorted for binary search"
);
let &Room { roomsince } = conn
let &Room { roomsince, .. } = conn
.rooms
.get(room_id)
.ok_or_else(|| err!("Missing connection state for {room_id}"))?;
debug_assert!(
*last_count > roomsince || *last_count == 0 || roomsince == 0,
"Stale room shouldn't be in the window"
);
if *membership == Some(MembershipState::Leave) {
return Ok(Some(response::Room {
initial: roomsince.eq(&0).then_some(true),
lists: lists.clone(),
membership: membership.clone(),
prev_batch: Some(conn.next_batch.to_string().into()),
limited: true,
required_state: vec![
services
.state_accessor
.room_state_get(room_id, &StateEventType::RoomMember, sender_user.as_str())
.map_ok(Event::into_format)
.await?,
],
..Default::default()
}));
}
let is_invite = *membership == Some(MembershipState::Invite);
let default_details = (0_usize, HashSet::new());
let (timeline_limit, required_state) = lists
@@ -84,41 +113,55 @@ pub(super) async fn handle(
})
.into();
let Ok(timeline) = timeline.await.transpose() else {
debug_error!(?room_id, "Missing timeline.");
return Ok(None);
};
let (timeline_pdus, limited, _lastcount) = timeline
.await
.flat_ok()
.unwrap_or_else(|| (Vec::new(), true, PduCount::default()));
let (timeline_pdus, limited, _lastcount) =
timeline.unwrap_or_else(|| (Vec::new(), true, PduCount::default()));
if roomsince != 0 && timeline_pdus.is_empty() && !is_invite {
return Ok(None);
}
let required_state = required_state
.into_iter()
.filter(|_| !timeline_pdus.is_empty())
.collect::<Vec<_>>();
let prev_batch = timeline_pdus
.first()
.map(at!(0))
.map(PduCount::into_unsigned)
.or_else(|| roomsince.ne(&0).then_some(roomsince))
.as_ref()
.map(ToString::to_string);
let bump_stamp = timeline_pdus
.iter()
.filter(|(_, pdu)| {
if *pdu.event_type() == TimelineEventType::RoomMember {
return pdu
.state_key()
.is_some_and(is_equal_to!(sender_user.as_str()));
}
DEFAULT_BUMP_TYPES
.binary_search(pdu.event_type())
.is_ok()
})
.fold(Option::<UInt>::None, |mut bump_stamp, (_, pdu)| {
let ts = pdu.origin_server_ts().0;
if bump_stamp.is_none_or(|bump_stamp| bump_stamp < ts) {
bump_stamp.replace(ts);
}
.map(at!(0))
.filter(|count| matches!(count, PduCount::Normal(_)))
.map(PduCount::into_unsigned)
.max()
.map(TryInto::try_into)
.flat_ok();
bump_stamp
});
let num_live: OptionFuture<_> = roomsince
.ne(&0)
.and_is(limited || timeline_pdus.len() >= timeline_limit)
.then(|| {
services
.timeline
.pdus(None, room_id, Some(roomsince.into()))
.count()
.map(TryInto::try_into)
.map(Result::ok)
})
.into();
let lazy = required_state
.iter()
@@ -183,14 +226,6 @@ pub(super) async fn handle(
})
.into();
let timeline = timeline_pdus
.iter()
.stream()
.filter_map(|item| ignored_filter(services, item.clone(), sender_user))
.map(at!(1))
.map(Event::into_format)
.collect();
let room_name = services
.state_accessor
.get_name(room_id)
@@ -235,15 +270,27 @@ pub(super) async fn handle(
.is_direct(room_id, sender_user)
.map(|is_dm| is_dm.then_some(is_dm));
let last_read_count = services
.user
.last_notification_read(sender_user, room_id);
let timeline = timeline_pdus
.iter()
.stream()
.filter_map(|item| ignored_filter(services, item.clone(), sender_user))
.map(at!(1))
.map(Event::into_format)
.collect();
let meta = join3(room_name, room_avatar, is_dm);
let events = join3(timeline, required_state, invite_state);
let events = join4(timeline, num_live, required_state, invite_state);
let member_counts = join(joined_count, invited_count);
let notification_counts = join(highlight_count, notification_count);
let notification_counts = join3(highlight_count, notification_count, last_read_count);
let (
(room_name, room_avatar, is_dm),
(timeline, required_state, invite_state),
(timeline, num_live, required_state, invite_state),
(joined_count, invited_count),
(highlight_count, notification_count),
(highlight_count, notification_count, _last_notification_read),
) = join4(meta, events, member_counts, notification_counts)
.boxed()
.await;
@@ -257,23 +304,21 @@ pub(super) async fn handle(
)
.await?;
let num_live = None; // Count events in timeline greater than global sync counter
Ok(Some(response::Room {
initial: Some(roomsince == 0),
initial: roomsince.eq(&0).then_some(true),
lists: lists.clone(),
membership: membership.clone(),
name: room_name.or(hero_name),
avatar: JsOption::from_option(room_avatar.or(heroes_avatar)),
is_dm,
heroes,
required_state,
invite_state: invite_state.flatten(),
prev_batch: prev_batch.as_deref().map(Into::into),
num_live: num_live.flatten(),
limited,
timeline,
bump_stamp,
heroes,
num_live,
joined_count,
invited_count,
unread_notifications: UnreadNotificationsCount { highlight_count, notification_count },
@@ -288,9 +333,9 @@ async fn calculate_heroes(
room_id: &RoomId,
room_name: Option<&DisplayName>,
room_avatar: Option<&MxcUri>,
) -> Result<(Option<Vec<response::Hero>>, Option<DisplayName>, Option<OwnedMxcUri>)> {
) -> Result<(Option<Heroes>, Option<DisplayName>, Option<OwnedMxcUri>)> {
const MAX_HEROES: usize = 5;
let heroes: Vec<_> = services
let heroes: Heroes = services
.state_cache
.room_members(room_id)
.ready_filter(|&member| member != sender_user)
+94 -85
View File
@@ -2,7 +2,7 @@ use std::cmp::Ordering;
use futures::{
FutureExt, StreamExt, TryFutureExt,
future::{OptionFuture, join3},
future::{OptionFuture, join5},
};
use ruma::{OwnedRoomId, UInt, events::room::member::MembershipState, uint};
use tuwunel_core::{
@@ -29,9 +29,8 @@ pub(super) async fn selector(
) -> (Window, ResponseLists) {
use MembershipState::*;
let SyncInfo { services, sender_user, request, .. } = sync_info;
let SyncInfo { services, sender_user, .. } = sync_info;
trace!(?request);
let mut rooms = services
.state_cache
.user_memberships(sender_user, Some(&[Join, Invite, Knock]))
@@ -48,6 +47,10 @@ pub(super) async fn selector(
.enumerate()
.for_each(|(i, room)| {
room.ranked = i;
conn.rooms
.entry(room.room_id.clone())
.or_default();
});
trace!(?rooms);
@@ -57,15 +60,15 @@ pub(super) async fn selector(
let window = select_window(sync_info, conn, rooms.iter(), &lists).await;
trace!(?window);
for room in &rooms {
conn.rooms
.entry(room.room_id.clone())
.or_default();
}
(window, lists)
}
#[tracing::instrument(
name = "window",
level = "debug",
skip_all,
fields(rooms = rooms.clone().count())
)]
async fn select_window<'a, Rooms>(
sync_info: SyncInfo<'_>,
conn: &Connection,
@@ -77,6 +80,8 @@ where
{
static FULL_RANGE: (UInt, UInt) = (UInt::MIN, UInt::MAX);
let SyncInfo { services, sender_user, .. } = sync_info;
let selections = lists
.keys()
.cloned()
@@ -95,49 +100,46 @@ where
.map(move |range| (id.clone(), range))
})
.flat_map(|(id, (start, end))| {
let list = rooms
rooms
.clone()
.filter(move |&room| room.lists.contains(&id));
let cycled = list.clone().all(|room| {
conn.rooms
.get(&room.room_id)
.is_some_and(|room| room.roomsince != 0)
});
list.enumerate()
.skip_while(move |&(i, room)| {
i < start
|| conn
.rooms
.get(&room.room_id)
.is_some_and(|conn_room| {
conn_room.roomsince >= room.last_count
|| (!cycled && conn_room.roomsince != 0)
})
.filter(move |&room| room.lists.contains(&id))
.filter(|&room| {
conn.rooms
.get(&room.room_id)
.is_some_and(|conn_room| {
conn_room.roomsince == 0 || room.last_count > conn_room.roomsince
})
})
.enumerate()
.skip_while(move |&(i, _)| i < start)
.take(end.saturating_add(1).saturating_sub(start))
.map(|(_, room)| (room.room_id.clone(), room.clone()))
});
})
.stream();
conn.subscriptions
let subscriptions = conn
.subscriptions
.iter()
.stream()
.broad_filter_map(async |(room_id, _)| {
filter_room_meta(sync_info, room_id)
.await
.then(|| WindowRoom {
room_id: room_id.clone(),
membership: None,
lists: Default::default(),
ranked: usize::MAX,
last_count: 0,
})
.into_option()?;
Some(WindowRoom {
room_id: room_id.clone(),
lists: Default::default(),
ranked: usize::MAX,
last_count: 0,
membership: services
.state_cache
.user_membership(sender_user, room_id)
.await,
})
})
.map(|room| (room.room_id.clone(), room))
.chain(selections.stream())
.collect()
.await
.map(|room| (room.room_id.clone(), room));
subscriptions.chain(selections).collect().await
}
#[tracing::instrument(
@@ -154,7 +156,7 @@ async fn match_lists_for_room(
) -> Option<WindowRoom> {
let SyncInfo { services, sender_user, .. } = sync_info;
let lists = conn
let (matched, lists) = conn
.lists
.iter()
.stream()
@@ -173,56 +175,74 @@ async fn match_lists_for_room(
.then(|| id.clone())
})
.collect::<ListIds>()
.map(|lists| (lists.is_empty().is_false(), lists))
.await;
let last_timeline_count: OptionFuture<_> = lists
.is_empty()
.is_false()
let last_notification: OptionFuture<_> = matched
.then(|| {
services
.timeline
.last_timeline_count(None, &room_id, None)
.map_ok(PduCount::into_unsigned)
.ok()
.user
.last_notification_read(sender_user, &room_id)
})
.into();
let last_account_count: OptionFuture<_> = lists
.is_empty()
.is_false()
.then(|| {
services
.account_data
.last_count(Some(room_id.as_ref()), sender_user, conn.next_batch)
.ok()
})
.into();
let last_receipt_count: OptionFuture<_> = lists
.is_empty()
.is_false()
let last_privateread: OptionFuture<_> = matched
.then(|| {
services
.read_receipt
.last_receipt_count(&room_id, sender_user.into(), conn.globalsince.into())
.map(Result::ok)
.last_privateread_update(sender_user, &room_id)
})
.into();
let (last_timeline_count, last_account_count, last_receipt_count) =
join3(last_timeline_count, last_account_count, last_receipt_count).await;
let last_receipt: OptionFuture<_> = matched
.then(|| {
services
.read_receipt
.last_receipt_count(&room_id, sender_user.into(), None)
.unwrap_or_default()
})
.into();
let last_account: OptionFuture<_> = matched
.then(|| {
services
.account_data
.last_count(Some(room_id.as_ref()), sender_user, Some(conn.next_batch))
.unwrap_or_default()
})
.into();
let last_timeline: OptionFuture<_> = matched
.then(|| {
services
.timeline
.last_timeline_count(None, &room_id, Some(conn.next_batch.into()))
.map_ok(PduCount::into_unsigned)
.unwrap_or_default()
})
.into();
let (last_timeline, last_notification, last_account, last_receipt, last_privateread) =
join5(last_timeline, last_notification, last_account, last_receipt, last_privateread)
.await;
Some(WindowRoom {
room_id: room_id.clone(),
membership,
lists,
ranked: 0,
last_count: [last_timeline_count, last_account_count, last_receipt_count]
.into_iter()
.map(Option::flatten)
.map(Option::unwrap_or_default)
.max()
.unwrap_or_default(),
last_count: [
last_timeline,
last_notification,
last_account,
last_receipt,
last_privateread,
]
.into_iter()
.map(Option::unwrap_or_default)
.filter(|count| conn.next_batch.ge(count))
.max()
.unwrap_or_default(),
})
}
@@ -243,15 +263,4 @@ where
})
}
fn room_sort(a: &WindowRoom, b: &WindowRoom) -> Ordering {
if a.membership != b.membership {
if a.membership == Some(MembershipState::Invite) {
return Ordering::Less;
}
if b.membership == Some(MembershipState::Invite) {
return Ordering::Greater;
}
}
b.last_count.cmp(&a.last_count)
}
fn room_sort(a: &WindowRoom, b: &WindowRoom) -> Ordering { b.last_count.cmp(&a.last_count) }
+1 -1
View File
@@ -4,7 +4,7 @@ use tuwunel_core::{self, Result};
use super::{Connection, SyncInfo};
#[tracing::instrument(level = "trace", skip_all)]
#[tracing::instrument(name = "to_device", level = "trace", skip_all, ret)]
pub(super) async fn collect(
SyncInfo { services, sender_user, sender_device, .. }: SyncInfo<'_>,
conn: &Connection,
+1 -1
View File
@@ -13,7 +13,7 @@ use tuwunel_core::{
use super::{Connection, SyncInfo, Window, extension_rooms_selector};
#[tracing::instrument(level = "trace", skip_all)]
#[tracing::instrument(name = "typing", level = "trace", skip_all, ret)]
pub(super) async fn collect(
sync_info: SyncInfo<'_>,
conn: &Connection,
+1 -1
View File
@@ -1,4 +1,4 @@
#![type_length_limit = "589824"] //TODO: REDUCE ME
#![type_length_limit = "262144"] //TODO: REDUCE ME
#![allow(clippy::toplevel_ref_arg)]
pub mod client;
+12
View File
@@ -4,6 +4,12 @@
pub trait BoolExt {
fn and<T>(self, t: Option<T>) -> Option<T>;
#[must_use]
fn and_is(self, b: bool) -> bool;
#[must_use]
fn and_if<F: FnOnce() -> bool>(self, f: F) -> bool;
fn and_then<T, F: FnOnce() -> Option<T>>(self, f: F) -> Option<T>;
#[must_use]
@@ -55,6 +61,12 @@ impl BoolExt for bool {
#[inline]
fn and<T>(self, t: Option<T>) -> Option<T> { self.then_some(t).flatten() }
#[inline]
fn and_if<F: FnOnce() -> Self>(self, f: F) -> Self { self.and_is(f()) }
#[inline]
fn and_is(self, b: Self) -> Self { self && b }
#[inline]
fn and_then<T, F: FnOnce() -> Option<T>>(self, f: F) -> Option<T> { self.then(f).flatten() }
+47 -18
View File
@@ -1,4 +1,5 @@
//! Extended external extensions to futures::FutureExt
#![allow(clippy::many_single_char_names, clippy::impl_trait_in_params)]
use std::marker::Unpin;
@@ -16,8 +17,6 @@ pub trait BoolExt
where
Self: Future<Output = bool> + Send,
{
type Result;
fn or<B>(self, b: B) -> impl Future<Output = bool> + Send
where
B: Future<Output = bool> + Send + Unpin,
@@ -46,15 +45,11 @@ impl<Fut> BoolExt for Fut
where
Fut: Future<Output = bool> + Send,
{
type Result = crate::Result<(), ()>;
fn or<B>(self, b: B) -> impl Future<Output = bool> + Send
where
B: Future<Output = bool> + Send + Unpin,
Self: Sized + Unpin,
{
let test = |test: bool| test.ok_or(Self::Result::Err(()));
select_ok([Left(self.map(test)), Right(b.map(test))]).map(|res| res.is_ok())
}
@@ -63,8 +58,6 @@ where
B: Future<Output = bool> + Send,
Self: Sized,
{
let test = |test: bool| test.ok_or(Self::Result::Err(()));
try_join(self.map(test), b.map(test)).map(|res| res.is_ok())
}
@@ -74,8 +67,6 @@ where
C: Future<Output = bool> + Send,
Self: Sized,
{
let test = |test: bool| test.ok_or(Self::Result::Err(()));
try_join3(self.map(test), b.map(test), c.map(test)).map(|res| res.is_ok())
}
@@ -86,8 +77,6 @@ where
D: Future<Output = bool> + Send,
Self: Sized,
{
let test = |test: bool| test.ok_or(Self::Result::Err(()));
try_join4(self.map(test), b.map(test), c.map(test), d.map(test)).map(|res| res.is_ok())
}
}
@@ -97,9 +86,7 @@ where
I: Iterator<Item = F> + Send,
F: Future<Output = bool> + Send,
{
type Result = crate::Result<(), ()>;
let args = args.map(|a| a.map(|a| a.ok_or(Result::Err(()))));
let args = args.map(|a| a.map(test));
try_join_all(args).map(|res| res.is_ok())
}
@@ -109,9 +96,51 @@ where
I: Iterator<Item = F> + Send,
F: Future<Output = bool> + Send + Unpin,
{
type Result = crate::Result<(), ()>;
let args = args.map(|a| a.map(|a| a.ok_or(Result::Err(()))));
let args = args.map(|a| a.map(test));
select_ok(args).map(|res| res.is_ok())
}
pub fn and4(
a: impl Future<Output = bool> + Send,
b: impl Future<Output = bool> + Send,
c: impl Future<Output = bool> + Send,
d: impl Future<Output = bool> + Send,
) -> impl Future<Output = bool> + Send {
a.and3(b, c, d)
}
pub fn and5(
a: impl Future<Output = bool> + Send,
b: impl Future<Output = bool> + Send,
c: impl Future<Output = bool> + Send,
d: impl Future<Output = bool> + Send,
e: impl Future<Output = bool> + Send,
) -> impl Future<Output = bool> + Send {
a.and2(b, c).and2(d, e)
}
pub fn and6(
a: impl Future<Output = bool> + Send,
b: impl Future<Output = bool> + Send,
c: impl Future<Output = bool> + Send,
d: impl Future<Output = bool> + Send,
e: impl Future<Output = bool> + Send,
f: impl Future<Output = bool> + Send,
) -> impl Future<Output = bool> + Send {
a.and3(b, c, d).and2(e, f)
}
pub fn and7(
a: impl Future<Output = bool> + Send,
b: impl Future<Output = bool> + Send,
c: impl Future<Output = bool> + Send,
d: impl Future<Output = bool> + Send,
e: impl Future<Output = bool> + Send,
f: impl Future<Output = bool> + Send,
g: impl Future<Output = bool> + Send,
) -> impl Future<Output = bool> + Send {
a.and3(b, c, d).and3(e, f, g)
}
fn test(test: bool) -> crate::Result<(), ()> { test.ok_or(()) }
+1 -1
View File
@@ -5,7 +5,7 @@ mod option_stream;
mod ready_eq_ext;
mod try_ext_ext;
pub use bool_ext::{BoolExt, and, or};
pub use bool_ext::{BoolExt, and, and4, and5, and6, and7, or};
pub use ext_ext::ExtExt;
pub use option_ext::OptionExt;
pub use option_stream::OptionStream;
+2
View File
@@ -20,4 +20,6 @@ where
if !self.engine.corked() {
self.engine.flush().expect("database flush error");
}
self.notify(key.as_ref());
}
+19 -5
View File
@@ -8,11 +8,12 @@ use std::{
use futures::pin_mut;
use serde::Serialize;
use tokio::sync::watch::{Sender, channel};
use tuwunel_core::implement;
use tuwunel_core::{debug, implement, smallvec::SmallVec};
use crate::keyval::{KeyBuf, serialize_key};
type Watchers = Mutex<BTreeMap<KeyBuf, Sender<()>>>;
type KeyVec = SmallVec<[KeyBuf; 1]>;
#[derive(Default)]
pub(super) struct Watch {
@@ -57,6 +58,14 @@ where
}
#[implement(super::Map)]
#[tracing::instrument(
level = "trace",
skip_all,
fields(
map = self.name(),
key = str::from_utf8(key.as_ref()).unwrap_or("<binary>"),
)
)]
pub(crate) fn notify<K>(&self, key: &K)
where
K: AsRef<[u8]> + Ord + ?Sized,
@@ -65,15 +74,20 @@ where
let mut watchers = self.watch.watchers.lock().expect("locked");
watchers
let num_notified = watchers
.range(range)
.rev()
.take_while(|(k, _)| key.as_ref().starts_with(k))
.filter_map(|(k, tx)| tx.send(()).is_err().then_some(k))
.cloned()
.collect::<Vec<_>>()
.collect::<KeyVec>()
.into_iter()
.for_each(|k| {
watchers.remove(&k);
.fold(0_usize, |num_notified, key| {
watchers.remove(&key);
num_notified.saturating_add(1)
});
if num_notified > 0 {
debug!(watchers = watchers.len(), num_notified, "notified");
}
}
+9
View File
@@ -200,6 +200,10 @@ pub(super) static MAPS: &[Descriptor] = &[
name: "roomuserid_lastprivatereadupdate",
..descriptor::RANDOM_SMALL
},
Descriptor {
name: "roomuserid_lastnotificationread",
..descriptor::RANDOM_SMALL
},
Descriptor {
name: "roomuserid_leftcount",
val_size_hint: Some(8),
@@ -337,6 +341,11 @@ pub(super) static MAPS: &[Descriptor] = &[
name: "userdeviceid_metadata",
..descriptor::RANDOM_SMALL
},
Descriptor {
name: "userdeviceconnid_conn",
block_size: 1024 * 16,
..descriptor::RANDOM_SMALL
},
Descriptor {
name: "userdeviceid_refresh",
..descriptor::RANDOM_SMALL
+2 -1
View File
@@ -169,10 +169,11 @@ pub async fn last_count<'a>(
&'a self,
room_id: Option<&'a RoomId>,
user_id: &'a UserId,
upper: u64,
upper: Option<u64>,
) -> Result<u64> {
type Key<'a> = (Option<&'a RoomId>, &'a UserId, u64, Ignore);
let upper = upper.unwrap_or(u64::MAX);
let key = (room_id, user_id, upper, Interfix);
self.db
.roomuserdataid_accountdata
+3 -2
View File
@@ -97,8 +97,9 @@ impl Data {
self.readreceiptid_readreceipt
.rev_keys_prefix(&key)
.ignore_err()
.ready_take_while(|(_, c, _): &Key<'_>| since.is_none_or(|since| since.gt(c)))
.ready_filter(|(_, _, u): &Key<'_>| user_id.as_ref().is_none_or(is_equal_to!(u)))
.ready_take_while(|(_, c, u): &Key<'_>| {
since.is_none_or(|since| since.gt(c)) && user_id.is_none_or(is_equal_to!(*u))
})
.map(|(_, c, _): Key<'_>| c)
.boxed()
.next()
+6 -2
View File
@@ -40,6 +40,7 @@ impl crate::Service for Service {
impl Service {
/// Replaces the previous read receipt.
#[tracing::instrument(skip(self), level = "debug", name = "set_receipt")]
pub async fn readreceipt_update(
&self,
user_id: &UserId,
@@ -58,6 +59,7 @@ impl Service {
}
/// Gets the latest private read receipt from the user in the room
#[tracing::instrument(skip(self), level = "debug", name = "get_private")]
pub async fn private_read_get(
&self,
room_id: &RoomId,
@@ -122,13 +124,13 @@ impl Service {
}
/// Sets a private read marker at PDU `count`.
#[tracing::instrument(skip(self), level = "debug")]
#[tracing::instrument(skip(self), level = "debug", name = "set_private")]
pub fn private_read_set(&self, room_id: &RoomId, user_id: &UserId, count: u64) {
self.db.private_read_set(room_id, user_id, count);
}
/// Returns the private read marker PDU count.
#[tracing::instrument(skip(self), level = "debug")]
#[tracing::instrument(skip(self), level = "debug", name = "get_private_count", ret)]
pub async fn private_read_get_count(
&self,
room_id: &RoomId,
@@ -140,12 +142,14 @@ impl Service {
}
/// Returns the PDU count of the last typing update in this room.
#[tracing::instrument(skip(self), level = "debug", name = "get_private_last", ret)]
pub async fn last_privateread_update(&self, user_id: &UserId, room_id: &RoomId) -> u64 {
self.db
.last_privateread_update(user_id, room_id)
.await
}
#[tracing::instrument(skip(self), level = "debug", name = "get_receipt_last", ret)]
pub async fn last_receipt_count(
&self,
room_id: &RoomId,
+4 -4
View File
@@ -22,7 +22,7 @@ use ruma::{
};
use tokio::sync::{Mutex, MutexGuard};
use tuwunel_core::{
Err, Error, Event, PduEvent, Result, implement,
Err, Error, Event, Result, implement,
utils::{
IterStream,
future::{BoolExt, TryExtExt},
@@ -513,12 +513,12 @@ async fn cache_insert(
fn get_space_child_events<'a>(
&'a self,
room_id: &'a RoomId,
) -> impl Stream<Item = PduEvent> + Send + 'a {
) -> impl Stream<Item = impl Event> + Send + 'a {
self.services
.state_accessor
.room_state_keys_with_ids(room_id, &StateEventType::SpaceChild)
.ready_filter_map(Result::ok)
.broad_filter_map(async move |(state_key, event_id): (_, OwnedEventId)| {
.broad_filter_map(async |(state_key, event_id): (_, OwnedEventId)| {
self.services
.timeline
.get_pdu(&event_id)
@@ -526,7 +526,7 @@ fn get_space_child_events<'a>(
.ok()
.await
})
.ready_filter_map(move |(state_key, pdu)| {
.ready_filter_map(|(state_key, pdu)| {
if let Ok(content) = pdu.get_content::<SpaceChildEventContent>() {
if content.via.is_empty() {
return None;
+5 -15
View File
@@ -1,7 +1,5 @@
use std::borrow::Borrow;
use futures::{Stream, StreamExt, TryFutureExt};
use ruma::{EventId, RoomId, events::StateEventType};
use ruma::{OwnedEventId, RoomId, events::StateEventType};
use serde::Deserialize;
use tuwunel_core::{
Result, err, implement,
@@ -82,16 +80,12 @@ pub fn room_state_full_pdus<'a>(
/// `state_key`).
#[implement(super::Service)]
#[tracing::instrument(skip(self), level = "debug")]
pub async fn room_state_get_id<Id>(
pub async fn room_state_get_id(
&self,
room_id: &RoomId,
event_type: &StateEventType,
state_key: &str,
) -> Result<Id>
where
Id: for<'de> Deserialize<'de> + Send + Sized + ToOwned,
<Id as ToOwned>::Owned: Borrow<EventId>,
{
) -> Result<OwnedEventId> {
self.services
.state
.get_room_shortstatehash(room_id)
@@ -103,15 +97,11 @@ where
/// `event_id` from the current state.
#[implement(super::Service)]
#[tracing::instrument(skip(self), level = "debug")]
pub fn room_state_keys_with_ids<'a, Id>(
pub fn room_state_keys_with_ids<'a>(
&'a self,
room_id: &'a RoomId,
event_type: &'a StateEventType,
) -> impl Stream<Item = Result<(StateKey, Id)>> + Send + 'a
where
Id: for<'de> Deserialize<'de> + Send + Sized + ToOwned + 'a,
<Id as ToOwned>::Owned: Borrow<EventId>,
{
) -> impl Stream<Item = Result<(StateKey, OwnedEventId)>> + Send + 'a {
self.services
.state
.get_room_shortstatehash(room_id)
+9 -21
View File
@@ -1,8 +1,8 @@
use std::{borrow::Borrow, ops::Deref, sync::Arc};
use std::{ops::Deref, sync::Arc};
use futures::{FutureExt, Stream, StreamExt, TryFutureExt, future::try_join, pin_mut};
use ruma::{
EventId, OwnedEventId, UserId,
OwnedEventId, UserId,
events::{
StateEventType,
room::member::{MembershipState, RoomMemberEventContent},
@@ -138,16 +138,12 @@ pub async fn state_get(
/// Returns a single EventId from `room_id` with key (`event_type`,
/// `state_key`).
#[implement(super::Service)]
pub async fn state_get_id<Id>(
pub async fn state_get_id(
&self,
shortstatehash: ShortStateHash,
event_type: &StateEventType,
state_key: &str,
) -> Result<Id>
where
Id: for<'de> Deserialize<'de> + Send + Sized + ToOwned,
<Id as ToOwned>::Owned: Borrow<EventId>,
{
) -> Result<OwnedEventId> {
let shorteventid = self
.state_get_shortid(shortstatehash, event_type, state_key)
.await?;
@@ -209,15 +205,11 @@ pub fn state_type_pdus<'a>(
/// Iterates the state_keys for an event_type in the state; current state
/// event_id included.
#[implement(super::Service)]
pub fn state_keys_with_ids<'a, Id>(
pub fn state_keys_with_ids<'a>(
&'a self,
shortstatehash: ShortStateHash,
event_type: &'a StateEventType,
) -> impl Stream<Item = (StateKey, Id)> + Send + 'a
where
Id: for<'de> Deserialize<'de> + Send + Sized + ToOwned + 'a,
<Id as ToOwned>::Owned: Borrow<EventId>,
{
) -> impl Stream<Item = (StateKey, OwnedEventId)> + Send + 'a {
let state_keys_with_short_ids = self
.state_keys_with_shortids(shortstatehash, event_type)
.unzip()
@@ -371,14 +363,10 @@ pub fn state_full_pdus(
/// Builds a StateMap by iterating over all keys that start
/// with state_hash, this gives the full state for the given state_hash.
#[implement(super::Service)]
pub fn state_full_ids<'a, Id>(
&'a self,
pub fn state_full_ids(
&self,
shortstatehash: ShortStateHash,
) -> impl Stream<Item = (ShortStateKey, Id)> + Send + 'a
where
Id: for<'de> Deserialize<'de> + Send + Sized + ToOwned + 'a,
<Id as ToOwned>::Owned: Borrow<EventId>,
{
) -> impl Stream<Item = (ShortStateKey, OwnedEventId)> + Send + '_ {
let shortids = self
.state_full_shortids(shortstatehash)
.ignore_err()
+6 -1
View File
@@ -24,7 +24,8 @@ impl crate::Service for Service {
db: Data {
userroomid_notificationcount: args.db["userroomid_notificationcount"].clone(),
userroomid_highlightcount: args.db["userroomid_highlightcount"].clone(),
roomuserid_lastnotificationread: args.db["userroomid_highlightcount"].clone(),
roomuserid_lastnotificationread: args.db["roomuserid_lastnotificationread"]
.clone(),
},
services: args.services.clone(),
}))
@@ -34,6 +35,7 @@ impl crate::Service for Service {
}
#[implement(Service)]
#[tracing::instrument(level = "debug", skip(self))]
pub fn reset_notification_counts(&self, user_id: &UserId, room_id: &RoomId) {
let count = self.services.globals.next_count();
@@ -52,6 +54,7 @@ pub fn reset_notification_counts(&self, user_id: &UserId, room_id: &RoomId) {
}
#[implement(Service)]
#[tracing::instrument(level = "debug", skip(self), ret)]
pub async fn notification_count(&self, user_id: &UserId, room_id: &RoomId) -> u64 {
let key = (user_id, room_id);
self.db
@@ -63,6 +66,7 @@ pub async fn notification_count(&self, user_id: &UserId, room_id: &RoomId) -> u6
}
#[implement(Service)]
#[tracing::instrument(level = "debug", skip(self), ret)]
pub async fn highlight_count(&self, user_id: &UserId, room_id: &RoomId) -> u64 {
let key = (user_id, room_id);
self.db
@@ -74,6 +78,7 @@ pub async fn highlight_count(&self, user_id: &UserId, room_id: &RoomId) -> u64 {
}
#[implement(Service)]
#[tracing::instrument(level = "debug", skip(self), ret)]
pub async fn last_notification_read(&self, user_id: &UserId, room_id: &RoomId) -> u64 {
let key = (room_id, user_id);
self.db
+191 -86
View File
@@ -1,20 +1,22 @@
mod watch;
use std::{
collections::BTreeMap,
sync::{Arc, Mutex as StdMutex},
collections::{BTreeMap, btree_map::Entry},
sync::Arc,
};
use futures::{FutureExt, Stream};
use ruma::{
DeviceId, OwnedDeviceId, OwnedRoomId, OwnedUserId, UserId,
DeviceId, OwnedDeviceId, OwnedRoomId, OwnedUserId, RoomId, UserId,
api::client::sync::sync_events::v5::{
ConnId as ConnectionId, ListId, Request, request,
request::{AccountData, E2EE, Receipts, ToDevice, Typing},
},
};
use serde::{Deserialize, Serialize};
use tokio::sync::Mutex as TokioMutex;
use tuwunel_core::{Result, err, implement, is_equal_to};
use tuwunel_database::Map;
use tuwunel_core::{Result, at, debug, err, implement, is_equal_to, utils::stream::TryIgnore};
use tuwunel_database::{Cbor, Deserialized, Map};
pub struct Service {
services: Arc<crate::services::OnceServices>,
@@ -22,7 +24,8 @@ pub struct Service {
db: Data,
}
pub struct Data {
struct Data {
userdeviceconnid_conn: Arc<Map>,
todeviceid_events: Arc<Map>,
userroomid_joined: Arc<Map>,
userroomid_invitestate: Arc<Map>,
@@ -36,24 +39,25 @@ pub struct Data {
roomusertype_roomuserdataid: Arc<Map>,
readreceiptid_readreceipt: Arc<Map>,
userid_lastonetimekeyupdate: Arc<Map>,
roomuserid_lastnotificationread: Arc<Map>,
}
#[derive(Debug, Default)]
#[derive(Debug, Default, Deserialize, Serialize)]
pub struct Connection {
pub lists: Lists,
pub rooms: Rooms,
pub subscriptions: Subscriptions,
pub extensions: request::Extensions,
pub globalsince: u64,
pub next_batch: u64,
pub lists: Lists,
pub extensions: request::Extensions,
pub subscriptions: Subscriptions,
pub rooms: Rooms,
}
#[derive(Clone, Debug, Default)]
#[derive(Clone, Copy, Debug, Default, Deserialize, Serialize)]
pub struct Room {
pub roomsince: u64,
}
type Connections = StdMutex<BTreeMap<ConnectionKey, ConnectionVal>>;
type Connections = TokioMutex<BTreeMap<ConnectionKey, ConnectionVal>>;
pub type ConnectionVal = Arc<TokioMutex<Connection>>;
pub type ConnectionKey = (OwnedUserId, OwnedDeviceId, Option<ConnectionId>);
@@ -65,6 +69,7 @@ impl crate::Service for Service {
fn build(args: &crate::Args<'_>) -> Result<Arc<Self>> {
Ok(Arc::new(Self {
db: Data {
userdeviceconnid_conn: args.db["userdeviceconnid_conn"].clone(),
todeviceid_events: args.db["todeviceid_events"].clone(),
userroomid_joined: args.db["userroomid_joined"].clone(),
userroomid_invitestate: args.db["userroomid_invitestate"].clone(),
@@ -78,6 +83,8 @@ impl crate::Service for Service {
roomusertype_roomuserdataid: args.db["roomusertype_roomuserdataid"].clone(),
readreceiptid_readreceipt: args.db["readreceiptid_readreceipt"].clone(),
userid_lastonetimekeyupdate: args.db["userid_lastonetimekeyupdate"].clone(),
roomuserid_lastnotificationread: args.db["roomuserid_lastnotificationread"]
.clone(),
},
services: args.services.clone(),
connections: Default::default(),
@@ -87,7 +94,178 @@ impl crate::Service for Service {
fn name(&self) -> &str { crate::service::make_name(std::module_path!()) }
}
#[implement(Service)]
#[tracing::instrument(level = "debug", skip(self))]
pub async fn clear_connections(
&self,
user_id: Option<&UserId>,
device_id: Option<&DeviceId>,
conn_id: Option<&ConnectionId>,
) {
self.connections
.lock()
.await
.retain(|(conn_user_id, conn_device_id, conn_conn_id), _| {
let retain = user_id.is_none_or(is_equal_to!(conn_user_id))
&& device_id.is_none_or(is_equal_to!(conn_device_id))
&& (conn_id.is_none() || conn_id == conn_conn_id.as_ref());
if !retain {
self.db
.userdeviceconnid_conn
.del((conn_user_id, conn_device_id, conn_conn_id));
}
retain
});
}
#[implement(Service)]
#[tracing::instrument(level = "debug", skip(self))]
pub async fn drop_connection(&self, key: &ConnectionKey) {
let mut cache = self.connections.lock().await;
self.db.userdeviceconnid_conn.del(key);
cache.remove(key);
}
#[implement(Service)]
#[tracing::instrument(level = "debug", skip(self))]
pub async fn load_or_init_connection(&self, key: &ConnectionKey) -> ConnectionVal {
let mut cache = self.connections.lock().await;
match cache.entry(key.clone()) {
| Entry::Occupied(val) => val.get().clone(),
| Entry::Vacant(val) => {
let conn = self
.db
.userdeviceconnid_conn
.qry(key)
.boxed()
.await
.deserialized::<Cbor<_>>()
.map(at!(0))
.map(TokioMutex::new)
.map(Arc::new)
.unwrap_or_default();
val.insert(conn).clone()
},
}
}
#[implement(Service)]
#[tracing::instrument(level = "debug", skip(self))]
pub async fn load_connection(&self, key: &ConnectionKey) -> Result<ConnectionVal> {
let mut cache = self.connections.lock().await;
match cache.entry(key.clone()) {
| Entry::Occupied(val) => Ok(val.get().clone()),
| Entry::Vacant(val) => self
.db
.userdeviceconnid_conn
.qry(key)
.await
.deserialized::<Cbor<_>>()
.map(at!(0))
.map(TokioMutex::new)
.map(Arc::new)
.map(|conn| val.insert(conn).clone()),
}
}
#[implement(Service)]
#[tracing::instrument(level = "debug", skip(self))]
pub async fn get_loaded_connection(&self, key: &ConnectionKey) -> Result<ConnectionVal> {
self.connections
.lock()
.await
.get(key)
.cloned()
.ok_or_else(|| err!(Request(NotFound("Connection not found."))))
}
#[implement(Service)]
#[tracing::instrument(level = "trace", skip(self))]
pub async fn list_loaded_connections(&self) -> Vec<ConnectionKey> {
self.connections
.lock()
.await
.keys()
.cloned()
.collect()
}
#[implement(Service)]
#[tracing::instrument(level = "trace", skip(self))]
pub fn list_stored_connections(&self) -> impl Stream<Item = ConnectionKey> {
self.db.userdeviceconnid_conn.keys().ignore_err()
}
#[implement(Service)]
#[tracing::instrument(level = "trace", skip(self))]
pub async fn is_connection_loaded(&self, key: &ConnectionKey) -> bool {
self.connections.lock().await.contains_key(key)
}
#[implement(Service)]
#[tracing::instrument(level = "trace", skip(self))]
pub async fn is_connection_stored(&self, key: &ConnectionKey) -> bool {
self.db.userdeviceconnid_conn.contains(key).await
}
#[inline]
pub fn into_connection_key<U, D, C>(user_id: U, device_id: D, conn_id: Option<C>) -> ConnectionKey
where
U: Into<OwnedUserId>,
D: Into<OwnedDeviceId>,
C: Into<ConnectionId>,
{
(user_id.into(), device_id.into(), conn_id.map(Into::into))
}
#[implement(Connection)]
#[tracing::instrument(level = "debug", skip(self, service))]
pub fn store(&self, service: &Service, key: &ConnectionKey) {
service
.db
.userdeviceconnid_conn
.put(key, Cbor(self));
debug!(
since = %self.globalsince,
next_batch = %self.next_batch,
"Persisted connection state"
);
}
#[implement(Connection)]
#[tracing::instrument(level = "debug", skip(self))]
pub fn update_rooms_prologue(&mut self, retard_since: Option<u64>) {
self.rooms.values_mut().for_each(|room| {
if let Some(retard_since) = retard_since {
if room.roomsince > retard_since {
room.roomsince = retard_since;
}
}
});
}
#[implement(Connection)]
#[tracing::instrument(level = "debug", skip_all)]
pub fn update_rooms_epilogue<'a, Rooms>(&mut self, window: Rooms)
where
Rooms: Iterator<Item = &'a RoomId> + Send + 'a,
{
window.for_each(|room_id| {
let room = self.rooms.entry(room_id.into()).or_default();
room.roomsince = self.next_batch;
});
}
#[implement(Connection)]
#[tracing::instrument(level = "debug", skip_all)]
pub fn update_cache(&mut self, request: &Request) {
Self::update_cache_lists(request, self);
Self::update_cache_subscriptions(request, self);
@@ -188,76 +366,3 @@ fn some_or_sticky<T: Clone>(target: Option<&T>, cached: &mut Option<T>) {
cached.replace(target.clone());
}
}
#[implement(Service)]
pub fn clear_connections(
&self,
user_id: Option<&UserId>,
device_id: Option<&DeviceId>,
conn_id: Option<&ConnectionId>,
) {
self.connections.lock().expect("locked").retain(
|(conn_user_id, conn_device_id, conn_conn_id), _| {
!(user_id.is_none_or(is_equal_to!(conn_user_id))
&& device_id.is_none_or(is_equal_to!(conn_device_id))
&& (conn_id.is_none() || conn_id == conn_conn_id.as_ref()))
},
);
}
#[implement(Service)]
pub fn drop_connection(&self, key: &ConnectionKey) {
self.connections
.lock()
.expect("locked")
.remove(key);
}
#[implement(Service)]
pub fn list_connections(&self) -> Vec<ConnectionKey> {
self.connections
.lock()
.expect("locked")
.keys()
.cloned()
.collect()
}
#[implement(Service)]
pub fn init_connection(&self, key: &ConnectionKey) -> ConnectionVal {
self.connections
.lock()
.expect("locked")
.entry(key.clone())
.and_modify(|existing| *existing = ConnectionVal::default())
.or_default()
.clone()
}
#[implement(Service)]
pub fn find_connection(&self, key: &ConnectionKey) -> Result<ConnectionVal> {
self.connections
.lock()
.expect("locked")
.get(key)
.cloned()
.ok_or_else(|| err!(Request(NotFound("Connection not found."))))
}
#[implement(Service)]
pub fn contains_connection(&self, key: &ConnectionKey) -> bool {
self.connections
.lock()
.expect("locked")
.contains_key(key)
}
#[inline]
pub fn into_connection_key<U, D, C>(user_id: U, device_id: D, conn_id: Option<C>) -> ConnectionKey
where
U: Into<OwnedUserId>,
D: Into<OwnedDeviceId>,
C: Into<ConnectionId>,
{
(user_id.into(), device_id.into(), conn_id.map(Into::into))
}
+5
View File
@@ -85,6 +85,11 @@ where
let roomuser_prefix = (room_id, user_id);
let typing_room_id = room_id.to_owned();
let watchers = [
// Notification clearance
self.db
.roomuserid_lastnotificationread
.watch_prefix(&roomuser_prefix)
.boxed(),
// Key changes
self.db
.keychangeid_userid