Dies ist eine alte Version des Dokuments!
HowTo Deploy Synapse with Multiple Workers using docker-compose
Status: Not working yet (25th of February 2022)
while trying to give my docker-compose setup two workers (multicontainer) - 1 for client, 1 for federation. While I made some progress today I am struggling with a strange problem:
- I use nginx reverse proxy to filter $request_uri and send it to one of the 3 upstreams.
- I have my nginx running and filtering, my workers and my master are up and running.
- But the instance isn't reachable (If I give my master the 8008-port back and don't send it through nginx any longer it's working again - but monolithic of course)
The Network Problem
- Checking my nginx log shows: A lot of
connect() failed (111: Connection refused) while connecting to upstreamwith every request. But ther workers themselves are reachable.- What is NOT reachable is
https://127.0.0.1:9093, i.e. the replication port. -->curl: (7) Failed to connect to 127.0.0.1 port 9093: Connection refused - But HTTP replication port is set in homeserver.yaml
Hint from Chat regarding external networks in docker: https://matrix.to/#/!ehXvUhWNASUkSLvAGP:matrix.org/$KUn6O3dNbSOrRMo0VNu_aBtzTRIYnj1Gr0AChGfux_0?via=matrix.org&via=matrix.breakpointingbad.com&via=libera.chat
- I don't see how an external network should help me. I of course don't use two or more docker-compose files. Ich want to enclose the whole setup in one seperate network and because in my setup all containers use the same network there shouldn't be a problem.
But the hint let me
docker-compose exec synapse bash.- There I
curl localhost:9093... and receive an 404 error page! 8-o curl localhost:9092or any other arbitrary port returns the expectedFailed to connect to localhost port 9092: Connection refused- I also can't reach the port 6379 of the redis container :-/ ... Maybe it's no surprise that this setup doesn't work ...
Leaving the container and opening a bash in one of the two workers.
- Here the same curl to port 9093 leads to ... Connection refused. As experienced. --> So the opened replication port isn't reached to the network?!
- As you will see down in my docker-compose.yml all containers share the default network.
- Changing into a bash inside one of the workers: I can't curl from the master`s bash to the ports of the workers.
- Having a look in docker-compose docs I reminded myself that you can call the other containers with e.g.
curl client_worker:8084- ... and in deed: I can reach it! So I can't use localhost or 127.0.0.1 or 0.0.0.0 for inter-container-communication but need to use the service name!
- Unfortunately that doesn't work in the nginx.conf :/
- Maybe I can solve this later by defining links (https://docs.docker.com/compose/networking/#links)?
Progress!
- Solved it: With the self defined
container_nameitem in the docker-compose.yml it is possible to access the containers in nginx.conf!- I updated the files down on this page.
- So far the setup produces a working matrix server, BUT:
- It seems that still the entire traffic is routed to the master process.
- All I get in the logs of the workers is
*** STARTING SERVER ***** matrix-federation | 2022-02-26 10:05:49,433 - root - 349 - WARNING - main- Server /usr/local/lib/python3.8/site-packages/synapse/app/generic_worker.py version 1.53.0 matrix-federation | 2022-02-26 10:05:49,650 - synapse.app - 49 - WARNING - sentinel- Failed to listen on 0.0.0.0, continuing because listening on [::]
- Redis outputs now
redis_1 | 1:M 26 Feb 2022 10:55:13.101 * 1 changes in 3600 seconds. Saving... redis_1 | 1:M 26 Feb 2022 10:55:13.594 * Background saving started by pid 19 redis_1 | 19:C 26 Feb 2022 10:55:14.103 * DB saved on disk redis_1 | 19:C 26 Feb 2022 10:55:14.104 * RDB: 0 MB of memory used by copy-on-write redis_1 | 1:M 26 Feb 2022 10:55:14.197 * Background saving terminated with success
- So it seems that Redis is now working but the workers seem to listen to the wrong bindings? Or are the mappings in nginx.conf not properly working?
- I have a lot of those errors in the nginx log. Don`t know if this has to do with it.
2022/02/26 11:20:48 [error] 32#32: *11814 open() "/etc/nginx/html/.well-known/matrix/server" failed (2: No such file or directory), client: 192.168.160.1, server: localhost, request: "GET /.well-known/matrix/server HTTP/1.1", host: "ismus.net" 192.168.160.1 - - [26/Feb/2022:11:20:48 +0000] "GET /.well-known/matrix/server HTTP/1.1" 404 153 "-" "Synapse/1.52.0" 192.168.160.1 - - [26/Feb/2022:11:20:48 +0000] "PUT /_matrix/federation/v1/send/1645518187107 HTTP/1.1" 200 21 "-" "Synapse/1.52.0"
My docker-compose.yml so far
version: '3'
services:
postgres:
restart: unless-stopped
networks:
- default
environment:
POSTGRES_PASSWORD: "[some-secret-password]"
POSTGRES_USER: synapse
POSTGRES_DB: synapse
POSTGRES_INITDB_ARGS: "--encoding='UTF8' --lc-collate='C' --lc-ctype='C'"
POSTGRES_HOST_AUTH_METHOD: trust
image: postgres:13-alpine
volumes:
- /volume1/docker/volumes/matrix-synapse/postgres:/var/lib/postgresql/data
nginx:
image: nginx:stable-alpine
volumes:
- /volume1/docker/volumes/matrix-synapse/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
ports:
- 8008:8008
networks:
- default
synapse:
image: matrixdotorg/synapse:latest
container_name: matrix-synapse
restart: "unless-stopped"
ports:
- 84:8448
- 8888:8888
environment:
- SYNAPSE_REPORT_STATS=no
- SYNAPSE_SERVER_NAME=ismus.net
- SYNAPSE_CONFIG_PATH=/data/homeserver-postgres.yaml
- TZ=Berlin/Europe
depends_on: ["postgres"]
volumes:
- /volume1/docker/volumes/matrix-synapse:/data
networks:
- default
client_worker:
image: matrixdotorg/synapse:latest
container_name: matrix-client
restart: "unless-stopped"
command:
- 'run'
- '--config-path=/data/homeserver-postgres.yaml'
- '--config-path=/data/workers/synchrotron-1.yaml' # Important! You can _NOT_ do this by `- SYNAPSE_CONFIG_PATH=/data/workers/synchrotron-1.yaml` in the environment section!
environment:
- SYNAPSE_REPORT_STATS=no
- SYNAPSE_SERVER_NAME=ismus.net
- SYNAPSE_WORKER=synapse.app.generic_worker
- TZ=Berlin/Europe
depends_on:
- redis
- synapse
volumes:
- /volume1/docker/volumes/matrix-synapse:/data
networks:
- default
federation_worker:
image: matrixdotorg/synapse:latest
container_name: matrix-federation
restart: "unless-stopped"
command:
- 'run'
- '--config-path=/data/homeserver-postgres.yaml'
- '--config-path=/data/workers/federation-1.yaml'
environment:
- SYNAPSE_REPORT_STATS=no
- SYNAPSE_SERVER_NAME=ismus.net
- SYNAPSE_WORKER=synapse.app.generic_worker
- TZ=Berlin/Europe
depends_on:
- redis
- synapse
volumes:
- /volume1/docker/volumes/matrix-synapse:/data
networks:
- default
redis:
image: "redis:latest"
restart: "unless-stopped"
networks:
- default
networks:
default:
The nginx.conf so far:
events {
worker_connections 1024;
}
http {
upstream synapse_master {
server matrix-synapse:8888;
}
upstream synapse_client {
server matrix-client:8084;
}
upstream synapse_federation {
server matrix-federation:8083;
}
map $request_uri $synapse_backend {
default synapse_master;
# Sync requests
~*^/_matrix/client/(v2_alpha|r0)/sync$ synapse_client;
~*^/_matrix/client/(api/v1|v2_alpha|r0)/events$ synapse_client;
~*^/_matrix/client/(api/v1|r0)/initialSync$ synapse_client;
~*^/_matrix/client/(api/v1|r0)/rooms/[^/]+/initialSync$ synapse_client;
# Federation requests
~*^/_matrix/federation/v1/event/ synapse_federation;
~*^/_matrix/federation/v1/state/ synapse_federation;
~*^/_matrix/federation/v1/state_ids/ synapse_federation;
~*^/_matrix/federation/v1/backfill/ synapse_federation;
~*^/_matrix/federation/v1/get_missing_events/ synapse_federation;
~*^/_matrix/federation/v1/publicRooms synapse_federation;
~*^/_matrix/federation/v1/query/ synapse_federation;
~*^/_matrix/federation/v1/make_join/ synapse_federation;
~*^/_matrix/federation/v1/make_leave/ synapse_federation;
~*^/_matrix/federation/v1/send_join/ synapse_federation;
~*^/_matrix/federation/v2/send_join/ synapse_federation;
~*^/_matrix/federation/v1/send_leave/ synapse_federation;
~*^/_matrix/federation/v2/send_leave/ synapse_federation;
~*^/_matrix/federation/v1/invite/ synapse_federation;
~*^/_matrix/federation/v2/invite/ synapse_federation;
~*^/_matrix/federation/v1/query_auth/ synapse_federation;
~*^/_matrix/federation/v1/event_auth/ synapse_federation;
~*^/_matrix/federation/v1/exchange_third_party_invite/ synapse_federation;
~*^/_matrix/federation/v1/user/devices/ synapse_federation;
~*^/_matrix/federation/v1/get_groups_publicised$ synapse_federation;
~*^/_matrix/key/v2/query synapse_federation;
# Inbound federation transaction request
~*^/_matrix/federation/v1/send/ synapse_federation;
# Client API requests
~*^/_matrix/client/(api/v1|r0|unstable)/publicRooms$ synapse_client;
~*^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/joined_members$ synapse_client;
~*^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/context/.*$ synapse_client;
~*^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/members$ synapse_client;
~*^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state$ synapse_client;
~*^/_matrix/client/(api/v1|r0|unstable)/account/3pid$ synapse_client;
~*^/_matrix/client/(api/v1|r0|unstable)/devices$ synapse_client;
~*^/_matrix/client/(api/v1|r0|unstable)/keys/query$ synapse_client;
~*^/_matrix/client/(api/v1|r0|unstable)/keys/changes$ synapse_client;
~*^/_matrix/client/versions$ synapse_client;
~*^/_matrix/client/(api/v1|r0|unstable)/voip/turnServer$ synapse_client;
~*^/_matrix/client/(api/v1|r0|unstable)/joined_groups$ synapse_client;
~*^/_matrix/client/(api/v1|r0|unstable)/publicised_groups$ synapse_client;
~*^/_matrix/client/(api/v1|r0|unstable)/publicised_groups/ synapse_client;
~*^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/event/ synapse_client;
~*^/_matrix/client/(api/v1|r0|unstable)/joined_rooms$ synapse_client;
~*^/_matrix/client/(api/v1|r0|unstable)/search$ synapse_client;
# Registration/login requests
~*^/_matrix/client/(api/v1|r0|unstable)/login$ synapse_client;
~*^/_matrix/client/(r0|unstable)/register$ synapse_client;
# Event sending requests
~*^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/redact synapse_client;
~*^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/send synapse_client;
~*^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state/ synapse_client;
~*^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$ synapse_client;
~*^/_matrix/client/(api/v1|r0|unstable)/join/ synapse_client;
~*^/_matrix/client/(api/v1|r0|unstable)/profile/ synapse_client;
}
server {
listen 8008;
listen [::]:8008;
server_name localhost;
clien_max_body_size 100M;
location ~* ^(\\/_matrix|\\/_synapse) {
proxy_pass http://$synapse_backend;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $remote_addr;
add_header Backend-Server $synapse_backend;
}
}
}
Old Outdated Information
As I experienced significant delay with my synapse instance when joining rooms on other instances like matrix.org I was told that I urgently need to split the cpu load of it into multiple processes. By this the load could be distributed to multiple cpu cores instead of only one that would limit the speed of processing of large room content. Well, until now I know next to nothing about so called workers and how to set those up in a dockerized environment. On its top: There's yet a poor amount of documentation or tutorials regarding this. Or at least I couldn't find it.
My efforts for now lead to some [general information about in synapse]. A few hints on how to configure separate workers in docker/docker-compose can be found in the official [README].
Meanwhile developers are obviously [working on a "batteries included" docker-image for multiprocess synapse] which already makes use of workers. But that's not for production at the moment unfortunately.
What I understand so far
- I have to add a separate synapse-service for every worker in the docker-compose.yml.
- The best recommendation I yet received about what workers I should setup for a start is "synchrotron and two federation readers".
- I guess I will start with this recommendation and when this works I may think about the other options.
- I should definitely switch from my minimal homeserver.yaml to a full commented version since it will lack the placeholder variables for docker
- every worker uses both the homeserver.yaml and a worker-config in e.g. the workers directory
- I have to work out how the reverse proxy of my DSM will distribute http requests between the workers as it is not so easy to configure as a full nginx.
- I guess everything gets in over 8448 and has to be distributed to the workers over the local ports? I.e. no further ports have to be exposed publicy? I'm not sure how to manage this except by adding another nginx to the docker-compose.yml. At least there is a [sample file for the simple nginx setup] ...
- redis is for shared database communication between the workers. Not sure yet how that works but maybe that will work automatically after activating redis in the homeserver.yaml.
n>
homer