Dies ist eine alte Version des Dokuments!
HowTo Deploy with Multiple Workers using
Status: Finally working!!! (1st of March 2022)
I can finally announce a working docker-compose setup with 2 workers! That is up to now one worker for client related requests and one for the federation requests. Of course the amount of workers is expandable and only depends on how you route the requests to the workers and the main process using your reverse proxy (here: nginx).
Scope
I document my relevant config files except my homeserver.yaml. For briefness I will only show the small parts of it that are relevant for having the workers to communicate successfully with the main process. It's assumed that you have an already working homeserver.yaml based on a monolithic synapse setup. I will try to comment some critical details so that you will know what to adapt and what to adopt.
My docker-compose.yml
The base of all
Please not that the container_name fields are crucial for the setup to work. You need them to address the containers between each other.
version: '3'
services:
postgres:
restart: unless-stopped
networks:
- default
environment:
POSTGRES_PASSWORD: "[some-secret-password]"
POSTGRES_USER: synapse
POSTGRES_DB: synapse
POSTGRES_INITDB_ARGS: "--encoding='UTF8' --lc-collate='C' --lc-ctype='C'"
POSTGRES_HOST_AUTH_METHOD: trust
image: postgres:13-alpine
volumes:
- /volume1/docker/volumes/matrix-synapse/postgres:/var/lib/postgresql/data
nginx:
image: nginx:stable-alpine
container_name: matrix-nginx
volumes:
- /volume1/docker/volumes/matrix-synapse/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
ports:
- 8008:8008 ## Those outbound ports fit the needs of my environment where 8008 leads to another proxy server that serves to port 80, 443 and 8448
- 84:8484
depends_on: ## That's actually very important! The startup order __must__ be: redis, postgres, synapse, worker[1:n], nginx
- synapse
- federation_worker
- client_worker
networks:
- default
synapse:
image: matrixdotorg/synapse:latest
container_name: matrix-synapse
restart: "unless-stopped"
environment:
- SYNAPSE_REPORT_STATS=no
- SYNAPSE_SERVER_NAME=ismus.net # of course you must adjust this to your domain
- SYNAPSE_CONFIG_PATH=/data/homeserver-postgres.yaml
- TZ=Berlin/Europe
depends_on: ["postgres"]
volumes:
- /volume1/docker/volumes/matrix-synapse:/data
networks:
- default
client_worker:
image: matrixdotorg/synapse:latest
container_name: matrix-client
restart: "unless-stopped"
command:
- 'run'
- '--config-path=/data/homeserver-postgres.yaml'
- '--config-path=/data/workers/synchrotron-1.yaml' # Important! You can _NOT_ do this by `- SYNAPSE_CONFIG_PATH=/data/workers/synchrotron-1.yaml` in the environment section!
environment:
- SYNAPSE_REPORT_STATS=no
- SYNAPSE_SERVER_NAME=ismus.net
- SYNAPSE_WORKER=synapse.app.generic_worker
- TZ=Berlin/Europe
depends_on:
- synapse
volumes:
- /volume1/docker/volumes/matrix-synapse:/data
networks:
- default
federation_worker:
image: matrixdotorg/synapse:latest
container_name: matrix-federation
restart: "unless-stopped"
command:
- 'run'
- '--config-path=/data/homeserver-postgres.yaml'
- '--config-path=/data/workers/federation-1.yaml'
environment:
- SYNAPSE_REPORT_STATS=no
- SYNAPSE_SERVER_NAME=ismus.net
- SYNAPSE_WORKER=synapse.app.generic_worker
- TZ=Berlin/Europe
depends_on:
- client-worker
volumes:
- /volume1/docker/volumes/matrix-synapse:/data
networks:
- default
redis:
image: "redis:latest"
restart: "unless-stopped"
networks:
- default
networks:
default:
The nginx.conf so far:
The nginx has the task to distribute the different kinds of request between the workers. I had not much idea how to config nginx before I found a template which looked much alike the following. But I still had a lot to correct and find out to make it work properly. If you want to re-balance the work between the two or even more workers you will have to alter the map block and if needed add further upstreams per worker.
As described above I had the special case that in my environment there's another proxy server on top that is responsible for domain and ssl so that I only needed the nginx container to distribute requests and reach traffic through between this on top reverse proxy and the synapse processes. If you want to use the nginx container as fully responsible server have a look into the official synapse documentation and add the few missing parts.
events {
worker_connections 1024;
}
http {
upstream synapse_master {
server matrix-synapse:8888; # Here neither localhost nor 127.0.0.1 or 0.0.0.0 worked for me.
}
upstream synapse_client {
server matrix-client:8084; # But the container_name labels are resolved to the local ips of the containers.
}
upstream synapse_federation {
server matrix-federation:8083; # That makes the docker-setup quite fail-safe in case of eventually changing network conditions, I guess.
}
map_hash_bucket_size 128;
map $request_uri $synapse_backend {
default synapse_master; # Important: Makes safe that everything _not_ covered by the regex will go to the master process by default and by this: won't be lost!
# The requests are basically copy paste from the [official docs](https://matrix-org.github.io/synapse/latest/workers.html).
# Sync requests
"~^/_matrix/client/(v2_alpha|r0)/sync$" synapse_client;
"~^/_matrix/client/(api/v1|v2_alpha|r0)/events$" synapse_client;
"~^/_matrix/client/(api/v1|r0)/initialSync$" synapse_client;
"~^/_matrix/client/(api/v1|r0)/rooms/[^/]+/initialSync$" synapse_client;
# Federation requests
"~^/_matrix/federation/v1/event/" synapse_federation;
"~^/_matrix/federation/v1/state/" synapse_federation;
"~^/_matrix/federation/v1/state_ids/" synapse_federation;
"~^/_matrix/federation/v1/backfill/" synapse_federation;
"~^/_matrix/federation/v1/get_missing_events/" synapse_federation;
"~^/_matrix/federation/v1/publicRooms" synapse_federation;
"~^/_matrix/federation/v1/query/" synapse_federation;
"~^/_matrix/federation/v1/make_join/" synapse_federation;
"~^/_matrix/federation/v1/make_leave/" synapse_federation;
"~^/_matrix/federation/v1/send_join/" synapse_federation;
"~^/_matrix/federation/v2/send_join/" synapse_federation;
"~^/_matrix/federation/v1/send_leave/" synapse_federation;
"~^/_matrix/federation/v2/send_leave/" synapse_federation;
"~^/_matrix/federation/v1/invite/" synapse_federation;
"~^/_matrix/federation/v2/invite/" synapse_federation;
"~^/_matrix/federation/v1/query_auth/" synapse_federation;
"~^/_matrix/federation/v1/event_auth/" synapse_federation;
"~^/_matrix/federation/v1/exchange_third_party_invite/" synapse_federation;
"~^/_matrix/federation/v1/user/devices/" synapse_federation;
"~^/_matrix/federation/v1/get_groups_publicised$" synapse_federation;
"~^/_matrix/key/v2/query" synapse_federation;
# Inbound federation transaction request
"~^/_matrix/federation/v1/send/" synapse_federation;
# Client API requests
"~^/_matrix/client/(api/v1|r0|unstable)/publicRooms$" synapse_client;
"~^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/joined_members$" synapse_client;
"~^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/context/.*$" synapse_client;
"~^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/members$" synapse_client;
"~^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state$" synapse_client;
"~^/_matrix/client/(api/v1|r0|unstable)/account/3pid$" synapse_client;
"~^/_matrix/client/(api/v1|r0|unstable)/devices$" synapse_client;
"~^/_matrix/client/(api/v1|r0|unstable)/keys/query$" synapse_client;
"~^/_matrix/client/(api/v1|r0|unstable)/keys/changes$" synapse_client;
"~^/_matrix/client/versions$" synapse_client;
"~^/_matrix/client/(api/v1|r0|unstable)/voip/turnServer$" synapse_client;
"~^/_matrix/client/(api/v1|r0|unstable)/joined_groups$" synapse_client;
"~^/_matrix/client/(api/v1|r0|unstable)/publicised_groups$" synapse_client;
"~^/_matrix/client/(api/v1|r0|unstable)/publicised_groups/" synapse_client;
"~^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/event/" synapse_client;
"~^/_matrix/client/(api/v1|r0|unstable)/joined_rooms$" synapse_client;
"~^/_matrix/client/(api/v1|r0|unstable)/search$" synapse_client;
# Registration/login requests
"~^/_matrix/client/(api/v1|r0|unstable)/login$" synapse_client;
"~^/_matrix/client/(r0|unstable)/register$" synapse_client;
# Event sending requests
"~^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/redact" synapse_client;
"~^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/send" synapse_client;
"~^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state/" synapse_client;
"~^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$" synapse_client;
"~^/_matrix/client/(api/v1|r0|unstable)/join/" synapse_client;
"~^/_matrix/client/(api/v1|r0|unstable)/profile/" synapse_client;
}
server {
listen 8008;
listen [::]:8008;
listen 8484;
listen [::]:8484
server_name localhost;
client_max_body_size 100M;
location ~ ^(/_matrix|/_synapse|client) {
proxy_pass http://$synapse_backend;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $remote_addr;
}
}
}
Worker 1
synchrotron-1.yaml
worker_app: synapse.app.generic_worker
worker_name: worker1
worker_replication_host: matrix-synapse # Again: Also here it's important to use the container_name
worker_replication_http_port: 9093
worker_listeners:
- type: http
port: 8084
resources:
- names:
- client
worker_log_config: /data/workers/logs/Worker1_log_config.yaml # if you like to have a seperate log for each worker you will need this config. But I understand that you can also use the already existing config of your main synapse process.
Worker 2
federation-1.yaml
worker_app: synapse.app.generic_worker
worker_name: worker2
worker_replication_host: matrix-synapse
worker_replication_http_port: 9093
worker_listeners:
- type: http
port: 8083
resources:
- names:
- federation
worker_log_config: /data/workers/logs/Worker2_log_config.yaml
Log config
Here you have an example how a WorkerX_log_config.yaml could look like. If something doesn't work maybe check the indention first after copypasting ...
version: 1
formatters:
precise:
format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s- $
filters:
context:
(): synapse.util.logcontext.LoggingContextFilter
request: ""
handlers:
file:
class: logging.handlers.RotatingFileHandler
formatter: precise
filename: /data/workers/logs/Worker1.log
maxBytes: 104857600
backupCount: 10
filters: [context]
encoding: utf8
level: DEBUG
console:
class: logging.StreamHandler
formatter: precise
level: INFO
loggers:
synapse:
level: INFO
synapse.storage.SQL:
level: INFO
synapse.app.generic_worker:
level: DEBUG
root:
level: INFO
handlers: [file, console]
homeserver.yaml
Last but not least the excerpts from the homeserver.yaml:
## Ports ##
listeners:
-
port: 8448
bind_addresses: ['::']
type: http
tls: false
x_forwarded: true
resources:
- names: [client]
compress: true
- names: [federation] # Federation APIs
compress: false
- port: 8888
tls: false
bind_addresses: ['::']
type: http
x_forwarded: true
resources:
- names: [client]
compress: true
- names: [federation]
compress: false
# The HTTP replication port
- port: 9093
bind_address: '::'
type: http
resources:
- names: [replication]
##################################################
snipsnap
##################################################
# Worker
worker_replication_secret: "[some long and super secret password you don't need to memorize or copy because main and workers share it already]"
redis:
enabled: true
host: matrix-redis
port: 6379
At least for me this Setup does work. I hope it is of help to you as I spent more than three whole days to get it too work. I searched a lot in the web about setting up workers in a docker-compose context, I knew only few about the single components and tried a lot pieces and hints that didn't work in the end.
Most of the time it was either
- something with the regex
- or with howto correctly address the different containers between each other. As mentioned before the only stable way to solve this turned out to be the container_name tags that resolved to the current ip-adresses even after down and up again the docker-compose environment.
And now good luck with your own efforts :)
Status: Not working yet (25th of February 2022)
while trying to give my docker-compose setup two workers (multicontainer) - 1 for client, 1 for federation. While I made some progress today I am struggling with a strange problem:
- I use nginx reverse proxy to filter $request_uri and send it to one of the 3 upstreams.
- I have my nginx running and filtering, my workers and my master are up and running.
- But the instance isn't reachable (If I give my master the 8008-port back and don't send it through nginx any longer it's working again - but monolithic of course)
The Network Problem
- Checking my nginx log shows: A lot of
connect() failed (111: Connection refused) while connecting to upstreamwith every request. But ther workers themselves are reachable.- What is NOT reachable is
https://127.0.0.1:9093, i.e. the replication port. -->curl: (7) Failed to connect to 127.0.0.1 port 9093: Connection refused - But HTTP replication port is set in homeserver.yaml
Hint from Chat regarding external networks in docker: https://matrix.to/#/!ehXvUhWNASUkSLvAGP:matrix.org/$KUn6O3dNbSOrRMo0VNu_aBtzTRIYnj1Gr0AChGfux_0?via=matrix.org&via=matrix.breakpointingbad.com&via=libera.chat
- I don't see how an external network should help me. I of course don't use two or more docker-compose files. Ich want to enclose the whole setup in one seperate network and because in my setup all containers use the same network there shouldn't be a problem.
But the hint let me
docker-compose exec synapse bash.- There I
curl localhost:9093... and receive an 404 error page! 8-o curl localhost:9092or any other arbitrary port returns the expectedFailed to connect to localhost port 9092: Connection refused- I also can't reach the port 6379 of the redis container :-/ ... Maybe it's no surprise that this setup doesn't work ...
Leaving the container and opening a bash in one of the two workers.
- Here the same curl to port 9093 leads to ... Connection refused. As experienced. --> So the opened replication port isn't reached to the network?!
- As you will see down in my docker-compose.yml all containers share the default network.
- Changing into a bash inside one of the workers: I can't curl from the master`s bash to the ports of the workers.
- Having a look in docker-compose docs I reminded myself that you can call the other containers with e.g.
curl client_worker:8084- ... and in deed: I can reach it! So I can't use localhost or 127.0.0.1 or 0.0.0.0 for inter-container-communication but need to use the service name!
- Unfortunately that doesn't work in the nginx.conf :/
- Maybe I can solve this later by defining links (https://docs.docker.com/compose/networking/#links)?
Progress!
- Solved it: With the self defined
container_nameitem in the docker-compose.yml it is possible to access the containers in nginx.conf!- It's important to define the dependencies of the containers because else the setup won't work after restarting (network wouldn't resolve properly and break at least nginx).
- I updated the files down on this page.
- So far the setup produces a working matrix server, BUT:
- It seems that still the entire traffic is routed to the master process.
- All I get in the logs of the workers is
*** STARTING SERVER ***** matrix-federation | 2022-02-26 10:05:49,433 - root - 349 - WARNING - main- Server /usr/local/lib/python3.8/site-packages/synapse/app/generic_worker.py version 1.53.0 matrix-federation | 2022-02-26 10:05:49,650 - synapse.app - 49 - WARNING - sentinel- Failed to listen on 0.0.0.0, continuing because listening on [::]
- Redis outputs now
redis_1 | 1:M 26 Feb 2022 10:55:13.101 * 1 changes in 3600 seconds. Saving... redis_1 | 1:M 26 Feb 2022 10:55:13.594 * Background saving started by pid 19 redis_1 | 19:C 26 Feb 2022 10:55:14.103 * DB saved on disk redis_1 | 19:C 26 Feb 2022 10:55:14.104 * RDB: 0 MB of memory used by copy-on-write redis_1 | 1:M 26 Feb 2022 10:55:14.197 * Background saving terminated with success
- So it seems that Redis is now working but the workers seem to listen to the wrong bindings? Or are the mappings in nginx.conf not properly working?
- I have a lot of those errors in the nginx log. Don`t know if this has to do with it.
2022/02/26 11:20:48 [error] 32#32: *11814 open() "/etc/nginx/html/.well-known/matrix/server" failed (2: No such file or directory), client: 192.168.160.1, server: localhost, request: "GET /.well-known/matrix/server HTTP/1.1", host: "ismus.net" 192.168.160.1 - - [26/Feb/2022:11:20:48 +0000] "GET /.well-known/matrix/server HTTP/1.1" 404 153 "-" "Synapse/1.52.0" 192.168.160.1 - - [26/Feb/2022:11:20:48 +0000] "PUT /_matrix/federation/v1/send/1645518187107 HTTP/1.1" 200 21 "-" "Synapse/1.52.0"
- The current version of the nginx.conf and docker-compose.yml
- made it possible to start setup
- nginx finally sent the requests corresponding to the regex to the upstream addresses
- but while I could receive a message from a account of the matrix.org network I couldn't send an answer back.
- This issue might help describes sth similar and to check some config files.
- the logs of the workers stayed unchanged, while the master log showed that it communicated with the replication port ...
- so I'm still unsure if the workers actually do something or not :/
- Summary: So there's some progress but there's still something wrong with the workers or with the outbound communication.
Old Outdated Information
As I experienced significant delay with my synapse instance when joining rooms on other instances like matrix.org I was told that I urgently need to split the cpu load of it into multiple processes. By this the load could be distributed to multiple cpu cores instead of only one that would limit the speed of processing of large room content. Well, until now I know next to nothing about so called workers and how to set those up in a dockerized environment. On its top: There's yet a poor amount of documentation or tutorials regarding this. Or at least I couldn't find it.
My efforts for now lead to some [general information about in synapse]. A few hints on how to configure separate workers in docker/docker-compose can be found in the official [README].
Meanwhile developers are obviously [working on a "batteries included" docker-image for multiprocess synapse] which already makes use of workers. But that's not for production at the moment unfortunately.
What I understand so far
- I have to add a separate synapse-service for every worker in the docker-compose.yml.
- The best recommendation I yet received about what workers I should setup for a start is "synchrotron and two federation readers".
- I guess I will start with this recommendation and when this works I may think about the other options.
- I should definitely switch from my minimal homeserver.yaml to a full commented version since it will lack the placeholder variables for docker
- every worker uses both the homeserver.yaml and a worker-config in e.g. the workers directory
- I have to work out how the reverse proxy of my DSM will distribute http requests between the workers as it is not so easy to configure as a full nginx.
- I guess everything gets in over 8448 and has to be distributed to the workers over the local ports? I.e. no further ports have to be exposed publicy? I'm not sure how to manage this except by adding another nginx to the docker-compose.yml. At least there is a [sample file for the simple nginx setup] ...
- redis is for shared database communication between the workers. Not sure yet how that works but maybe that will work automatically after activating redis in the homeserver.yaml.
n>
homer