HowTo Deploy with Multiple Workers using

Dies ist eine alte Version des Dokuments!

I can finally announce a working docker-compose setup with 2 workers! That is up to now one worker for client related requests and one for the federation requests. Of course the amount of workers is expandable and only depends on how you route the requests to the workers and the main process using your reverse proxy (here: nginx).

I document my relevant config files except my homeserver.yaml. For briefness I will only show the small parts of it that are relevant for having the workers to communicate successfully with the main process. It's assumed that you have an already working homeserver.yaml based on a monolithic synapse setup. I will try to comment some critical details so that you will know what to adapt and what to adopt.

The base of all

Please not that the container_name fields are crucial for the setup to work. You need them to address the containers between each other.

version: '3'

services:
  postgres:
    restart: unless-stopped
    networks:
      - default
    environment:
      POSTGRES_PASSWORD: "[some-secret-password]"
      POSTGRES_USER: synapse
      POSTGRES_DB: synapse
      POSTGRES_INITDB_ARGS: "--encoding='UTF8' --lc-collate='C' --lc-ctype='C'"
      POSTGRES_HOST_AUTH_METHOD: trust
    image: postgres:13-alpine
    volumes:
      - /volume1/docker/volumes/matrix-synapse/postgres:/var/lib/postgresql/data

  nginx:
    image: nginx:stable-alpine
    container_name: matrix-nginx
    volumes:
      - /volume1/docker/volumes/matrix-synapse/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - 8008:8008   ## Those outbound ports fit the needs of my environment where 8008 leads to another proxy server that serves to port 80, 443 and 8448
      - 84:8484
    depends_on:     ## That's actually very important! The startup order __must__ be: redis, postgres, synapse, worker[1:n], nginx
      - synapse
      - federation_worker
      - client_worker
    networks:
      - default


  synapse:
    image: matrixdotorg/synapse:latest
    container_name: matrix-synapse
    restart: "unless-stopped"
    environment:
      - SYNAPSE_REPORT_STATS=no
      - SYNAPSE_SERVER_NAME=ismus.net     # of course you must adjust this to your domain
      - SYNAPSE_CONFIG_PATH=/data/homeserver.yaml
      - TZ=Berlin/Europe
    depends_on: ["postgres"]
    volumes:
      - /volume1/docker/volumes/matrix-synapse:/data
    networks:
      - default
      
  client_worker:
    image: matrixdotorg/synapse:latest
    container_name: matrix-client
    restart: "unless-stopped"
    command:
      - 'run'
      - '--config-path=/data/homeserver-postgres.yaml'
      - '--config-path=/data/workers/synchrotron-1.yaml' # Important! You can _NOT_ do this by `- SYNAPSE_CONFIG_PATH=/data/workers/synchrotron-1.yaml` in the environment section!
    environment:
      - SYNAPSE_REPORT_STATS=no
      - SYNAPSE_SERVER_NAME=ismus.net
      - SYNAPSE_WORKER=synapse.app.generic_worker
      - TZ=Berlin/Europe
    depends_on:
      - synapse
    volumes:   
      - /volume1/docker/volumes/matrix-synapse:/data
    networks:
      - default

  federation_worker:
    image: matrixdotorg/synapse:latest
    container_name: matrix-federation
    restart: "unless-stopped"
    command:
      - 'run'  
      - '--config-path=/data/homeserver-postgres.yaml'
      - '--config-path=/data/workers/federation-1.yaml'
    environment:
      - SYNAPSE_REPORT_STATS=no
      - SYNAPSE_SERVER_NAME=ismus.net
      - SYNAPSE_WORKER=synapse.app.generic_worker
      - TZ=Berlin/Europe
    depends_on: 
      - client-worker
    volumes:
      - /volume1/docker/volumes/matrix-synapse:/data
    networks:
      - default
    
  redis:
    image: "redis:latest"
    restart: "unless-stopped"
    networks:
      - default
    
    
networks:
  default:

The nginx has the task to distribute the different kinds of request between the workers. I had not much idea how to config nginx before I found a template which looked much alike the following. But I still had a lot to correct and find out to make it work properly. If you want to re-balance the work between the two or even more workers you will have to alter the map block and if needed add further upstreams per worker.

As described above I had the special case that in my environment there's another proxy server on top that is responsible for domain and ssl so that I only needed the nginx container to distribute requests and reach traffic through between this on top reverse proxy and the synapse processes. If you want to use the nginx container as fully responsible server have a look into the official synapse documentation and add the few missing parts.

events {
        worker_connections 1024;
}

http {

upstream synapse_master {
        server matrix-synapse:8888;            # Here neither localhost nor 127.0.0.1 or 0.0.0.0 worked for me.
}

upstream synapse_client {
        server matrix-client:8084;             # But the container_name labels are resolved to the local ips of the containers.
}

upstream synapse_federation {                  
        server matrix-federation:8083;         # That makes the docker-setup quite fail-safe in case of eventually changing network conditions, I guess.
}

map_hash_bucket_size 128;

map $request_uri $synapse_backend {
        default synapse_master;                # Important: Makes safe that everything _not_ covered by the regex will go to the master process by default and by this: won't be lost!
                                               # The requests are basically copy paste from the [official docs](https://matrix-org.github.io/synapse/latest/workers.html).

        # Sync requests
        "~^/_matrix/client/(v2_alpha|r0)/sync$" synapse_client;
        "~^/_matrix/client/(api/v1|v2_alpha|r0)/events$" synapse_client;
        "~^/_matrix/client/(api/v1|r0)/initialSync$" synapse_client;
        "~^/_matrix/client/(api/v1|r0)/rooms/[^/]+/initialSync$" synapse_client;

        # Federation requests
        "~^/_matrix/federation/v1/event/" synapse_federation;
        "~^/_matrix/federation/v1/state/" synapse_federation;
        "~^/_matrix/federation/v1/state_ids/" synapse_federation;
        "~^/_matrix/federation/v1/backfill/" synapse_federation;
        "~^/_matrix/federation/v1/get_missing_events/" synapse_federation;
        "~^/_matrix/federation/v1/publicRooms" synapse_federation;
        "~^/_matrix/federation/v1/query/" synapse_federation;
        "~^/_matrix/federation/v1/make_join/" synapse_federation;
        "~^/_matrix/federation/v1/make_leave/" synapse_federation;
        "~^/_matrix/federation/v1/send_join/" synapse_federation;
        "~^/_matrix/federation/v2/send_join/" synapse_federation;
        "~^/_matrix/federation/v1/send_leave/" synapse_federation;
        "~^/_matrix/federation/v2/send_leave/" synapse_federation;                                           
        "~^/_matrix/federation/v1/invite/" synapse_federation;
        "~^/_matrix/federation/v2/invite/" synapse_federation;
        "~^/_matrix/federation/v1/query_auth/" synapse_federation;
        "~^/_matrix/federation/v1/event_auth/" synapse_federation;
        "~^/_matrix/federation/v1/exchange_third_party_invite/" synapse_federation;
        "~^/_matrix/federation/v1/user/devices/" synapse_federation;
        "~^/_matrix/federation/v1/get_groups_publicised$" synapse_federation;
        "~^/_matrix/key/v2/query" synapse_federation;
        
        # Inbound federation transaction request
        "~^/_matrix/federation/v1/send/" synapse_federation;

        # Client API requests
        "~^/_matrix/client/(api/v1|r0|unstable)/publicRooms$" synapse_client;
        "~^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/joined_members$" synapse_client;
        "~^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/context/.*$" synapse_client;
        "~^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/members$" synapse_client;
        "~^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state$" synapse_client;
        "~^/_matrix/client/(api/v1|r0|unstable)/account/3pid$" synapse_client;
        "~^/_matrix/client/(api/v1|r0|unstable)/devices$" synapse_client;
        "~^/_matrix/client/(api/v1|r0|unstable)/keys/query$" synapse_client;                                 
        "~^/_matrix/client/(api/v1|r0|unstable)/keys/changes$" synapse_client;
        "~^/_matrix/client/versions$" synapse_client;
        "~^/_matrix/client/(api/v1|r0|unstable)/voip/turnServer$" synapse_client;
        "~^/_matrix/client/(api/v1|r0|unstable)/joined_groups$" synapse_client;
        "~^/_matrix/client/(api/v1|r0|unstable)/publicised_groups$" synapse_client;
        "~^/_matrix/client/(api/v1|r0|unstable)/publicised_groups/" synapse_client;
        "~^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/event/" synapse_client;
        "~^/_matrix/client/(api/v1|r0|unstable)/joined_rooms$" synapse_client;
        "~^/_matrix/client/(api/v1|r0|unstable)/search$" synapse_client;

        # Registration/login requests
        "~^/_matrix/client/(api/v1|r0|unstable)/login$" synapse_client;
        "~^/_matrix/client/(r0|unstable)/register$" synapse_client;
        # Event sending requests
        "~^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/redact" synapse_client;
        "~^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/send" synapse_client;
        "~^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state/" synapse_client;
        "~^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$" synapse_client;
        "~^/_matrix/client/(api/v1|r0|unstable)/join/" synapse_client;
        "~^/_matrix/client/(api/v1|r0|unstable)/profile/" synapse_client;                     
}

        server {
                listen 8008;
                listen [::]:8008;
                listen 8484;
                listen [::]:8484
                server_name localhost;
                client_max_body_size 100M;

                location ~ ^(/_matrix|/_synapse|client) {
                        proxy_pass http://$synapse_backend;
                        proxy_set_header Host $host;
                        proxy_set_header X-Forwarded-Proto $scheme;
                        proxy_set_header X-Forwarded-For $remote_addr;
                }

        }


}

while trying to give my docker-compose setup two workers (multicontainer) - 1 for client, 1 for federation. While I made some progress today I am struggling with a strange problem:

I use nginx reverse proxy to filter $request_uri and send it to one of the 3 upstreams.
I have my nginx running and filtering, my workers and my master are up and running.
But the instance isn't reachable (If I give my master the 8008-port back and don't send it through nginx any longer it's working again - but monolithic of course)
The Network Problem
Checking my nginx log shows: A lot ofconnect() failed (111: Connection refused) while connecting to upstream with every request. But ther workers themselves are reachable.
- What is NOT reachable ishttps://127.0.0.1:9093, i.e. the replication port. -->curl: (7) Failed to connect to 127.0.0.1 port 9093: Connection refused
- But HTTP replication port is set in homeserver.yaml
Hint from Chat regarding external networks in docker: https://matrix.to/#/!ehXvUhWNASUkSLvAGP:matrix.org/$KUn6O3dNbSOrRMo0VNu_aBtzTRIYnj1Gr0AChGfux_0?via=matrix.org&via=matrix.breakpointingbad.com&via=libera.chat
- I don't see how an external network should help me. I of course don't use two or more docker-compose files. Ich want to enclose the whole setup in one seperate network and because in my setup all containers use the same network there shouldn't be a problem.
- But the hint let me docker-compose exec synapse bash.
  - There I curl localhost:9093 ... and receive an 404 error page! 8-o
  - curl localhost:9092 or any other arbitrary port returns the expected Failed to connect to localhost port 9092: Connection refused
  - I also can't reach the port 6379 of the redis container :-/ ... Maybe it's no surprise that this setup doesn't work ...
- Leaving the container and opening a bash in one of the two workers.
  - Here the same curl to port 9093 leads to ... Connection refused. As experienced. --> So the opened replication port isn't reached to the network?!
  - As you will see down in my docker-compose.yml all containers share the default network.
  - Changing into a bash inside one of the workers: I can't curl from the master`s bash to the ports of the workers.
- Having a look in docker-compose docs I reminded myself that you can call the other containers with e.g. curl client_worker:8084
  - ... and in deed: I can reach it! So I can't use localhost or 127.0.0.1 or 0.0.0.0 for inter-container-communication but need to use the service name!
  - Unfortunately that doesn't work in the nginx.conf :/
  - Maybe I can solve this later by defining links (https://docs.docker.com/compose/networking/#links)?
    Progress!
- Solved it: With the self defined container_nameitem in the docker-compose.yml it is possible to access the containers in nginx.conf!
  - It's important to define the dependencies of the containers because else the setup won't work after restarting (network wouldn't resolve properly and break at least nginx).
  - I updated the files down on this page.
  - So far the setup produces a working matrix server, BUT:
    
    It seems that still the entire traffic is routed to the master process.
    
    All I get in the logs of the workers is
    *** STARTING SERVER ***** matrix-federation | 2022-02-26 10:05:49,433 - root - 349 - WARNING - main- Server /usr/local/lib/python3.8/site-packages/synapse/app/generic_worker.py version 1.53.0 matrix-federation | 2022-02-26 10:05:49,650 - synapse.app - 49 - WARNING - sentinel- Failed to listen on 0.0.0.0, continuing because listening on [::]
    
    Redis outputs now
    redis_1 | 1:M 26 Feb 2022 10:55:13.101 * 1 changes in 3600 seconds. Saving... redis_1 | 1:M 26 Feb 2022 10:55:13.594 * Background saving started by pid 19 redis_1 | 19:C 26 Feb 2022 10:55:14.103 * DB saved on disk redis_1 | 19:C 26 Feb 2022 10:55:14.104 * RDB: 0 MB of memory used by copy-on-write redis_1 | 1:M 26 Feb 2022 10:55:14.197 * Background saving terminated with success
    
    So it seems that Redis is now working but the workers seem to listen to the wrong bindings? Or are the mappings in nginx.conf not properly working?
    
    I have a lot of those errors in the nginx log. Don`t know if this has to do with it.
    2022/02/26 11:20:48 [error] 32#32: *11814 open() "/etc/nginx/html/.well-known/matrix/server" failed (2: No such file or directory), client: 192.168.160.1, server: localhost, request: "GET /.well-known/matrix/server HTTP/1.1", host: "ismus.net" 192.168.160.1 - - [26/Feb/2022:11:20:48 +0000] "GET /.well-known/matrix/server HTTP/1.1" 404 153 "-" "Synapse/1.52.0" 192.168.160.1 - - [26/Feb/2022:11:20:48 +0000] "PUT /_matrix/federation/v1/send/1645518187107 HTTP/1.1" 200 21 "-" "Synapse/1.52.0"
  - The current version of the nginx.conf and docker-compose.yml
    
    made it possible to start setup
    
    nginx finally sent the requests corresponding to the regex to the upstream addresses
    
    but while I could receive a message from a account of the matrix.org network I couldn't send an answer back.
    
    This issue might help describes sth similar and to check some config files.
    
    the logs of the workers stayed unchanged, while the master log showed that it communicated with the replication port ...
    
    so I'm still unsure if the workers actually do something or not :/
  - Summary: So there's some progress but there's still something wrong with the workers or with the outbound communication.

As I experienced significant delay with my synapse instance when joining rooms on other instances like matrix.org I was told that I urgently need to split the cpu load of it into multiple processes. By this the load could be distributed to multiple cpu cores instead of only one that would limit the speed of processing of large room content. Well, until now I know next to nothing about so called workers and how to set those up in a dockerized environment. On its top: There's yet a poor amount of documentation or tutorials regarding this. Or at least I couldn't find it.

My efforts for now lead to some [general information about in synapse]. A few hints on how to configure separate workers in docker/docker-compose can be found in the official [README].

Meanwhile developers are obviously [working on a "batteries included" docker-image for multiprocess synapse] which already makes use of workers. But that's not for production at the moment unfortunately.

I have to add a separate synapse-service for every worker in the docker-compose.yml.
The best recommendation I yet received about what workers I should setup for a start is "synchrotron and two federation readers".
- I guess I will start with this recommendation and when this works I may think about the other options.
I should definitely switch from my minimal homeserver.yaml to a full commented version since it will lack the placeholder variables for docker
- every worker uses both the homeserver.yaml and a worker-config in e.g. the workers directory
I have to work out how the reverse proxy of my DSM will distribute http requests between the workers as it is not so easy to configure as a full nginx.
- I guess everything gets in over 8448 and has to be distributed to the workers over the local ports? I.e. no further ports have to be exposed publicy? I'm not sure how to manage this except by adding another nginx to the docker-compose.yml. At least there is a [sample file for the simple nginx setup] ...
redis is for shared database communication between the workers. Not sure yet how that works but maybe that will work automatically after activating redis in the homeserver.yaml.