Unterschiede
Hier werden die Unterschiede zwischen zwei Versionen angezeigt.
| Beide Seiten der vorigen Revision Vorhergehende Überarbeitung Nächste Überarbeitung | Vorhergehende Überarbeitung | ||
| howto_deploy_synapse_with_multiple_workers [2022/02/26 12:37] – homer | howto_deploy_synapse_with_multiple_workers [2023/05/24 22:16] (aktuell) – [Log config] homer | ||
|---|---|---|---|
| Zeile 1: | Zeile 1: | ||
| < | < | ||
| # HowTo Deploy Synapse with Multiple Workers using docker-compose | # HowTo Deploy Synapse with Multiple Workers using docker-compose | ||
| - | ## Status: | + | ## Status: |
| - | while trying | + | The update |
| - | - I use nginx reverse proxy to filter $request_uri and send it to one of the 3 upstreams. | + | |
| - | - I have my nginx running and filtering, my workers and my master are up and running. | + | |
| - | - But the instance isn't reachable (If I give my master the 8008-port back and don't send it through nginx any longer it's working again - but monolithic of course) | + | |
| - | ### The Network Problem | + | |
| - | - Checking my nginx log shows: A lot of`connect() failed (111: Connection refused) while connecting to upstream` with every request. But ther workers themselves are reachable. | + | |
| - | - What is NOT reachable is`https://127.0.0.1: | + | |
| - | - But HTTP replication port is set in homeserver.yaml | + | |
| - | - Hint from Chat regarding external networks in docker: https://matrix.to/#/ | + | |
| - | | + | |
| - | - But the hint let me `docker-compose exec synapse bash`. | + | |
| - | - There I `curl localhost: | + | |
| - | - `curl localhost: | + | |
| - | - I also can't reach the port 6379 of the redis container :-/ ... Maybe it's no surprise that this setup doesn' | + | |
| - | + | ||
| - | - Leaving the container and opening a bash in one of the two workers. | + | |
| - | - Here the same curl to port 9093 leads to ... Connection refused. As experienced. --> So the opened replication port isn't reached to the network?! | + | |
| - | - As you will see down in my docker-compose.yml all containers share the default network. | + | |
| - | - Changing into a bash inside one of the workers: I can't curl from the master`s bash to the ports of the workers. | + | |
| - | - Having a look in docker-compose docs I reminded myself that you can call the other containers with e.g. `curl client_worker: | + | |
| - | - ... and in deed: I can reach it! So I can't use localhost or 127.0.0.1 or 0.0.0.0 for inter-container-communication but need to use the service name! | + | |
| - | - Unfortunately that doesn't work in the nginx.conf :/ | + | |
| - | - Maybe I can solve this later by defining links (https:// | + | |
| - | #### Progress! | + | |
| - | - Solved it: With the self defined `container_name`item in the docker-compose.yml it is possible to access the containers in nginx.conf! | + | |
| - | - I updated the files down on this page. | + | |
| - | - So far the setup produces a working matrix server, BUT: | + | |
| - | - It seems that still the entire traffic is routed to the master process. | + | |
| - | - All I get in the logs of the workers is | + | |
| - | ``` | + | |
| - | *** STARTING SERVER ***** | + | |
| - | matrix-federation | + | |
| - | matrix-federation | + | |
| - | ``` | + | |
| - | - Redis outputs now | + | |
| - | ``` | + | |
| - | redis_1 | + | |
| - | redis_1 | + | |
| - | redis_1 | + | |
| - | redis_1 | + | |
| - | redis_1 | + | |
| - | ``` | + | |
| - | - So it seems that Redis is now working but the workers seem to listen to the wrong bindings? Or are the mappings in nginx.conf not properly working? | + | |
| - | - I have a lot of those errors in the nginx log. Don`t know if this has to do with it. | + | |
| - | ``` | + | |
| - | 2022/02/26 11:20:48 [error] 32#32: *11814 open() "/ | + | |
| - | 192.168.160.1 - - [26/ | + | |
| - | 192.168.160.1 - - [26/ | + | |
| - | ``` | + | |
| + | ## Status: Finally working!!! (1st of March 2022) | ||
| + | I can finally announce a working docker-compose setup with 2 workers for [Matrix-Synapse](https:// | ||
| + | That is up to now one worker for client related requests and one for the federation requests. Of course the amount of workers is expandable and only depends on how you route the requests to the workers and the main process using your reverse proxy (here: nginx). | ||
| + | ## Scope | ||
| + | I document my relevant config files except my homeserver.yaml. For briefness I will only show the small parts of it that are relevant for having the workers to communicate successfully with the main process. It's assumed that you have an already working homeserver.yaml based on a monolithic synapse setup. I will try to comment some critical details so that you will know what to adapt and what to adopt. | ||
| - | ### My docker-compose.yml | + | If you're looking for an example of synapse with workers _without_ docker you might want to visit https:// |
| + | ### Thnx | ||
| + | ... go out to Sloth (@sleuth: | ||
| + | ### My docker-compose.yml | ||
| + | #### The base of all | ||
| + | Please note that the container_name fields are crucial for the setup to work. You need them to address the containers between each other. | ||
| ``` | ``` | ||
| version: ' | version: ' | ||
| Zeile 74: | Zeile 36: | ||
| nginx: | nginx: | ||
| image: nginx: | image: nginx: | ||
| + | container_name: | ||
| volumes: | volumes: | ||
| - / | - / | ||
| ports: | ports: | ||
| - | - 8008:8008 | + | - 8008: |
| + | - 84:8484 | ||
| + | depends_on: | ||
| + | - synapse | ||
| + | - federation_worker | ||
| + | - client_worker | ||
| networks: | networks: | ||
| - default | - default | ||
| Zeile 86: | Zeile 54: | ||
| container_name: | container_name: | ||
| restart: " | restart: " | ||
| - | ports: | ||
| - | - 84:8448 | ||
| - | - 8888:8888 | ||
| environment: | environment: | ||
| - SYNAPSE_REPORT_STATS=no | - SYNAPSE_REPORT_STATS=no | ||
| - | - SYNAPSE_SERVER_NAME=ismus.net | + | - SYNAPSE_SERVER_NAME=ismus.net |
| - SYNAPSE_CONFIG_PATH=/ | - SYNAPSE_CONFIG_PATH=/ | ||
| - TZ=Berlin/ | - TZ=Berlin/ | ||
| depends_on: [" | depends_on: [" | ||
| - | volumes: | + | |
| - / | - / | ||
| networks: | networks: | ||
| Zeile 114: | Zeile 79: | ||
| - TZ=Berlin/ | - TZ=Berlin/ | ||
| depends_on: | depends_on: | ||
| - | - redis | ||
| - synapse | - synapse | ||
| volumes: | volumes: | ||
| Zeile 135: | Zeile 99: | ||
| - TZ=Berlin/ | - TZ=Berlin/ | ||
| depends_on: | depends_on: | ||
| - | - redis | + | - client-worker |
| - | | + | |
| volumes: | volumes: | ||
| - / | - / | ||
| Zeile 155: | Zeile 118: | ||
| ``` | ``` | ||
| ### The nginx.conf so far: | ### The nginx.conf so far: | ||
| + | The nginx has the task to distribute the different kinds of request between the workers. I had not much idea how to config nginx before I found a template which looked much alike the following. But I still had a lot to correct and find out to make it work properly. If you want to re-balance the work between the two or even more workers you will have to alter the map block and if needed add further upstreams per worker. | ||
| + | |||
| + | As described above I had the special case that in my environment there' | ||
| ``` | ``` | ||
| Zeile 164: | Zeile 130: | ||
| upstream synapse_master { | upstream synapse_master { | ||
| - | server matrix-synapse: | + | server matrix-synapse: |
| } | } | ||
| upstream synapse_client { | upstream synapse_client { | ||
| - | server matrix-client: | + | server matrix-client: |
| } | } | ||
| - | upstream synapse_federation { | + | upstream synapse_federation { |
| - | server matrix-federation: | + | server matrix-federation: |
| } | } | ||
| + | map_hash_bucket_size 128; | ||
| map $request_uri $synapse_backend { | map $request_uri $synapse_backend { | ||
| - | default synapse_master; | + | default synapse_master; |
| + | # The requests are basically copy paste from the [official docs](https:// | ||
| # Sync requests | # Sync requests | ||
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| # Federation requests | # Federation requests | ||
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| + | |||
| # Inbound federation transaction request | # Inbound federation transaction request | ||
| - | ~*^/ | + | |
| # Client API requests | # Client API requests | ||
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | # Registration/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| + | # Registration/ | ||
| + | " | ||
| + | " | ||
| # Event sending requests | # Event sending requests | ||
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| - | ~*^/ | + | |
| } | } | ||
| Zeile 245: | Zeile 213: | ||
| listen 8008; | listen 8008; | ||
| listen [::]:8008; | listen [::]:8008; | ||
| + | listen 8484; | ||
| + | listen [::]:8484 | ||
| server_name localhost; | server_name localhost; | ||
| - | | + | |
| - | location ~* ^(\\/_matrix|\\/_synapse) { | + | location ~ ^(/ |
| proxy_pass http:// | proxy_pass http:// | ||
| proxy_set_header Host $host; | proxy_set_header Host $host; | ||
| proxy_set_header X-Forwarded-Proto $scheme; | proxy_set_header X-Forwarded-Proto $scheme; | ||
| proxy_set_header X-Forwarded-For $remote_addr; | proxy_set_header X-Forwarded-For $remote_addr; | ||
| - | add_header Backend-Server $synapse_backend; | ||
| } | } | ||
| + | | ||
| + | |||
| + | | ||
| + | location / { | ||
| + | proxy_pass http:// | ||
| + | proxy_set_header Host $host; | ||
| + | proxy_set_header X-Forwarded-Proto $scheme; | ||
| + | proxy_set_header X-Forwarded-For $remote_addr; | ||
| + | } | ||
| + | | ||
| + | # ALTERNATIVELY!! you can redirect from your domain to any url you want as e.g. a matrix webclient | ||
| + | location / { | ||
| + | return 301 https:// | ||
| + | } | ||
| + | |||
| + | |||
| + | |||
| } | } | ||
| Zeile 261: | Zeile 247: | ||
| } | } | ||
| ``` | ``` | ||
| + | ### Worker 1 | ||
| + | #### synchrotron-1.yaml | ||
| + | ``` | ||
| + | worker_app: synapse.app.generic_worker | ||
| + | worker_name: | ||
| + | |||
| + | worker_listeners: | ||
| + | - type: http | ||
| + | port: 8084 | ||
| + | resources: | ||
| + | - names: [client] | ||
| + | - type: http | ||
| + | port: 9094 | ||
| + | resources: | ||
| + | - names: [replication] | ||
| + | |||
| + | send_federation: | ||
| + | |||
| + | worker_log_config: | ||
| + | ``` | ||
| + | |||
| + | ### Worker 2 | ||
| + | #### federation-1.yaml | ||
| + | ``` | ||
| + | worker_app: synapse.app.generic_worker | ||
| + | worker_name: | ||
| + | |||
| + | worker_listeners: | ||
| + | - type: http | ||
| + | port: 8083 | ||
| + | resources: | ||
| + | - names: [federation] | ||
| + | - type: http | ||
| + | port: 9092 | ||
| + | resources: | ||
| + | - names: [replication] | ||
| + | |||
| + | send_federation: | ||
| + | |||
| + | worker_log_config: | ||
| + | ``` | ||
| + | ### Log config | ||
| + | Here you have an example how a WorkerX_log_config.yaml could look like. If something doesn' | ||
| + | ``` | ||
| + | version: 1 | ||
| + | |||
| + | formatters: | ||
| + | precise: | ||
| + | | ||
| + | |||
| + | filters: | ||
| + | context: | ||
| + | (): synapse.util.logcontext.LoggingContextFilter | ||
| + | request: "" | ||
| + | | ||
| + | handlers: | ||
| + | file: | ||
| + | class: logging.handlers.RotatingFileHandler | ||
| + | formatter: precise | ||
| + | filename: / | ||
| + | maxBytes: 104857600 | ||
| + | backupCount: | ||
| + | filters: [context] | ||
| + | encoding: utf8 | ||
| + | level: DEBUG | ||
| + | console: | ||
| + | | ||
| + | formatter: precise | ||
| + | level: INFO | ||
| + | | ||
| + | loggers: | ||
| + | synapse: | ||
| + | level: INFO | ||
| + | | ||
| + | synapse.storage.SQL: | ||
| + | level: INFO | ||
| + | | ||
| + | synapse.app.generic_worker: | ||
| + | level: DEBUG | ||
| + | root: | ||
| + | level: INFO | ||
| + | handlers: [file, console] | ||
| + | ``` | ||
| + | |||
| + | |||
| + | ### homeserver.yaml | ||
| + | Last but not least the excerpts from the homeserver.yaml: | ||
| + | |||
| + | ``` | ||
| + | ## Ports ## | ||
| + | | ||
| + | listeners: | ||
| + | - | ||
| + | port: 8448 | ||
| + | bind_addresses: | ||
| + | type: http | ||
| + | tls: false | ||
| + | x_forwarded: | ||
| + | resources: | ||
| + | - names: [client] | ||
| + | compress: true | ||
| + | - names: [federation] | ||
| + | compress: false | ||
| + | |||
| + | - port: 8888 | ||
| + | tls: false | ||
| + | bind_addresses: | ||
| + | type: http | ||
| + | x_forwarded: | ||
| + | |||
| + | resources: | ||
| + | - names: [client] | ||
| + | compress: true | ||
| + | - names: [federation] | ||
| + | compress: false | ||
| + | | ||
| + | # The HTTP replication port | ||
| + | - port: 9093 | ||
| + | bind_address: | ||
| + | type: http | ||
| + | resources: | ||
| + | - names: [replication] | ||
| + | |||
| + | ################################################## | ||
| + | snipsnap | ||
| + | ################################################## | ||
| + | |||
| + | # Worker | ||
| + | worker_replication_secret: | ||
| + | |||
| + | # The following block was added to be compatible with v1.84.0 which broke my former config on the 24th of May 2023. Check also the Worker configs above! | ||
| + | instance_map: | ||
| + | main: | ||
| + | host: matrix-synapse | ||
| + | port: 9093 | ||
| + | tls: false | ||
| + | generic_worker2: | ||
| + | host: matrix-federation | ||
| + | port: 9092 | ||
| + | tls: false | ||
| + | generic_worker1: | ||
| + | host: matrix-client | ||
| + | port: 9094 | ||
| + | tls: false | ||
| + | |||
| + | |||
| + | redis: | ||
| + | enabled: true | ||
| + | host: matrix-redis | ||
| + | port: 6379 | ||
| + | ``` | ||
| + | |||
| + | At least for me this Setup does work. I hope it is of help to you as I spent more than three whole days to get it too work. I searched a lot in the web about setting up workers in a docker-compose context, I knew only few about the single components and tried a lot pieces and hints that didn't work in the end. | ||
| + | |||
| + | Most of the time it was either | ||
| + | - something with the regex | ||
| + | - or with howto correctly address the different containers between each other. As mentioned before the only stable way to solve this turned out to be the container_name tags that resolved to the current ip-adresses even after down and up again the docker-compose environment. | ||
| + | |||
| + | And now good luck with your own efforts :) | ||
| + | |||
| + | |||
| + | ## For documentation and transparency | ||
| + | ### or What Happened Before ... | ||
| + | |||
| + | ## Status: Not working yet (25th of February 2022) | ||
| + | while trying to give my docker-compose setup two workers (multicontainer) - 1 for client, 1 for federation. While I made some progress today I am struggling with a strange problem: | ||
| + | - I use nginx reverse proxy to filter $request_uri and send it to one of the 3 upstreams. | ||
| + | - I have my nginx running and filtering, my workers and my master are up and running. | ||
| + | - But the instance isn't reachable (If I give my master the 8008-port back and don't send it through nginx any longer it's working again - but monolithic of course) | ||
| + | ### The Network Problem | ||
| + | - Checking my nginx log shows: A lot of`connect() failed (111: Connection refused) while connecting to upstream` with every request. But ther workers themselves are reachable. | ||
| + | - What is NOT reachable is`https:// | ||
| + | - But HTTP replication port is set in homeserver.yaml | ||
| + | - Hint from Chat regarding external networks in docker: https:// | ||
| + | - I don't see how an external network should help me. I of course don't use two or more docker-compose files. Ich _want_ to enclose the whole setup in one seperate network and because in my setup all containers use the same network there shouldn' | ||
| + | - But the hint let me `docker-compose exec synapse bash`. | ||
| + | - There I `curl localhost: | ||
| + | - `curl localhost: | ||
| + | - I also can't reach the port 6379 of the redis container :-/ ... Maybe it's no surprise that this setup doesn' | ||
| + | |||
| + | - Leaving the container and opening a bash in one of the two workers. | ||
| + | - Here the same curl to port 9093 leads to ... Connection refused. As experienced. --> So the opened replication port isn't reached to the network?! | ||
| + | - As you will see down in my docker-compose.yml all containers share the default network. | ||
| + | - Changing into a bash inside one of the workers: I can't curl from the master`s bash to the ports of the workers. | ||
| + | - Having a look in docker-compose docs I reminded myself that you can call the other containers with e.g. `curl client_worker: | ||
| + | - ... and in deed: I can reach it! So I can't use localhost or 127.0.0.1 or 0.0.0.0 for inter-container-communication but need to use the service name! | ||
| + | - Unfortunately that doesn' | ||
| + | - Maybe I can solve this later by defining links (https:// | ||
| + | #### Progress! | ||
| + | - Solved it: With the self defined `container_name`item in the docker-compose.yml it is possible to access the containers in nginx.conf! | ||
| + | - It's important to define the dependencies of the containers because else the setup won't work after restarting (network wouldn' | ||
| + | - I updated the files down on this page. | ||
| + | - So far the setup produces a working matrix server, BUT: | ||
| + | - It seems that still the entire traffic is routed to the master process. | ||
| + | - All I get in the logs of the workers is | ||
| + | ``` | ||
| + | *** STARTING SERVER ***** | ||
| + | matrix-federation | ||
| + | matrix-federation | ||
| + | ``` | ||
| + | - Redis outputs now | ||
| + | ``` | ||
| + | redis_1 | ||
| + | redis_1 | ||
| + | redis_1 | ||
| + | redis_1 | ||
| + | redis_1 | ||
| + | ``` | ||
| + | - So it seems that Redis is now working but the workers seem to listen to the wrong bindings? Or are the mappings in nginx.conf not properly working? | ||
| + | - I have a lot of those errors in the nginx log. Don`t know if this has to do with it. | ||
| + | ``` | ||
| + | 2022/02/26 11:20:48 [error] 32#32: *11814 open() "/ | ||
| + | 192.168.160.1 - - [26/ | ||
| + | 192.168.160.1 - - [26/ | ||
| + | ``` | ||
| + | - The current version of the nginx.conf and docker-compose.yml | ||
| + | - made it possible to start setup | ||
| + | - nginx finally sent the requests corresponding to the regex to the upstream addresses | ||
| + | - but while I could receive a message from a account of the matrix.org network I couldn' | ||
| + | - [This issue](https:// | ||
| + | - the logs of the workers stayed unchanged, while the master log showed that it communicated with the replication port ... | ||
| + | - so I'm still unsure if the workers actually do something or not :/ | ||
| + | - __Summary: | ||
| + | |||
| + | |||
| + | |||
| ## Old Outdated Information | ## Old Outdated Information | ||
homer