cluster.mdx

title	Deploying a distributed Restate cluster using Docker
description	Learn how to deploy a Restate cluster using Docker.
pagination_next
pagination_prev

import Admonition from '@theme/Admonition'; import {Step} from "../../src/components/Stepper";

Deploying a distributed Restate cluster using Docker

This guide shows how to deploy a distributed Restate cluster consisting of 3 nodes using Docker and Docker compose.

- [Docker](https://docs.docker.com/get-started/get-docker/) - [Docker compose](https://docs.docker.com/compose/install/)

To deploy a 3 node distributed Restate cluster, create a file docker-compose.yml and run docker compose up.

x-environment: &common-env
  RESTATE_CLUSTER_NAME: "restate-cluster"
  # Every node runs every role
  RESTATE_ROLES: '["admin", "worker", "log-server", "metadata-server"]'
  # For more on logging, see: https://docs.restate.dev/operate/monitoring/logging
  RESTATE_LOG_FILTER: "restate=info"
  RESTATE_BIFROST__DEFAULT_PROVIDER: "replicated"
  RESTATE_BIFROST__REPLICATED_LOGLET__DEFAULT_LOG_REPLICATION: 2  # We require minimum of 2 nodes to accept writes
  RESTATE_METADATA_SERVER__TYPE: "replicated"
  # The addresses where nodes can reach each other over the "internal" Docker Compose network
  RESTATE_METADATA_CLIENT__ADDRESSES: '["http://restate-1:5122","http://restate-2:5122","http://restate-3:5122"]'
  # Partition snapshotting, see: https://docs.restate.dev/operate/snapshots
  RESTATE_WORKER__SNAPSHOTS__DESTINATION: "s3://restate/snapshots"
  RESTATE_WORKER__SNAPSHOTS__SNAPSHOT_INTERVAL_NUM_RECORDS: "1000"
  RESTATE_WORKER__SNAPSHOTS__AWS_REGION: "local"
  RESTATE_WORKER__SNAPSHOTS__AWS_ENDPOINT_URL: "http://minio:9000"
  RESTATE_WORKER__SNAPSHOTS__AWS_ALLOW_HTTP: true
  RESTATE_WORKER__SNAPSHOTS__AWS_ACCESS_KEY_ID: "minioadmin"
  RESTATE_WORKER__SNAPSHOTS__AWS_SECRET_ACCESS_KEY: "minioadmin"

x-defaults: &defaults
  image: docker.restate.dev/restatedev/restate:1.2
  extra_hosts:
    - "host.docker.internal:host-gateway"

services:
  restate-1:
    <<: *defaults
    ports:
      - "8080:8080"  # Ingress
      - "9070:9070"  # Admin
      - "5122:5122"  # Node-to-node communication
    environment:
      <<: *common-env
      RESTATE_NODE_NAME: restate-1
      RESTATE_FORCE_NODE_ID: 1
      RESTATE_ADVERTISED_ADDRESS: "http://restate-1:5122"  # Other Restate nodes must be able to reach us using this address
      RESTATE_AUTO_PROVISION: "true"                       # Only the first node provisions the cluster

  restate-2:
    <<: *defaults
    ports:
      - "25122:5122"
      - "29070:9070"
      - "28080:8080"
    environment:
      <<: *common-env
      RESTATE_NODE_NAME: restate-2
      RESTATE_FORCE_NODE_ID: 2
      RESTATE_ADVERTISED_ADDRESS: "http://restate-2:5122"
      RESTATE_AUTO_PROVISION: "false"

  restate-3:
    <<: *defaults
    ports:
      - "35122:5122"
      - "39070:9070"
      - "38080:8080"
    environment:
      <<: *common-env
      RESTATE_NODE_NAME: restate-3
      RESTATE_FORCE_NODE_ID: 3
      RESTATE_ADVERTISED_ADDRESS: "http://restate-3:5122"
      RESTATE_AUTO_PROVISION: "false"

  minio:
    image: quay.io/minio/minio
    entrypoint: "/bin/sh"
    # Ensure a bucket called "restate" exists on startup:
    command: "-c 'mkdir -p /data/restate && /usr/bin/minio server --quiet /data'"
    ports:
      - "9000:9000"

The cluster uses the replicated Bifrost provider and replicates log writes to a minimum of 2 nodes. Since we are running with 3 nodes, the cluster can tolerate 1 node failure without becoming unavailable. By default, partition state is replicated to all workers (though each partition has only one acting leader at a time).

The replicated metadata cluster consists of all nodes since they all run the metadata-server role. Since the replicated metadata cluster requires a majority quorum to operate, the cluster can tolerate 1 node failure without becoming unavailable.

Take a look at the cluster deployment documentation for more information on how to configure and deploy a distributed Restate cluster. In this example we also deployed a Minio server to host the cluster snapshots bucket. Visit Snapshots to learn more about whis is strongly recommended for all clusters.

You can check the status of the cluster by running the `restatectl status` command on any of the started Restate servers. Note, it might take a few seconds until the cluster has fully started and the status is available.

```shell
docker compose exec restate-1 restatectl status
```

<Step stepLabel="3" title={<a href={"/get_started/quickstart"}>Start a local Restate service}/>

You can register the service endpoint at any of the started Restate nodes since they all run the `admin` role.

```shell
restate dp register http://host.docker.internal:9080
```

Or alternatively you can open the Restate UI at http://localhost:9080 and register the service endpoint there.

You can invoke the registered service at any of the started Restate nodes since they all run the ingress.

```shell
curl localhost:8080/Greeter/greet -H 'content-type: application/json' -d '"Sarah"' &&
curl localhost:28080/Greeter/greet -H 'content-type: application/json' -d '"Bob"' &&
curl localhost:38080/Greeter/greet -H 'content-type: application/json' -d '"Eve"'
```

Try killing and restarting one of the Restate nodes and see how the cluster reacts. ```shell docker compose kill restate-1 && sleep 5 && docker compose up -d restate-1 ``` Try instructing the partition processors to create a snapshot of their state in the object store bucket: ```shell docker compose exec restate-1 restatectl snapshot create ``` Navigate to the Minio console at [http://localhost:9000](http://localhost:9000) and browse the bucket contents (default credentials: `minioadmin`/`minioadmin`).

Here are some next steps for you to try:

• Try to configure a 5 server Restate cluster that can tolerate up to 2 server failures.
• Trim the logs (either manually, or by setting up automatic trimming) before adding more nodes.
• Try to deploy a 3 server Restate cluster using Kubernetes.