Skip to main content

Temporal Server self-hosted production deployment

note

The information in this page is being dispersed into Knowledge base articles, Cluster concept guide, and the Cluster deployment guide.

Temporal Server​

Temporal Server is a Go application which you can import or run as a binary (we offer builds with every release).

If you are running only the Go binary, Go is not required.

But if you are building Temporal or running it from source, Go v1.17+ is required.

While Temporal can be run as a single Go binary, we recommend that production deployments of Temporal Server should deploy each of the 4 internal services separately (if you are using Kubernetes, one service per pod) so they can be scaled independently in the future.

See Clusters concept guide for more info.

In practice, this means you will run each container with a flag specifying each service, e.g.

docker run
    # persistence/schema setup flags omitted
    -e SERVICES=history \                      -- Spinup one or more of: history, matching, worker, frontend
    -e LOG_LEVEL=debug,info \                           -- Logging level
    -e DYNAMIC_CONFIG_FILE_PATH=config/foo.yaml         -- Dynamic config file to be watched
    temporalio/server:<tag>

See the Docker source file for more details.

Each release also ships a Server with Auto Setup Docker image that includes an auto-setup.sh script we recommend using for initial schema setup of each supported database. You should familiarize yourself with what auto-setup does, as you will likely be replacing every part of the script to customize for your own infrastructure and tooling choices.

Though neither are blessed for production use, you can consult our Docker-Compose repo or Helm Charts for more hints on configuration options.

The information in this page is being dispersed into Knowledge base articles, Cluster concept guide, and the Cluster deployment guide.

:::

Minimum Requirements​

Kubernetes is not required for Temporal, but it is a popular deployment platform anyway. We do maintain a Helm chart you can use as a reference, but you are responsible for customizing it to your needs.

Configuration​

At minimum, the development.yaml file needs to have the global and persistence parameters defined.

The Server configuration reference has a more complete list of possible parameters.

Before you deploy: Reminder on shard count

A huge part of production deploy is understanding current and future scale - the number of shards can't be changed after the cluster is in use so this decision needs to be upfront. Shard count determines scaling to improve concurrency if you start getting lots of lock contention. The default numHistoryShards is 4; deployments at scale can go up to 500-2000 shards. Please consult our configuration docs and check with us for advice if you are worried about scaling.

Scaling and Metrics​

The requirements of your Temporal system will vary widely based on your intended production workload. You will want to run your own proof of concept tests and watch for key metrics to understand the system health and scaling needs.

All metrics emitted by the server are listed in Temporal's source. There are also equivalent metrics that you can configure from the client side. At a high level, you will want to track these 3 categories of metrics:

  • Service metrics: For each request made by the service handler we emit service_requests, service_errors, and service_latency metrics with type, operation, and namespace tags. This gives you basic visibility into service usage and allows you to look at request rates across services, namespaces and even operations.
  • Persistence metrics: The Server emits persistence_requests, persistence_errors and persistence_latency metrics for each persistence operation. These metrics include the operation tag such that you can get the request rates, error rates or latencies per operation. These are super useful in identifying issues caused by the database.
  • Workflow Execution stats: The Server also emits counters for when Workflow Executions are complete. These are useful in getting overall stats about Workflow Execution completions. Use workflow_success, workflow_failed, workflow_timeout, workflow_terminate and workflow_cancel counters for each type of Workflow Execution completion. These include the namespace tag.

Please request any additional information in our forum. Key discussions are here:

Checklist for Scaling Temporal​

Temporal is highly scalable due to its event sourced design. We have load tested up to 200 million concurrent Workflow Executions. Every shard is low contention by design, and it is very difficult to oversubscribe to a Task Queue in the same cluster. With that said, here are some guidelines to some common bottlenecks:

  • Database. The vast majority of the time the database will be the bottleneck. We highly recommend setting alerts on schedule_to_start_latency to look out for this. Also check if your database connection is getting saturated.
  • Internal services. The next layer will be scaling the 4 internal services of Temporal (Frontend, Matching, History, and Worker). Monitor each accordingly. The Frontend Service is more CPU bound, whereas the History and Matching Services require more memory. If you need more instances of each service, spin them up separately with different command line arguments. You can learn more cross referencing our Helm chart with our Server Configuration reference.
  • See Platform limits for other limits you will want to keep in mind when doing system design, including event history length.

Please see the dedicated docs on Tuning and Scaling Workers.

FAQs​

FAQ: Autoscaling Workers based on Task Queue load​

Temporal does not yet support returning the number of tasks in a task queue. The main technical hurdle is that each task can have its own ScheduleToStart timeout, so just counting how many tasks were added and consumed is not enough.

This is why we recommend tracking schedule_to_start_latency for determining if the task queue has a backlog (aka your Workflow and Activity Workers are under-provisioned for a given Task Queue). We do plan to add features that give more visibility into the task queue state in the future.

FAQ: High Availability cluster configuration​

You can set up a high availability deployment by running more than one instance of the server. Temporal also handles membership and routing. You can find more details in the clusterMetadata section of the Server Configuration reference.

clusterMetadata:
  enableGlobalNamespace: false
  failoverVersionIncrement: 10
  masterClusterName: "active"
  currentClusterName: "active"
  clusterInformation:
    active:
      enabled: true
      initialFailoverVersion: 0
      rpcAddress: "127.0.0.1:7233"

FAQ: Multiple deployments on a single cluster​

You may sometimes want to have multiple parallel deployments on the same cluster, for example:

  • when you want to split Temporal deployments based on namespaces, e.g. staging/dev/uat, or for different teams who need to share common infrastructure.
  • when you need a new deployment to change numHistoryShards.

You can skip the following procedure if your server is running v1.19 or later. The v1.19 release ensures that the membership from different clusters does not combine.

We recommend not doing this if you can avoid it. If you need to do it anyway, double-check the following:

  • Have a separate persistence (database) for each deployment
  • Cluster membership ports should be different for each deployment (they can be set through environment variables). For example:
    • Temporal1 services can have 7233 for frontend, 7234 for history, 7235 for matching
    • Temporal2 services can have 8233 for frontend, 8234 for history, 8235 for matching
  • There is no need to change gRPC ports.

More details about the reason here.

Securing Temporal​

Please see our dedicated docs on Temporal Server Security.

Debugging Temporal​

Debugging Temporal Server Configs​

Recommended configuration debugging techniques for production Temporal Server setups:

Debugging Workflows​

We recommend using Temporal Web to debug your Workflow Executions in development and production.

Tracing Workflows​

Temporal Web's tracing capabilities mainly track activity execution within a Temporal context. If you need custom tracing specific for your usecase, you should make use of context propagation to add tracing logic accordingly.

Further things to consider​

danger

This document is still being written, and we would welcome your questions and contributions.

Please search for these topics in our forum or ask on Slack.

Temporal Antipatterns​

Please request elaboration on any of these.

  • Trying to implement a queue in a Workflow (because people hear we replace queues)
  • Serializing massive amounts of state into and out of the workflow.
  • Treating everything as rigid/linear sequence of steps instead of dynamic logic
  • Implementing a DSL which is actually just a generic schema based language
  • Polling in activities instead of using signals
  • Blocking on incredibly long RPC requests and not using heartbeats
  • Failing/retrying workflows without a specific business reason

Temporal Best practices​

Please request elaboration on any of these.

  • Mapping things to entities instead of traditional service design
  • Testing: unit, integration
  • Retries: figuring out right values for timeouts
  • Versioning
  • The Workflow is Temporal's fundamental unit of scalability - break things into workflows to scale, don't try to stuff everything in one workflow!

External Runbooks​

Third party content that may help: