How to monitor Temporal Cloud metrics

Beyond the metrics provided by the Temporal SDKs, some key metrics exist only in Temporal Cloud. You can use your own observability tool to query an endpoint and review Namespace metrics.

To ensure security of your metrics, a CA certificate dedicated to observability is required. Only clients that use certificates signed by that CA, or that chain up to the CA, can query the metrics endpoint. For more information about CA certificates in Temporal Cloud, see Certificate requirements.

Configure a metrics endpoint using Temporal Cloud UI

note

To view and manage third-party integration settings, your user account must have the Global Admin Role.

To assign a certificate and generate your metrics endpoint, follow these steps:

1. In Temporal Cloud UI, click Settings.
2. On the Settings page, click Integrations.
3. In the Observability card, click Configure Observability. (If observability is already configured, the Edit link appears instead.)
4. In Certificates, paste a base64-encoded CA certificate PEM block.
5. Click Generate endpoint.

After the page refreshes, the new metrics endpoint appears below Endpoint, in the form https://<account-id>.tmprl.cloud/prometheus. Use the endpoint to configure your observability tool—for example, Grafana with this dashboard. For more information, see Set up Grafana with Temporal Cloud.

You can also query via the Prometheus HTTP API at URLs like:

https://<account-id>.tmprl.cloud/prometheus/api/v1/query?query=temporal_cloud_v0_state_transition_count

For example:

$ curl --cert client.pem --key client-key.pem "https://<account-id>.tmprl.cloud/prometheus/api/v1/query?query=temporal_cloud_v0_state_transition_count" | jq .
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "temporal_cloud_v0_state_transition_count",
          "__rollup__": "true",
          "operation": "WorkflowContext",
          "temporal_account": "a2dd6",
          "temporal_namespace": "mynamespace.a2dd6",
          "temporal_service_type": "history"
        },
        "value": [
          1672347471.2,
          "0"
        ]
      },
      ...
}

Configure a metrics endpoint using tcld

To add a certificate to a metrics endpoint, use tcld account metrics accepted-client-ca addtcld account metrics accepted-client-ca add

How to add end-entity certificates to the metrics endpoint of a Temporal Cloud account using tcld.

Learn more.

To enable a metrics endpoint, use tcld account metrics enabletcld account metrics enable

How to enable the metrics endpoint for a Temporal Cloud account using tcld.

Learn more.

To disable a metrics endpoint, use tcld account metrics disabletcld account metrics disable

How to disable the metrics endpoint for a Temporal Cloud account using tcld.

Learn more.

Available performance metrics

Temporal tracks the following metrics for your various Namespaces.

• temporal_cloud_v0_frontend_service_error_count
• temporal_cloud_v0_frontend_service_request_count
• temporal_cloud_v0_poll_success_count
• temporal_cloud_v0_poll_success_sync_count
• temporal_cloud_v0_poll_timeout_count
• temporal_cloud_v0_service_latency_bucket
• temporal_cloud_v0_service_latency_count
• temporal_cloud_v0_service_latency_sum
• temporal_cloud_v0_state_transition_count
• temporal_cloud_v0_total_action_count
• temporal_cloud_v0_workflow_cancel_count
• temporal_cloud_v0_workflow_continued_as_new_count
• temporal_cloud_v0_workflow_failed_count
• temporal_cloud_v0_workflow_success_count
• temporal_cloud_v0_workflow_terminate_count
• temporal_cloud_v0_workflow_timeout_count

Metrics for all Namespaces in your account are available from the metrics endpoint. The temporal_namespace label identifies the Namespace that is associated with each metric so that each user can build their own dashboard to meet their needs.

Metrics lag real-time performance by approximately one minute.

We retain raw metrics for seven days.