diff --git a/dictionary-octopus.txt b/dictionary-octopus.txt index 2c9618ef4b..2992565889 100644 --- a/dictionary-octopus.txt +++ b/dictionary-octopus.txt @@ -345,6 +345,7 @@ nlog nmap noconsolelogging nodir +noeviction nologo nologs noninteractive diff --git a/src/pages/docs/administration/high-availability/polling-tentacles-with-ha/index.mdx b/src/pages/docs/administration/high-availability/polling-tentacles-with-ha/index.mdx new file mode 100644 index 0000000000..dd94e46384 --- /dev/null +++ b/src/pages/docs/administration/high-availability/polling-tentacles-with-ha/index.mdx @@ -0,0 +1,21 @@ +--- +layout: src/layouts/Default.astro +pubDate: 2023-01-01 +modDate: 2026-10-06 +title: Polling Tentacles with HA +navTitle: Overview +navSection: Polling Tentacles with HA +description: How to connect Polling Tentacles to an Octopus High Availability cluster, using multi-node support with Redis or by polling every node. +navOrder: 50 +--- + +Listening Tentacles require no special configuration for Octopus High Availability. Polling Tentacles and Kubernetes agents, however, poll a server at regular intervals to check if there are any tasks waiting for the Tentacle to perform. In a High Availability scenario, a Polling Tentacle needs to be able to collect work queued by any Octopus Server node in your configuration. To configure the Kubernetes agent with Octopus High Availability, see [Kubernetes agent HA Cluster Support](/docs/infrastructure/deployment-targets/kubernetes/kubernetes-agent/ha-cluster-support). + +## Connecting Polling Tentacles + +There are two ways to connect Polling Tentacles to an HA cluster: + +- **[Multi-node support for Polling Tentacles](/docs/administration/high-availability/polling-tentacles-with-ha/multi-node-polling-tentacles)** (recommended). The nodes share a queue of pending requests in Redis, so any node can hand work to any Tentacle. Each Polling Tentacle only needs to poll a single load-balanced address, and you do not need to update your Tentacles when you add or remove a node. +- **[Polling every node](/docs/administration/high-availability/polling-tentacles-with-ha/poll-every-node)**. Each Polling Tentacle polls every node in your cluster, using a unique address or a unique port for each node. This does not need Redis, but you need to register every node with every Polling Tentacle, and update your Tentacles whenever you add or remove a node. + +We recommend multi-node support for Polling Tentacles. Poll every node only if you cannot run Redis, or your Octopus Server version does not support multi-node support for Polling Tentacles. diff --git a/src/pages/docs/administration/high-availability/polling-tentacles-with-ha/multi-node-polling-tentacles.md b/src/pages/docs/administration/high-availability/polling-tentacles-with-ha/multi-node-polling-tentacles.md new file mode 100644 index 0000000000..7416597bb4 --- /dev/null +++ b/src/pages/docs/administration/high-availability/polling-tentacles-with-ha/multi-node-polling-tentacles.md @@ -0,0 +1,282 @@ +--- +layout: src/layouts/Default.astro +pubDate: 2026-09-28 +modDate: 2026-10-06 +title: Multi-node support for Polling Tentacles +description: Use Redis to let Polling Tentacles connect to any node in an Octopus High Availability cluster through a single load-balanced address. +navOrder: 10 +--- + +In an Octopus High Availability (HA) cluster, a Polling Tentacle normally has to [poll every Octopus Server node](/docs/administration/high-availability/polling-tentacles-with-ha/poll-every-node). Work for a Tentacle is queued in memory on the node that runs the task, and only that node can hand it to the Tentacle. So each Tentacle needs a unique address or port for every node, and you need to update every Tentacle when you add or remove a node. + +Multi-node support for Polling Tentacles removes that restriction. The nodes share a pending request queue stored in Redis, so a request queued by any node can be collected by whichever node the Tentacle is connected to. Each Tentacle only needs to poll a single address, which a load balancer spreads across all the nodes. + +:::div{.hint} +Multi-node support for Polling Tentacles is available from Octopus Server 2026.4.6342. +::: + +## How it works + +When multi-node support for Polling Tentacles is turned on: + +- Each node stores the requests it queues for Polling Tentacles in Redis, instead of in its own memory. When a Tentacle polls a node, that node collects the next request for the Tentacle from Redis, sends it, and returns the response to the node that queued it. +- Requests stored in Redis are compressed and encrypted with your [Master Key](/docs/security/data-encryption). +- Small data streams travel inside the encrypted request in Redis. Larger data streams are written to a `DataStreams` directory in the [cluster shared directory](#cluster-shared-storage) so every node can read them. Packages that are already on shared storage, such as the shared package cache, are read from where they are and are not copied. +- Tentacle communication logs, shown on the deployment target's **Connectivity** page, are collected from every active node, not only the node you are connected to. + +Listening Tentacles are not affected. + +## Requirements + +To use multi-node support for Polling Tentacles, you need: + +- An Octopus HA cluster where every node runs a version of Octopus Server that supports the feature. +- A [Redis instance](#redis-requirements) that every node can reach, configured the way Octopus needs. +- A [cluster shared directory](#cluster-shared-storage) on storage every node can read and write. +- A [TCP load balancer](#load-balancer) in front of the nodes' Polling Tentacle port (`10943` by default). + +### Redis requirements \{#redis-requirements} + +We have tested multi-node support for Polling Tentacles with Redis 8.0.3, and recommend Redis 8.0 or later. Earlier versions may work, but we have not tested them. + +Octopus uses Redis as a short-lived queue, not a database. Redis must hold data in memory only: + +- **Turn off persistence.** Do not use RDB snapshots or AOF. +- **Do not use replication or automatic failover.** Replication is asynchronous, so a promoted replica can bring back requests that a node has already collected, and they would be sent to the Tentacle again. +- **Set the eviction policy to `noeviction`.** Evicting keys would silently drop requests. + +Octopus detects when Redis loses all of its data, for example when it restarts. Each node checks for this every minute, so it can take up to a minute to notice. It fails the requests that were in flight at the time and then decides whether to retry them. New requests work again as soon as Redis is back. Octopus cannot detect a partial restore, which is why persistence and replication must be off. + +A single Redis node started with these options meets the requirements: + +```bash +redis-server --save "" --appendonly no --maxmemory-policy noeviction --requirepass "your-secret-password" +``` + +If you use a managed Redis service, choose a tier or configuration without persistence and replicas, and set the eviction policy to `noeviction`. + +### Cluster shared storage \{#cluster-shared-storage} + +Every node must be able to read the data streams written by the other nodes, so Octopus stores them in the cluster shared directory. If multi-node support for Polling Tentacles is on, but neither a cluster shared directory nor an executions cluster shared directory is configured, Octopus Server fails to start with this error: + +```text +Multi-node support for polling tentacles is enabled, but no cluster shared directory has been configured. +``` + +Set the cluster shared directory with the [path command](/docs/administration/octopus.server.exe-command-line/path): + +```powershell +Octopus.Server.exe path --instance="OctopusServer" --clusterShared \\OctoShared\OctopusData +``` + +Octopus stores transient execution data, which is only needed while tasks run, in these folders in the cluster shared directory: + +- `DataStreams`, for data streams sent to Polling Tentacles +- `DataBus`, used internally by Octopus Server +- `SharedPackageCache`, for the package cache + +To keep transient execution data on separate storage, such as faster storage that does not need to be backed up, use `--executionsClusterShared` instead of, or as well as, `--clusterShared`. Octopus then uses the same folders in the executions cluster shared directory. Both must point to storage every node can read and write. + +If you are running the [Octopus Server Linux container](#linux-container) or the [Helm chart](#helm-chart), configure this with the settings in those sections instead. + +### Load balancer \{#load-balancer} + +Put a load balancer in front of the Polling Tentacle port on every node that processes tasks. The load balancer must: + +- Pass TCP traffic straight through. Octopus terminates TLS and authenticates Tentacles with certificates, so the load balancer must not terminate TLS. +- Route to every node that processes tasks. You do not need to include [UI-only nodes](/docs/installation/octopus-server-linux-container/octopus-in-kubernetes#ui-and-backend-nodes). + +You do not need session affinity. Any node can serve any Tentacle. + +## Turn on multi-node support for Polling Tentacles + +Multi-node support for Polling Tentacles is turned on when a Redis connection string is configured, and turned off when it is not. Configure **every node** in the cluster with the same connection string. + +The value is a [StackExchange.Redis connection string](https://stackexchange.github.io/StackExchange.Redis/Configuration.html), for example: + +```text +your-redis-host:6380,password=your-secret-password,ssl=true +``` + +You can set the connection string in any of the following ways. If more than one is set, the environment variable takes precedence over the configuration file. + +| Method | Name | +| --- | --- | +| Command line | `Octopus.Server configure --multiNodePollingTentaclesRedisConnectionString=""` | +| Environment variable | `OCTOPUS_MULTI_NODE_POLLING_TENTACLES_REDIS_CONNECTION_STRING` | +| Server configuration file key | `Octopus.Communications.MultiNodePollingTentaclesRedisConnectionString` | + +### Windows and Linux servers + +1. Make sure the [cluster shared directory](#cluster-shared-storage) is configured. +1. On each node, run the [configure command](/docs/administration/octopus.server.exe-command-line/configure): + + ```powershell + Octopus.Server.exe configure --instance="OctopusServer" --multiNodePollingTentaclesRedisConnectionString="your-redis-host:6380,password=your-secret-password,ssl=true" + ``` + + The command checks that the connection string is valid before saving it. The value is treated as sensitive, so it is masked in the command's output. +1. Restart each node. The setting takes effect when Octopus Server starts. +1. [Check the connection to Redis](#check-redis). +1. [Point your Polling Tentacles at the load balancer](#register-polling-tentacles). + +### Octopus Server Linux container \{#linux-container} + +Set these environment variables on every Octopus Server container: + +| Name | Value | +| --- | --- | +| `OCTOPUS_MULTI_NODE_POLLING_TENTACLES_REDIS_CONNECTION_STRING` | Your Redis connection string. | +| `CLUSTER_SHARED_CONFIG` | `CLUSTER_SHARED` for a new installation, or `SEPARATE_VOLUMES_WITH_CLUSTER_SHARED` to keep the existing `/repository`, `/artifacts`, `/taskLogs`, and `/eventExports` volumes of an existing installation. | + +Then mount `/clusterShared` on storage every node can read and write. Octopus writes data streams to `/clusterShared/DataStreams`, or to `/executionsClusterShared/DataStreams` if `USE_EXECUTIONS_CLUSTER_SHARED` is `True`. See [cluster shared configuration](/docs/installation/octopus-server-linux-container#cluster-shared-configuration) for what each `CLUSTER_SHARED_CONFIG` value does. + +The container checks these settings when it starts: + +- If `OCTOPUS_MULTI_NODE_POLLING_TENTACLES_REDIS_CONNECTION_STRING` is set and `CLUSTER_SHARED_CONFIG` is `SEPARATE_VOLUMES`, the container stops with an error. +- If `OCTOPUS_MULTI_NODE_POLLING_TENTACLES_REDIS_CONNECTION_STRING` is set and `CLUSTER_SHARED_CONFIG` is not set, the container logs a warning. Octopus Server then fails to start unless a cluster shared or executions cluster shared directory was already configured. + +### Helm chart \{#helm-chart} + +The [Octopus Deploy Helm chart](https://github.com/OctopusDeploy/helm-charts/tree/main/charts/octopus-deploy) can configure multi-node support for Polling Tentacles, the cluster shared volume, and the load balancer for you. It can also run Redis in your cluster, configured to meet the [Redis requirements](#redis-requirements): + +```yaml +octopus: + clusterShared: + mode: SEPARATE_VOLUMES_WITH_CLUSTER_SHARED + multiNodePollingTentacles: + enabled: true +redis: + enabled: true +``` + +This example uses `SEPARATE_VOLUMES_WITH_CLUSTER_SHARED`, which keeps the existing volumes of an installation you are moving to multiple nodes. For a new installation, we recommend `CLUSTER_SHARED`, which stores everything in a single cluster shared volume. See [cluster shared configuration](/docs/installation/octopus-server-linux-container#cluster-shared-configuration). + +The in-cluster Redis is a single pod. Requests that are in flight when it restarts fail, and new requests work again once it is back. + +By default, the in-cluster Redis has no memory limit, so the `noeviction` policy never applies and Redis can grow until the pod runs out of memory and restarts. Set `redis.maxMemory`, for example to `200mb`. When Redis reaches it, new requests are rejected instead of queued requests being evicted. If you also set a memory limit in `redis.resources`, set `redis.maxMemory` below it. + +To use your own Redis instead, provide the connection string: + +```yaml +octopus: + clusterShared: + mode: SEPARATE_VOLUMES_WITH_CLUSTER_SHARED + multiNodePollingTentacles: + enabled: true + redis: + connectionString: "your-redis-host:6380,password=your-secret-password,ssl=true" +``` + +This setting is under `octopus.multiNodePollingTentacles`, not the top-level `redis` key, which only controls the in-cluster Redis. Leave `redis.enabled` set to `false`. If it is `true`, the chart ignores your connection string and uses the in-cluster Redis. + +The chart will not render if multi-node support for Polling Tentacles is on but `octopus.clusterShared.mode` is not set, or if there is neither an in-cluster Redis nor a connection string. + +When the feature is on, the chart creates a `LoadBalancer` service, named `-octopus-deploy-polling-tentacles` by default, which passes Tentacle traffic through to any node. Point your Polling Tentacles at this service's address. + +The chart's per-node services and ingresses are still created, so existing Tentacles that poll every node keep working. For all the chart's settings, including load balancer annotations and supplying the connection string from your own secret, see the [chart's README](https://github.com/OctopusDeploy/helm-charts/tree/main/charts/octopus-deploy#multi-node-polling-tentacles). + +## Check the connection to Redis \{#check-redis} + +After the nodes restart, send a `GET` request to `/api/serverstatus/redis` on each node. The endpoint does not need an API key: + +```bash +curl https://your-octopus-url/api/serverstatus/redis +``` + +The response tells you whether that node can use Redis: + +| Property | Meaning when `true` | +| --- | --- | +| `IsEnabled` | A Redis connection string is configured, so the feature is on. | +| `IsConfigured` | Octopus could create a Redis connection from the connection string. | +| `IsReachable` | Octopus connected to Redis and ran a command. | + +All three values should be `true` on every node. Because the request goes through your web load balancer, you might need to send it to each node's own address to check every node. + +## Point Polling Tentacles at the load balancer \{#register-polling-tentacles} + +Register each Polling Tentacle with the load balancer's address as its only comms address. Use `--server` for the Octopus Web Portal address, and `--server-comms-address` for the Polling Tentacle load balancer: + +```bash +tentacle register-with --server="https://your-octopus-url" --apiKey="API-YOUR-KEY" --comms-style="TentacleActive" --server-comms-address="https://your-polling-load-balancer:10943" --environment="Production" --role="web-server" +``` + +Then restart the Tentacle: + +```bash +tentacle service --restart +``` + +To point an existing Polling Tentacle at the load balancer: + +1. Add the load balancer with the [poll-server](/docs/administration/tentacle.exe-command-line/poll-server) command: + + ```bash + tentacle poll-server --server="https://your-octopus-url" --apiKey="API-YOUR-KEY" --server-comms-address="https://your-polling-load-balancer:10943" + ``` + + The Tentacle reuses the subscription ID it already has for your Octopus Server, so Octopus still sees it as the same Tentacle. + +1. Remove the per-node entries with the `clear-trusted-servers` command, keeping the load balancer: + + ```bash + tentacle clear-trusted-servers --keep="https://your-polling-load-balancer:10943" + ``` + + This removes every trusted server whose address is not listed in `--keep`. Each address must match the stored address exactly, so use the same scheme, host, and port you passed to `--server-comms-address`. For example, `https://your-polling-load-balancer` does not match `https://your-polling-load-balancer:10943`. If the Tentacle also trusts another Octopus Server, add that server's address to `--keep` as a comma-separated list. + + The `clear-trusted-servers` command needs Tentacle 8.1.1713 or later. On an older Tentacle, upgrade it first. + +1. Restart the Tentacle: + + ```bash + tentacle service --restart + ``` + +Do not use `configure --reset-trust` for this. It removes the load balancer entry and the Tentacle's subscription ID as well, so you would need to register the Tentacle again. + +You can run the first step on its own and remove the per-node entries later. A Tentacle that polls the load balancer and the individual nodes at the same time works, because each request is collected by only one connection. The extra connections add traffic but do not change how tasks run. + +Tentacles that still poll every node individually keep working while multi-node support for Polling Tentacles is on, so you can move them to the load balancer at your own pace. + +### Kubernetes agents + +Kubernetes agents also poll for work, and can use the load balancer the same way. When multi-node support for Polling Tentacles is on, the Kubernetes agent creation wizard asks for a single Communications URL instead of one for each node. To learn how to set this URL, and how to move an existing agent to the load balancer, see [Kubernetes agent HA Cluster Support](/docs/kubernetes/targets/kubernetes-agent/ha-cluster-support#multi-node-support-for-polling-tentacles). + +## Turn off multi-node support for Polling Tentacles + +To turn the feature off, clear the connection string on every node and restart them: + +```powershell +Octopus.Server.exe configure --instance="OctopusServer" --multiNodePollingTentaclesRedisConnectionString= +``` + +If the `OCTOPUS_MULTI_NODE_POLLING_TENTACLES_REDIS_CONNECTION_STRING` environment variable is still set, the feature stays on and the command logs a warning. Remove the environment variable as well. + +Before you turn the feature off, make sure every Polling Tentacle and Kubernetes agent polls each node individually, as described in [Polling every node](/docs/administration/high-availability/polling-tentacles-with-ha/poll-every-node). Otherwise, tasks run by a node that a Tentacle is not polling will wait for that Tentacle until they time out. + +## Troubleshooting + +**Octopus Server does not start, and reports that no cluster shared directory has been configured.** +Configure a [cluster shared directory](#cluster-shared-storage) on storage every node can access, then start the node again. + +**The configure command reports that the Redis connection string is not valid.** +Check the value follows the [StackExchange.Redis connection string format](https://stackexchange.github.io/StackExchange.Redis/Configuration.html). Wrap the whole value in quotes so your shell does not split it on commas. + +**`IsReachable` is `false`.** +Check the node can reach the Redis host and port through any firewalls, that the password is correct, and that `ssl=true` is set if your Redis requires TLS. + +**Tentacles fail to connect through the load balancer.** +Check the load balancer passes TCP traffic straight through on the Polling Tentacle port, and does not terminate TLS. + +**Deployments to Polling Tentacles fail or wait, only on some nodes.** +Check every node is configured with the same Redis connection string, and that each node's `/api/serverstatus/redis` response is `true` for all three values. + +## Learn more + +- [Polling Tentacles with HA](/docs/administration/high-availability/polling-tentacles-with-ha) +- [Octopus Server Linux container](/docs/installation/octopus-server-linux-container) +- [Octopus Server in Kubernetes](/docs/installation/octopus-server-linux-container/octopus-in-kubernetes) +- [Configure command](/docs/administration/octopus.server.exe-command-line/configure) +- [Path command](/docs/administration/octopus.server.exe-command-line/path) diff --git a/src/pages/docs/administration/high-availability/polling-tentacles-with-ha.mdx b/src/pages/docs/administration/high-availability/polling-tentacles-with-ha/poll-every-node.md similarity index 71% rename from src/pages/docs/administration/high-availability/polling-tentacles-with-ha.mdx rename to src/pages/docs/administration/high-availability/polling-tentacles-with-ha/poll-every-node.md index b2c6b1f7e1..190838b164 100644 --- a/src/pages/docs/administration/high-availability/polling-tentacles-with-ha.mdx +++ b/src/pages/docs/administration/high-availability/polling-tentacles-with-ha/poll-every-node.md @@ -1,26 +1,22 @@ --- layout: src/layouts/Default.astro pubDate: 2023-01-01 -modDate: 2026-08-20 -title: Polling Tentacles with HA -description: With Octopus High Availability, Polling Tentacles must poll all Octopus Server nodes in your configuration. -navOrder: 50 +modDate: 2026-10-06 +title: Polling every node +description: Connect Polling Tentacles to an Octopus High Availability cluster without Redis, by registering every node with every Polling Tentacle. +navOrder: 20 --- -Listening Tentacles require no special configuration for Octopus High Availability. Polling Tentacles and Kubernetes agents, however, poll a server at regular intervals to check if there are any tasks waiting for the Tentacle to perform. In a High Availability scenario Polling Tentacles must poll all Octopus Server nodes in your configuration. To configure the Kubernetes agent with Octopus High Availability, see [Kubernetes agent HA Cluster Support](/docs/infrastructure/deployment-targets/kubernetes/kubernetes-agent/ha-cluster-support). +If you cannot use [multi-node support for Polling Tentacles](/docs/administration/high-availability/polling-tentacles-with-ha/multi-node-polling-tentacles), configure each Polling Tentacle to poll every node in your Octopus High Availability (HA) cluster. Work for a Tentacle is queued in memory on the node that runs the task, and only that node can hand it to the Tentacle, so the Tentacle must be able to reach each node individually. -## Connecting Polling Tentacles - -While a Tentacle could poll a load balancer in an Octopus High Availability cluster, there is a risk, depending on your load balancer configuration, that the Tentacle will not poll all servers in a timely manner. - -We recommend two options when configuring Polling Tentacles to connect to your Octopus High Availability cluster: +There are two ways to do this: - Using a **unique address**, and the same listening port (`10943` by default) for each node. - Using the same address and a **unique port** for each node. -These are discussed further in the next sections. +With either option, you need to register every node with every Polling Tentacle, and update your Tentacles whenever you add or remove a node. -### Using a unique address +## Using a unique address In this scenario, no load balancer is required. Instead, each Octopus node would be configured to listen on the same port (`10943` by default) for inbound traffic. In addition, each node would be able to be reached directly by your Polling Tentacle on a unique address for the node. @@ -38,15 +34,15 @@ The important thing to remember is that each node should be using a **unique add **Tip:** A Polling Tentacle will connect to the Octopus Rest API over ports 80 or 443 when it is registering itself with the Octopus Server. After that, it will connect over port `10943` (by default) with the Octopus Server node. -It's important to ensure that any firewalls also allow port 80 or 443 for the initial Tentacle registration. +It is important to ensure that any firewalls also allow port 80 or 443 for the initial Tentacle registration. ::: -### Using a unique port +## Using a unique port In this scenario, a type of [Network Address Translation (NAT)](https://en.wikipedia.org/wiki/Network_address_translation) is leveraged by using the same address and **unique ports**, usually routed through a load balancer or other network device. Each Octopus node would be configured to listen on a different port (starting at `10943` by default) for inbound traffic. :::div{.hint} -The advantage of using unique ports is that the Polling Tentacle doesn't need to know each node's address, only the port. The address translation is handled by the load balancer. This allows each node to have a private IP address, with no public access from outside your network required. +The advantage of using unique ports is that the Polling Tentacle does not need to know each node's address, only the port. The address translation is handled by the load balancer. This allows each node to have a private IP address, with no public access from outside your network required. ::: Imagine a three-node HA cluster. For each one, we expose a different port to listen on using the [Octopus.Server configure command](/docs/administration/octopus.server.exe-command-line/configure): @@ -70,7 +66,7 @@ The important thing to remember is that each node should be using the **same add There are two options to add Octopus Servers to a Polling Tentacle: the command line, or editing the Tentacle.config file directly. -Both methods need the Tentacle service restarted afterwards. A running Tentacle doesn't pick up a new server address until it restarts. +Both methods need the Tentacle service restarted afterwards. A running Tentacle does not pick up a new server address until it restarts. **Command line:** @@ -80,7 +76,7 @@ The command line is the preferred option. Run the command once per server; an ex C:\Program Files\Octopus Deploy\Tentacle>Tentacle poll-server --server=https://your-octopus-url --apikey=API-YOUR-KEY ``` -Once you've added every node in your cluster, restart the Tentacle service: +Once you have added every node in your cluster, restart the Tentacle service: ```text C:\Program Files\Octopus Deploy\Tentacle>Tentacle service --restart @@ -90,7 +86,7 @@ For more information on these commands, see the [Tentacle poll-server](/docs/adm **Tentacle.config:** -Alternatively you can edit Tentacle.config directly to add each Octopus Server (this is interpreted as a JSON array of servers). This method isn't recommended, as editing the JSON array by hand is error-prone. Restart the Tentacle service after you've saved your changes. +Alternatively you can edit Tentacle.config directly to add each Octopus Server (this is interpreted as a JSON array of servers). This method is not recommended, as editing the JSON array by hand is error-prone. Restart the Tentacle service after you have saved your changes. ```xml diff --git a/src/pages/docs/administration/octopus.server.exe-command-line/configure.md b/src/pages/docs/administration/octopus.server.exe-command-line/configure.md index d89f6b5438..b77c31de42 100644 --- a/src/pages/docs/administration/octopus.server.exe-command-line/configure.md +++ b/src/pages/docs/administration/octopus.server.exe-command-line/configure.md @@ -1,7 +1,7 @@ --- layout: src/layouts/Default.astro pubDate: 2023-01-01 -modDate: 2023-01-01 +modDate: 2026-09-29 title: Configure description: Configure this Octopus instance navOrder: 31 @@ -56,6 +56,15 @@ Where [] is any of: 'https://+:443/OctopusComms'); set to blank to disable websockets. Refer to https://o- c.to/WebSocketComms. + --multiNodePollingTentaclesRedisConnectionString=VALUE + Sets the Redis connection string used by + multi-node support for polling tentacles, + which allows polling tentacles to connect to + any node in cluster (e.g. via a load + balancer). Setting a value enables the + feature; set to blank to disable it. Every + node in the cluster must be configured with + the same value. --webListenPrefixes=VALUE Comma-separated list of HTTP.sys listen prefixes (e.g., 'http://localhost/octopus') diff --git a/src/pages/docs/administration/octopus.server.exe-command-line/path.md b/src/pages/docs/administration/octopus.server.exe-command-line/path.md index a5c5d5ea6f..b2a8882333 100644 --- a/src/pages/docs/administration/octopus.server.exe-command-line/path.md +++ b/src/pages/docs/administration/octopus.server.exe-command-line/path.md @@ -1,7 +1,7 @@ --- layout: src/layouts/Default.astro pubDate: 2023-01-01 -modDate: 2023-01-01 +modDate: 2026-09-29 title: Path description: Set the file paths that Octopus will use for storage navOrder: 160 @@ -29,7 +29,39 @@ Where [] is any of: is not running. This directory should not be shared between nodes. --clusterShared=VALUE Set the root path where shared files will be - stored for Octopus clusters + stored for Octopus clusters. Set to blank to + clear it, so that relative paths resolve under + the Home directory again. + --executionsClusterShared=VALUE + Set the root path where transient execution + files will be stored for Octopus clusters. + + When configured, Octopus stores transient + execution data (DataStreams, PackageCache, and + DataBus) here instead of in the Cluster Shared + directory. This data is generated during tasks + like deployments and runbooks and is not long- + lived. As with Cluster Shared, this path must + be on a shared volume accessible to all nodes + in the cluster. + + Configuration is typically not required, as + Octopus automatically falls back to the + Cluster Shared directory or the Home directory + if no path is specified. + + Note: Before changing this setting, ensure + that no tasks are running on the Octopus + Servers. Additionally, the previously used + directories (DataBus, DataStreams, + PackageCache) in the ClusterShared directory or + the Home directory will no longer be used and + can be manually deleted since they will not be + cleaned up automatically. + + Set to blank to clear it, so that transient + execution data falls back to the Cluster + Shared or Home directory. --nugetRepository=VALUE Set the package path for the built-in package repository @@ -69,3 +101,15 @@ octopus.server path --imports \\Octoshared\OctopusData\Imports octopus.server path --eventExports \\Octoshared\OctopusData\EventExports octopus.server path --telemetry \\Octoshared\OctopusData\Telemetry ``` + +This example stores transient execution data, such as data streams for [multi-node support for Polling Tentacles](/docs/administration/high-availability/polling-tentacles-with-ha/multi-node-polling-tentacles), on separate shared storage: + +```text +octopus.server path --executionsClusterShared \\OctoFastShared\OctopusExecutions +``` + +This example clears the executions cluster shared directory, so transient execution data is stored in the cluster shared directory again: + +```text +octopus.server path --executionsClusterShared= +``` diff --git a/src/pages/docs/installation/octopus-server-linux-container/index.mdx b/src/pages/docs/installation/octopus-server-linux-container/index.mdx index a9c543fd24..21389db392 100644 --- a/src/pages/docs/installation/octopus-server-linux-container/index.mdx +++ b/src/pages/docs/installation/octopus-server-linux-container/index.mdx @@ -1,7 +1,7 @@ --- layout: src/layouts/Default.astro pubDate: 2023-01-01 -modDate: 2024-07-15 +modDate: 2026-09-29 title: Octopus Server Linux Container description: Run Octopus Deploy in the official Docker Linux container. Pull the image, configure environment variables, connect a SQL Server database, and get started. navOrder: 8 @@ -82,6 +82,9 @@ Read the Docker [docs](https://docs.docker.com/engine/reference/commandline/run/ |**ADMIN_EMAIL**|The email associated with the admin user account| |**TASK_CAP**|Sets the task cap for this node. If not specified, the default is 5.| |**DISABLE_DIND**|The Linux image will by default attempt to run Docker-in-Docker to support [execution containers for workers](/docs/projects/steps/execution-containers-for-workers). This requires the image to be launched with [privileged permissions](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities). Setting `DISABLE_DIND` to `Y` prevents Docker-in-Docker from being run when the container is booted.| +|**CLUSTER_SHARED_CONFIG**|Sets how Octopus stores the files that every node in a cluster needs to share. Valid values are `CLUSTER_SHARED`, `SEPARATE_VOLUMES_WITH_CLUSTER_SHARED`, and `SEPARATE_VOLUMES`, and they are not case-sensitive. Any other value stops the container with an error. If not set, a new installation uses the `/repository`, `/artifacts`, `/taskLogs`, and `/eventExports` volumes, and an existing installation keeps its current paths. See [cluster shared configuration](#cluster-shared-configuration).| +|**USE_EXECUTIONS_CLUSTER_SHARED**|Set to `True` to store transient execution data (the `SharedPackageCache`, `DataBus`, and `DataStreams` folders) in the `/executionsClusterShared` volume instead of in `/clusterShared`. The value is not case-sensitive. Only valid when `CLUSTER_SHARED_CONFIG` is `SEPARATE_VOLUMES_WITH_CLUSTER_SHARED` or `CLUSTER_SHARED`. With any other `CLUSTER_SHARED_CONFIG` value, including not set, the container stops with an error. See [cluster shared configuration](#cluster-shared-configuration) for why you might use this.| +|**OCTOPUS_MULTI_NODE_POLLING_TENTACLES_REDIS_CONNECTION_STRING**|The Redis connection string that turns on [multi-node support for Polling Tentacles](/docs/administration/high-availability/polling-tentacles-with-ha/multi-node-polling-tentacles). Requires `CLUSTER_SHARED_CONFIG` to be `SEPARATE_VOLUMES_WITH_CLUSTER_SHARED` or `CLUSTER_SHARED`.| ### Exposed container ports @@ -106,11 +109,34 @@ Read the Docker [docs](https://docs.docker.com/engine/reference/commandline/run/ | **/taskLogs** | Path where task logs are stored | Shared storage | | **/eventExports** | Path where event audit logs are exported | Shared storage | | **/cache** | Path where cached files e.g., signature and delta files (used for package acquisition), are stored | Host filesystem or container | +| **/clusterShared** | Cluster shared directory. Used when `CLUSTER_SHARED_CONFIG` is `SEPARATE_VOLUMES_WITH_CLUSTER_SHARED` or `CLUSTER_SHARED` | Shared storage | +| **/executionsClusterShared** | Transient execution data, in the `SharedPackageCache`, `DataBus`, and `DataStreams` folders. Used when `USE_EXECUTIONS_CLUSTER_SHARED` is `True` | Shared storage | :::div{.hint} **Note:** We recommend using shared storage when mounting the volumes for files that must be shared between multiple octopus container nodes, e.g., artifacts, packages, task logs, and event exports. ::: +### Cluster shared configuration \{#cluster-shared-configuration} + +The `CLUSTER_SHARED_CONFIG` environment variable sets which volumes Octopus uses for the files that every node needs to share. When it is set, the container applies it each time it starts, not only on first start, so you can change it on an existing installation. + +The values are not case-sensitive. + +| Value | When to use it | Volumes used | +| --- | --- | --- | +| `CLUSTER_SHARED` | Recommended for new installations. | A single `/clusterShared` volume: packages in `/clusterShared/Packages`, artifacts in `/clusterShared/Artifacts`, task logs in `/clusterShared/TaskLogs`, event exports in `/clusterShared/EventExports`, imports in `/clusterShared/Imports`, telemetry in `/clusterShared/Telemetry`, and transient execution data in `/clusterShared/SharedPackageCache`, `/clusterShared/DataBus`, and `/clusterShared/DataStreams`. The `/repository`, `/artifacts`, `/taskLogs`, and `/eventExports` volumes are not used. | +| `SEPARATE_VOLUMES_WITH_CLUSTER_SHARED` | Recommended if you are moving an existing installation to multiple nodes and want to keep your existing volumes, rather than moving their contents into a single volume. | `/repository`, `/artifacts`, `/taskLogs`, and `/eventExports`, plus `/clusterShared`. Octopus stores transient execution data in `/clusterShared` instead of in `/cache` and the Octopus home directory: the package cache in `/clusterShared/SharedPackageCache`, DataBus in `/clusterShared/DataBus`, and DataStreams in `/clusterShared/DataStreams`. Imports and telemetry are also stored in `/clusterShared/Imports` and `/clusterShared/Telemetry`. | +| `SEPARATE_VOLUMES` | To go back to the layout without a cluster shared directory, for example if you tried one of the other values and want to undo it. | `/repository`, `/artifacts`, `/taskLogs`, and `/eventExports`, with no cluster shared directory. Any cluster shared or executions cluster shared directory configured previously is removed. Multi-node support for Polling Tentacles cannot be used with this value. | +| Not set | To keep the current configuration. | On a new installation, `/repository`, `/artifacts`, `/taskLogs`, and `/eventExports`. On an existing installation, the container does not change any paths, so whatever was configured previously, including any cluster shared directory, is kept. For example, if you used `CLUSTER_SHARED` and then remove the variable, Octopus keeps using `/clusterShared`. | + +In every mode, `/cache` is still used as the node's own cache directory. + +Switching an existing installation to `CLUSTER_SHARED` does not move existing packages, artifacts, or logs into `/clusterShared`, so Octopus stops finding them unless you move them yourself. To add a cluster shared directory to an existing installation, use `SEPARATE_VOLUMES_WITH_CLUSTER_SHARED`. + +To keep transient execution data on different storage, such as faster storage that does not need to be backed up, set `USE_EXECUTIONS_CLUSTER_SHARED` to `True` and mount `/executionsClusterShared` on storage every node can read and write. Octopus then uses the same `SharedPackageCache`, `DataBus`, and `DataStreams` folders in `/executionsClusterShared`. + +[Multi-node support for Polling Tentacles](/docs/administration/high-availability/polling-tentacles-with-ha/multi-node-polling-tentacles) needs a cluster shared directory. If `OCTOPUS_MULTI_NODE_POLLING_TENTACLES_REDIS_CONNECTION_STRING` is set and `CLUSTER_SHARED_CONFIG` is `SEPARATE_VOLUMES`, the container stops with an error. The container only checks the environment variable, not a connection string set with the `configure` command. + ## Upgrading When the volumes are externally mounted to the host filesystem, upgrades between Octopus versions are much easier. We can picture the upgrade process with a container as being similar to [moving a standard Octopus Server](/docs/administration/managing-infrastructure/moving-your-octopus/move-the-database-and-server) since containers, being immutable, don't themselves get updated. diff --git a/src/pages/docs/installation/octopus-server-linux-container/octopus-in-kubernetes.mdx b/src/pages/docs/installation/octopus-server-linux-container/octopus-in-kubernetes.mdx index ff31f9ccfc..fba9f3c244 100644 --- a/src/pages/docs/installation/octopus-server-linux-container/octopus-in-kubernetes.mdx +++ b/src/pages/docs/installation/octopus-server-linux-container/octopus-in-kubernetes.mdx @@ -1,7 +1,7 @@ --- layout: src/layouts/Default.astro pubDate: 2023-01-01 -modDate: 2024-04-23 +modDate: 2026-09-29 title: Run Octopus Server in Kubernetes description: Install Octopus Server into a Kubernetes cluster using the official Linux container. Choose a single-node setup or High Availability across multiple pods. navOrder: 40 @@ -150,6 +150,10 @@ spec: Unlike the Octopus Web Portal, Polling Tentacles must be able to connect to each Octopus node individually to pick up new tasks. Our Octopus HA cluster assumes two nodes, therefore a load balancer is required for each node to allow direct access. +:::div{.hint} +If you turn on [multi-node support for Polling Tentacles](/docs/administration/high-availability/polling-tentacles-with-ha/multi-node-polling-tentacles), Polling Tentacles can connect to any node through a single TCP load balancer instead, so you do not need to add each node to each Polling Tentacle. The [Helm chart](#helm-chart) can create this load balancer for you. +::: + The following YAML creates load balancers with separate public IPs for each node. They direct web traffic to each node on port `80`, Polling Tentacle traffic on port `10943`, and gRPC traffic on port `8443`. The `octopus-0` load balancer: diff --git a/src/pages/docs/kubernetes/targets/kubernetes-agent/ha-cluster-support.md b/src/pages/docs/kubernetes/targets/kubernetes-agent/ha-cluster-support.md index b416eedd42..1f176b9e7a 100644 --- a/src/pages/docs/kubernetes/targets/kubernetes-agent/ha-cluster-support.md +++ b/src/pages/docs/kubernetes/targets/kubernetes-agent/ha-cluster-support.md @@ -1,7 +1,7 @@ --- layout: src/layouts/Default.astro pubDate: 2024-05-14 -modDate: 2026-01-15 +modDate: 2026-10-06 title: HA Cluster Support description: How to install/update the agent when running Octopus in an HA Cluster navOrder: 50 @@ -10,12 +10,29 @@ navOrder: 50 ## Octopus Deploy HA Cluster -Similarly to Polling Tentacles, the Kubernetes agent must have a URL for each individual node in the HA Cluster so that it receive commands from all clusters. These URLs must be provided when registering the agent or some deployments may fail depending on which node the tasks are executing. +Similarly to Polling Tentacles, the Kubernetes agent polls Octopus Server for work, so in an HA Cluster it must be able to receive commands from every node. There are two ways to do this: + +- Turn on [multi-node support for Polling Tentacles](/docs/administration/high-availability/polling-tentacles-with-ha/multi-node-polling-tentacles). The agent only needs a single URL, such as a load balancer in front of the nodes. See [Multi-node support for Polling Tentacles](#multi-node-support-for-polling-tentacles). +- Give the agent a URL for each individual node in the HA Cluster. These URLs must be provided when registering the agent or some deployments may fail depending on which node the tasks are executing. To read more about selecting the right URL for your nodes, see [Polling Tentacles and Kubernetes agents with HA](/docs/administration/high-availability/polling-tentacles-with-ha). +## Multi-node support for Polling Tentacles + +When [multi-node support for Polling Tentacles](/docs/administration/high-availability/polling-tentacles-with-ha/multi-node-polling-tentacles) is turned on, the nodes share pending requests through Redis, so the agent can collect work queued by any node from whichever node it connects to. The agent only needs a single Communications URL. + +The Kubernetes agent creation wizard detects that the feature is on, and does not show the extra page that asks for a URL for each node. Instead, it uses a single **Octopus Deploy Server Communications URL**. By default, this is your Octopus Server URL with port `10943`. To use your [Polling Tentacle load balancer](/docs/administration/high-availability/polling-tentacles-with-ha/multi-node-polling-tentacles#load-balancer) instead, select **Advanced Setup** in the wizard and enter its address, for example `https://your-polling-load-balancer:10943/`. + +Because the agent connects through a single URL, you do not need to update it when you add or remove nodes. + +:::div{.warning} +If you turn off multi-node support for Polling Tentacles, configure every agent with a URL for each node first. Otherwise, tasks run by a node the agent is not connected to wait for the agent until they time out. +::: + ## Agent Installation on an HA Cluster +This section is for HA Clusters that do not use multi-node support for Polling Tentacles. + ### Octopus Deploy 2024.3+ To make things easier, Octopus will detect when it's running HA and show an extra configuration page in the Kubernetes agent creation wizard which asks you to give a unique URL for each cluster node. @@ -48,7 +65,7 @@ The new property name is `agent.serverCommsAddresses`. Note that "Addresses" is ## Upgrading the Agent after Adding/Removing Cluster nodes -If you add or remove cluster nodes, you need to update your agent's configuration so that it continues to connect to all nodes in the cluster. To do this, you can simply run a helm upgrade command with the urls of all current cluster nodes. The agent will take remove any old urls and replace them with the provided ones. +If you are not using multi-node support for Polling Tentacles and you add or remove cluster nodes, you need to update your agent's configuration so that it continues to connect to all nodes in the cluster. To do this, you can simply run a helm upgrade command with the urls of all current cluster nodes. The agent will take remove any old urls and replace them with the provided ones. ```bash helm upgrade --atomic \