Use gateways to expose an integration

Plus tier

This feature is available in HCP Vault Secrets Plus tier.

Some rotating secret use cases involve integrations with services that aren't exposed to the internet, for example, your local on-premises PostgreSQL cluster. To accommodate these use cases, HashiCorp introduced a tool named vault-secrets-gateway that makes the service accessible to HCP Vault Secrets. This is a binary that you can run within your network, where it's able to communicate with the service you want to auto-rotate secrets for.

Key concepts

Gateways are running instances of vault-secrets-gateway.
Gateways pools are collections of gateways that run within the same network. These gateway pools can be attached to integrations.
Integrations manage the authentication and connection details that HCP Vault Secrets uses to access third-party providers to create and revoke credentials.

How do gateways work?

Gateways connect to HCP and identify themselves as a gateway for a specific gateway pool. Gateway pool IDs can be provided when creating an integration in HCP Vault Secrets. When HCP Vault Secrets needs to perform a rotation operation for an integration that has an associated gateway pool, instead of trying to connect directly to the third-party provider defined in the integration, HCP Vault Secrets reaches out to the gateway. HCP Vault Secrets can't initiate connections to the gateway directly, but it can send messages to any gateways that have already connected to HCP and are waiting for a response to its question "does anyone need to connect to my integration?"

Gateway pool setup

There is currently no support for creating gateway-pools via the UI.

Run hcp vault-secrets gateway-pools create to create a gateway pool.

Sensitive data

The creds.json generated includes the client_secret, used for the gateway to connect to HCP, and is considered sensitive information.

$ hcp vault-secrets gateway-pools create --description "test gateway" my-gateway-pool --output-dir ./gateway
Gateway Pool Name:          my-gateway-pool
Gateway Pool Resource Name: secrets/project/ad203a6b-b390-46a5-b9b7-98d192be2159/gateway-pool/my-gateway-pool
Gateway Pool Resource ID:   72eaad2f-0958-46ea-8cc4-c75ff5382446
Description:                test gateway

The command will create a configuration file and a credentials file in the given output directory.

Create environment variables required by the gateway.

$ hcp vault-secrets gateway-pools read my-gateway-pool --format=json > gatewayResponse.json \
GATEWAY_RESOURCE_NAME=$(jq -r .gateway_pool.resource_name < gatewayResponse.json) \
GATEWAY_SERVICE_PRINCIPAL_ID=$(jq -r .gateway_pool.service_principal_id < gatewayResponse.json | sed 's/-[0-9][0-9]*@.*$//')

Use the POST /secrets/2023-11-28/organizations/{org_id}/projects/{proj_id}/gateway-pools endpoint to create a gateway pool.

$ curl \
--location "https://api.cloud.hashicorp.com/secrets/2023-11-28/organizations/${HCP_ORG_ID}/projects/${HCP_PROJ_ID}/gateway-pools" \
--request POST \
--data '{"name": "my-gateway-pool", "description": "test gateway"}' \
--header "Authorization: Bearer ${HCP_API_TOKEN}" > gatewayResponse.json

Example output:

{
    "gateway_pool": {
        "organization_id": "f4ad6462-8846-4798-8923-b4b397b982b4",
        "project_id": "63674848-312a-48b4-ad28-e86793eb0496",
        "name": "my-gateway-pool",
        "description": "test gateway",
        "resource_name": "secrets/project/63674848-312a-48b4-ad28-e86793eb0496/gateway-pool/my-gateway-pool",
        "resource_id": "1e76229a-6136-4ba4-81d8-b90ac87fda2c",
        "created_at": "2024-11-20T20:29:53.049515Z",
        "updated_at": null,
        "created_by": {
            "name": "test",
            "type": "TYPE_SERVICE",
            "email": ""
        },
        "updated_by": null,
        "service_principal_id": "my-test-gateway-6afea9de-459202@63674848-312a-48b4-ad28-e86793eb0496"
    },
    "client_id": "sNvydiLmJj3hjDU3QQTsmme35jxXMHyX",
    "client_secret": "3D91JOnhkW15CbL_cZrOFpMvK_0ysbRzbLcBhLrX8vbPtaG9tZEqyffwajUg24QV"
}

Create environment variables required by the gateway.

$ GATEWAY_RESOURCE_NAME=$(jq -r .gateway_pool.resource_name < gatewayResponse.json)
GATEWAY_CLIENT_ID=$(jq -r .client_id < gatewayResponse.json)
GATEWAY_CLIENT_SECRET=$(jq -r .client_secret < gatewayResponse.json)
GATEWAY_SERVICE_PRINCIPAL_ID=$(jq -r .gateway_pool.service_principal_id < gatewayResponse.json | sed 's/-[0-9][0-9]*@.*$//')

Create a directory to store the credentials file (creds.json), and configuration (config.hcl) files.
```
$ mkdir -p gateway
```

Create a credentials file using the client_id and client_secret from the CreateGatewayPool response.

Sensitive data

The creds.json generated includes the client_secret, used for the gateway to connect to HCP, and is considered sensitive information.

$ cat - > gateway/creds.json <<EOF
{
    "project_id": "$HCP_PROJECT_ID",
    "scheme": "service_principal_creds",
    "oauth": {
        "client_id": "$GATEWAY_CLIENT_ID",
        "client_secret": "$GATEWAY_CLIENT_SECRET"
    }
}
EOF

Create the configuration file from the CreateGatewayPool response.

$ cat - > gateway/config.hcl <<EOF
cloud {
  cred_file     = "./creds.json"
  resource_name = "$GATEWAY_RESOURCE_NAME"
}
EOF

Installation

The gateway can be installed using Helm for Kubernetes environments, as a stand-alone Docker container, or manually by downloading the binary.

Add the Helm repository.

$ helm repo add hashicorp https://helm.releases.hashicorp.com

Install the helm chart.

$ helm install gateway hashicorp/vault-secrets-gateway

Start the gateway

The vault-secrets-gateway needs two things to run:

the resource name of a gateway pool
a way to authenticate to HCP, in the form of a credentials file or workload identity federation

Create a directory to store the credentials file and copy creds.json to it.
```
$ mkdir -p ./creds && cp gateway/creds.json ./creds
```

Create a Kubernetes secret.

$ kubectl create secret generic my-hcp-creds --from-file=./creds/

Install the helm chart.

$ helm install gateway hashicorp/vault-secrets-gateway \
    --set hcpGatewayPoolResourceName="$GATEWAY_RESOURCE_NAME" \
    --set credsSecretName=my-hcp-creds

The creds.json approach requires storing the service principle credentials in a file on disk. A better option is to use Workload Identity Federation.

Configure workload identity federation in HCP using the .gateway_pool.service_principal_id from the CreateGatewayPool response.

$ hcp iam workload-identity-providers create-oidc k8s_pod --allowed-audience vault-secrets \
  --issuer "$(kubectl get --raw /.well-known/openid-configuration | jq -r .issuer)" \
  --conditional-access 'jwt_claims.sub == "system:serviceaccount:default:gateway-vault-secrets-gateway"' \
  --service-principal iam/project/$HCP_PROJECT_ID/service-principal/$GATEWAY_SERVICE_PRINCIPAL_ID

Deploy the helm chart.

$ helm install gateway hashicorp/vault-secrets-gateway \
  --set hcpGatewayPoolResourceName="$GATEWAY_RESOURCE_NAME" \
  --set hcpWorkloadIdentityProviderResourceName=iam/project/$HCP_PROJECT_ID/service-principal/$GATEWAY_SERVICE_PRINCIPAL_ID/workload-identity-provider/k8s_pod

This requires that the Kubernetes cluster exposes at least the paths .well-known/openid-configuration and /openid/v1/jwks so that HCP can verify the claim.

Start the gateway using the configuration and credentials files created in the gateway directory earlier.

$ docker run --rm --name vault-secrets-gateway \
    --add-host=host.docker.internal:host-gateway \
    -v $(pwd)/gateway:/gateway \
    -w /gateway hashicorp/vault-secrets-gateway \
  /bin/vault-secrets-gateway run --config-file config.hcl

Note

When launching a docker container this way, the local host machine itself can be addressed as host.docker.internal. So for example, if the gateway will be used in order to rotate PostgreSQL user passwords, and the PostgreSQL cluster is running on the same host as the gateway docker container, then the PostgreSQL URL used in the integration should use host.docker.internal as the hostname part of the URL.

If there's no need for the gateway to connect to the local host machine on which the docker container is running, the --add-host option may be omitted.

Additional example

Here is a more complete configuration file example showing some optional configuration.

gateway {
  log_level = "trace"
  meta_config {
    use_hostname = true
  }
  metadata = {
    foo = "bar"
  }
  http_port = 8080
  cert_path = "./cached-cert.pem"
}

cloud {
  cred_file     = "./creds.json"
  resource_name = "secrets/project/63674848-312a-48b4-ad28-e86793eb0496/gateway-pool/my-gateway-pool"
}

log_level controls what log messages are emitted
use_hostname adds a metadata "hostname" field to the gateway telemetry shared with HCP Vault Secrets; this can be useful for debugging, but is turned off by default for those who consider this too sensitive
metadata defines key-value metadata pairs that are shared with HCP Vault Secrets, which are visible when doing a list-gateways request; this can be useful for debugging and identifying which gateway is which
http_port exposes an HTTP listener that provides /liveness and /ready endpoints; this is mostly intended for use with the helm chart
cert_path specifies where to write the X.509 certificate used by the gateway

Gateway certificates

If cert_path isn't provided as configuration, the following warning log message will appear in vault-secrets-gateway's output:

no cert_path specified in config, will not persist fetched certificate

As part of its self-initialization, once vault-secrets-gateway authenticates to HCP, it will request an X.509 certificate from HCP Vault Secrets. Ideally, this certificate would be persisted to disk so that if restarted we can avoid generating a new cert, but it's not essential, so we don't block startup if we can't persist it.

Why is this certificate needed?

It's used to encrypt the channel created between HCP Vault Secrets and vault-secrets-gateway. The HTTP requests vault-secrets-gateway makes to HCP don't go to HCP Vault Secrets directly. Instead, both vault-secrets-gateway and HCP Vault Secrets speak to a broker running within HCP. Using the certificate ensures that the broker is unable to see any of the content of the communication, it only gets to see the wrapper information necessary to route between them.

Commands

create - Create a new Vault Secrets Gateway Pool.
list - List Vault Secrets gateway pools.
list-gateways - List Vault Secrets gateway pools gateways.
read - Read a Vault Secrets gateway pool.
update - Update a Vault Secrets gateway pool.
delete - Delete a Vault Secrets gateway pool.