Protecting APIs Deployed Behind Istio Service Mesh
Learn how to install and configure SecureAuth's Istio Authorizer to protect APIs and services deployed behind the Istio Service Mesh. You can learn, for example, how to deploy and protect HTTP and gRPC services.
Install Istio Authorizer
Installation Alternative
As an alternative to this procedure, you can install both Istio Service Mesh and Istio Authorizer using the SecureAuth on k8s repository which provides simple deployment commands you can use out of the box.
Prerequisites
Kubernetes version 1.19 or subsequent
Kubernetes cluster is set up.
Setting Up k8s Cluster
You can set up a Kubernetes cluster locally using kind.
Go version below 1.17
GO111MODULE="on" go get sigs.k8s.io/kind@v0.9.0 && kind create cluster
Go version 1.17 or above
GO111MODULE="on" go install sigs.k8s.io/kind@ && kind create cluster
Istio version 1.9 or subsequent is installed and up and running.
Istio Sidecar is installed.
Injecting Istio Sidecar
To configure Istio proxies, you need to inject Istio Sidecar to pods running in your Istio Service Mesh. It can be done either automatically at pod creation and using admission controller, or manually at any given time.
SecureAuth recommends using automatic sidecar injection and setting the
istio-injection
flag toenabled
for any namespace that contains the services you wish to protect with Istio Authorizer.Helm version 3.0 or subsequent
Create Istio Authorizer in SecureAuth
In the workspace of your choice, select Authorization > Gateways> CREATE GATEWAY.
Select Istio, enter the name and the description for your authorizer, and select NEXT.
In the Gateway Management view, go to the QUICK START tab and proceed as follows:
Install Istio Authorizer using Helm chart:
helm repo add acp https://charts.cloudentity.io && helm repo update helm upgrade --install istio-authorizer acp/istio-authorizer \ --set clientCredentials.clientID={yourClientIdentifier} \ --set clientCredentials.clientSecret={yourClientSecret} \ --set issuerURL={yourIssuerURL} \ --set "discovery.namespaces={default}" \ --namespace acp-istio-authorizer \ --create-namespace
You can find this command with the arguments filled for you in the QUICKSTART view for your authorizer instance.
The
clientCredentials.clientID
,clientCredentials.clientSecret
, andissuerURL
parameters are used in the client credentials OAuth grant type to authenticate your authorizer's requests to, for example, fetch authorization policies from SecureAuth.The
clientCredentials.clientID
argument should point to the client identifier of the client application created for your authorizer in the System workspace of your tenant. You can find the client identifier in the Settings view for your authorizer.The
clientCredentials.clientSecret
argument should point to the client secret of the client application created for your authorizer in the System workspace of your tenant. You can find the client secret in the Settings view for your authorizer.Credentials security
For production environments, you should create Kubernetes secrets manually that are responsible for storing your credentials values.
To increase security of secrets stored in your repository, it is recommended to encrypt your Kubernetes secrets. You can use tools like Mozilla SOPS or Bitnami Sealed Secrets to encrypt your secrets.
When the secrets are applied to your Kubernetes deployment, the secrets are visible as plain text. Anyone who is authorized to create a Pod in a namespace can read any secret in that namespace; this includes indirect access such as the ability to create a Deployment. To mitigate the risks, Kubernetes recommends to:
Enable encryption at Rest for secrets.
Enable or configure RBAC rules that restrict reading data in secrets.
Where appropriate, use mechanisms such as RBAC to limit which principals are allowed to create or replace secrets.
To learn more, visit Kubernetes secrets documentation.
The
issuerURL
argument should point to the issuer URL of your Istio Authorizer client application created within the System workspace of your tenant. You can find the issuer URL in the Settings view for your authorizer. If you are using a vanity domain for your SecureAuth tenant and it is impossible to retrieve the tenant's and server's identifier from the URL, provide values for thetenantID
andserverID
parameters in yourvalues.yaml
file.The
discovery.namespaces={default}
parameter is responsible for providing the namespaces where Istio Authorizer performs the service discovery. By default, Istio Authorizer performs discovery only in thedefault
namespace. You can add a comma-separated list of namespaces as the value for thediscovery.namespaces
parameter provided that those namespaces already exist, for example,discovery.namespaces={default,example1,example2}
. If you wish to know more, see the API Discovery Configuration for Istio Authorizer article.Tip
If you need to change your Istio Authorizer settings, see configuration reference.
Result:Istio Authorizer scans deployments in configured namespaces and funnels information about discovered APIs to SecureAuth. Note that if you didn't deploy any services yet, there's nothing to discover. To quickly deploy a sample HTTP/gPRC service, follow:
Add Istio Authorizer as Extension Provider to Istio
To delegate access control enforcement to your Istio Authorizer, you need to define it as an external authorization provider for your Istio Service Mesh.
Edit the mesh configuration using the following command:
kubectl edit configmap istio -n istio-system
Define SecureAuth Authorizer using extension providers, for example:
data: mesh: |- extensionProviders: - name: "acp-authorizer" envoyExtAuthzGrpc: service: "istio-authorizer.acp-istio-authorizer.{cluster-name}" port: "9001"
The
{cluster-name}
variable should point to your cluster name, for example,svc.cluster.local
.Istio Extension Providers
Since v1.14.0, Istio added an example extension provider to their mesh configuration. When adding the
acp-authorizer
extension provider to the Istio configmap, make sure that you do not duplicate theextensionProviders
entry as it may result in the authorizer or your additional extension providers not working properly.Restart Istio to apply the changes:
kubectl rollout restart deployment/istiod -n istio-system
Deploy and Protect Services
With Istio Authorizer installed and attached to Istio Service Mesh as an external authorization provider, you can:
Parsing Request Bodies
For HTTP services, SecureAuth supports parsing request bodies. It can be used, for example, to write an authorization policy that checks request bodies.
If you wish to parse request bodies, you need to set the parseBody.enabled
flag to true
in the values.yaml file for your authorizer.
API Discovery Configuration for Istio Authorizer
SecureAuth Istio Authorizer supports automatic service discovery based on the OpenAPI specification.
Services Hosting OpenAPI Endpoints
For services that host an OpenAPI endpoint, it is possible to provide this endpoint's path as the value for the openAPIEndpoint
parameter in Istio Authorizer's configuration. By doing so, you can instruct Istio Authorizer to add this path to the whitelist enabling the API discovery functionality to work without being blocked by the authorization layer.
A service can use the services.k8s.cloudentity.com/spec-url
annotation on a deployed k8s resource to specify a URL where its OpenAPI or Proto specification is available, for example:
kind: Deployment metadata: name: hello labels: app: hello namespace: default annotations: services.k8s.cloudentity.com/spec-url: "https://raw.githubusercontent.com/OAI/OpenAPI-Specification/master/examples/v3.0/petstore.yaml"
Istio Authorizer scans k8s deployments and, once it has found the annotation described above, it fetches the specification, parses it to get a list of APIs that a service is exposing, and then it's sending this information to SecureAuth.
By default, Istio Authorizer is configured to perform service discovery only in the default namespace. To make Istio Authorizer perform the service discovery in other namespaces, edit the values.yaml file. In th data.config
section, add your namespaces:
discovery: namespaces: - default - namespace1 - namespace2
With the above settings, Istio performs service discovery in all of the following namespaces: default
, namespace1
, and namespace2
.
To learn how you can configure API discovery for Configuration ReferenceIstio Authorizer, see its configuration reference.
Deploy and Protect HTTP Services
Prerequisites
Istio Authorizer is installed
Deploy Sample Service
Deploy a sample HTTP service using the following command:
kubectl apply -f - <<EOF apiVersion: v1 kind: ServiceAccount metadata: name: httpbin --- apiVersion: v1 kind: Service metadata: name: httpbin spec: selector: app: httpbin ports: - port: 80 name: http --- apiVersion: apps/v1 kind: Deployment metadata: name: httpbin labels: app: httpbin annotations: services.k8s.cloudentity.com/spec-url: "http://httpbin.org/spec.json" spec: replicas: 1 selector: matchLabels: app: httpbin template: metadata: labels: app: httpbin spec: serviceAccountName: httpbin containers: - name: httpbin image: "kennethreitz/httpbin" ports: - containerPort: 80 name: http EOF
Connect Service
There are two ways to connect Istio API groups to SecureAuth services: starting from the gateway to be connected or starting from the service that you want to connect.
From Gateway
From the list of available gateways, select your newly-created Istio gateway and go to its APIs tab.
A list of imported API groups opens.
From the list of API groups available, select an API group and, from its drop-down menu, pick a service to which you'd like to connect the API group.
Note
You can connect the API group to an existing service or a new one you create, both options available from the same service drop-down menu.
From Service
Select APIs from the left sidebar and go to the AUTHORIZATION tab.
Pick a service that you want to connect and select ADD GATEWAY API for the selected service.
In the Connect Istio API Group popup window, select an API gateway and an API group to be connected. Click CONNECT to proceed.
Result:In the APIS tab of the Gateway Management view, you can see specific API groups integrated to services.
Apply Policy
Once SecureAuth has discovered the APIs deployed behind your Istio gateway, you can protect those APIs with an SecureAuth policy.
Create an API policy:
Select APIs from the left sidebar and go to the AUTHORIZATION tab.
Select any Istio-protected API with authorization status Unrestricted. The policy selection form opens.
In the policy selection form, select a policy from the dropdown list and click Update to proceed.
Result:You have successfully assigned a policy to your Istio API.
Test Enforcement
To test your deployed and protected service, change the variables and execute the command:
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.9/samples/sleep/sleep.yaml export SLEEP_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name}) kubectl exec -it $SLEEP_POD -c sleep curl {YOUR_SERVICE_URL}/{ENDPOINT}
Example:
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.9/samples/sleep/sleep.yaml export SLEEP_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name}) kubectl exec -it $SLEEP_POD -c sleep curl http://httpbin:80/deny
Verify that the request is blocked or passing in accordance with your applied policy.
Deploy and Protect gRPC Services
About gRPC
gRPC is an open source Remote Procedure Call (RPC) framework that allows you to efficiently connect services to data centers while maintaining a high performance results. gRPC uses protocol buffers as both its Interface Definition Language (IDL) and as its underlying message interchange format.
It makes it easier to create a distributed applications and services system with gRPC, as your client applications can directly call methods on the server application that resides on a different machine as they were local objects. Most of the RPCs, including gRPC, are based on the solution to define a service and to specify the methods that can be called remotely with their parameters and return types. On the server side (in this case, SecureAuth, the server implements those interfaces and runs a gRPC server that handles the requests coming from client applications.
Prerequisites
Istio Authorizer is installed
Deploy Sample Service
Deploy a sample grpc service using the following command:
kubectl apply -f https://raw.githubusercontent.com/cloudentity/acp-on-k8s/master/examples/fortune-teller/deployment.yaml
Connect Service
There are two ways to connect Istio API groups to SecureAuth services: starting from the gateway to be connected or starting from the service that you want to connect.
From Gateway
From the list of available gateways, select your newly-created Istio gateway and go to its APIs tab.
A list of imported API groups opens.
From the list of API groups available, select an API group and, from its drop-down menu, pick a service to which you'd like to connect the API group.
Note
You can connect the API group to an existing service or a new one you create, both options available from the same service drop-down menu.
From Service
Select APIs from the left sidebar and go to the AUTHORIZATION tab.
Pick a service that you want to connect and select ADD GATEWAY API for the selected service.
In the Connect Istio API Group popup window, select an API gateway and an API group to be connected. Click CONNECT to proceed.
Result:In the APIS tab of the Gateway Management view, you can see specific API groups integrated to services.
Apply Sample Policy
Select APIs from the left sidebar and go to the AUTHORIZATION tab.
Select a service protected by Istio and any API with authorization status None.
In the Edit API popup window, select Policy from the dropdown list and click Update to proceed.
Result:You have successfully assigned a policy to your API.
Test Enforcement
To test your deployed and protected service, change the variables and execute the command:
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.9/samples/sleep/sleep.yaml export SLEEP_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name}) kubectl exec -it $SLEEP_POD -c sleep curl {YOUR_SERVICE_URL}/{ENDPOINT}
Example:
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.9/samples/sleep/sleep.yaml export SLEEP_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name}) kubectl exec -it $SLEEP_POD -c sleep curl http://httpbin:80/deny
Note
grpcurl
is using reflection to be able to call gRPC service endpoints. By default the authorizer blocks calls to unknown APIs (APIs that were not discovered). To be able to use the grpcurl
run the authorizer with the enforcement.allow_unkown
value set to true
in your Istio Authorizer installation values.yaml
file. Use this for testing purposes only.
Configure Istio Authorizer
Istio Authorizer configuration changes are done by adjusting the Istio Authorizer's values.yaml file and upgrading your Helm Chart release. To know which settings are available for your authorizer, see the Configuration reference section.
Commonly used properties like, for example, issuerURL
, client credentials, and more are easily available top-level properties.
The following snippet shows a simple configuration with some of the popular options tuned.
parseBody: enabled: true discovery: enabled: true namespaces: - default extraConfig: acp: reload_interval: 1m0s # reload interval enforcement: allow_unkown: false
Values Case
Note that top-level settings are provided using camelCase and settings in the extraConfig
value are provided with underscores.
If you want to change more advanced settings of the authorizer, you can use the extraConfig
value. Properties in the extraConfig
value correspond to the raw configuration passed to the istio-authorizer binary.
Configuration Order
Properties defined through top-level properties take precedence over configuration provided in the extraConfig
value.
You can apply your changes to the configuration using the helm upgrade release-name chart-name -f ValuesYamlFile
. See example below:
helm upgrade istio-authorizer acp/istio-authorizer \ --values ./values/istio-authorizer.yaml \ --namespace acp-istio-authorizer \ --timeout 5m \ --wait
Tip
If you need help with upgrading Helm Charts, use the helm upgrade --help
command in your terminal.
Configuration Reference
# acp acp: reload_interval: 1m0s # reload interval reload_timeout: 30s # reload configuration timeout issuer_url: https://localhost:8443/sample/system # issuer url client_id: bqesdrc4m4co2s81mpu0 # client id client_secret: LH6mAb6PNljvjYMIF-A5RP2bElA5a5bnQah8sG0fsLA # client secret tenant_id: "" # tenant id server_id: "" # server id # http client http_client: timeout: 10s # http client timeout retry_wait_min: 0s # minimum time to wait between retries retry_wait_max: 0s # maximum time to wait between retries retry_max: 0 # maximum number of retries root_ca: "" # root ca that this client should trust (defaults to system root ca) insecure_skip_verify: false # disable cert verification disable_follow_redirects: false # disable follow redirects disable_retry: true # disable retry # metrics metrics: enabled: false # enable metrics endpoint port: 9000 # metrics endpoint port # analytics analytics: enabled: true # when enabled, events are sent to audit log # event format event_format: include_policy_output: false # when enabled, policy evaluation output is sent to audit log # sampling sampling: probability: 1 # Probability of an event to be published (0.0-1.0) batch_inverval: 1s # Max duration to wait for a batch to publish batch_limit: 100 # Max number of events in a batch limit: 5 # Max number of batches per second to be published timeout: 5s # Timeout for a single batch to send workers: 8 # Number of sending workers # cache cache: ttl: 10s # ttl max_size: 100 # max size # logging config logging: level: info # log level severity # token echange config token_exchange: enabled: false # enable token exchange # cache cache: ttl: 1m0s # ttl max_size: 1000 # max size # inject config (supported only for istio authorizer) inject: mode: "" # Defines what token should be sent to the target service when token is exchanged # headers config headers: exchanged_token: "" # Defines the name of the header that contains an exchanged token. original_token: "" # Defines the name of the header that contains an original token. strip_bearer: false # Allows to strip the bearer prefix in headers # enforcement config enforcement: allow_unknown: false # allow requests with no matching rule # discovery config discovery: enabled: true # when true, API discovery is enabled timeout: 10s # discovery process timeout interval: 30s # how often discovery is performed disable_proxy: false # disable proxy in discovery open_api_endpoint: "" # if your service is self-hosting an OpenAPI endpoint, you need to provide the path to this endpoint to enable Istio authorizer to call this endpoint and automatically discover your APIs # discover services in a given namespaces namespaces: - default grpc_reflection_calls: false # allow grpc reflection calls # http server http_server: port: 9002 # http port dangerous_disable_tls: true # diables TLS # certificate configuration certificate: password: "" # key passphrase cert_path: "" # path to the certificate PEM file key_path: "" # path to the key PEM file cert: "" # base64 encoded cert PEM key: "" # base64 encoded key PEM generated_key_type: "" # type for generated key if cert and key are not provided (rsa or ecda) client_auth_type: 0 # client auth type # grpc server grpc_server: port: 9001 # gRPC port # target service config target_service: # inject config inject: # headers config headers: custom_data_prefix: x-output- # custom data header name prefix used to propage global variables defined in a policy auth_ctx: x-auth-ctx # header name containing base64 encoded authentication context json object trust_domain: cluster.local kubeconfig: "" # absolute path to the kubeconfig file
Import Istio Authorizer Configuration Declaratively
Declarative Configuration Import for SecureAuth platform
To learn about declarative configuration import for SecureAuth configuration, see the Declarative configuration import for SecureAuth article.
Prerequisites
All prerequisites for Declarative configuration import for SecureAuth are met.
Istio version 1.9 or subsequent is installed and up and running.
Istio Authorizer is added as an extension provider for your Istio Service Mesh.
For instructions on how to define Istio Authorizer as an external authorization provider for Istio, see Installing Istio Authorizer guide.
Service of your choice deployed in your Kubernetes cluster
You need to deploy a service in order to be able to import APIs to your SecureAuth instance and apply access control to protect them.
Tip
If you do not have a deployment-ready service at hand, you can deploy, for example, a httpbin service which can be found in the acp-on-k8s GitHub repository . For more instructions regarding protecting HTTP services, see Deploying and protecting HTTP services.
Import Istio Authorizer
Edit your
acp-cd
Helm Chart'svalues.yaml
file and provide your configuration definition for Istio Authorizer and its client application.Tip
To learn how to import authorizers to SecureAuth, see Authorizers section of the Declarative configuration import for SecureAuth article.
data: gateways: - tenant_id: {tid} authorization_server_id: {aid} id: sample-istio-authorizer client_id: istio-authorizer-client client_secret: cf53caab-fb55-40a7-8c63-620ce11102e2 name: Sample Istio Authorizer description: Sample imported Istio Authorizer type: istio create_and_bind_services_automatically: true clients: - tenant_id: {tid} authorization_server_id: system client_id: istio-authorizer-client client_secret: cf53caab-fb55-40a7-8c63-620ce11102e2 token_endpoint_auth_method: client_secret_basic system: false trusted: false client_name: Istio Authorizer client application_type: web grant_types: - client_credentials response_types: - code scopes: - introspect_tokens - push_gateway_requests - read_gateway_configuration - write_gateway_configuration
Upgrade your Helm Chart.
Result: Your Istio Authorizer is imported to SecureAuth and a client application for the authorizer is created in the System workspace of your tenant.
Install Istio Authorizer in k8s
Install Istio Authorizer using its Helm Chart and adjust its configuration to your needs. You do not need to manually create your authorizer in SecureAuth's UI as you had already imported the authorizer in the previous step of this guide.
Import API Groups and Services
To learn how to import API groups and services and to know what are the possible scenarios that differ between your authorizer configuration for service discovery, get familiar with the API groups and services section of the Declarative configuration import for SecureAuth article.
For Istio Authorizer, API group identifier and service identifiers are Base64 URL encoded SPIFFE IDs which are built being based on the service account of the protected pod. To avoid using long and unreadable API groups identifiers, SecureAuth supports Base64 encoding of SPIFFE IDs.
You can define your API group identifier using the encSpiffeID
function as shown below:
gateway_api_groups: - id: "{{ encSpiffeID \"spiffe://{domain}/ns/{namespace}/sa/{service_account}\" }}"
You can define your service identifier using the encServiceID
function as shown below:
services: - id: "{{ encServiceID \"spiffe://{domain}/ns/{namespace}/sa/{service_account}\" }}"
Prepare your
values.yaml
file.If you disabled service discovery for your authorizer, an import example can look like the following:
data: gateway_api_groups: - tenant_id: {tid} server_id: {aid} id: "{{ encSpiffeID \"spiffe://cluster.local/ns/default/sa/httpbin\" }}" service_id: httpbin-service gateway_id: sample-istio-authorizer name: default/httpbin_api_group apis: - method: GET path: /deny - method: GET path: /anything services: - id: httpbin-service tenant_id: {tid} authorization_server_id: {aid} gateway_id: sample-istio-authorizer name: default/httpbin_service apis: - method: GET path: /deny can_have_policy: true policy_id: block_api - method: GET path: /anything can_have_policy: true policy_id: block_api
You can see that in the example above, the service identifier is not encoded. When you encode your API group identifier, you do not need to encode the service ID. You can encode it if you wish, but then, in the
service_id
parameter of thegateway_api_groups
object, you need to provide the encoded service identifier as well.If the service discovery setting is enabled for your authorizer, an import example can look like the following:
data: services: - id: "{{ encServiceID \"spiffe://cluster.local/ns/default/sa/httpbin\" }}" tenant_id: {tid} authorization_server_id: {aid} gateway_id: sample-istio-authorizer name: default/httpbin_service apis: - method: GET path: /deny can_have_policy: true policy_id: block_api - method: GET path: /anything can_have_policy: true policy_id: block_api
You can see that to avoid long and unreadable service ID, it is encoded using the
{{ encServiceID }}
function.Policies Assignment
The above example includes policies already assigned to your APIs. You can assign policies when importing services as long as they already exist within SecureAuth or are imported as a part of the same
values.yaml
file.Upgrade your Helm Chart.
Result:Your API groups and services are imported to SecureAuth and visible in the APIs view.
If you imported the definition for services with policies assigned, your APIs are already protected. If not, you can assign them manually.
Troubleshooting
RBAC: Access Denied
Example:
$ curl -v http://httpbin.default:80/get ... < HTTP/1.1 403 Forbidden RBAC: access denied
If your calls to protected APIs always result with the RBAC: access denied
response, verify your Istio configuration for Istio authorization policies and Istio configuration (configmap). Such response may be a result of Istio changing the authorization policy from, for example, ALLOW
or CUSTOM
to DENY
:
│ 2022-10-07T13:49:40.972592Z error authorization Processed authorization policy: failed to parse CUSTOM action, will generate a deny all config: available providers are [] but found "acp-authorizer" │
In case of the SecureAuth Istio Authorizer, such situation may happen if you duplicate the extensionProviders
configuration in your Istio configmap when adding the acp-authorizer
extension provider.
Incorrect Configmap Example:
apiVersion: v1 data: mesh: |- extensionProviders: - name: "acp-authorizer" envoyExtAuthzGrpc: service: "istio-authorizer.acp-istio-authorizer.svc.cluster.local" port: 9001 accessLogFile: /dev/stdout defaultConfig: discoveryAddress: istiod.istio-system.svc:15012 proxyMetadata: {} tracing: zipkin: address: zipkin.istio-system:9411 enablePrometheusMerge: true extensionProviders: - envoyOtelAls: port: 4317 service: opentelemetry-collector.istio-system.svc.cluster.local name: otel [...]
In the example above, notice how the extensionProviders
entry is added twice to the Istio configuration. Because of this, Istio treats your configuration for the acp-authorizer
as invalid and switches the following Istio authorization policy for the SecureAuth Istio Authorizer from the CUSTOM
action to the DENY_ALL
action:
Spec: Action: CUSTOM Provider: Name: acp-authorizer Rules: Events: <none>
Tip
This authorization policy is added for you automatically to Istio as part of the SecureAuth Istio Authorizer Helm Chart installation process.
Correct Configmap Example:
apiVersion: v1 data: mesh: |- accessLogFile: /dev/stdout defaultConfig: discoveryAddress: istiod.istio-system.svc:15012 proxyMetadata: {} tracing: zipkin: address: zipkin.istio-system:9411 enablePrometheusMerge: true extensionProviders: - envoyOtelAls: port: 4317 service: opentelemetry-collector.istio-system.svc.cluster.local name: otel - name: "acp-authorizer" envoyExtAuthzGrpc: service: "istio-authorizer.acp-istio-authorizer.svc.cluster.local" port: 9001 [...]