mutual Transport Layer Security (mTLS)

mTLS provides encryption between sidecar-injected, istio-enabled services. By default, traffic among our three services of customer, preference, and recommendation is in “clear text” as they just use HTTP. This means that another team, with access to your cluster, could deploy their own service and attempt to sniff the traffic flowing through the system. To make that point, open up two command shells where one is using tcpdump to sniff traffic while the other is performing a curl command.

Shell 1:

PREFPOD=$(oc get pod -n tutorial -l app=preference -o 
'jsonpath={.items[0].metadata.name}')

oc exec -it $PREFPOD -n tutorial -c istio-proxy /bin/bash

sudo tcpdump -A -s 0 
'tcp port 8080 and (((ip[2:2]-((ip[0]&0xf)<<2))-
                                  ((tcp[12]&0xf0)>>2))!= 0)'

Shell 2:

PREFPOD=$(oc get pod -n tutorial -l app=preference -o 
'jsonpath={.items[0].metadata.name}')

oc exec -it $PREFPOD -n tutorial -c preference /bin/bash

curl recommendation:8080

The results for Shell 1 should look similar to the following:

..:...:.HTTP/1.1 200 OK
content-length: 47
x-envoy-upstream-service-time: 0
date: Mon, 24 Dec 2018 17:16:13 GMT
server: envoy

recommendation v1 from '66b7c9779c-75fpl': 345

And the results for Shell 2 will be:

recommendation v1 from '66b7c9779c-75fpl': 345

The curl command works and the tcpdump command outputs the results in clear text as seen in Figure 7-1.

Figure 7-1. Three shells before mTLS policy

Enabling mTLS in Istio uses the combination of Policy and DestinationRule objects. The Policy declaration is as follows:

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "default"
  namespace: "tutorial"
spec:
  peers:
  - mtls: {}

This applies to all services in the tutorial namespace. You can also optionally set a “mode” of PERMISSIVE versus STRICT. which allows for both mTLS and non-mTLS traffic, useful for scenarios where the sidecar, istio-proxy, has not yet been applied to all of your services. See the documentation on mTLS Migration for an example of how to have both legacy no-sidecar services as well as services that have the sidecar applied.

For now, our example services of customer, preference, and recommendation all have the sidecar injected, so we can apply mTLS across the whole of the tutorial namespace.

In a third shell, apply the policy manifest that was provided in the Istio Tutorial when you git cloned the repository.

Shell 3:

oc apply -n tutorial -f istiofiles/authentication-enable-tls.yml

Now, apply the DestinationRule that enables mTLS amongst the services in the tutorial namespace

apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "default"
  namespace: "tutorial"
spec:
  host: "*.tutorial.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

Shell 3:

oc apply -n tutorial -f istiofiles/destination-rule-tls.yml

And in Shell 2 run your curl command again, with successful results as seen in Figure 7-2:

Figure 7-2. Three shells after mTLS DestinationRule

You should notice that the tcpdump shell is no longer providing clear text and your curl command executes successfully. You can also use the istioctl tool to verify if mTLS is enabled:

istioctl authn tls-check | grep tutorial

Now it is time to test from the external world’s perspective. In Shell 2, exit from the preference container back to your host OS. Then curl the customer endpoint, which results in “Empty reply from server”:

curl customer-tutorial.$(minishift ip).nip.io
curl: (52) Empty reply from server

This particular external URL was generated via an OpenShift Route and minishift leverages a special service called nip.io for DNS resolution. Now that you have enabled mTLS, you need to leverage a gateway to achieve end-to-end encryption. Istio has its own ingress gateway, aptly named Istio Gateway, a solution that exposes a URL external to the cluster and supports Istio features such as monitoring, traffic management, and policy.

To set up the Istio Gateway for the customer service, create the Gateway and its supporting VirtualService objects

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: customer-gateway
  namespace: tutorial
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: customer
  namespace: tutorial
spec:
  hosts:
  - "*"
  gateways:
  - customer-gateway
  http:
  - match:
    - uri:
        exact: /
    route:
    - destination:
        host: customer
        port:
          number: 8080

And you can apply these manifests:

oc apply -f istiofiles/gateway-customer.yml

On minishift or minikube, the Istio Gateway service exposes a NodePort to make it visible outside the cluster. This port is available via minishift or minikube’s IP address:

INGRESS_PORT=$(oc -n istio-system get service
                                      istio-ingressgateway 
 -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')

GATEWAY_URL=$(minishift ip):$INGRESS_PORT

curl http://${GATEWAY_URL}/

Issuing curl commands via the gateway works as expected as seen in Figure 7-3.

Figure 7-3. Three shells with Istio customer gateway

Clean up:

oc delete -n tutorial -f istiofiles/gateway-customer.yml
oc delete -n tutorial -f istiofiles/destination-rule-tls.yml
oc delete -n tutorial -f istiofiles/authentication-
                                    enable-tls.yml

And return to the original invocation mechanism with the OpenShift Route:

oc expose service customer

curl customer-tutorial.$(minishift ip).nip.io

Access Control with Mixer Policy

Istio’s Mixer Policy service allows you to construct a series of rules that ensure the various microservices that make up your application follow an approved invocation path. In the case of the example services, it is expected that customer calls preference and then preference calls recommendation, in that specific order. Therefore there are some alternative paths that are specifically denied:

• customer is not allowed to call recommendation

• preference is not allowed to call customer

• recommendation is not allowed to call customer

• recommendation is not allowed to call preference

Istio has some additional objects or Kinds involved in this sort of access control: denier, checknothing, and rule.

Before you apply the denier and rules, first explore the potential hazard associated with customer, preference, and recommendation as it may not be apparent at first glance.

First, grab the recommendation pod’s name/id and exec into its business logic container:

RECPOD=$(oc get pod -n tutorial -l app=recommendation -o 
'jsonpath={.items[0].metadata.name}')

oc exec -it $RECPOD -n tutorial -c recommendation /bin/bash

Next, curl the customer service and see that it succeeds, because all the services are visible to one another by default (as expected in a Kubernetes/OpenShift cluster):

curl customer:8080
customer => preference => recommendation v2 from
                                         '7cbd9f9c79': 23

Also curl the preference service:

curl preference:8080
preference => recommendation v1 from '66b7c9779c': 152

And for the sake of argument, the business/organization has determined that only the customer → preference → recommendation invocation path is the correct one. You can close down the others with the denier and rules. The listing of these rules is a bit long but make sure to check the manifest before applying it.

The syntax for the rules are straightforward, based on source and destination labels. You can check your pod labels:

oc get pods -n tutorial --show-labels

This has an output similar to the following (output truncated here for formatting reasons):

NAME                         READY STATUS   LABELS
customer-6564ff969f          2/2   Running  app=customer,
                                                version=v1
preference-v1-5485dc6f49     2/2   Running  app=preference,
                                                version=v1
recommendation-v1-66b7c9779c 2/2   Running  app=recommendation,
                                                version=v1
recommendation-v2-7cbd9f9c79 2/2   Running  app=recommendation,
                                                version=v2

And you can apply these rules with the following command:

oc -n tutorial apply -f 
istiofiles/acl-deny-except-
           customer2preference2recommendation.yml

Now when you exec into recommendation and attempt a curl of preference:

curl preference:8080
PERMISSION_DENIED:do-not-pass-go.denier.tutorial:
Customer -> Preference -> Recommendation ONLY

Make sure to double-check that your normal invocation path continues to execute as expected:

curl customer-tutorial.$(minishift ip).nip.io
customer => preference => recommendation v2 from
                                         '7cbd9f9c79': 238

Use the describe verb for the kubectl or oc tool to see the rules you have in place:

oc get rules
NAME                              AGE
no-customer-to-recommendation     3m
no-preference-to-customer         3m
no-recommendation-to-customer     3m
no-recommendation-to-preference   3m

oc describe rule no-preference-to-customer
...
Spec:
  Actions:
    Handler:  do-not-pass-go.denier
    Instances:
      just-stop.checknothing
  Match:  source.labels["app"]=="preference" &&
   destination.labels["app"] == "customer"
Events:   <none>

You can remove these rules to return to original state:

oc delete rules --all -n tutorial

Istio’s Mixer also supports a whitelist and blacklist mechanism involving the listchecker and listentry objects. If you are interested in that capability check out the Istio Tutorial and/or the Istio Documentation.

Role-Based Access Control (RBAC)

Istio includes a Role-Based Access Control (RBAC) authorization feature that can be used to further constrain which services (e.g., customer, preference, recommendation) are accessible by particular users.

Make sure there are no pre-existing DestinationRule, VirtualService, Gateway, or Policy objects:

oc get destinationrule -n tutorial
oc get virtualservice -n tutorial
oc get gateway -n tutorial
oc get policy -n tutorial

Setting up Istio’s RBAC support is simple enough using the RbacConfig object:

apiVersion: "rbac.istio.io/v1alpha1"
kind: RbacConfig
metadata:
  name: default
spec:
  mode: 'ON_WITH_INCLUSION'
  inclusion:
    namespaces: ["tutorial"]

Where mode can be:

• OFF: Istio authorization is disabled.

• ON: Istio authorization is enabled for all services in the mesh.

• ON_WITH_INCLUSION: Enabled only for services and namespaces specified in the inclusion field.

• ON_WITH_EXCLUSION: Enabled for all services in the mesh except the services and namespaces specified in the exclusion field.

That is the line required to apply/create the RbacConfig object:

oc create -f istiofiles/authorization-enable-rbac.yml
                        -n tutorial

Now if you curl your customer endpoint, you will receive “RBAC: access denied”:

curl customer-tutorial.$(minishift ip).nip.io
RBAC: access denied

Istio’s RBAC uses a deny-by-default strategy, meaning that nothing is permitted until you explicitly define an access-control policy to grant access to any service. To reopen the customer endpoint to end-user traffic, create a ServiceRole and a ServiceRoleBinding:

apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRole
metadata:
  name: service-viewer
  namespace: tutorial
spec:
  rules:
  - services: ["*"]
    methods: ["GET"]
    constraints:
    - key: "destination.labels[app]"
      values: ["customer", "recommendation", "preference"]
---
apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRoleBinding
metadata:
  name: bind-service-viewer
  namespace: tutorial
spec:
  subjects:
  - user: "*"
  roleRef:
    kind: ServiceRole
    name: "service-viewer"

Apply it:

oc -n tutorial apply -f 
istiofiles/namespace-rbac-policy.yml

Now try your curl command again:

curl customer-tutorial.$(minishift ip).nip.io
customer => preference => recommendation v1 from
                                            '66b7c9779c': 36

Istio’s ServiceRole object allows you to specify which services are protected either by naming them directly or using the destination.labels[app] constraint demonstrated in the example. You can also specify which methods are allowed such as GET versus POST. The ServiceRoleBinding object allows you to specify which users are permitted. In the current case, user: “*” with no additional properties means that any user is allowed to access these services.

The concept of users and user management has always been unique per organization, often unique per application. In the case of a Kubernetes cluster, your cluster administrator will likely have a preferred strategy for user authentication and authorization.

Istio has support for user authentication and authorization via JWT (JSON Web Token). From the “Introduction to JSON Web Tokens” page: “JSON Web Token (JWT) is an open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object.” To leverage JWT, you will need a JWT issuer like auth0.com, or perhaps a local service based on the open source software project called Keycloak, which supports OpenID Connect, OAuth 2.0, and SAML 2.0.

Setup and configuration of a JWT Issuer is beyond the scope of this book, but more information can be found at the Istio Tutorial.

In addition, Istio Security has more information about Istio’s security capabilities.

Conclusion

You have now taken a relatively quick tour through some of the capabilities of Istio service mesh. You saw how this service mesh can solve distributed systems problems in cloud native environments, and how Istio concepts like observability, resiliency, and chaos injection can be immediately beneficial to your current application.

Moreover, Istio has capabilities beyond those we discussed in this book. If you’re interested, we suggest that you explore the following topics more deeply:

• Policy enforcement

• Mesh expansion

• Hybrid deployments

• Phasing in Istio into an existing environment

• Gateway/Advanced ingress

Istio is also evolving at a rapid rate. To keep up with the latest developments, we suggest that you keep an eye on the upstream community project page as well as Red Hat’s evolving Istio Tutorial.