Truefoundry Docs

Truefoundry uses Istio as the ingress controller. It, by default, provisions a single external load-balancer for one Kubernetes cluster. The load-balancer is provisioned automatically by the tfy-istio-ingress helm chart installed by TrueFoundry which creates a Kubernetes service of type LoadBalancer. This is fundamentally an ingress gateway which automatically brings up the load balancer in the respective cloud provider.

Istio is currently mandatory for the Truefoundry components to work. We will be adding support for other ingress controllers in the future.

You can find the configuration of the tfy-ingress-gateway in Deployments > Helm > tfy-istio-ingress (Make sure you are filtering for the desired cluster)

You can click on the three dots to understand the configuration.

If you want to modify any of your load-balancer settings, you will have to edit this configuration and deploy the helm chart. The load-balancer settings are configured using the annotations on the gateway object. The annotations vary based on the cloud provider and you can find the corresponding cloud specific documentation below.

Changing some of the settings might cause the load balancer to be recreated and you will have to remap your DNS. This can bring down the services temporarily - so be careful with the changes you make or consult with the Truefoundry team.

Getting the load balancer IP address

Using the below command you can get the load balancer IP address. You need to point this IP address to your domain in your DNS provider as an A record. For AWS - this will be a CNAME record.

kubectl get svc -n istio-system tfy-istio-ingress

Modifying your load balancer configuration

Below are the condiguration for the loadbalancers for the different cloud providers.

AWS
GCP
Azure
On-Prem

Below are the various load balancer types that are supported for AWS.

AWS Network Load Balancer

By default, the istio gateway creates a network load balancer for the ingress traffic. Functionality of the load balancer can be modified using the NLB annotations listed hereDefault annotations for the NLB are as follows:

"service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy": "ELBSecurityPolicy-TLS13-1-2-2021-06"
"service.beta.kubernetes.io/aws-load-balancer-name": "CLUSTER_NAME"
"service.beta.kubernetes.io/aws-load-balancer-type": "external"
"service.beta.kubernetes.io/aws-load-balancer-scheme": "internet-facing"
"service.beta.kubernetes.io/aws-load-balancer-ssl-ports": "https"
"service.beta.kubernetes.io/aws-load-balancer-alpn-policy": "HTTP2Preferred"
"service.beta.kubernetes.io/aws-load-balancer-backend-protocol": "tcp"
"service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags":
  cluster-name=CLUSTER_NAME,  truefoundry.com/managed=true,
  owner=Truefoundry, application=tfy-istio-ingress
"service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled": "true"

Making the load balancer internal

Switching from public to internal load balancer will cause the load balancer to be recreated and you will have to remap your DNS.

To make the load balancer internal, add the following annotation:

"service.beta.kubernetes.io/aws-load-balancer-scheme": "internal"

Restricting the access to the load balancer to a specific IP address

To restrict the access to the load balancer to a specific IP address, add the following annotation:

"service.beta.kubernetes.io/aws-load-balancer-source-ranges": "10.10.0.0/16","10.11.0.0/16"

Adding multiple certificates to the load balancer

To add multiple certificates to the load balancer, add the following annotation:

"service.beta.kubernetes.io/aws-load-balancer-ssl-cert": "arn:aws:acm:us-east-2:XXXXXXX:certificate/XXXXX-XXXXX-XXXXXX,arn:aws:acm:us-east-2:XXXXXXX:certificate/XXXXX-XXXXX-XXXXXX"

To add multiple domains to the load Balancer, just add it in the hosts section. You can also use * to match all subdomains.

hosts:
  - '*.tfy.example.com'
  - '*.ml.example.com'

AWS Application Load Balancer

To use an application load balancer for the ingress traffic, make the following changes in the tfy-istio-ingress helm chart values. Ensure you have replaced cluster name, load balancer name, certificate arn and ingress class name is alb. Functionality of the load balancer can be modified using the ALB annotations listed here

ALB doesn’t support SSH servers in the platform.

Replacing the load balancer from NLB to ALB will cause the load balancer to be recreated and you will have to remap your DNS.

  alb:
    ingress:
      enabled: true
      annotations:
        alb.ingress.kubernetes.io/tags: cluster-name=CLUSTER_NAME, name=LOAD_BALANCER_NAME
        alb.ingress.kubernetes.io/scheme: internet-facing
        alb.ingress.kubernetes.io/certificate-arn: >-
          CERTIFICATE_ARN
        alb.ingress.kubernetes.io/load-balancer-name: LOAD_BALANCER_NAME
      ingressClassName: alb
  gateway:
    service:
      type: NodePort
  tfyGateway:
    name: tfy-wildcard
    spec:
      servers:
        - port:
            name: http-tfy-wildcard
            number: 80
            protocol: HTTP
          hosts:
            - '*'
      selector:
        istio: tfy-istio-ingress

Making the load balancer internal

Switching from public to internal load balancer will cause the load balancer to be recreated and you will have to remap your DNS.

To make the load balancer internal, add the following annotation:

"alb.ingress.kubernetes.io/scheme": "internal"

Restricting the access to the load balancer to a specific IP address

To restrict the access to the load balancer to a specific IP address, add the following annotation:

"alb.ingress.kubernetes.io/source-ranges": "10.10.0.0/16","10.11.0.0/16"

Adding multiple certificates to the load balancer

To add multiple certificates to the load balancer, add the following annotation:

"alb.ingress.kubernetes.io/certificate-arn": "arn:aws:acm:us-east-2:XXXXXXX:certificate/XXXXX-XXXXX-XXXXXX,arn:aws:acm:us-east-2:XXXXXXX:certificate/XXXXX-XXXXX-XXXXXX"

To add multiple domains to the load Balancer, just add it in the hosts section. You can also use * to match all subdomains.

hosts:
  - '*.tfy.example.com'
  - '*.ml.example.com'

When running TrueFoundry on GCP, istio creates a network load balancer for the ingress traffic. Functionality of the load balancer can be modified using the network load balancer annotations listed hereBy default no annotations are added to the load balancer gateway in GCP.

Making the loadbalancer internal

YAML

gateway:
  annotations:
    networking.gke.io/load-balancer-type: "Internal"

To add multiple domains to the load Balancer, just add it in the hosts section. You can also use * to match all subdomains.

hosts:
  - '*.tfy.example.com'
  - '*.ml.example.com'

To restrict the access to the load balancer to a specific IP address, you need to modify the gateway.service spec

gateway:
  service:
    loadBalancerSourceRanges:
      - 10.20.0.0/16
      - 10.10.0.0/16

When running TrueFoundry on Azure, istio creates a load balancer for the ingress traffic. Functionality of the load balancer can be modified using the load balancer annotations listed hereBy default no annotations are added to the load balancer gateway in Azure.

Making the load balancer internal

YAML

gateway:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"

To add multiple domains to the load Balancer, just add it in the hosts section. You can also use * to match all subdomains.

hosts:
  - '*.tfy.example.com'
  - '*.ml.example.com'

To restrict the access to the load balancer to a specific IP address, you need to modify the gateway.service spec

gateway:
  service:
    loadBalancerSourceRanges:
      - 10.20.0.0/16
      - 10.10.0.0/16

When running TrueFoundry on On-Prem, istio creates a load balancer for the ingress traffic. There are two ways to run this mode.

Assign IP address to the load balancer

By default load balancer will be created without an external IP address in the case of generic cluster.
Run metalLB to assign an IP address to the load balancer.
Ensure that load balancer has an external IP address in its status. You can check that by running the following command:

kubectl get svc -n istio-system tfy-istio-ingress

Once the load balancer gets the IP address assign in its status, modify your DNS to point to the IP address.

Use existing load balancer

If you already have provisioned a load balancer outside of the k8s cluster, you can use istio gateway as a backend to route traffic.
Modify the gateway configuration to use nodeport service type.

  gateway:
    service:
      type: NodePort

Modify your external load balancer to point to all (or few) nodes at the port that comes up in the nodeport service. You can check that by running the following command:

kubectl get svc -n istio-system tfy-istio-ingress

If you want to terminate TLS at the istio layer you can use the port 443 as the backend port otherwise use the port 80.

To add multiple domains to the load Balancer, just add it in the hosts section. You can also use * to match all subdomains.

hosts:
  - '*.tfy.example.com'
  - '*.ml.example.com'

To restrict the access to the load balancer to a specific IP address, you need to modify the gateway.service spec

gateway:
  service:
    loadBalancerSourceRanges:
      - 10.20.0.0/16
      - 10.10.0.0/16

Adding base domain to the TrueFoundry platform

If you want to add the base domain(s) to the TrueFoundry platform to expose your services via your domain, you can do so by editing the Cluster Page from the Platform section and adding the domain in the Base Domain URLs field. You can also add multiple domains.

Adding base domain for workbench config

If you want to use notebooks and SSH servers, you need to add the domain URL for them in the cluster’s page by enabling workbench config toggle.

If you have a wildcard domain, you can add the base domain as notebook.tfy.example.com and ssh.tfy.example.com with port 80 if *.tfy.example.com is your base domain. Feel free to customize the domain names as per your needs.

Adding base domain for spark

If you want to expose your spark job, you need to add the domain URL in the cluster’s page by enabling spark config toggle.

Deploy multiple load balancers

Each installation of tfy-istio-ingress creates a load balancer. If you want to deploy multiple multiple load-balancers, for e.g. one internal and one external, you can clone the current tfy-istio-ingress application in the same namespace istio-system, change the tfyGateway.Name to something else other then default tfy-wildcard and update the tfyGateway.spec.Selector with the new name of the application. For e.g. if you clone the tfy-istio-ingress a new application with the name tfy-istio-ingress-1 will be created , update the tfyGateway.Name to a new name and the tfyGateway.spec.Selector to

tfyGateway:
  spec:
    selector:
      app: "tfy-istio-ingress-1"

Once the ingress is installed, it will automatically create another loadbalancer whose IP you can get using

kubectl get svc -n istio-system

Add authentication to all services behind a load balancer

We can configure Istio to apply authentication at a gateway level. This will work only if you are accessing the service using the DNS provided in Istio and not access the service directly from within the cluster. This process is a bit complicated, and you should only do this if you really want to enable authentication at an istio gateway level.

Istio will validate if the JWT is valid. If not valid, it will return an Unauthorized Error.

Create a RequestAuthentication resource to ensure that the JWT issuer and Audience are correct.

Authentication will be only done if there is an Authorization header. This is pass-through if no Authorization header is present in the Request or it gets an empty string after removing the prefix.

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: tfy-oauth2
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: tfy-istio-ingress
  jwtRules:
  - issuer: "truefoundry.com"
    fromHeaders:
    - name: Authorization
      prefix: "Bearer "
    audiences:
      - <tenant_name_in_truefoundry>
    jwksUri: https://login.truefoundry.com/.well-known/jwks.json
    forwardOriginalToken: true

Create an AuthorizationPolicy that will reject any requests with an empty JWT.

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: tfy-oauth2
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: tfy-istio-ingress
  action: DENY
  rules:
  - from:
    - source:
        notRequestPrincipals: ["*"]
    to:
      - operation:
          ports:
            - "443" 

You can read the Istio docs: https://istio.io/latest/docs/reference/config/security/request_authentication/ for further customization or making it work with your own IdP.

Platform

​Getting the load balancer IP address

​Modifying your load balancer configuration

​Making the load balancer internal

​Restricting the access to the load balancer to a specific IP address

​Adding multiple certificates to the load balancer

​Making the load balancer internal

​Restricting the access to the load balancer to a specific IP address

​Adding multiple certificates to the load balancer

​Making the loadbalancer internal

​Making the load balancer internal

​Adding base domain to the TrueFoundry platform

​Deploy multiple load balancers

​Add authentication to all services behind a load balancer

​Istio will validate if the JWT is valid. If not valid, it will return an Unauthorized Error.

Getting the load balancer IP address

Modifying your load balancer configuration

Making the load balancer internal

Restricting the access to the load balancer to a specific IP address

Adding multiple certificates to the load balancer

Making the load balancer internal

Restricting the access to the load balancer to a specific IP address

Adding multiple certificates to the load balancer

Making the loadbalancer internal

Making the load balancer internal

Adding base domain to the TrueFoundry platform

Deploy multiple load balancers

Add authentication to all services behind a load balancer

Istio will validate if the JWT is valid. If not valid, it will return an Unauthorized Error.