Networking overview

In-cluster networking

Chef 360 Platform’s embedded cluster built-in network provider is Calico. Calico is a layer 3 container networking solution that routes packets to pods.

For additional information on in-cluster networking, see Container Network Interface (CNI).

Note

Chef 360 Platform doesn’t support IP in IP routing with Calico.

Controller-worker communication

One goal of k0s is to allow for the deployment of an isolated control plane which may prevent the establishment of an IP route between controller nodes and the pod network. To enable this communication path (which is mandated by conformance tests), k0s deploys the Konnectivity service to proxy traffic from the API server (control plane) into the worker nodes. This ensures that Chef 360 Platform can always fulfill all the Kubernetes API functionalities, but still operate the control plane in total isolation from the workers.

Note

To allow Konnectivity agents running on the worker nodes to establish a connection, configure your firewalls for outbound access with port 8132. Moreover, configure your firewalls for outbound access on port 6443 to access Kube-API from the worker nodes.

Required ports and protocols

Protocol	Port	Service	Direction	Notes
TCP	2380	etcd peers	controller <-> controller
TCP	6443	kube-apiserver	worker, CLI => controller	Authenticated Kube API using Kube TLS client certs, ServiceAccount tokens with RBAC
TCP	179	kube-router	worker <-> worker	BGP routing sessions between peers
UDP	4789	Calico	worker <-> worker	Calico VXLAN overlay
TCP	10250	kubelet	controller, worker => host `*`	Authenticated kubelet API for the controller node `kube-apiserver` (and `heapster`/`metrics-server` addons) using TLS client certs
TCP	9443	k0s-api	controller <-> controller	k0s controller join API, TLS with token auth
TCP	8132	konnectivity	worker <-> controller	Konnectivity is used as “reverse” tunnel between kube-apiserver and worker kubelets
TCP	112	keepalived	controller <-> controller	Only required for control plane load balancing VRRP instances for IP address 224.0.0.18. 224.0.0.18 is a multicast IP address defined in [RFC 3768].

You also need to enable all traffic to and from the [podCIDR and serviceCIDR] subnets on nodes with a worker role.

Note

Required ports and protocols cover the Chef 360 Platform k0s runtime, but doesn’t include ports for applications.

iptables

iptables can work in two distinct modes, legacy and nftables. k0s auto detects the mode and prefers nftables.

To check which mode k0s is configured with, run:

ls -lah /var/lib/k0s/bin/

The iptables link target reveals the mode which k0s selected.

k0s has the same logic as other Kubernetes components. Use the following commands to make sure all components use the same mode:

kube-proxy:
- nsenter -t $(pidof kube-proxy) -m iptables -V
kube-router:
- nsenter -t $(pidof kube-router) -m /sbin/iptables -V
calico:
- nsenter -t $(pidof -s calico-node) -m iptables -V

Different iptables versions have known incompatibility issues.

k0s ships a version of iptables that’s tested to interoperate with all other Kubernetes components it ships with. However, if you have other tooling (firewalls etc.) on your hosts that uses iptables and the host iptables version is different from the version k0s (and other Kubernetes components) ships with, it may cause networking issues.

Firewalld

If you are using firewalld on your hosts, ensure it’s configured to use the same FirewallBackend as Chef 360 Platform and other Kubernetes components use. Otherwise, networking will be broken in various ways.

Here’s an example configuration for a tested working networking setup:

[root@rhel-test ~]# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0
  sources: 10.244.0.0/16 10.96.0.0/12
  services: cockpit dhcpv6-client ssh
  ports: 80/tcp 6443/tcp 8132/tcp 10250/tcp 179/tcp 179/udp
  protocols:
  forward: no
  masquerade: yes
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules: