Architecting and Operating OpenShift Cluster

Cluster

The set of machines in the cluster. i.e. the Masters and the Nodes.

Master

A controller of the OpenShift Container Platform cluster. Note that the master may not be a node in the cluster, and thus, may not have IP connectivity to the pods.

Node

Group of containers running on a node, managed by OpenShift Container Platform.

Service

Abstraction that presents a unified network interface that is backed by one or more pods.

Router

A web proxy that can map various URLs and paths into OpenShift Container Platform services to allow external traffic to travel into the cluster.

Node Address

The IP address of a node. This is assigned and managed by the owner of the network to which the node is attached. Must be reachable from any node in the cluster (master and client).

Pod Address

The IP address of a pod. These are assigned and managed by OpenShift Container Platform. By default they are assigned out of the 10.128.0.0/14 network. Only reachable from the client nodes.

Service Address

An IP address that represents the service, and is mapped to a pod address internally. These are assigned and managed by OpenShift Container Platform. By default they are assigned out of the 172.30.0.0/16 network. Only reachable from the client nodes.

The following diagram shows all of the pieces involved with external access:

In the default configuration, the cluster network is the 10.128.0.0/14 network, and node allocated /23 subnets (i.e., 10.128.0.0/23, 10.128.2.0/23, 10.128.4.0/23, and so on). This means that the cluster network has 512 subnets available to assign to nodes, and a given node is allocated 510 addresses that it can assign to the containers running on it. The size and address range of the cluster network are configurable, as is the host subnet size.

[root@master ~]# view /etc/origin/master/master-config.yaml
networkConfig:
  clusterNetworks:
  - cidr: 10.128.0.0/14
    hostSubnetLength: 9
  externalIPNetworkCIDRs:
  - 0.0.0.0/0
  networkPluginName: redhat/openshift-ovs-subnet
  serviceNetworkCIDR: 172.30.0.0/16

[root@master ~]# oc get hostsubnet
NAME                     HOST                     HOST IP         SUBNET          EGRESS CIDRS   EGRESS IPS
infra.myopenshift.com    infra.myopenshift.com    192.168.23.52   10.129.0.0/23   []             []
master.myopenshift.com   master.myopenshift.com   192.168.23.31   10.128.0.0/23   []             []
node.myopenshift.com     node.myopenshift.com     192.168.23.51   10.130.0.0/23   []             []
node2.myopenshift.com    node2.myopenshift.com    192.168.23.32   10.128.2.0/23   []             []
node3.myopenshift.com    node3.myopenshift.com    192.168.23.33   10.131.0.0/23   []             []
node4.myopenshift.com    node4.myopenshift.com    192.168.23.34   10.129.2.0/23   []             []

OpenShift SDN creates and configures three network devices:

br0

the OVS bridge device that pod containers will be attached to. OpenShift SDN also configures a set of non-subnet-specific flow rules on this bridge.

tun0

an OVS internal port (port 2 on br0). This gets assigned the cluster subnet gateway address, and is used for external network access. OpenShift SDN configures netfilter and routing rules to enable access from the cluster subnet to the external network via NAT.

vxlan_sys_4789

The OVS VXLAN device (port 1 on br0), which provides access to containers on remote nodes. Referred to as vxlan0 in the OVS rules.

For each Pod in the Node, the local OpenShift creates a vethXX interface and assign it to the OVS br0. The vxlan_sys_4789 of br0 is the interface that defines the VXLAN tunnels, or the overlay network, that enables the communication between local Pods with Pods in remote Nodes. This interface is known as vxlan0 interface inside the OVS and that is the name used in the OpenFlow entries. The tun0 interface gets the local cluster network subnet gateway address. This is the interface that provide NAT access from the cluster network subnet to the external network. In additional to the local cluster network subnet gateway address, on each Node the Kubernetes Service objects network is also pointed to the tun0 interface.

[root@node ~]# ifconfig tun0
tun0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.129.0.1  netmask 255.255.254.0  broadcast 10.129.1.255
        ...

[root@node ~]# ip route
default via 192.168.16.1 dev ens192
10.128.0.0/14 dev tun0 scope link                                     # This sends all pod traffic into OVS
169.254.0.0/16 dev ens192 scope link metric 1002                      # This is for Zeroconf
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1      # Docker's private IPs... used only by things directly configured by docker; not OpenShift
172.30.0.0/16 dev tun0                                                # To the Kubernetes Service objects network
192.168.16.0/20 dev ens192 proto kernel scope link src 192.168.23.31  # The physical interface on the local subnet
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 # Linux virtual network switch (use NAT mode) for hypervisor

The default OpenShift Router is one or more Router Pods running on Infrastructure Nodes (infra.myopenshift.com) and is deployed as a Deployment Config (deploymentconfig.apps.openshift.io/router). Sharing the Network Namespace enables these Router Pods to receive traffic over the host-network. By default, the OpenShift Router listens on TCP ports 80 (HTTP), 443 (HTTPS), and 1936 (HAProxy Stats). Once the traffic arrives to the Pod, it will match the corresponding Route object.

[root@master ~]# oc get all --selector='router=router' -n default -o wide
NAME                 READY     STATUS    RESTARTS   AGE       IP              NODE                    NOMINATED NODE
pod/router-1-qv5f7   1/1       Running   1          5d        192.168.23.52   infra.myopenshift.com   <none>

NAME                             DESIRED   CURRENT   READY     AGE       CONTAINERS   IMAGES                                                   SELECTOR
replicationcontroller/router-1   1         1         1         5d        router       registry.redhat.io/openshift3/ose-haproxy-router:v3.11   deployment=router-1,deploymentconfig=router,router=router

NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                   AGE       SELECTOR
service/router   ClusterIP   172.30.210.30   <none>        80/TCP,443/TCP,1936/TCP   5d        router=router

NAME                                        REVISION   DESIRED   CURRENT   TRIGGERED BY
deploymentconfig.apps.openshift.io/router   1          1         1         config

• Pod to pod in the same node

  eth0 (pod) \(\rightarrow\) vethA \(\rightarrow\) br0 \(\rightarrow\) vethB \(\rightarrow\) eth0 (pod)

• Pod to pod in the different nodes

  eth0 (pod) \(\rightarrow\) vethA \(\rightarrow\) br0 \(\rightarrow\) vxlan0 \(\rightarrow\) network \(\rightarrow\) vxlan0 \(\rightarrow\) br0 \(\rightarrow\) vethB \(\rightarrow\) eth0 (pod)

• Pod to external host

  eth0 (pod) \(\rightarrow\) vethA \(\rightarrow\) br0 \(\rightarrow\) tun0 \(\rightarrow\) \(\texttt{SNAT} \atop \texttt{(MASQUERADE)}\) \(\rightarrow\) eth0 (phy.) \(\rightarrow\) Host

oc login -u system -p admin
oc project hello

Configure the externalIPNetworkCIDRs parameter in the /etc/origin/master/master-config.yaml file as shown (default is 0.0.0.0/0):

networkConfig:
  externalIPNetworkCIDRs:
  - <ip_address>/<cidr>

Restart the OpenShift Container Platform master service to apply the changes.

master-restart api
master-restart controllers

In the server:

# Enable NFS service and allow iptable rules
systemctl enable nfs-server
systemctl enable rpcbind
systemctl enable nfs-lock
systemctl enable nfs-idmap

vi /etc/sysconfig/nfs
# ...
# LOCKD_TCPPORT=32803
# LOCKD_UDPPORT=32769
# MOUNTD_PORT=892
# STATD_PORT=662

grep -w -e 111 -e 2049 /etc/services

systemctl restart rpcbind
systemctl restart nfs-server
systemctl restart nfs-lock
systemctl restart nfs-idmap

rpcinfo -p | grep -E '(rquota|mount|nlock)'

iptables -I INPUT -i ens192 -p tcp -s 192.168.23.0/24 -m multiport --dport 111,2049,32803,32769,892,662 -j ACCEPT
iptables -I INPUT -i ens192 -p udp -s 192.168.23.0/24 -m multiport --dport 111,2049,32803,32769,892,662 -j ACCEPT

iptables-save

# Setup /etc/exports & enable anonymity access
mkdir -p /ose/public
setfacl -m u:nobody:rwx /ose/public
getfacl /ose/public
ls -lZd /ose/public
vi /etc/exports
# /ose/public 192.168.23.0/24(rw,all_squash,anonuid=99,anongid=99) *(ro)

# Exporting the share:
exportfs -r
# '-r' re-exports entries in /etc/exports and sync /var/lib/nfs/etab with /etc/exports.

# Restart the NfS services:
systemctl restart nfs-server

In the client:

mount -t nfs nfs.myopenshift.com:/ose/public /mnt
(hostname; date) >/mnt/test.txt

You must first create an object definition for the PVs (Persistent Volumes):

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs01 
spec:
  capacity:
    storage: 30Gi 
  accessModes:
  - ReadWriteOnce 
  nfs: 
    path: /ose/public
    server: nfs.myopenshift.com
  persistentVolumeReclaimPolicy: Retain 

Create the PV:

oc create -f ./nfs-pv.yaml
oc get pv

The next steps can be to create a PVC (persistent volume claims), to provide a convenient method for sharing a volume across a project:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-claim01
spec:
  accessModes:
    - ReadWriteOnce 
  resources:
    requests:
      storage: 1Gi

Create the PVC:

oc create -f ./nfs-pvc.yaml
oc get pvc
# NAME          STATUS    VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
# nfs-claim01   Bound     nfs01     30Gi       RWO                           11s

Add the volume to pod in management console, OpenShift will terminate old pods and create new pods (rolling out).

Install the module:

apt-get install python3 python3-pip
pip3 --proxy http://user:pass@192.168.19.19:8888 install psycopg2-binary

Copy & query the PostgreSQL: see scripts in web-conf.