Container Access to External Services in CCE Clusters
In VPC route networking mode, IP address network segment of a pod is usually different from that of a node.
Under the default cluster configuration, when pods in non-HostNetwork mode access IPs within private IP address ranges (10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16), the source IP of the packet is the pod IP. This results in network unavailability when accessing extra-cluster virtual machines or services.
For these cases, we need to configure ip-masq-agent to meet network access requirements.
Key terms
- NAT (Network Address Translation) It is a method of remapping one IP address to another by modifying the source and/or destination address information in the IP header. It is typically handled by devices performing IP route.
- Masquerade A form of NAT, typically used for many-to-one address translation, where multiple source IP addresses are hidden behind a single address, usually the device performing IP route. In Kubernetes, this is the node's IP address.
- CIDR (Classless Inter-Domain Routing) Based on variable-length subnet masks, prefixes of any length can be specified. CIDR introduces a new IP address notation, now commonly called CIDR notation, where a suffix is added after the address or route prefix to indicate the number of digits of prefix (e.g., 192.168.2.0/24).
Introduction to principle
ip-masq-agent is deployed as a daemonset, launching an agent program on each node. By configuring iptables rules, the agent program ?can select to use the IP address of the container itself or the IP address masqueraded as the node as the source address of the egress traffic when sending the container traffic to the target outside the range of the cluster node IP and the cluster IP. The agent program reloads its configuration from /etc/config/ip-masq-agent every 60 seconds to update iptables rules.
The iptables rules added by the agent program introduce a custom chain named IP-MASQ-AGENT to the POSTROUTING chain, determining whether outgoing traffic should be masqueraded by matching rules in the IP-MASQ-AGENT chain.
K8S Cluster CCE deploys the ip-masq-agent component by default. By default, the agent configuration treats the three private IP ranges specified by RFC 1918 as non-masqueraded CIDRs. These ranges are 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16. By default, the agent program also treats link-local address (169.254.0.0/16) as non-masqueraded CIDR.
Advanced configuration
The configuration of ip-masq-agent is sourced from configMap and can be viewed or modified via kubectl edit cm cce-ip-masq-agent -n kube-system. After modifying the configuration, changes take effect after a period (~60s by default), and the agent program automatically updates iptables rules.
Common configuration options are as follows:
- masqueradeCIDRs []string: Used to specify the masqueraded range
- nonMasqueradeCIDRs []string: Used to specify the non-masqueraded range
- masqOutBound bool: Exclude IPv4 nonMasqueradeCIDRs, whether to masquerade all other ranges
- masqOutBoundIPv6 bool: Exclude IPv6 nonMasqueradeCIDRs, whether to masquerade all other ranges
- masqLinkLocal bool: Whether the traffic is masqueraded or not 169.254.0.0/16
- masqLinkLocalIPv6 bool: Whether the traffic is masqueraded or not fe80::/10
- resyncInterval string: Time interval for the agent to reload the configuration
In the IP-MASQ-AGENT chain, rules from masqueradeCIDRs have higher priority than those from nonMasqueradeCIDRs. Configuring masqueradeCIDRs typically targets disguising specific IPs or network sections within nonMasqueradeCIDRs. Therefore, configurations for masqueradeCIDRs take precedence over those for nonMasqueradeCIDRs, and nonMasqueradeCIDRs configurations take priority over masqOutBound configurations.
Common scenarios
Pods access extra-cluster IP addresses within the same VPC
Within a VPC, in addition to K8S Cluster CCE nodes, there are numerous extra-cluster virtual machines or services.
Security groups of these virtual machines or services sometimes only allow packets whose source IPs are in the VPC address ranges to pass through.
When pods access these virtual machines or services, packets use pod IPs as the source IPs. Since pod IP address ranges typically differ from VPC address ranges, packets may be discarded by security groups of destination virtual machines or services.
Thus, we must configure the masqueradeCIDRs field of ip-masq-agent to enable pods to perform SNAT using their node's IP when accessing these virtual machines.
For example, in network topology, 10.10.10.10 represents a BCC address. Configuration reference:
Containers only masquerade themselves as nodes for certain addresses
Pods access IP addresses within peering connections VPCs
After establishing peering connections between two VPCs, under the default cluster network configuration, when a pod in VPC-1 accesses an address in the peer VPC-2, the source IP is the pod IP. When the destination machine receives the packet and sends a response packet, the destination IP is the pod IP. Since the pod IP address range differs from VPC-1's address range, response packet may be lost due to missing peering connection routes for the pod's address range.
Thus, we need to configure ip-masq-agent to enable pods to perform SNAT using their node's IP when accessing these virtual machines, allowing packets to pass through the peering connection gateway smoothly.
For example, in the network topology, 10.10.10.10 is the address of a BCC in the peer VPC of VPC where the cluster is located in the peering connection. We need to set the masqueradeCIDRs field of ip-masq-agent, with reference configuration:
Containers only masquerade themselves as nodes for certain addresses
Pods access public network address
Containers access the public network using the node's IP
Once a public IP is configured for a node, pods on that node can access the public network via SNAT using the node's IP address.
In this case, we need to configure the masqOutBound attributes of ip-masq-agent to true, with example reference:
Container masquerading as node identity
Containers cooperate with NAT gateway to access the public network with gateway IP.
In addition to configuring public IPs for individual nodes, pods can also access the public network via NAT gateway and VPC route.
In this case, refer to the example configuration: