CCE IP Masquerade Agent User Guide

This document details how to configure cce-ip-masq-agent, allowing users to flexibly combine various configurations to meet identity selection requirements when container traffic egresses the node.

Quick start

Key terms

• NAT (Network Address Translation) It is a method of remapping one IP address to another by modifying the source and/or destination address information in the IP header. It is typically handled by devices performing IP route.
• Masquerade A form of NAT, typically used for many-to-one address translation, where multiple source IP addresses are hidden behind a single address, usually the device performing IP route. In Kubernetes, this is the node's IP address.
• CIDR (Classless Inter-Domain Routing) Based on variable-length subnet masks, prefixes of any length can be specified. CIDR introduces a new IP address notation, now commonly called CIDR notation, where a suffix is added after the address or route prefix to indicate the number of digits of prefix (e.g., 192.168.2.0/24).

Introduction to principle

cce-ip-masq-agent is deployed as a daemonset, launching an agent program on each node. By configuring iptables rules, the agent program can select to use the IP address of the container itself or the IP address masqueraded as the node as the source address of the egress traffic when sending the container traffic to the target outside the range of the cluster node IP and the cluster IP. The agent program reloads its configuration from /etc/config/ip-masq-agent every 60 seconds to update iptables rules.

The iptables rules added by the agent program introduce a custom chain named IP-MASQ-AGENT to the POSTROUTING chain, determining whether outgoing traffic should be masqueraded by matching rules in the IP-MASQ-AGENT chain.

Installation steps

1. Plugin selection

cce-ip-masq-agent is an extension component of CCE. The first step is to manually select the installation on the CCE Console -> Helm -> Helm Template -> Baidu AI Cloud Template. As shown above, click the Install button to enter the Plugin Installation page.

2. Installation parameters

The second step of plugin installation is configuring necessary parameters:

• Instance name: Helm instance name of the plugin supports customization. Recommended settings: cce-ip-masq-agent.
• Deployment cluster: Choose the target cluster for installing the plugin.
• Namespace: It's recommended to install CCE system components in the kube-system namespace.
• Template content: The current version of template content is used to configure the image version for cce-ip-masq-agent. The following versions are recommended:

1CNIMasqAgentImage: registry.baidubce.com/cce-plugin-pro/cce-cni:v1.3.4

By default, the agent configuration excludes the three private IP ranges specified by RFC 1918—10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16—from masquerading. Additionally, it excludes the link-local address range (169.254.0.0/16) from masquerading by default.

Note: When modifying the cce-ip-masq-agent configuration, users need to ensure both the container network segments and node network segments are within non-masqueraded CIDRs to avoid container network connectivity issues.

Advanced configuration

The configuration of cce-ip-masq-agent is sourced from configMap and can be viewed or modified via kubectl edit cm cce-ip-masq-agent -n kube-system. After modifying the configuration, changes take effect after a period (~60s by default), and the agent program automatically updates iptables rules.

Common configuration options are as follows:

• masqueradeCIDRs []string: Used to specify the masqueraded range
• nonMasqueradeCIDRs []string: Used to specify the non-masqueraded range
• masqOutBound bool: Exclude IPv4 nonMasqueradeCIDRs, whether to masquerade all other ranges
• masqOutBoundIPv6 bool: Exclude IPv6 nonMasqueradeCIDRs, whether to masquerade all other ranges
• masqLinkLocal bool: Whether the traffic is masqueraded or not 169.254.0.0/16
• masqLinkLocalIPv6 bool: Whether the traffic is masqueraded or not fe80::/10
• resyncInterval string: Time interval for the agent to reload the configuration

In the IP-MASQ-AGENT chain, rules generated by masqueradeCIDRs always take precedence over those from nonMasqueradeCIDRs. Typically, configuring masqueradeCIDRs is intended to masquerade specific IP addresses or network segments contained within nonMasqueradeCIDRs.

Typical scenarios

Containers egress nodes with their own identity

Configuration:

1    nonMasqueradeCIDRs:
2      - 10.0.0.0/8
3      - 172.16.0.0/12
4      - 100.64.230.0/24
5      - 192.168.0.0/16
6      - fc00::/7
7    masqOutBound: false
8    masqOutBoundIPv6: true
9    masqLinkLocal: false
10    masqLinkLocalIPv6: false
11    resyncInterval: 60s

IP-MASQ-AGENT chain rules:

1Chain IP-MASQ-AGENT (1 references)
2 pkts bytes target     prot opt in     out     source               destination
3    0     0 RETURN     all  --  *      *       0.0.0.0/0            169.254.0.0/16       /* ip-masq-agent: local traffic is not subject to MASQUERADE */
4    0     0 RETURN     all  --  *      *       0.0.0.0/0            10.0.0.0/8           /* ip-masq-agent: local traffic is not subject to MASQUERADE */
5    1    71 RETURN     all  --  *      *       0.0.0.0/0            172.16.0.0/12        /* ip-masq-agent: local traffic is not subject to MASQUERADE */
6    0     0 RETURN     all  --  *      *       0.0.0.0/0            100.64.230.0/24      /* ip-masq-agent: local traffic is not subject to MASQUERADE */
7    0     0 RETURN     all  --  *      *       0.0.0.0/0            192.168.0.0/16       /* ip-masq-agent: local traffic is not subject to MASQUERADE */

In this setup, containers usually access the public network via a NAT gateway and VPC route.

Container masquerading as node identity

This configuration is typically applied in the following scenarios:

• Containers hope to access the public network using the node's public IP

Configuration:

1  nonMasqueradeCIDRs:
2      - 10.0.0.0/8
3      - 172.16.0.0/12
4      - 100.64.230.0/24
5      - 192.168.0.0/16
6      - fc00::/7
7    masqOutBound: true
8    masqOutBoundIPv6: true
9    masqLinkLocal: false
10    masqLinkLocalIPv6: false
11    resyncInterval: 60s

IP-MASQ-AGENT chain rules:

1Chain IP-MASQ-AGENT (1 references)
2 pkts bytes target     prot opt in     out     source               destination
3    0     0 RETURN     all  --  *      *       0.0.0.0/0            169.254.0.0/16       /* ip-masq-agent: local traffic is not subject to MASQUERADE */
4    0     0 RETURN     all  --  *      *       0.0.0.0/0            10.0.0.0/8           /* ip-masq-agent: local traffic is not subject to MASQUERADE */
5    4   240 RETURN     all  --  *      *       0.0.0.0/0            172.16.0.0/12        /* ip-masq-agent: local traffic is not subject to MASQUERADE */
6    0     0 RETURN     all  --  *      *       0.0.0.0/0            100.64.230.0/24      /* ip-masq-agent: local traffic is not subject to MASQUERADE */
7    0     0 RETURN     all  --  *      *       0.0.0.0/0            192.168.0.0/16       /* ip-masq-agent: local traffic is not subject to MASQUERADE */
8    1    76 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ip-masq-agent: outbound traffic is subject to MASQUERADE (must be last in chain) */

With this configuration, containers can access the public network as long as the nodes themselves can connect to the public network.

Containers only masquerade themselves as nodes for certain addresses

This configuration is typically applied in the following scenarios:

• Intra-cluster containers hope to access virtual machines or services that are in the same VPC as the current cluster and have allow lists or security groups (for example, only IP in VPC is allowed to access)
• Intra-cluster containers hope to access another VPC that establishes peering connection or private line with the current VPC or services on virtual machine in virtual private cloud space

For instance, in a user's network topology, 10.10.10.10/32 serves as a VIP for a particular service and is accessible only from a specific node.

Configuration:

1    masqueradeCIDRs:
2      - 10.10.10.10/32
3    nonMasqueradeCIDRs:
4      - 10.0.0.0/8
5      - 172.16.0.0/12
6      - 100.64.230.0/24
7      - 192.168.0.0/16
8      - fc00::/7
9    masqOutBound: false
10    masqOutBoundIPv6: true
11    masqLinkLocal: false
12    masqLinkLocalIPv6: false
13    resyncInterval: 60s

IP-MASQ-AGENT chain rules:

1Chain IP-MASQ-AGENT (1 references)
2 pkts bytes target     prot opt in     out     source               destination
3    0     0 MASQUERADE  all  --  *      *       0.0.0.0/0            10.10.10.10          /* ip-masq-agent: specific outbound traffic is subject to MASQUERADE */
4    0     0 RETURN     all  --  *      *       0.0.0.0/0            169.254.0.0/16       /* ip-masq-agent: local traffic is not subject to MASQUERADE */
5    0     0 RETURN     all  --  *      *       0.0.0.0/0            10.0.0.0/8           /* ip-masq-agent: local traffic is not subject to MASQUERADE */
6  225 15788 RETURN     all  --  *      *       0.0.0.0/0            172.16.0.0/12        /* ip-masq-agent: local traffic is not subject to MASQUERADE */
7    0     0 RETURN     all  --  *      *       0.0.0.0/0            100.64.230.0/24      /* ip-masq-agent: local traffic is not subject to MASQUERADE */
8    0     0 RETURN     all  --  *      *       0.0.0.0/0            192.168.0.0/16       /* ip-masq-agent: local traffic is not subject to MASQUERADE */