This document shows in detail how to configure ip-masq-agent. Users can flexibly combine various configurations to choose the identity when the container traffic flows out of the node.

Quick Start

Key Terms

• NAT (Network Address Resolution) is a method to remap an IP address to another IP address by modifying the source and/or destination address information in the IP address header. It is usually performed by the device that performs IP routing.
• A form of Masquerade NAT is usually used to perform many-to-one address translation, in which multiple source IP addresses are hidden behind a single address which is usually the device that performs IP routing. In Kubernetes, this address is the IP address of the node.
• CIDR (Classless Inter-Domain Routing) bases on a variable-length subnet mask, allowing you to specify a prefix of any length. CIDR introduces a new IP address representation method, which is now commonly referred to as CIDR notation, in which a suffix is added after the address or routing prefix to indicate the number of bits of the prefix, such as 192.168.2.0/24.

Principle Introduction

ip-masq-agentip-masq-agent is deployed as a daemonset and start an agent on each node. By configuring iptables rules, the agent can choose to use the container IP address or the node-masqueraded IP address as the source address of the outbound traffic when sending container traffic to the cluster node IP and the destination outside the cluster IP range. The agent reloads its configuration from /etc/config/ip-masq-agent every 60 seconds to update the iptables rules.

The iptables rule configured by the agent is to add a custom chain IP-MASQ-AGENT to the POSTROUTING chain. By judging whether each rule in the IP-MASQ-AGENT chain matches, it is determined whether the outbound traffic needs to be disguised.

The CCE cluster deploysip-masq-agentcomponents by default. By default, the agent is configured to treat the three private IP ranges specified by RFC 1918 as non-masquerading CIDRs. These ranges are 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16. By default, the agent also treats the link-local address (169.254.0.0/16) as a non-masquerading CIDR.

Advanced Configuration

The configuration of ip-masq-agentcomes from kubectl edit cm cce-ip-masq-agent -n kube-system. You can use kubectl ed (...) to view or modify the configuration. After a period (about 60s in the default configuration) from the configuration modification, the configuration takes effect. The agent will automatically update the iptables rules.

Commonly-used configuration options are as follows:

• masqueradeCIDRs []string: Used to specify the masquerad range.
• nonMasqueradeCIDRs []string: Used to specify non-masquerade range.
• masqOutBound bool: Exclude IPv4 nonMasqueradeCIDRs, whether to masquerade all other ranges.
• masqOutBoundIPv6 bool: Exclude IPv6 nonMasqueradeCIDRs, whether to masquerade all other ranges.
• masqLinkLocal bool: Whether to masquerade the traffic to 169.254.0.0/16
• masqLinkLocalIPv6 bool: Whether to masquerade the traffic fe80::/10
• resyncInterval string: The interval at which the agent reloads the configuration

Note that in the IP-MASQ-AGENT chain, the rules generated by masqueradeCIDRs always precede the rules generated by nonMasqueradeCIDRs. The configuration of masqueradeCIDRs is usually to masquerade a specific IP address or network IP address range in nonMasqueradeCIDRs.

Typical Scenarios

The container exits the node with its identity.

Configuration:

    nonMasqueradeCIDRs:
      - 10.0.0.0/8
      - 172.16.0.0/12
      - 100.64.230.0/24
      - 192.168.0.0/16
      - fc00::/7
    masqOutBound: false
    masqOutBoundIPv6: true
    masqLinkLocal: false
    masqLinkLocalIPv6: false
    resyncInterval: 60s

IP-MASQ-AGENT Chain Rules:

Chain IP-MASQ-AGENT (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  *      *       0.0.0.0/0            169.254.0.0/16       /* ip-masq-agent: local traffic is not subject to MASQUERADE */
    0     0 RETURN     all  --  *      *       0.0.0.0/0            10.0.0.0/8           /* ip-masq-agent: local traffic is not subject to MASQUERADE */
    1    71 RETURN     all  --  *      *       0.0.0.0/0            172.16.0.0/12        /* ip-masq-agent: local traffic is not subject to MASQUERADE */
    0     0 RETURN     all  --  *      *       0.0.0.0/0            100.64.230.0/24      /* ip-masq-agent: local traffic is not subject to MASQUERADE */
    0     0 RETURN     all  --  *      *       0.0.0.0/0            192.168.0.0/16       /* ip-masq-agent: local traffic is not subject to MASQUERADE */

In this configuration, the container usually uses the NAT gateway and VPC routing to access the public network.

The container masquerades as a node identity.

This configuration is usually applied for the following scenarios:

• The container wants to access the public network via the public IP address of the node

Configuration:

  nonMasqueradeCIDRs:
      - 10.0.0.0/8
      - 172.16.0.0/12
      - 100.64.230.0/24
      - 192.168.0.0/16
      - fc00::/7
    masqOutBound: true
    masqOutBoundIPv6: true
    masqLinkLocal: false
    masqLinkLocalIPv6: false
    resyncInterval: 60s

IP-MASQ-AGENT Chain Rules:

Chain IP-MASQ-AGENT (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  *      *       0.0.0.0/0            169.254.0.0/16       /* ip-masq-agent: local traffic is not subject to MASQUERADE */
    0     0 RETURN     all  --  *      *       0.0.0.0/0            10.0.0.0/8           /* ip-masq-agent: local traffic is not subject to MASQUERADE */
    4   240 RETURN     all  --  *      *       0.0.0.0/0            172.16.0.0/12        /* ip-masq-agent: local traffic is not subject to MASQUERADE */
    0     0 RETURN     all  --  *      *       0.0.0.0/0            100.64.230.0/24      /* ip-masq-agent: local traffic is not subject to MASQUERADE */
    0     0 RETURN     all  --  *      *       0.0.0.0/0            192.168.0.0/16       /* ip-masq-agent: local traffic is not subject to MASQUERADE */
    1    76 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ip-masq-agent: outbound traffic is subject to MASQUERADE (must be last in chain) */

In this configuration, the container can access the public network as long as the node can access the public network.

The container masquerades as a node only for some addresses.

This configuration is usually applied for the following scenarios:

• Containers in the cluster want to access virtual machines or services that are in the same VPC and have a whitelist or security group. For example, the access is possible only via the IP in the VPC
• ontainers in the cluster want to access services on a virtual machine in another VPC or private network space that already builds a peer-to-peer connection or express tunnel with the current VPC

For example, in the user's network topology, 10.10.10.10/32 is a service VIP, which is accessible only from the node.

Configuration:

    masqueradeCIDRs:
      - 10.10.10.10/32
    nonMasqueradeCIDRs:
      - 10.0.0.0/8
      - 172.16.0.0/12
      - 100.64.230.0/24
      - 192.168.0.0/16
      - fc00::/7
    masqOutBound: false
    masqOutBoundIPv6: true
    masqLinkLocal: false
    masqLinkLocalIPv6: false
    resyncInterval: 60s

IP-MASQ-AGENT Chain Rules:

Chain IP-MASQ-AGENT (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MASQUERADE  all  --  *      *       0.0.0.0/0            10.10.10.10          /* ip-masq-agent: specific outbound traffic is subject to MASQUERADE */
    0     0 RETURN     all  --  *      *       0.0.0.0/0            169.254.0.0/16       /* ip-masq-agent: local traffic is not subject to MASQUERADE */
    0     0 RETURN     all  --  *      *       0.0.0.0/0            10.0.0.0/8           /* ip-masq-agent: local traffic is not subject to MASQUERADE */
  225 15788 RETURN     all  --  *      *       0.0.0.0/0            172.16.0.0/12        /* ip-masq-agent: local traffic is not subject to MASQUERADE */
    0     0 RETURN     all  --  *      *       0.0.0.0/0            100.64.230.0/24      /* ip-masq-agent: local traffic is not subject to MASQUERADE */
    0     0 RETURN     all  --  *      *       0.0.0.0/0            192.168.0.0/16       /* ip-masq-agent: local traffic is not subject to MASQUERADE */