{"id":1313,"date":"2016-10-13T14:20:53","date_gmt":"2016-10-13T12:20:53","guid":{"rendered":"https:\/\/elkano.org\/blog\/?p=1313"},"modified":"2017-01-17T10:27:20","modified_gmt":"2017-01-17T09:27:20","slug":"venet-type-access-linux-containers-lxc","status":"publish","type":"post","link":"https:\/\/elkano.org\/blog\/venet-type-access-linux-containers-lxc\/","title":{"rendered":"venet access like in Linux Containers (LXC)"},"content":{"rendered":"

I’ve been using OpenVZ containers in Proxmox for a while, and after upgrading to Proxmox 4 OpenVZ has been removed in favor of LXC containers. Although LXC containers have a lot of great features, the way they access to the network is not very good if you have untrusted users using them, because the network device in the container is attached directly to the bridge.<\/p>\n

I liked a lot the venet devices in OpenVZ because with this devices the container has only access to the layer 3 of the network. In this post I tried to get venet like access with LXC containers, for that I used ebtables rules to limit this access.<\/p>\n

In my case I created a new bridge (vmbr02) in a separate vlan (550) for the containers. The idea is that the proxmox node is going to be the gateway in layer 2 and layer 3 for the container and the MAC of the container is going to be masqueraded with the MAC of the bridge in the proxmox node. Because I have already a gateway configured in the proxmox node I’m going to create a new route table with the configuration of this network.<\/p>\n

First we add the IP (10.10.10.10) to the new bridge in the proxmox node. This IP will be the gateway in the container.<\/p>\n

sudo ip addr add 10.10.10.10\/32 dev vmbr02v550 \r\nsudo ip route add 10.10.10.0\/24 dev vmbr02v550 src 10.10.10.10\r\n<\/pre>\n
Create the new route table for the new network in vlan 550<\/p>\n
sudo echo "550 vlan550" >> \/etc\/iproute2\/rt_tables\r\n<\/pre>\n
Populate the new table with the real gateway of this network (10.10.10.1)<\/p>\n
sudo ip route add throw 10.10.10.0\/24 table 550\r\nsudo ip route add  default via 10.10.10.1 table 550\r\n<\/pre>\n
Add a rule for the traffic comming up from the bridge lookup the table 550:<\/p>\n
\r\nsudo ip rule add from 31.193.227.0\/24 iif vmbr02v550 lookup vlan550\r\n<\/pre>\n
Enable ip forwarding because the proxmox node will be the router for the LXC container. Because we only want ip forwarding in the bridge interface we can enable it only for this interface:<\/p>\n
\r\nsudo systectl -w net.ipv4.conf.vmbr02v550.forwarding= 1\r\n<\/pre>\n
or<\/p>\n
\r\necho 1 > \/proc\/sys\/net\/ipv4\/conf\/vmbr02v550\/forwarding\r\n<\/pre>\n
and to be permanent between reboots:<\/p>\n
\r\necho "net.ipv4.conf.vmbr02v550.forwarding= 1" >> \/etc\/sysctl.conf\r\n<\/pre>\n
I added this iptables rule because I’dont want the server be accessed with this IP address, this IP is going to use only to provide networking to the containers.<\/p>\n
iptables -A INPUT -d 10.10.10.10 -j REJECT\r\n<\/pre>\n
At this point we have to have the LXC container created. In the following example we see a LXC with id 150:<\/p>\n
$ sudo pct list\r\nVMID       Status     Lock         Name                \r\n150        stopped                 pruebas-lxc         \r\n<\/pre>\n
The bridge vmbr02v550 has one port connected to the physical network (bond0.550) and one virtual port connected to the container (veth150i1). Note that the container must be running to see the virtual port up:<\/p>\n
$ sudo brctl show vmbr02v550\r\nbridge name\tbridge id\t\tSTP enabled\tinterfaces\r\nvmbr02v550\t\t8000.002590911cee\tno\t\tbond0.550\r\n\t\t\t\t\t\t\tveth150i1\r\n<\/pre>\n
Now it’s time to put some ebtables rules to limit the traffic forwarded in the bridge, these are general rules used for all the containers:<\/p>\n
# ebtables rules in the table filter and chain forward\r\nebtables -A FORWARD -o veth+ --pkttype-type multicast -j DROP #1\r\nebtables -A FORWARD -i veth+ --pkttype-type multicast -j DROP #2\r\nebtables -A FORWARD -o veth+ --pkttype-type broadcast -j DROP #3\r\nebtables -A FORWARD -i veth+ --pkttype-type broadcast -j DROP #4\r\nebtables -A FORWARD -p IPv4 -i veth+ -o bond0.550 --ip-dst 10.10.10.0\/24 -j ACCEPT #5\r\nebtables -A FORWARD -p IPv4 -i bond0.550 -o veth+ --ip-dst 10.10.10.0\/24 -j ACCEPT #6\r\nebtables -A FORWARD -o veth+ -j DROP #6\r\nebtables -A FORWARD -i veth+ -j DROP #7\r\n<\/pre>\n
In this rules we use the expression veth+ to refer to all LXC virtual ports that can be connected to the bridge. The short explanation of each rule is the following:<\/p>\n
    \n
  • #1 and #2 => Stop all multicast layer 3 packets delivered from an to the LXC virtual ports<\/li>\n
  • #2 and #3 => Stop all broadcast layer 3 packets delivered from and to the LXC virtual ports<\/li>\n
  • #5 and #6 => Only allow forwarding\u00a0if the source IP or the destination IP \u00a0is in the LXC containers network.<\/li>\n
  • #6 and #7 => Drop all packets that don’t met the above rules. We don’t want any packets from layer 2.<\/li>\n<\/ul>\n
    We need some ebtables rules in the nat table, but in this case these rules are set for each container. For this example we suppose the following:<\/p>\n
      \n
    • Proxmox node MAC address:  0:25:90:91:1c:ee<\/li>\n
    • LXC real MAC address:  66:36:61:62:32:31<\/li>\n
    • LXC IP address:  10.10.10.11<\/li>\n<\/ul>\n
      \r\n#Ebtables rule to translate the packets that must be delivered to the LXC container IP to its real MAC\r\nebtables -t nat -A PREROUTING -p IPv4 -d 0:25:90:91:1c:ee -i bond0.550 --ip-dst 10.10.10.11 -j dnat --to-dst 66:36:61:62:32:31 --dnat-target ACCEPT\r\n#Ebtables rule to reply to the ARP requests looking for the MAC of the LXC container with the MAC of the host:\r\nebtables -t nat -A PREROUTING -i bond0.550 -p ARP --arp-op Request --arp-ip-dst 10.10.10.11 -j arpreply --arpreply-mac 0:25:90:91:1c:ee\r\n# I preffer the ebtables rule, but the above can be addressed too with the following arp command:\r\n# arp -i vmbr02v550 -Ds 10.10.10.11 vmbr02v550 pub\r\n#Ebtables rule to mask the LXC container MAC with the MAC of the host\r\nebtables -t nat -A POSTROUTING -s 66:36:61:62:32:31 -o bond0.550 -j snat --to-src 0:25:90:91:1c:ee --snat-arp --snat-target ACCEPT\r\n<\/pre>\n
      And we are done! with this configuration the LXC container, although connected directly to the linux bridge, it has a limited access to the network.<\/p>\n","protected":false},"excerpt":{"rendered":"
      I’ve been using OpenVZ containers in Proxmox for a while, and after upgrading to Proxmox 4 OpenVZ has been removed in favor of LXC containers. Although LXC containers have a lot of great features, the way they access to the network is not very good if you have untrusted users using them, because the network […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[128,7],"tags":[190,187,189,35,188],"_links":{"self":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/posts\/1313"}],"collection":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/comments?post=1313"}],"version-history":[{"count":1,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/posts\/1313\/revisions"}],"predecessor-version":[{"id":1318,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/posts\/1313\/revisions\/1318"}],"wp:attachment":[{"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/media?parent=1313"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/categories?post=1313"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/elkano.org\/blog\/wp-json\/wp\/v2\/tags?post=1313"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}