| Table of Contents |
|---|
Introduction
By default, your containers are accessible only from the host. For serious use you will want to expose some containers to the outside world. There are various ways of doing this. Currently I have settled on the following,
...
macvlan with Additional IP - allows you to have, a dedicated network interfaces (to the outside world) but actually only use one real physical network card. Unlike using a bridge this will not have the cpu overhead and need for your network card to work in promiscuous mode. This article builds on the work done in the introductory LXC article.
I actually use both multiple techniques together.
| Warning |
|---|
Make sure to change the password or better remove the default ubuntu account generated by the lxc creation script before making the container accessible to the Internet. |
UFW
UFW in the Host
UFW is a great simple firewall, but at this point I do not recommend installing on your host if you intend to use port forwarding as there may be conflicts. Second, port forwarding using UFW is overly complex and seems like a hack versus it being very simple with IP Tables.
If you insist on using UFW, make sure to change the setting to not drop forwarded packets. I will revisit this later as I do like UFW. Perhaps I can ask the developers to make port forwarding more straight-forward.
UFW in a Container
Also, firewalls work at the kernel level. So you should not be installing UFW or even IP Tables inside of a container.
I will revisit this topic but believe it is due to modules not loading inside of containers /etc/modules and the container not being able to modify it.
| Code Block | ||
|---|---|---|
| ||
sudo ufw allow 22
ERROR: initcaps
[Errno 2] modprobe: ERROR: ../libkmod/libkmod.c:556 kmod_search_moddep() could not open moddep file '/lib/modules/3.13.0-57-generic/modules.dep.bin'
ip6tables v1.4.21: can't initialize ip6tables table `filter': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded. |
Trying to enable UFW inside of a container results in a a kernel needs to be upgraded error.
This will get better over time, ie compared to Solaris where you just tell the container to use a public IP address and the macvlan or bridging is done for you.
Port Forwarding using IP Tables
You might want to use one IP Address on the host and then map specific ports out from the containers. As a pre-requisite you will need to setup Static LXC Assigned IP address.
There are a number of ways to do this but I favour iptables.
...
| Code Block | ||
|---|---|---|
| ||
netstat -an | grep LISTEN | grep 80 tcp6 0 0 fe80::2cd7:eff:fea3::53 :::* LISTEN |
Next determine your hosts network card name associated with the hosts IP address,
| Code Block | ||
|---|---|---|
| ||
ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:0d:3a:02:e6:8a
inet addr:10.0.0.4 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::20d:3aff:fe02:e68a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:964479 errors:0 dropped:0 overruns:0 frame:0
TX packets:1199824 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:333377198 (333.3 MB) TX bytes:1757835361 (1.7 GB)
# ... more get's displayed |
In this case, I know my host's public IP6 address (put reference of how to convert to ip4 or reverse lookup on dns IP6 address) is fe80::20d:3aff:fe02:e68a/64 and see it as the first entry. This let's us know the network card name is eth0. In Ubuntu it will normally be eth0 or ens33.
While on the host issue these commands, (TBD, look at making own named chain to distinguish the rules)
...
Now traffic on port 80 on the host will be forwarded to port 80 in the container IP specified. You can see your rules, (note I got to try below review output again on a clean machine).
| Code Block | ||
|---|---|---|
| ||
sudo iptables -t nat -L -n -v
Chain PREROUTING (policy ACCEPT 15 packets, 957 bytes)
pkts bytes target prot opt in out source destination
1 64 DNAT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 to:10.0.3.10:80
Chain INPUT (policy ACCEPT 1 packets, 229 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
695K 53M MASQUERADE all -- * * 10.0.3.0/24 !10.0.3.0/24 |
...
| Code Block | ||
|---|---|---|
| ||
sudo iptables -t nat -D PREROUTING -p tcp -i eth0 --dport 22 -j DNAT --to-destination 10.0.3.11:23 |
bridge with Additional IP Address
...
How to adjust LXD,
Also rever to my own article on bridge networking.
macvlan with Additional IP Address
Here's the Use Case,
| Panel |
|---|
For further isolation you may have purchased an additional IP address. For example run Apache on your host using port 80 and then also run Apache inside one of your containers also requiring port 80. Since it is not possible to have port 80 twice on the same IP you opt to purchase a second IP. In most hosting services this is not expensive, but they will likely not give you a dedicated network card. In this example we purchase an additional static IP from the hosting company and use the same network card as the host. |
| Note |
|---|
| Hopefully this will get better with LXD, but this is quite cumbersome compared to Solaris where you just tell the container to use a public IP address. |
The most viable options, I understand are using bridge or a dedicated vlan.
With macvlan you configure the container to directly use the public IP address without the overhead of changing the network card to promiscuous mode. Once and have a lower CPU overhead. Once setup the macvlan gets it's own MAC address. This only works if there are no restrictions on the network which set's static IPs based on the hosts' MAC address. Usually this is only the case with the initial primary IP provided by the hosting company.
The containers can reach the network and each other, but not the host. Even though the host may be on the same network. I am not sure why this is the case (maybe security?) but do not have a need to solve this use case at the moment. Macvlan has many modes, but from my readings bridge mode is most appropriate.This is by design of macvlan. The container network cards (Sub-interfaces) on the same host (parent) cannot communicate with each other. ALL frames from sub-interfaces are forward out through the parent interface. Even physcial switches that reflect the frames back in will get dropped.
If you had previously assigned a static IP to the container using /etc/lxc/dnsmasq.conf make sure to remove the entry (I believe you also need to restart the host).
| Note |
|---|
| One interesting limitation |
...
| is that |
...
| the containers cannot resolve by DNS to the |
...
| original Public IP |
...
| directly used by the host. I don't have a use case requiring this, so will resolve iif/when needed. |
You must have,
- 1 macvlan mapped to 1 container interface
- each interface must have different static IP addresses
macvlan mac address
The first thing to do is to create a mac address for the macvlan interface we will create on the host.
...
| Warning |
|---|
You can not use the same MAC interface on multiple containers on the same host. Otherwise, you will not be able to start you container and receive the error message about your interface already being in use. |
Command Line macvlan
You can use the command line inside your container to quickly create a macvlan and test, but it will disappear after reboot,
| Code Block | ||
|---|---|---|
| ||
ip link add mvlan0 link eth0 address 8a:38:2a:cc:d7:aa type macvlan mode bridge ifconfig mvlan0 up |
Create a Permanent macvlan on the Host
Add to the bottom of the /etc/network/interfaces file of the host,
...
Notice mvlan0 is present with the hardware address you specified.
Connect Container to macvlan on Host
Now one container may connect to the mvlan0 interface on the host and they will get their IPs directly from the same network connected to the host (if DHCP) or you can assign static IPs inside the container that are reserved for you.
...
| Note |
|---|
There is probably a way to setup the static ip address here in this config file for the container. I am pretty sure I saw an option. However, in the vein of individual container control, I would rather set that at the container level using steps outlined below. |
Next, modify the interfaces file within the container to your selected static IP address. I find the fastest is from within the Host OS using root, /var/lib/lxc/<container name>/rootfs/etc/network/interfaces which in this case would be, /var/lib/lxc/web/rootfs/etc/network/interfaces,
| Code Block | ||
|---|---|---|
| ||
# This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). # The loopback network interface auto lo iface lo inet loopback auto eth0 iface eth0 inet static address 64.73.220.117 gateway 64.73.220.1 netmask 255.255.255.0 dns-nameservers 216.15.129.205 216.15.129.206 auto eth1 iface eth1 inet dhcp |
Above we now have to two interfaces. That The static interface is the public IP address purchased called eth0 and second is the internal LXC assigned address. The other entries can be obtained from Add Additional IP Addresses to Ubuntu Server.
Update dnsmasq
Make sure to check your dnsmasq and make modifications accordingly. Using the example since we modified a container that was already using dnsmasq we needed to change the original /etc/lxc/dnsmasq.conf,
...
| Include Page | ||||
|---|---|---|---|---|
|
Multiple Interfaces
In most cases you will want multiple interfaces. In this example, we built a front-end container called "web", gave it a public IP address using mavlan. In addition, we create an "app" container which has an lxc internal IP address. In order for web and app to communicate, web must have a second interface that also uses an lxc provided internal IP address.
...
I got this all working, but need to document.
Firewall
UFW does not work inside of LXC because it is run at kernel level. UFW does not work with LXC well because I can't port forward easily and IP Tables seems the way to got for now. Should make a request to UFW to fix that... in any event, need to figure out how to firewall Host and containers.
macvtap
This looks promising... The most prominent user of macvtap interfaces seems to be libvirt/KVM, which allows guests to be connected to macvtap interfaces. Doing so allows for (almost) bridged-like behaviour of guests but without the need to have a real bridge on the host, as a regular ethernet interface can be used as the macvtap's lower device.
References
Networking - https://help.ubuntu.com/lts/serverguide/lxc.html#lxc-network
...
More in depth and discusses outbound NAT so containers can communicate to other container public IPs -http://blog.codeaholics.org/2013/giving-dockerlxc-containers-a-routable-ip-address/
Bridge versus Macvlan - http://hicu.be/bridge-vs-macvlan