1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
3 .. (c) Prakash Ramchandran
5 ======================================
6 Exploring IPv6 in Container Networking
7 ======================================
9 This document is the summary of how to use IPv6 with Docker.
11 The defualt Docker container uses 172.17.0.0/24 subnet with 172.17.0.1 as gateway.
12 So IPv6 network needs to be enabled and configured before we can use it with IPv6
15 We will describe how to use IPv6 in Docker in the following 5 sections:
17 1. Install Docker Community Edition (CE)
19 3. Design Simple IPv6 Topologies
21 5. Challenges in Production Use
23 -------------------------------------
24 Install Docker Community Edition (CE)
25 -------------------------------------
27 **Step 1.1**: Download Docker (CE) on your system from [1]_.
29 For Ubuntu 16.04 Xenial x86_64, please refer to [2]_.
31 **Step 1.2**: Refer to [3]_ to install Docker CE on Xenial.
33 **Step 1.3**: Once you installed the docker, you can verify the standalone
34 default bridge nework as follows:
39 NETWORK ID NAME DRIVER SCOPE
40 b9e92f9a8390 bridge bridge local
41 74160ae686b9 host host local
42 898fbb0a0c83 my_bridge bridge local
43 57ac095fdaab none null local
47 * the details may be different with different network drivers.
48 * User-defined bridge networks are the best when you need multiple containers
49 to communicate on the same Docker host.
50 * Host networks are the best when the network stack should not be isolated from
51 the Docker host, but you want other aspects of the container to be isolated.
52 * Overlay networks are the best when you need containers running on different
53 Docker hosts to communicate, or when multiple applications work together
55 * Macvlan networks are the best when you are migrating from a VM setup or need
56 your containers to look like physical hosts on your network, each with a
58 * Third-party network plugins allow you to integrate Docker with specialized
59 network stacks. Please refer to [4]_.
63 # This will have docker0 default bridge details showing
64 # ipv4 172.17.0.1/16 and
65 # ipv6 fe80::42:4dff:fe2f:baa6/64 entries
68 11: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
69 link/ether 02:42:4d:2f:ba:a6 brd ff:ff:ff:ff:ff:ff
70 inet 172.17.0.1/16 scope global docker0
71 valid_lft forever preferred_lft forever
72 inet6 fe80::42:4dff:fe2f:baa6/64 scope link
73 valid_lft forever preferred_lft forever
75 Thus we see here a simple defult ipv4 networking for docker. Inspect and verify
76 that IPv6 address is not listed here showing its enabled but not used by
77 default docker0 bridge.
79 You can create user defined bridge network using command like ``my_bridge``
80 below with other than default, e.g. 172.18.0.0/24 here. **Note** that ``--ipv6``
85 $ sudo docker network create \
87 --subnet=172.18.0.0/24 \
88 --gaeway= 172.18.0.1 \
91 $ docker network inspect bridge
95 "Id": "b9e92f9a839048aab887081876fc214f78e8ce566ef5777303c3ef2cd63ba712",
96 "Created": "2017-10-30T23:32:15.676301893-07:00",
105 "Subnet": "172.17.0.0/16",
106 "Gateway": "172.17.0.1"
118 "ea76bd4694a8073b195dd712dd0b070e80a90e97b6e2024b03b711839f4a3546": {
120 "EndpointID": "b04dc6c5d18e3bf4e4201aa8ad2f6ad54a9e2ea48174604029576e136b99c49d",
121 "MacAddress": "02:42:ac:11:00:02",
122 "IPv4Address": "172.17.0.2/16",
127 "com.docker.network.bridge.default_bridge": "true",
128 "com.docker.network.bridge.enable_icc": "true",
129 "com.docker.network.bridge.enable_ip_masquerade": "true",
130 "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
131 "com.docker.network.bridge.name": "docker0",
132 "com.docker.network.driver.mtu": "1500"
138 $ sudo docker network inspect my_bridge
142 "Id": "898fbb0a0c83acc0593897f5af23b1fe680d38b804b0d5a4818a4117ac36498a",
143 "Created": "2017-07-16T17:59:55.388151772-07:00",
152 "Subnet": "172.18.0.0/16",
153 "Gateway": "172.18.0.1"
170 You can note that IPv6 is not enabled here yet as seen through network inspect.
171 Since we have only IPv4 installed with Docker, we will move to enable IPv6 for
172 Docker in the next step.
178 Verifyig IPv6 with Docker involves the following steps:
180 **Step 2.1**: Enable ipv6 support for Docker
182 In the simplest term, the first step is to enable IPv6 on Docker on Linux hosts.
183 Please refer to [5]_:
185 * Edit ``/etc/docker/daemon.json``
186 * Set the ``ipv6`` key to true.
194 **Step 2.1.1**: Set up IPv6 addressing for Docker in ``daemon.json``
196 If you need IPv6 support for Docker containers, you need to enable the option
197 on the Docker daemon ``daemon.json`` and reload its configuration, before
198 creating any IPv6 networks or assigning containers IPv6 addresses.
200 When you create your network, you can specify the ``--ipv6`` flag to enable
201 IPv6. You can't selectively disable IPv6 support on the default bridge network.
203 **Step 2.1.2**: Enable forwarding from Docker containers to the outside world
205 By default, traffic from containers connected to the default bridge network is
206 not forwarded to the outside world. To enable forwarding, you need to change
207 two settings. These are not Docker commands and they affect the Docker host's
210 * Setting 1: Configure the Linux kernel to allow IP forwarding:
214 $ sysctl net.ipv4.conf.all.forwarding=1
216 * Setting 2: Change the policy for the iptables FORWARD policy from DROP to ACCEPT.
220 $ sudo iptables -P FORWARD ACCEPT
222 These settings do not persist across a reboot, so you may need to add them to
225 **Step 2.1.3**: Use the default bridge network
227 The default bridge network is considered a legacy detail of Docker and is not
228 recommended for production use. Configuring it is a manual operation, and it
229 has technical shortcomings.
231 **Step 2.1.4**: Connect a container to the default bridge network
233 If you do not specify a network using the ``--network`` flag, and you do
234 specify a network driver, your container is connected to the default bridge
235 network by default. Containers connected to the default bridge network can
236 communicate, but only by IP address, unless they are linked using the legacy
239 **Step 2.1.5**: Configure the default bridge network
241 To configure the default bridge network, you specify options in ``daemon.json``.
242 Here is an example of ``daemon.json`` with several options specified. Only
243 specify the settings you need to customize.
248 "bip": "192.168.1.5/24",
249 "fixed-cidr": "192.168.1.5/25",
250 "fixed-cidr-v6": "2001:db8::/64",
252 "default-gateway": "10.20.1.1",
253 "default-gateway-v6": "2001:db8:abcd::89",
254 "dns": ["10.20.1.2","10.20.1.3"]
257 Restart Docker for the changes to take effect.
259 **Step 2.1.6**: Use IPv6 with the default bridge network
261 If you configure Docker for IPv6 support (see **Step 2.1.1**), the default
262 bridge network is also configured for IPv6 automatically. Unlike user-defined
263 bridges, you cannot selectively disable IPv6 on the default bridge.
265 **Step 2.1.7**: Reload the Docker configuration file
269 $ systemctl reload docker
271 **Step 2.1.8**: You can now create networks with the ``--ipv6`` flag and assign
272 containers IPv6 addresses.
274 **Step 2.1.9**: Verify your host and docker networks
279 CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
280 ea76bd4694a8 registry:2 "/entrypoint.sh /e..." x months ago Up y months 0.0.0.0:4000->5000/tcp registry
283 NETWORK ID NAME DRIVER SCOPE
284 b9e92f9a8390 bridge bridge local
285 74160ae686b9 host host local
286 898fbb0a0c83 my_bridge bridge local
287 57ac095fdaab none null local
289 **Step 2.1.10**: Edit ``/etc/docker/daemon.json`` and set the ipv6 key to true.
299 **Step 2.1.11**: Reload the Docker configuration file.
303 $ sudo systemctl reload docker
305 **Step 2.1.12**: You can now create networks with the ``--ipv6`` flag and
306 assign containers IPv6 addresses using the ``--ip6`` flag.
310 $ sudo docker network create --ipv6 --driver bridge alpine-net--fixed-cidr-v6 2001:db8:1/64
312 # "docker network create" requires exactly 1 argument(s).
313 # See "docker network create --help"
315 Earlier, user was allowed to create a network, or start the daemon, without
316 specifying an IPv6 ``--subnet``, or ``--fixed-cidr-v6`` respectively, even when
317 using the default builtin IPAM driver, which does not support auto allocation
318 of IPv6 pools. In another word, it was an incorrect configurations, which had
319 no effect on IPv6 stuff. It was a no-op.
321 A fix cleared that so that Docker will now correctly consult with the IPAM
322 driver to acquire an IPv6 subnet for the bridge network, when user did not
325 If the IPAM driver in use is not able to provide one, network creation would
326 fail (in this case the default bridge network).
328 So what you see now is the expected behavior. You need to remove the ``--ipv6``
329 flag when you start the daemon, unless you pass a ``--fixed-cidr-v6`` pool. We
330 should probably clarify this somewhere.
332 The above was found on following Docker.
342 Server Version: 17.06.1-ce-rc1
344 Root Dir: /var/lib/docker/aufs
345 Backing Filesystem: extfs
347 Dirperm1 Supported: false
348 Logging Driver: json-file
349 Cgroup Driver: cgroupfs
352 Network: bridge host macvlan null overlay
353 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
356 Default Runtime: runc
357 Init Binary: docker-init
358 containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
359 runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
360 init version: 949e6fa
365 Kernel Version: 3.13.0-88-generic
366 Operating System: Ubuntu 16.04.2 LTS
370 Total Memory: 11.67GiB
372 ID: HS5N:T7SK:73MD:NZGR:RJ2G:R76T:NJBR:U5EJ:KP5N:Q3VO:6M2O:62CJ
373 Docker Root Dir: /var/lib/docker
374 Debug Mode (client): false
375 Debug Mode (server): false
376 Registry: https://index.docker.io/v1/
380 Live Restore Enabled: false
382 **Step 2.2**: Check the network drivers
384 Among the 4 supported drivers, we will be using user-defined bridge-network [6]_.
386 -----------------------------
387 Design Simple IPv6 Topologies
388 -----------------------------
390 **Step 3.1**: Creating IPv6 user-defined subnet.
392 Let's create a Docker with IPv6 subnet:
396 $ sudo docker network create \
399 --subnet=172.18.0.0/16 \
400 --subnet=fcdd:1::/48 \
401 --gaeway= 172.20.0.1 \
404 # Error response from daemon:
406 cannot create network 8957e7881762bbb4b66c3e2102d72b1dc791de37f2cafbaff42bdbf891b54cc3 (br-8957e7881762): conflicts with network
407 no matching subnet for range 2002:ac14:0000::/48
409 # try changing to ip-addess-range instead of subnet for ipv6.
410 # networks have overlapping IPv4
412 NETWORK ID NAME DRIVER SCOPE
413 b9e92f9a8390 bridge bridge local
414 74160ae686b9 host host local
415 898fbb0a0c83 my_bridge bridge local
416 57ac095fdaab none null local
417 no matching subnet for gateway 172.20.01
419 # So finally making both as subnet and gateway as 172.20.0.1 works
421 $ sudo docker network create \
424 --subnet=172.20.0.0/16 \
425 --subnet=2002:ac14:0000::/48 \
426 --gateway=172.20.0.1 \
428 898fbb0a0c83acc0593897f5af23b1fe680d38b804b0d5a4818a4117ac36498a (br-898fbb0a0c83):
430 Since lxdbridge used the ip range on the system there was a conflict.
431 This brings us to question how do we assign IPv6 and IPv6 address for our solutions.
437 For best practices, please refer to [7]_.
439 Use IPv6 Calcualtor at [8]_.
441 * For IPv4 172.16.0.1 = 6to4 prefix 2002:ac10:0001::/48
442 * For IPv4 172.17.01/24 = 6to4 prefix 2002:ac11:0001::/48
443 * For IPv4 172.18.0.1 = 6to4 prefix 2002:ac12:0001::/48
444 * For IPv4 172.19.0.1 = 6to4 prefix 2002:ac13:0001::/48
445 * For IPv4 172.20.0.0 = 6to4 prefix 2002:ac14:0000::/48
447 To avoid overlaping IP's, let's use the .20 in our design:
451 $ sudo docker network create \
454 --subnet=172.20.0.0/24 \
455 --subnet=2002:ac14:0000::/48
461 052da268171ce47685fcdb68951d6d14e70b9099012bac410c663eb2532a0c87
464 NETWORK ID NAME DRIVER SCOPE
465 b9e92f9a8390 bridge bridge local
466 74160ae686b9 host host local
467 898fbb0a0c83 my_bridge bridge local
468 052da268171c my_ipv6_bridge bridge local
469 57ac095fdaab none null local
471 # Note the first 16 digits is used here as network id from what we got
472 # whaen we created it.
474 $ docker network inspect my_ipv6_bridge
477 "Name": "my_ipv6_bridge",
478 "Id": "052da268171ce47685fcdb68951d6d14e70b9099012bac410c663eb2532a0c87",
479 "Created": "2018-03-16T07:20:17.714212288-07:00",
488 "Subnet": "172.20.0.0/16",
489 "Gateway": "172.20.0.1"
492 "Subnet": "2002:ac14:0000::/48"
511 * IPv6 flag is ebnabled and that IPv6 range is listed besides Ipv4 gateway.
512 * We are mapping IPv4 and IPv6 address to simplify assignments as per "Best
513 Pratice Document" [7]_.
515 Testing the solution and topology:
519 $ sudo docker run hello-world
522 This message shows that your installation appears to be working correctly.
524 To generate this message, Docker took the following steps:
526 1. The Docker client contacted the Docker daemon.
527 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
528 3. The Docker daemon created a new container from that image which runs the
529 executable that produces the output you are currently reading.
530 4. The Docker daemon streamed that output to the Docker client, which sent it
533 To try something more ambitious, you can run an Ubuntu container with:
537 $ docker run -it ubuntu bash
539 root@62b88b030f5a:/# ls
540 bin dev home lib64 mnt proc run srv tmp var
541 boot etc lib media opt root sbin sys usr
543 On terminal it appears that the docker is functioning normally.
545 Let's now push to see if we can use the ``my_ipv6_bridge`` network.
546 Please refer to "User-Defined Bridge" [9]_.
548 ++++++++++++++++++++++++++++++++++++++++++++
549 Connect a container to a user-defined bridge
550 ++++++++++++++++++++++++++++++++++++++++++++
552 When you create a new container, you can specify one or more ``--network``
553 flags. This example connects a Nginx container to the ``my-net`` network. It
554 also publishes port 80 in the container to port 8080 on the Docker host, so
555 external clients can access that port. Any other container connected to the
556 ``my-net`` network has access to all ports on the my-nginx container, and vice
561 $ docker create --name my-nginx \
566 To connect a running container to an existing user-defined bridge, use the
567 ``docker network connect`` command. The following command connects an
568 already-running ``my-nginx`` container to an already-existing ``my_ipv6_bridge``
573 $ docker network connect my_ipv6_bridge my-nginx
575 Now we have connected the IPv6-enabled network to ``mynginx`` conatiner. Let's
576 start and verify its IP Address:
581 CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
582 df1df6ed3efb alpine "ash" 4 hours ago Up 4 hours alpine1
583 ea76bd4694a8 registry:2 "/entrypoint.sh /e..." 9 months ago Up 4 months 0.0.0.0:4000->5000/tcp registry
585 The ``nginx:latest`` image is not runnung, so let's start and log into it.
589 $ docker images | grep latest
590 REPOSITORY TAG IMAGE ID CREATED SIZE
591 nginx latest 73acd1f0cfad 2 days ago 109MB
592 alpine latest 3fd9065eaf02 2 months ago 4.15MB
593 swaggerapi/swagger-ui latest e0b4f5dd40f9 4 months ago 23.6MB
594 ubuntu latest d355ed3537e9 8 months ago 119MB
595 hello-world latest 1815c82652c0 9 months ago 1.84kB
597 Now we do find the ``nginx`` and let`s run it
601 $ docker run -i -t nginx:latest /bin/bash
602 root@bc13944d22e1:/# ls
603 bin dev home lib64 mnt proc run srv tmp var
604 boot etc lib media opt root sbin sys usr
607 Open another terminal and check the networks and verify that IPv6 address is
608 listed on the container:
613 CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
614 bc13944d22e1 nginx:latest "/bin/bash" About a minute ago Up About a minute 80/tcp loving_hawking
615 df1df6ed3efb alpine "ash" 4 hours ago Up 4 hours alpine1
616 ea76bd4694a8 registry:2 "/entrypoint.sh /e..." 9 months ago Up 4 months 0.0.0.0:4000->5000/tcp registry
623 NETWORK ID NAME DRIVER SCOPE
624 b9e92f9a8390 bridge bridge local
625 74160ae686b9 host host local
626 898fbb0a0c83 my_bridge bridge local
627 052da268171c my_ipv6_bridge bridge local
628 57ac095fdaab none null local
631 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
632 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
633 inet 127.0.0.1/8 scope host lo
634 valid_lft forever preferred_lft forever
635 inet6 ::1/128 scope host
636 valid_lft forever preferred_lft forever
637 2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
638 link/ether 8c:dc:d4:6e:d5:4b brd ff:ff:ff:ff:ff:ff
639 inet 10.0.0.80/24 brd 10.0.0.255 scope global dynamic eno1
640 valid_lft 558367sec preferred_lft 558367sec
641 inet6 2601:647:4001:739c:b80a:6292:1786:b26/128 scope global dynamic
642 valid_lft 86398sec preferred_lft 86398sec
643 inet6 fe80::8edc:d4ff:fe6e:d54b/64 scope link
644 valid_lft forever preferred_lft forever
645 11: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
646 link/ether 02:42:4d:2f:ba:a6 brd ff:ff:ff:ff:ff:ff
647 inet 172.17.0.1/16 scope global docker0
648 valid_lft forever preferred_lft forever
649 inet6 fe80::42:4dff:fe2f:baa6/64 scope link
650 valid_lft forever preferred_lft forever
651 20: br-052da268171c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
652 link/ether 02:42:5e:19:55:0d brd ff:ff:ff:ff:ff:ff
653 inet 172.20.0.1/16 scope global br-052da268171c
654 valid_lft forever preferred_lft forever
655 inet6 2002:ac14::1/48 scope global
656 valid_lft forever preferred_lft forever
657 inet6 fe80::42:5eff:fe19:550d/64 scope link
658 valid_lft forever preferred_lft forever
659 inet6 fe80::1/64 scope link
660 valid_lft forever preferred_lft forever
662 Note that on the 20th entry we have the ``br-052da268171c`` with IPv6
663 ``inet6 2002:ac14::1/48`` scope global, which belongs to root@bc13944d22e1.
665 At this time we have been able to provide a simple Docker with IPv6 solution.
667 +++++++++++++++++++++++++++++++++++++++++++++++++
668 Disconnect a container from a user-defined bridge
669 +++++++++++++++++++++++++++++++++++++++++++++++++
671 If another route needs to be added to ``nginx``, you need to modify the routes:
675 # using ip route commands
678 default via 10.0.0.1 dev eno1 proto static metric 100
679 default via 10.0.0.1 dev wlan0 proto static metric 600
680 10.0.0.0/24 dev eno1 proto kernel scope link src 10.0.0.80
681 10.0.0.0/24 dev wlan0 proto kernel scope link src 10.0.0.38
682 10.0.0.0/24 dev eno1 proto kernel scope link src 10.0.0.80 metric 100
683 10.0.0.0/24 dev wlan0 proto kernel scope link src 10.0.0.38 metric 600
684 10.0.8.0/24 dev lxdbr0 proto kernel scope link src 10.0.8.1
685 169.254.0.0/16 dev lxdbr0 scope link metric 1000
686 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
687 172.18.0.0/16 dev br-898fbb0a0c83 proto kernel scope link src 172.18.0.1
688 172.20.0.0/16 dev br-052da268171c proto kernel scope link src 172.20.0.1
689 192.168.99.0/24 dev vboxnet1 proto kernel scope link src 192.168.99.1
691 If the routes are correctly updated you should be able to see ``nginx`` web
692 page on link ``http://172.20.0.0.1``
694 We now have completed the exercise.
696 To disconnect a running container from a user-defined bridge, use the
697 ``docker network disconnect`` command. The following command disconnects the
698 ``my-nginx`` container from the ``my-net`` network.
702 $ docker network disconnect my_ipv6_bridge my-nginx
704 The IPv6 Docker we used is for demo purpose only. For real production we need
705 to follow one of the IPv6 solutions we have come across.
707 ----------------------------
708 Challenges in Production Use
709 ----------------------------
711 The link "here" [10]_ discusses the details of the use of ``nftables`` which
712 is nextgen ``iptables``, and tries to build production worthy Docker for IPv6
719 .. [1] https://www.docker.com/community-edition#/download
720 .. [2] https://store.docker.com/editions/community/docker-ce-server-ubuntu
721 .. [3] https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-docker-ce-1
722 .. [4] https://docs.docker.com/network/network-tutorial-host/#other-networking-tutorials
723 .. [5] https://docs.docker.com/config/daemon/ipv6/
724 .. [6] https://docs.docker.com/network/
725 .. [7] https://networkengineering.stackexchange.com/questions/119/ipv6-address-space-layout-best-practices
726 .. [8] http://www.gestioip.net/cgi-bin/subnet_calculator.cgi
727 .. [9] https://docs.docker.com/network/bridge/#use-ipv6-with-the-default-bridge-network
728 .. [10] https://stephank.nl/p/2017-06-05-ipv6-on-production-docker.html