Network2
Introduction
Last updated: Michael Stone 05:11, 18 July 2009 (UTC)
This document proposes a design for networking based on Scott's Network Principles. It then explores and elaborates the design with analysis, example configuration, and experimental results after which it concludes by crediting those who have contributed to the design and by explaining future work inspired by current results.
- (If you've contributed and don't see your name, don't fret -- just add yourself with a word or two explaining your contribution!)
Some important quality criteria to consider while reading it include:
- primum, non nocere
- People usually think that sufferance of free software is voluntary but this is not so for our users.
- First, do no harm.
- no lock-in
- Does the success of the design depend on any ideas which lack pre-existing interoperating implementations?
- What existing software is it presently incompatible with? What does that cost to change?
- no ponies
- How well does the design conform to the physical and social realities which define its niche?
- For example: bandwidth, latency, error, ignorance, interdiction, authority, autonomy...
When judging, please also note that the design is not yet complete in several important respects:
- it has only a stub of a bandwidth model,
- its self-test algorithm is not yet written, (though good diagnostic primitives are systematically identified)
- it lacks truly clear implementation guidance and comprehensive sample code, and
- there are unresolved questions about
- how routing and timeouts should be configured so that peers search their target address space in a useful fashion
- how communications security might best be provided.
Design
Protocols
We take IPv6 and DNS as the basis of our system and rely heavily the Linux Documentation Project's IPv6 documentation.
A couple of key facts and expectations inform the rest of this design, like:
- hosts will have multiple interfaces,
- interfaces will have multiple addresses,
- DNS queries (used via getaddrinfo()) will return multiple results
- these results will be sorted in a sane order, and
- hosts will choose routes for packets based on how specifically the routes match the destination and on any QoS information available to the routing node.
Peer IPv6 Configuration
Your job is to be an IPv6 node. Consequently, when you bring up your interfaces,
- You might discover an IPv6 router advertising on one of your links.
- (See sysctl net.ipv6.conf.all.accept_ra and related variables.)
- You might try out dhcp6c.
- You might have some kind of IPv4 connectivity. If so, connect to the Internet or to other internetworks of your choice.
- Use dnshash to add guessable link-local addresses to all your links.
Server IPv6 Configuration
Your job is to be an IPv6 router and a DNS server. One of several situations might obtain:
- You might discover an IPv6 router advertising one or more IPv6 prefixes on your outbound link(s).
- You might have some kind of IPv4 connectivity. If so, connect to the Internet or to other internetworks of your choice.
- You might be under a tree. If so, generate a Unique Local Address prefix.
- (Use dnshash to add guessable link-local addresses to all your links?)
When done, use radvd or dhcp6d to share addresses.
Server DNS Configuration
One of the server's most important jobs is to get itself on appropriate internetworks so that it can dynamically map stable (DNS) names to unstable names (IPv6 addresses) for itself and its peers.
Unfortunately, the most reliable and secure means of updating these mappings is likely to be bespoke -- RFC 2136 is not widely implemented and specifies no concrete security protocol while DNSSEC seems immature at present.
Consequently, I propose the following strawman update protocol -- exchange an RFC-2136 UPDATE packet and response over your favorite authenticated RPC protocol with the nameserver.
(My favorite protocol for this sort of thing is currently "json-over-SSH-to-python-and-make", but variations (ucspi-ssl, 9p, etc.) make me smile.)
(Other possibilities: maybe DNSSEC isn't so hard? Maybe DNSCurve will be usable? See ipcheck and ddclient for contemporary work...)
Peer DNS Configuration
Peers which have been registered with one or more servers need to update those servers when their addresses change using the protocol described above.
Security Ideas
- Spoofing, Integrity, Confidentiality. See communications security and petnames for some background. A very rough road along which something reasonable might lie:
- Use physical introduction to CNAME cscott.michael.laptop.org to <key>.cscott.laptop.org.
- Then, my dnscurve-compatible DNS resolver will refuse to give me addresses unless the nameserver I contact for cscott proves knowledge of cscott's private key.
- Then I have a nice basis with which to configure IPsec security associations.
- System Integrity
- DoS
Analysis
Bandwidth Usage
Several important numbers that we need to predict and to measure:
tx == transmit, rx == receive, btx == broadcast btx/tx/rx - ICMPv6+IPv6+phys - router discovery (RD) btx/rx - ICMPv6+IPv6+phys - duplicate address detection (DAD) tx/rx - ICMPv6+IPv6+phys - NS neighbor discovery (ND) tx/rx - UDP+IPv6+phys - DNS query tx/rx - JSON+SSH+TCP+IPv6+phys - DNS update where "phys" describes the equations' dependence on the "physical" layer's frame overhead and MTU notable "phys" layers: Ethernet -- ad-hoc wifi, infra wifi, 802.11s mesh, switch, hub TLS+UDP+IPv4 -- openvpn L2TP+IPsec+IPv4 -- raccoon, isakmpd, openswan, etc. UDP+IPv4 -- teredo
Debugging Techniques
Start recording a typescript so that we can see what you did.
TESTDIR=`pwd`/testing mkdir -p $TESTDIR && cd TESTDIR script ulimit -c unlimited
Check that you've got the right DNS name for the person you want to talk to.
NAME=the.right.person echo $NAME > peer
Dump your addresses, routes, and perhaps your open connections.
hostname --fqdn | tee host ip addr show | tee addrs ip route show | tee ipv4_routes ip -6 route show | tee ipv6_routes netstat -anp | tee conns
If you have wireless devices,
iwconfig | tee iwconfig iwlist scan | tee iwlist_scan
Fire up tcpdump:
tcpdump -w packets -s0 &
Resolve that name to addresses. Check that the addresses seem sane.
dnshash lookup $NAME | tee peer_addrs_dnshash dig $NAME | tee peer_addrs_dig
See who's answering broadcasts:
ping6 -I $IFACE ff02::1
Route to the addresses:
ping6 -I $IFACE $ADDR | tee ping traceroute6 $ADDR | tee traceroute tracepath6 $ADDR | tee tracepath
Connect to the address:
nc6 $ADDR $PORT # echo "SSH-2.0-Hi" | nc6 $ADDR 22 # printf "GET / HTTP/1.0\r\n\r\n" | nc6 $ADDR 80 # ssh $ADDR # curl -I http://$ADDR/ # ...
Conduct a bandwidth test:
iperf -c -V $ADDR
Collect logs from your application and send them to developers:
kill -SIGINT %1 cd .. tar c $TESTDIR | lzma -c > logs.tar.lzma
Self-Test Algorithm
As we gain experience with the system, we'll write a decision-list here which inspects the output of the diagnostic procedures listed above and which identifies the proximate cause of networking failure based on those results.
Advice for Coders
There are two critical changes that you'll need to make to your design in order to really make it sing.
First, you'll want to add some mechanism for your users to type in hostnames that they want you to connect to. This lets them do all sorts of cool stuff like:
- copy-and-paste links from websites or cerebro
- type in names from a physical display like a blackboard or a handout,
Second, you'll want to be prepared to re-resolve names in order to get fresh addresses each time your connectivity changes. For the time being, you should do this by calling libc's getaddrinfo() function.
Third, go check out SCTP (wikipedia, man page). It's support for multi-homing, multi-streaming with and without ordering guarantees, and for updating the addresses you're using to talk to your peer on the fly seem particularly serendipitous.
Advice for Deployers
Ask your ISPs to provide IPv6 prefixes or tunnel endpoints. After all -- if none of their customers ask, then what incentive will they ever have to upgrade?
Failing that, see if you (or a local university?) can afford a public IPv4 address -- even if it's dynamic. If so, you can be many sorts of tunnel endpoint.
Regardless, if you manage to get a globally reachable IPv6 address by any means, then you can provide a DNS server for your kids and it can direct them to one another and to any other services that you feel like pointing them at.
Experiments
Link-local configuration
Try out dnshash on an isolated access point, ad-hoc network, switch, or hub.
Observations: very pleasant!
VPN server configuration
In this experiment, we're going to configure openvpn and radvd on a machine (teach.laptop.org) with a public IPv4 address. Truthfully, this combination is probably overkill, but the task of constructing it seemed like it might to offer valuable experience, e.g. for someone who wants to bridge multiple kinds of tunnel endpoint or who wants to load-balance lots of peers between a couple of endpoints.
# Install our VPN and route advertisement software.
apt-get install openvpn radvd
# yum -y install openvpn radvd
# add nobody:nobody
groupadd nobody
useradd nobody
usermod -a -G nobody nobody
# Configure radvd
cat > /etc/radvd.conf <<EOF
interface tap0
{
AdvSendAdvert on;
MinRtrAdvInterval 30;
MaxRtrAdvInterval 100;
prefix 1234:db8:1:0::/64
{
AdvOnLink on;
};
};
EOF
# enable forwarding everywhere
sysctl -w net.ipv6.conf.all.forwarding=1
# flush the forwarding table
ip6tables -F FORWARD
# really, I /want/ a multi-user version of
# openvpn --dev tap --user nobody --group nobody --verb 6
# but I'm not sure how to get that. instead, I'll use some fake keys and no ciphers.
mkdir -P keys && cd keys
wget http://teach.laptop.org/~mstone/sample-keys.tar.bz2
tar xf sample-keys.tar.bz2 && cd sample-keys
# create a multi-user tunnel
openvpn --mode server --client-to-client --dev tap --user nobody --group nobody --verb 6 --opt-verify --tls-server --client-connect /bin/true --auth-user-pass-optional --duplicate-cn --auth-user-pass-verify /bin/true via-env --dh ./dh1024.pem --ca ./ca.crt --cert client.crt --key client.key --script-security 3 --auth none --cipher none &
# at any rate, bring up the interface so that we get link-local addresses
ip link set tap0 up
# turn on the route advertisement daemon
radvd -d 5 -m stderr &
VPN client configuration
The purpose of this experiment was to test the VPN configuration described immediately above.
# install vpn client apt-get install openvpn # yum -y install openvpn # add nobody:nobody groupadd nobody useradd nobody usermod -a -G nobody nobody # download fake keys. mkdir -P keys && cd keys wget http://teach.laptop.org/~mstone/sample-keys.tar.bz2 tar xf sample-keys.tar.bz2 && cd sample-keys # connect to the vpn openvpn --user nobody --group nobody --dev tap --remote teach.laptop.org --tls-client --ca ca.crt --cert ./client.crt --key client.key --auth none --cipher none & # bring up the interface ip link set tap0 up # find other people ping6 -I tap0 ff02::1 # if using dnshash, attach dnshash attach <your>.<domain>.<name> # ... test, as described above ...
Observations:
- TLS imposes a high latency cost, even with null algorithms.
- TAP devices work rather nicely.
- Be careful of firewall rules!
- radvd is perhaps unnecessary with a single virtual ethernet -- dnshash "suffices" -- though it might be useful for routing between several load-balanced ethernets.
- The default IP sorting rules and route priorities mean that it may take a long time for a connecting app like ssh or nc6 to connect to the /correct/ dnshash address.
Credits
- Michael Stone [none] (writing)
- C. Scott Ananian [OLPC] (architecture,teaching)
- John Watlington [OLPC] (architecture)
- Robert McQueen [Collabora] (prior work,critique)
- Dafydd Harries [Collabora] (prior work,critique)
- Polychronis Ypodimatopolous [MIT] (prior work,critique)
- Cortland Setlow [Tower Research Capital] (testing)
- Andres Ambrois [] (design,testing)
- Benjamin Schwartz [Harvard] (critique,publicity)
- Tabitha Roder [] (testing)
Future Work
- Per-host networks and per-app IPs and names.
- Sample code.
- Designs for higher protocols like discovery, presence, and health.