These are unedited transcripts and may contain errors.
Plenary session
14 October, 2013, at 4 p.m.:
BENNO OVEREINDER: Welcome for the second part of the plenary session on Monday. This is for the BCOP, can I have a different slide set? This is for the BoF this afternoon, also and the lightning talk. For the, this session, we have two plenary speakers and three lightning talks, University of Princeton University, I have to say correctly, he will give a presentation on SGN in the Internet Exchange. Software defined networks, people say a lot about it, they either like it or or not, they say it's already around for years. Others say well, it's really new and hot. I think it definitely solves some interesting problems and Laurent will present one of these. The floor is yours:
LAURENT VANBEVER: Most of you will know how to operate a BGP network. I would like you to think for one minute about all the problems and the frustration that you have when you do so. So maybe you want to configure in bound traffic so you have ten routers, you have a couple of 100s of prefixes and you have requirements that forces you to configure and the pro prefix basis. This will be really painful to configure, and it will be really slow because the configuration will be insanely big and you will have problem adapting your network to the dynamics of the traffic. But maybe you have other problems like convergence, you realise one of the peering dies in your network, it takes ten minutes for the network to converge, and obviously customers not very happy about that. Maybe yet another, you have a peer setting up a different route and putting it to your interface and using you as a free transit and to detect that or prevent that from happening in the first place. In this presentation, I will explain you how deploying SDN at Internet Exchange point can actually alleviate some of these problems but also enable new inter?domain applications.
So I would like to start with a comparison between BGP and SDN so it will be rough along three axis. First, forward paradigm. Forwarding control, how do you push something in the FIB when you are using BGP RSDN, and where do you push that, so where do you have something to say so where do you influence the traffic?
So BGP, it's pretty easy, destination based, are indirect so in BGP I want to send traffic on the right, I have to configure BGP in such a way it will compute the pass on the right. So I know the pass but I need to figure out what is the configuration that will lead me there. So it's an indirect way to programme the table. And the influences is only local, I need a BGP session to do something.
With SDN you are not tied to a destination based forwarding any more, you can do it if you want but you are not forced to do it. The forwarding control is direct, you want to send traffic on the right you push that in the FIB and you are using open API to do that, in this case I assume OpenFlow but any can give you that. The control can be global, if you want to do something remotely, if you are not physically connected to an ISP in this case you can still have an impact by controlling the control over an interface.
So why XIP? We think are a really good place to deploy new inter?domain features, large number of participants, in AMS?IX more than 600 of them, carry an insane amount of traffic, is flowing in AMS?IX during peak hour and really a hot bed of innovation so all the work about route server has served out of the AMS?IX community. So bottom line is even a single deployment can have a large impact. So what we do is we take an XIP and we document it with SDN capabilities and that will enable us to have fine grain inter?domain policies and of course, we know that XIP are really, well, critical point in the Internet hierarchy so we need to be very cautious about what we are doing there and we want the platform to scale so we need to be able to deal with millions of routes in the control plane so we keep that in mind.
It's useful for you to know what is hard or impossible to do with BGP and SDN can give you. I have only listed here a couple of examples, there are more, and I have classified the examples in four different classes.
So the first one is security. So what you can do with an SDX is prevent participation communication in easier man, you can say two ports will never be able to exchange any packet and do not use VLAN or configuration to do that. It's built in in the switch. You can prevent or block policy violation. Forwarding optimisation, traffic steering, you can steer the traffic for a single flow towards a /32, even TCP port number and you can bring that traffic to a middle box or any port so it's very fine grained traffic steering. Traffic offloading, for instance, is when you have traffic from A to C that goes via B, all of them are connected to the XIP, the traffic will cross multiple times the XIP so with an SDX we can limit or completely remove forewarning. Yet another example are peering, for instance, we can do applications specific peering, can peer with Netflix or remote control. If you are CDN you can announce your prefix from the ISP, all the traffic will arrive there and you can do your magic using the forwarding capabilities of SDN.
So the talk will be divided in three parts, first I will present the architecture in the control plane and data plane, inbound traffic engineering, in easy manner, I challenge you to do that with BGP and fast convergence. So with SDN you can converge in less than one second. With any router, even old one. And I will say how to do that.
So let's start with the architecture. So I suppose that everyone here knows what an XIP is but it is a large Layer 2 domain where participants come and bring their router and they connect the router to a switch and then they establish A BGP sessions in order to accentuate ability and information and you have two types of sessions, private session directly between edge router and session to potentially a route server in order to reduce the amount of sessions that an operator needs to unfortunately manually configure in most of the cases.
And what is very important is that the IP traffic is flowing directly between the routers and not going through the route server in this case so the route server is an off past entity completely contra plane entity. So with respect to that, in SDX, we require you to have an SDN enabled forwarding plane so in this case I am assuming OpenFlow again, and you can imagine others. We need a controller so we take a route server and we document it with SDN capabilities and what we call SDX controller.
So, once we have this, participants can know right forwarding rules directly in the data plane but I don't know if you guys have ever played with OpenFlow but it's a very, very low level API so we don't want the participants to directly manipulate rules because they will make mistakes and it will never work. Instead of that we give the participants the ability to right forwarding policies using eye level language so the participants will use this language to write the policy. For any of you have written Cisco or Juniper, it is pretty basic, some action on the track that has been matched. You can match on a lot of fields so on router you can cannot match on Layer 2 fields because it works on layer 3. Here you can. You can work in conjunction of fields so it's pretty powerful. What you can do is either drop that traffic, forward the traffic or rewrite anything in the header as well. For instance, what I can do is I can match all the traffic that is going to one of Google's prefixes, that is coming from one source, I can rewrite something and can forward to another port. So I can write these kind of policies easily.
So, in the SDX every participants will know they don't have to communicate these policies between themselves obviously and then they will send these to the SDX controller and you can think of the language as being C, for instance, and of the SDX controller as being the compiler which will compile down all these policies and will basically output OpenFlow role in the switch at the end of the day.
Obviously building such a platform is very challenging but I want to give you an insight on why it is possible to do so?so I will describe four challenges and the solution that we have come up.
First one authentication, you can give remote access to the existing controller. Obviously, we don't want anyone in the world to be able to do something on Google traffic, for instance, so if you want to put a policy you have to prove me that you own the prefixes for which you want to push a policy on and here what we do is we are leveraging the RPKI system which is currently being deployed in order to secure BGP.
How prevent phishing, rewrote the source McI can pretend traffic is coming from another participants and maybe I will not get billed if I do that, obviously don't want to do that. We are using basic stuff, access lists which will limit what each participant is able to do. If you write or rewrite source Mc, only for your own and not for any other value.
So do we avoid clashes between different policy. It's likely that different participants in the XIP will write policies on the same prefix for instance. So here we are using the idea of having a virtual topology so each participant will have a different view of the XIP, it's personal view and he will write this policies on top of this virtual view and in order to compose policies that are on different participants we follow the forwarding paths so for instance if, A is forwarding traffic to B and then B is forwarding traffic to C, what we do is that we will take the policy of A, sequentially compose that with the policy of B and with the result of policy C, so we follow the path of the traffic.
Scalability: Big issue, how do we manage millions of forwarding entries, OpenFlow switch state?of?the?art, 100s of thousands of ?? after it stopped. So we need obviously to do something here. And here, the idea is that we will leverage actually the routers, so in an XIP you have routers and then you have a switch. So what we do is we use the routers in order to help us, so the routers will take care of all the work for IP prefixes. And then so this part of the policies that match on IP prefixes will be made in the router and we can do that even using ?? BGP. And then in the switch we will only do the part of the policies that is static, that really requires a switch to do that. So by combining routers and the switch you can scale to millions of entries. I am being a bit high level here, I don't have the time to enter into details, please come and talk to me and I will explain you better.
Let's look at two applications that you can use. First one is inbound traffic in generic. So let's consider the simple case. I have an XIP on your left, that is composing of four routers, two of B and one of A and one of C. And the BGP topology is on your right and is basically B is the provider of A and C. B is owning two prefixes, one /24, two /24 is giving that to ?? so, nothing really fancy here. What is the problem is that what if you want to implement this inbound policy of B. Let's say you are using car done of this is the output of the tool. The tool is telling you you need to send all the one /24 traffic coming from A on the left, the/ ?? the dot 2/traffic coming from C on the right, so the third line in which you have to send the traffic going to two /24 coming from AT&T prefix on the right. I challenge to you do that with BGP. So it's pretty challenging.
So why is that? Well BGP provides you with ?? observe I say inference decision. And that ?? implementing that policy will be configuration intensive in any case, play with the Net, call the other guy and if I tag the route with that BGP please increase the local pref, you will have to do this kind of stuff. It's even impossible to implement all the requirements because in BGP as I said this BGP is destination based, here requirements that forces me to define policy based on the source address so I want the other guy, the one I am sending the route to, to change based on source address. I cannot do that.
So, in any case the outcome sun predictable, there is absolutely no guarantee that the remote guy will comply, will not listen or care about your AS path, so you have no guarantee that it will work. And so if you are an engineer you have no choice but to try and pray that it will give you the right results which makes it impossible you to adapt the traffic part.
The nice thing about having and not ability to, if you think about it, traffic engineering policy it's forwarding policy, it's not a routing policy in this case. So here there is really a one?to?one mapping between the forwarding ?? inbound policy on the left and the SDX policy on the right, so B will simply write the few lines of code on the right and will give that to the SDX controller and the policy will be applied and you can change that every second if it wants to because there does not need to have any route exchange in BGP for that to apply.
Fast convergence. How do we enable peering convergence after failure?
As you probably know or have experienced, BGP is pretty slow to converge. Let's consider this simple case: I have four routers, the two on the left are owned by A and the two on right are owned by B. And B is the provider of A. So B is giving a full routing table to A. And the requirement is that B2 is a backup router so it can only be used if B1 is dead, otherwise B2 should not be used and you have four BGP sessions between the routers.
So, the router exchange the route, A1 and 2 will run the BGP decision process so you will end up with a forwarding table in A1 and 2 will look like that, so 500,000 prefixes, entries with all the ?? pointing to B1. So obviously what happens is that B1 dies, so what the router has to do now is update it's firing table entry by entry, one by one and set B1 to B2 for 500K entries. So people have measured the time it takes to update and what they realise, 150 microseconds if you multiply this by 500,000 entries, you will end up with 75 seconds. So in the best case it will take the router 75 seconds to update all his entries.
So how do we do sub second convergence on the SDX, where the controller is acting as route server. So the route server will receive two times 500K routes. And A will tell it I want B1 as primary router and 2 as backup. So there is the controller will act as route server and will send 500K routes to A 1 and A 2, they all point to B 16789 in the initial case nothing has changed with respect to B before. Now B1 dies. So what can happen ?? what will happen is that SDX controller will realise B1 is dead and as soon as it realises this, it will push this in the data plane. So what this two rule says is all the traffic that is coming from A1 and A2 and that is going to B1, will rewrite the destination MAC and we immediate send that to B2. I have created shortcut, all the traffic is directly sent to B 2, I only need to update two entries and I have upgraded one million forwarding pass with two entries.
So, here, basically what is convergence time, well the number of edge routers that you have, here I have two so it's two, times 150 microseconds this does not change with SDX, plus the time it will take for the controller to realise there is a failure and push two rules. Here I am being cautious and saying it will take about 30 and 50 milliseconds. I am expecting that number to be really slower.
?? faster, sorry.
So the total convergence now it's 30 to 50 milliseconds. So I have converged my two routers in 30 milliseconds. The nice thing it works for most of the peering links out there, if you look for instance at AMS?IX, most of the participants have two interface. It does not interfere at all with the participants' policy, even if you don't want the language, you can already do that and will already give a new service to your customer. What I like it does not require any other change. If you have all 7,200 Cisco router you can still converge as fast, without being the price of ?? it's pretty cool, I think.
So, to conclude, I would like to point out that we do not only provide additional services to the participants that are connected to SDX platform we can help the XIP operator themselves, also these guys have some issues, relating to broadcast traffic or unwanted traffic, so what you can do with SDN you will add rows in the switch that capture all that traffic and send everything to the controller where you will most likely drop everything there which means you are no longer dependent on the configuration of the port to protect you, if one guy is pretending the ?? it will not work any more and will not create the problem we all know of.
You can also enable fine grain traffic engineering, load balancing, you are not tied any more to Layer 2 or 3 technologies, you can do anything you want as long as you know the forwarding path, just in general improper the infrastructure operation, get rid of spanning tree, without requiring VLANs, you are not limited by the A VLANs. So it's not entirely a science fiction talk, we have running code, first deployment site, so the prototype is currently implemented on top of pie on this. We have partner with a large regional in Atlanta in US so we are mostly doing this in the US we know the XIP committee is way more lively here in Europe so we are interested in extending our reach in trying to do something with the XIP here in Europe. So if you are an XIP or interested even to try this in the lab so you don't have to try this on production network but you are interested in the technology, just please come and talk to me. We are also open to peering requests, if you have peering points in the US close to Atlanta, it might be interesting for to you peer with us. We also have a survey on?line, so if you download my slides and click on this link. Here we are interested in knowing a bit more about what is the problems of the XIP but also of the participants of the XIP. It does not cost you much to tell us your problem and maybe we will solve them for free, so I don't think you have much to lose here.
I will finish here and be happy to take questions if you have ?? thank you.
(Applause)
SHANE KERR: I have a question. So, what do you view as a migration strategy if an exchange point wanted to deploy such a technology because it requires participation of the members as well as just providing the infrastructure?
LAURENT VANBEVER: Yes, you are right. So, I think we can end partial deployment for some application, for the fast convergence application as long as your two interfaces with connected to the switch you can beneficiate from that service. You will need at least two participants to be connected to the SDN platform but for some applications not true. I would say it's on case by case basis but what you must know is that OpenFlow switch also can be included in Layer 2 domain, can act as normal switch so you can partially deploy this switch in your architecture and then move participants towards these switches and provide these services on a gradual basis.
SHANE KERR: . Is that the approach that you are taking with your existing deployment in Atlanta?
LAURENT VANBEVER: No, we are doing another network next to the deployment.
SHANE KERR: OK. Thank you.
BENNO OVEREINDER: Any other questions?
Mike Hughes: I have run exchanges before, one of the big things that came in my mind was the level of fine grain control, and changing the default behaviour. One of the things that we are very aware of at the moment is privacy concerns. Does not ring alarm bells for you that this makes it very, very easy for people that believe in systemic monitoring to turn up at an exchange and give them another tool? Is my question clear enough for you? Van yes, I think so, you are right that it can ?? it simplifies this type of monitoring, you are right. I would say it's still possible to do it even today, so even if you don't have an SDN I can still monitor the traffic so you are right that we give yourself an extra weapon to monitor the traffic there. So I mean, I cannot say more about that. Yes, you are right, as long as you can push a rule that send part of your traffic to middle box and bring back to another port, so if I don't trust the XIP to not do that, then I would not use this type of services. With respect to ?? for the people who believes that the XIP now is influencing the past somehow, it's really the participants policies that are being given to the SDX controller and if you think about it today, when you are using a route server, you already give your policy to the route server as well using this BGP community, that says I don't want this to be announced to that guy, I want to two times to this participants, you are in a sense giving your policies as well, in a policy point of view I don't think we are worse today. For the monitoring you have a point.
Mike Hughes: Even the control of third parties over the policy /SW?RBLGS so could turn up and say hello, I am man in a nice dark suit with this warrant and I want you to go and install these policies that make this happen on the SDN controller?
LAURENT VANBEVER: Yes.
Mike Hughes: The XIP is then interfering in the participants' policies and track and that makes the XIP in the eyes of the participants untrustworthy, some of this is good and some dangerous.
LAURENT VANBEVER: OK. I will be happy to discuss further on this /SH*UFPLT but yes, you are right, my point would be if you have a warrant today I am pretty sure they will do it anyway.
Mike Hughes: Exactly it makes it easier. What you find is things like taps and that you tend to know about and people turn up and do them actually at the IXP, particularly in something that so far I don't know about people in this room but if you are connected to an XIP at the moment, raise your hands if you implicitly trust that XIP to deliver your traffic. One person only implicitly trusts it. Let me rephrase the question: If you are connected to an Internet Exchange, do you implicitly trust that exchange to deliver a packet that you inject into it? Wow, no confidence in Internet Exchanges then. Brilliant. Why aren't people raising their hands? It's a switch. You deserve to lose, cheers.
LAURENT VANBEVER: I agree with the comment. I think it's ?? I mean, the argument is valid but it's too bad that because it makes the network easier to manage, that we don't want to do that because if we have this, then it will be easier for someone with a warrant to do that as well. So you have to think about your day?to?day life as an operator, so, I agree there are issues but I would not want to hamper the innovation in management just on that basis. So that would be my argument.
BENNO OVEREINDER: Three people at the mic, I want to close the mic. So go ahead.
AUDIENCE SPEAKER: Just the same bell I want to ring. With this technique it's not only to possible to do passive measurement but active man in the middle attacks against participants, you can send to middle box and we already have discussions in Germany with G 10 loss that the v6 is not allowed to talk about what the secret services are doing there. So this will just have a huge impact.
LAURENT VANBEVER: You are right. But such as BGP today you can inject something in BGP and if you don't have, make ?? the route will go through. As I mentioned, we are using RPKI techniques to validate the policies before we push them in the switch.
AUDIENCE SPEAKER: It's not about ?? they can be forced to do it and with BGP you see it, with this technique you don't see it, there are hidden in the background ??
LAURENT VANBEVER: Not all the policy will be seen, some of them will be. Because you go through a route server, so not everything is done in the data plane.
AUDIENCE SPEAKER: Still it's not TCP part specific or something like that.
LAURENT VANBEVER: Yes.
AUDIENCE SPEAKER: From the Amsterdam Internet Exchange, so first of all thank you for the good words and presentation. I would like to ask from the XIP side of things how do you think ?? how much would, would it be scapable for a controller to forward rules for 700 customers and 70,000 routes, for example?
LAURENT VANBEVER: Yes, I do. So, in an XIP all the rules will be proactive, the OpenFlow and the scale that comes with OpenFlow, you have this packet and the concern needs to react based on the packet. Here it's like the Google network, it's proactive push, so the XIP will proactively push the rule but never to the controller so that problem goes away. You can scale, if you look at can deal with millions of flows per second, if your data plane is big enough. I think there are ways to scale. And anyway, logically centralised does not mean physically centralised, a controller that is composed of ten ?? in order to scale if you are concerned about this. My answer would be that yes and anyway on a single switch, you will maybe have 48 policy points, 96, something like that. So I don't expect on one switch to have 10,000 of entries to be pushed, and again, I want to point out that we are using the routers as well so prefixes will be maintained on the routers and on the switch actually you do not need to maintain prefixes.
AUDIENCE SPEAKER: Have you tested the kind of scalability?
LAURENT VANBEVER: We are currently testing it. I don't have any numbers to give you. If you are interested, we can definitely test it together, I would be happy to know your requirements, for instance, and design of the network, to give you numbers.
AUDIENCE SPEAKER: Thanks.
RUEDIGER VOLK: Deutsche Telekom. I would like, first, to comment towards Mike's question; let me say I trust back to back connection, single circuit to appear, definitely more than any IXP, and we are not seeing this or that kind of underlying that, well, OK, sometimes there are problems when you inject a moderately complex system into the interconnect, and I certainly would trust an SDN based IXP much less than the traditional simple ones today. And Laurent, for the claim that well, OK, operational simplicity actually is an argument, it will be pretty hard to convince me that the rules injected by multiple parties into a common switching fabric are not going to interfere in ways that really result in something that can be considered operationally simple.
LAURENT VANBEVER: OK. So two points: First one, by default, the SDX does not do anything else than the Layer 2 switch and the controller is just a plain route server so if you don't want to use this technology you are not forced to use that. One XIP deployed that and you don't want this, you say I don't want to use that feature, don't activate it, ever. If you are really concerned and that comes to the first question, you can imagine that you will have some devices that are SDN based and others that are not. If you don't want it, you will ask to be connected to the non?SDN device so that you are sure that no one can push any rule in that device. I would say if you are really concerned about this, don't use it but we have ways. For the ones that aren't interested they can use it on the side for the participants ?? with the participants that are also interested. So the default is to be Layer 2 switch and the route server, plain BGP, nothing fancy there.
BENNO OVEREINDER: Thank you for this good discussion.
LAURENT VANBEVER: Thank you.
(Applause)
BENNO OVEREINDER: So, next speaker is Georgios Smaragdakis, talking about enabling ISP and CDN collaboration.
GEORGIOS SMARAGDAKIS: Thank you very much, involving researches from Deutsche Telekom labs in Berlin and doesn't of course represent the official opinion about any of the parties.
Before coming into the technical details about the presentation I want to give the new setting of the, we are operating in. So today there are a lot of big and smaller CDNs, huge amount of traffic that peer in different points in the Internet with other ISPs. They can use ISPs or multiple presence of of course this creates a lot of problems. In order to monitor traffic that flows within your network. In the good days over provision buy more bandwidth and create new peering points but as this becomes more costly and everybody tries to reduce the operational costs it's not any more the solution.
The other thing we have in the toolbox is traffic engineer. You are adjust your route in a way that you better influence the traffic that flows within your network. There are two obstacles here: One is this process is very slow, as the previous speaker mentioned. And can also create escalations and to make things worse for ISPs it's very difficult to predict whereby the next traffic, nobody predicted Facebook or NetFlix, now approximately 30% of the traffic in the US networks. When you try to solve a problem you create another one. The operational people tell you we increase the traffic in a peering point for 10 gigabit to 100 and it takes 30 to 40 minutes to actually reach the same consistency because applications will try to find bandwidth.
Today we have the problem that the traditional engineering doesn't work, if you see it from the perspective of ISP an ISPs have lost the control of their own traffic. And then you see all this in the popular place where the COs of main companies go to Google and say we are losing revenues here, so we have to negotiate. The problem is you go to them and say of course we have to negotiate but we offer you free mail server ?? service, which is gmail, free video, which is YouTube, then probably you have to pay us and at this point ISPs have a problem to negotiate, so why I am signature here and discuss about the collaboration, if I have nothing to offer and have no problems.
Well the problems that CDN have problems. You have to go back to the initial architectureual ?? so a client wants to download the content goes first to DNS and going to the DNS of the CDN and says I want to get this object. So the CDN decides which server will deliver the content to the user. The first problem is DNS does not know where is the user. Because the delegation doesn't work. Secondly, there is so much in the network the DNS has no way to find out active measurements where the user S the second is the CDN operates its own ?? and most of the times they make wrong decisions, that is because they send traffic through ?? the third thing is that ?? the cost reduction for ISPs also for CDNs, cannot continue deploying hundreds of thousands of servers any more.
What we want for this collaboration ?? and we try to take advantage of diversity that exists in the network, not only in terms of path but also in terms of server capacity and also we would like to find a way that ISP can communicate with the CDN and can community with the ISP without ??
So what we call this approach content aware traffic engineering and it has some requirements. The first is has to be an on?line process, we don't rely on BGP and convergence, and we don't need any reconfiguration of the network, we don't want to put more boxes and more complexity and has to be transparent to the user so doesn't have down loud something in order to activate this application.
Throughout the talk I am going to go through the measurements that we did in operational network as well as in CDNs and I am going to very briefly highlight what is the other theory that makes all these scheme works and I am going to describe the development and the architecture of a prototype that we use in order to establish this collaboration and I will conclude with some results with operational network.
So, what we did is we used as I described before and we monitor close to the DNS the traffic that goes to the CDNs. In order to do this, we do rely only on what is the website as we know that most ?? in many websites, the content comes from different servers. So we did various techniques in order to map the IP of the server to the CDN, and I am not going to go into the details, I give here all the links in order to find out how we do this. Once you have this you go and monitor how much traffic flows within the network and originates from CDNs. So if you can see more than 10 percent of the traffic in the original from one CDN, this has increased by the way. If you consider the top ten CDNs, this can increase to 25% and if you go higher you can reach easily 40 or 60% today that flows from a type of CDN. Something had a we also saw is more than 40% of the traffic that is actually delivered at least in networks can be delivered from at least three different locations in the Internet, which means that we pretty much have the flexibility to choose what is the source of the traffic that flows within our network and we don't really do that now.
And if you focus only on the big CDNs you see because they have so much service out there locating so many networks then the content that flows to your customers can come from anywhere between 10 and 50 locations, including locations within your network. This is actually a diversity that hasn't been useilised by ISPs, to some extent by CDNs but problems I mentioned before. What we want to do a better server selection by using the DNS infrastructure of an ISP, most of the ISP still take 95?plus% of the queries and also we want to find a way that the ISPs and the CDN can negotiate how to operate a CDN within the network or how the traffic flows within their own network. And probably this will also reduce the need to actually increase the capacity in parts of the network.
So, what is a ?? out of the box for 20 years we think that the traffic engineering is the problem that your routing metrics in a way that you have a satisfactory traffic that flows within links, why here? So what we say is OK, let's fix the routing as it is now and has to do with policies and instead let's focus on how the traffic flows from the server to the user. Of ?? we can say the other part of the flow.
Now if do you this, you say OK, how much traffic I can influence for the CDNs that I can collaborate with. I fix all the other traffic, as background traffic, and then pretty much I try to find what is optimal way to rebalance the traffic that flows within my network by taking different servers.
And in a way just to illustrate this, if you have two CDNs, the green and red, right now you make the traffic from host ?? serve B and C but then you can actually rebalance this and then reduce the traffic that flows to specifically within your network by just selecting a different server.
I am not going to go into the details about why this works, but you have to believe me that this can reduce to on?line balancing problem that is very well studied in computer science and it shows even a simpler ?? can work. If you want to see how this will work in your network even before deploying it then then to do this even with linear programming formulation.
How we can realise this in operational network. What we need, except from the least of the servers that can deliver the content is pretty much an intelligent way to collect all the information that is now available in ISPs that includes the using lowerication, from radio servers, the capacity of the users and of the links and other information that actually flows within the network control plane.
Once we have this, then we know that if we have for an object we don't download it from one server but any other server in the CDN so having this ?? taking this into account, we can have a list of the potential servers that we can download the content, give this to what we call provider aided information system and then this machine writes based on some criteria. And of course, it is true that the DNS goes again back to the user as ?? so in this cartoon you can see instead of selecting host one, we can select host two which is within the network and giving better formance. If we /TWAOPBLT enable the /KHRAEUB ration with the CDN we don't have to change anything, we follow all the architecture that is in place now, so the provider, DNS will request the CDN authoritative DNS server about a server, then there is a private channel between the CDN and, where they negotiate how they are going to solve this problem. This can be done by request, because it's quite scaleable or it can be done frequently, for example, based on TTL. And now the CDN decides which server for its own reasons, are capable to serve the content. It gives this to ?? then they have a contract to say, OK, this is the stable matching problem that we are solving, for example we have to decide the first ?? the best for both of us and then the CDN is responsible to give back to the DNS to give it back to the user, so stays on the CDN side.
So let's see if we put all this together and we try in operational network what we get. So we consider for now the ten biggest CDNs, as I mentioned for more than 30 or 40%, we have try criteria. Of course any combination can be held here. First is utilisation, so we want to minimise the maximum utilisation ?? this is a bottleneck for everything that happens within the network or at least with the paths. Second is delay, want to minimise the delay between the user and the server. And the third thing is I just want to take the closest path.
So what we realised is that if you have 30 or 40% of the traffic that you can actually rebalance then you have significant improvements in many criteria. So if we just focus in the case where our objective is to minimise the maximum utilisation, then we can see you can save up to 40% reduction during the peak hour on most congested link. This means we have to delay the capacity update of this link for in the future. But, on the other side, we also improve other objectives, so for example, we can reduce the delay and also find servers that, in order to download content from that diverse closest path. If our objective is to minimise the traffic that flows within the network, then probably by utilising ?? by taking the path, then we can see that you can minimise the traffic that flows within the network up to 15%. So, if you multiply this with the capacity of the network you can save up to 15% of the traffic or tens of better bytes of traffic per day.
Now, the same happens if you choose to reduce the delay. Our measurements show that it is possible for approximately 24% of the traffic to reduce end?to?end delay by more than 60 milliseconds which means you don't have to introduce a new access technology but pretty much use the same infrastructure that you have but you ?? by doing better management of how you allocate users to servers.
And something else that is also very important is that with this technology we are able to push traffic from longer paths to shorter paths, which means that less traffic has to traverse the backbone as well as the peering points.
Now, so far, I assume that the servers are fixed and are given by the CDN. Of course we have seen new trend with Google or Netflix open connect that operators ?? not operators ?? CDNs want to deploy end service within the network of ISPs. And of course, they want to do that because they want to have local storage and network resources as well as like CPU. Of course this is very difficult to do because you don't know where is your demand, it takes time to deploy your servers and maybe visuallation is one of the solutions here. On service deployment probably you have to go into the XIP or data centre that is not necessarily very well connected with the customers in an ISP.
What has proposed is this new opportunity with function virtualisation which pretty much says we have basic functionality which gives the servers storage and switches and then you can run all your services like CDNs on top of this. And this is supported by more than 12 big ISPs. So the challenge here is to take what we have achieved before and try to do this within the virtualisation by also trying to put the cloud inside the network so the idea here and the vision is you have micro data centre in POPs you can give them to tenants like CDNs, that they want to run the applications and then charge them by the time that they use them.
So, what do we call this? It's a network platform, as I said. And this similar to what we do with ? engineering, so CDN comes and goes to something that we call resource broker and we say OK I want some slices for some time to run this application and I want, for example, 30 gigabit based in your network. So this will tell you OK, I have developable in this locations, give me the site specific locations. There is discussion between CDN and ISP, once ?? the licensed CDN then they start paying by the volume that they deliver to the user. Actually, this is also a way that, for example, if Netflix or somebody else wants to come in an ISP and says look, I don't have hardware, I don't know how many users I am examining to have, can you run on demand my serves for as long as I tell you, until I bring the real hardware? Or you can imagine cases like on?line gaming that can benefit from /SUFPLT now, once you have this in place, you can use this user to serve assignment optimisation where the users go to the DNS, then on their way to the CDN, then there is a negotiation of what is the best slice of the server to deliver the content of the user and finally, the user knows which server to contact.
Now, we do did this exercise with big CDN and ISP, by investing on installing virtual servers for some period of time, mainly on the big time, identical location within the network and this is only 50 out of 900 locations you can install it, then the benefits are huge. What we saw is something like 40 to 60% of the traffic of the CDN can be delivered locally in the same POP without the need to traverse the backbone.
So to summarise:
I show you some measurements we did in the big ISP that CDNs, the CDN traffic is really increasing. Today, it's anywhere between 30 and 50%. The there are opportunities for collaboration between ISP and CDN that has not been explored before and the main recipe here is to explore the server and path diversity in the network and the CDN and to find a way to collaborate by exchanging information without actually revealing any secret about the operation of the network.
A case study is traffic engineering is a big problem especially with the CDN traffic sifts, what happens today. We believe this is a way that all the involved parties, includes CDN, ISPs, content providers, as well as end users can be benefited. This is an ongoing project. You can find more information in this /SAOEUFPLT I also want to thank many collaborators and contributors of this project here. Thank you.
(Applause)
BENNO OVEREINDER: Any questions? Perfectly timed. We have enough time for questions.
AUDIENCE SPEAKER: First of all, an observation and then maybe a question. Looking at your presentation, I came to the conclusion that I have probably been coming to RIPE meetings for too long, and going for the archives, I found a presentation from RIPE 56, grandma is telling stories about the old times, which was about ISP?aided network selection for peer to peer /S*BGS held by somebody from Deutsche Telekom laboratories to Berlin, and that was about using the exact same technologies to optimise a way the nasty problem of peer to peer traffic. So, either this is worked perfectly and the peer to peer problem has gone away.
GEORGIOS SMARAGDAKIS: Not really.
AUDIENCE SPEAKER: If that is not true, I guess RFC 1925 applies, rule 11, which is that every ?? proposed again with a different name and different presentation regardless of whether it works, and that was my question. Thank you.
GEORGIOS SMARAGDAKIS: Yes, so I have an answer for this. First of all, I don't know when it was that RIPE 56. Secondly, this is specific to CDNs. So what you discuss about peer to peer cannot hold here because completely different topic and completely different problem. Well, with peer to peer there is the hope ??
AUDIENCE SPEAKER: I would suggest that you go and look up the slides, you will find it remarkably similar.
GEORGIOS SMARAGDAKIS: OK. With the peer to peer, because there is only the boot strap server, you cannot guarantee what we say here is /T* can actually be followed. Nobody actually manages the peers, if you talk about the hybrid it's completely different topic, actually don't hold.
AUDIENCE SPEAKER: So Dave Netflix, which you mentioned numerous times in your presentation. We have never spoken. I think probably this looks familiar because it's probably the same marketing collateral, but really, you know, you mention multiple things here, you know I will try to keep this brief, but I'd like to know what CDNs are demanding payment for you because I checked the peering pages of Google and Limelight and Netflix, we all say open, so nobody is demanding payment, right? If someone in this room knows differently I would love to hear from them. Can you comment on that? Is this something that comes up pretty often?
GEORGIOS SMARAGDAKIS: What I know is the big players like Google they have different policies depending on how much traffic you send to a network. If you are too big you go to a private peering. If you are medium, you go probably to an XIP and you have bilateral agreement. If you are small, you use something like route server. So of course these change over the time. So what I mentioned here is that there is a trend that CDNs want to deploy probably you are referring to the last part of my talk, right?
AUDIENCE SPEAKER: In the beginning, you referred to rue numeration in the reverse direction, you are saying from Google. In the end you talk about ref share, being a way that you are talking about extracting money from content in a way that is not necessarily proportionate with their traffic, so, you know, you are talking about two things. Frankly, I think you probably would have given us a much better presentation if you had kept the business part out of it.
GEORGIOS SMARAGDAKIS: I completely kept it out of this. I said this is a solution to enable this. I don't know how much revenue can have out of this and it's not my business, also. What I am saying if you want to deploy servers within a network, either hardware as for example Netflix is doing right now, or Google. In the next years we see with the licensed CDN you can going to develop software servers or talk about CDNs, then the way to do it, instead of throwing ?? you can actually either put in ?? locations and have the information of where the user is what is actually is interested in, to improve the assignment of users to servers, or you can have a more flexible model where we can jointly ISPs and CDNs decide where we are going to locate the serves. This is what I am saying.
AUDIENCE SPEAKER: You use terms like ref shares, the don't really ?? revenue ?? you used the term ref share.
GEORGIOS SMARAGDAKIS: I said this can enable a discussion about how you want to do revenue sharing. It also opens a discussion if an ISP has to deploy its own CDN or it has jointly to deploy a CDN with either for one application or next CDN with an ISP.
AUDIENCE SPEAKER: Thank you.
BENNO OVEREINDER: I want to close the queue. So last question.
AUDIENCE SPEAKER: Peter Losher, Internet systems consortium. In the earlier part of your presentation you were talking about the link between ISP DNS and the CDNs DNS and how that relates. So, what we are seeing more and more these days is that the big Anycast resolvers like open DNS and Google DNS, how would that affect this, because they are not necessarily in the ?? at the XIP level, and we have all seen the issues in regards to Anycast recursive servers and CDN locations.
GEORGIOS SMARAGDAKIS: Yes. As, you know, one of the big Anycast DNS providers like, I am talking about Google DNS, has already enabled the DNS ?? which means that if an ISP wants to propagate part of the information of the client all the way up to the data server it can happen. Now, the disadvantage with this approach yes, but definitely you can some part of the information. There is way to put information about the location of the user.
AUDIENCE SPEAKER: OK. Thanks.
BENNO OVEREINDER: Thank you. I would like to thank this speaker.
(Applause)
BENNO OVEREINDER: So next we go on for lightning talks. Will, go ahead.
AUDIENCE SPEAKER: We have Tore Anderson up first, and he is presenting about ?? deploying IPv6 in ten days, 24 media.
TORE ANDERSON: Hello. I will be quick. Earlier this month on the 3rd, I realised that I was coming here to this nice party and when you go to a party, you usually you bring something nice for your hosts, and then I was thinking what can I bring for the networking people of Greece that they would appreciate? So I was looking through my data centres and I noticed that I have a customer called 24 media hosted in them, and by their own statement they are the largest publishing or on?line publishing group in Greece. So, they do not have IPv6. Maybe I should have IPv6. So I asked them: What about deploying IPv6 in celebration of the RIPE community coming to Athens? And they loved the idea. Come on, let's do it. And then I had ten calendar days to go until yesterday. So the first step was to pick out an IPv6 prefix and deploy it to their front VLAN so it's very simple topology, two front end nodes, combined load balancers and web caches accelerateers running Linux. So, I set up with stateless auto configuration with adjustments and the machines got IPv6 just like that. So second step was to add configuration to the heartbeat failover software package so they actually got a service address that would failover automatically between the two nodes in case one failed and that was one line of config. So that was easy.
The third step was to change HA proxy which is the load balancer software we use, to listen on IPv6 socket instead of IPv4 socket. And that was one line of config that needed changing and on Linux IPv6 accept IPv4 connections as well, so that is all we had to do.
Now, at that point, their system were perfectly capable of responding to IPv6 queries. And we had ?? we added the various host names for their sites to test that it worked. And there was no reason why it shouldn't and it did work. The last finishing touches is to publish those AAAA records in DNS and that is where the real challenge began because all of this took less than one working day. And 24 media has two DNS providers, one is called rapid switch and they manage to add the AAAA records for the sub domains of their sites so they could add for www.domain dot TLD but not for just the domain. And since they have several of the sites that actually redirects you immediately to just the domain, that was kind of a problem. So then we asked rapid switch to please fix this. And also we left one site out, which was the biggest one, sport 24.gr because they wanted to verify that everything was OK before they did the biggest one. The other DNS provider was server lot of. They did not support adding AAAA records through their self?service web interface at all. We asked them to fix that.
Tuesday last week they responded. Rapid switch said this is a bug, we will have to fix this, what I liked and made they cautiously optimistic that they will be able to do this. Serve lot of said we don't support IPv6, we have no intention of doing it. Boo. In response to that, we thought maybe we should just kick out server lot of and move the zones to rapid switch or somewhere else we know it will be working. But at this point we had five days to go and we wanted more preparation so we ?? at that point we stopped trying to migrate or get those sites on?line, IPv6. But maybe we will do so later.
So on Friday last week, rapid switch managed to fix their bug and managed to add the AAAA records for only the pure domains as well. But we still kept sport 24, the G R out, because there was some big football match on on Sunday and wanted not to do any changes ahead of time. And today, they added the AAAAs for sport 24.gr as well and here we are. So one day over time, I would have hoped to be finished before I went on the plane here, but it's good enough.
So, this is what we managed to do. We got seven out of their nine sites on v6. And we blamed the DNS provider for the remaining two. And when we look at Erik's graph of how much sites in the Greek top 100 has v6, we only managed to more than double that number.
(Applause)
So, it's something, right? . It's a start. The sites you can see are various sports, news sites. And the one I think that is pure news sites, which is news 24/7 dot G R is one which was at serve lot of, so hopefully soon.
And that is basically my lightning talk. If there is any questions and if there is time for questions.
WILL HARGRAVE: A couple of minutes of questions if anybody has got any. Great. Thank you very much.
(Applause)
WILL HARGRAVE: Our next is Lee Howard and is he is going to talk about the state of IPv6 only. Do remember to rate the talks on the website. Thank you.
LEE HOWARD: I work for Time Warner Cable which does not do business in this content, I am delighted to be here to speak with people who do business here. The department I work in, we work on new technologies, things that are upcoming and moving towards the future so we are sort of settling down our IPv6 effort because it's big. But, we are working on a time machine to see it what it looks like when you have IPv6 only. I thought I would show you our time machine. We said let's ?? here is our configuration for our time machine, we basically blocked the v4 ether type on the cable modem. That will break IPv4 connectivity. We connected a host directly to our cable modem, we do did not do it on home gate way, just a device directly, just like you'd expect, you get a v6 address and IPv6 DNS on Windows, Mac that will not be a surprise for anybody. We did not test wi?fi devices because we don't have a wi?fi device on a cable modem, only on embedded routers so we didn't have one, but I didn't have to because I knew that Andrea had done at the last meeting you knew that iOS doesn't get ?? or router advertisements, so v6 ?? this is Android, it works gret over v6 as long as you don't need DNS and iOS sort of works, you have got the details there. What I want to know is what happens when you then go try to do things on the Internet without IPv6.
Any guesses? So, we have got an IPv6 only network here today. How many people have connected to that? Andrei has done a great job with it. The trouble is it has NAT 6?4 on it. It's v6 with crutches. This is v6 only. Hey, look at that, RIPE's website works when you have v6 only, congratulations, way to go, RIPE. The very astute observer will note it says your IP address is...
I cheated, RIPE's website does work, I took the screen shot from a different network while I was on the flight over here. But it does really work, I did test it.
So, great. RIPE's own site works. What else do we try? Let's try, some sites work and some sites kind of work. I have two websites or two screenshots were some sort of websites here, the one on the left with blueish writing is the content works but the CSS didn't come up. So, it's kind of like the web from 1995. It's kind of cool. On the right side, we have another one, it works great, you can view videos all day long on that particular website and you don't get ads because the ad network doesn't work v6.
(Applause)
Feels pretty good. So, so far, so good, we have hit some website and they worked pretty well. Here is one that doesn't work, this is Time Warner Cable's website. We are working on it. And we actually are working on it, we had a major organisational change and you don't care the long song and dance but getting there. What I do have is stickers. I put ?? it says legacy IP only, this product does not support the current generation of the IP V protocol" I put on table this morning, Sander Stefan has some others very similar and great for putting on vendor equipment, test the box it doesn't work, slap a sticker on it and send it back to the vendor.
Also works very well for blue rate players and game consols and anything else. I went to look for some more content. What other websites should I test, and I had a summer intern, there is a list of websites that support IPv6. What is the point of testing that don't claim v6 support. The world IPv6 launch. The ones in blue on that pie chart on the left are ones that support IPv6, 133 out of 200ish that we tested. So, more than half actually supported IPv6 of the websites that claimed to support IPv6. And another 25 of them sort of ?? those are the ones that had some graphics didn't come up or the CSS didn't show and so 23, those that had major problems, what is really troubling is 25% of the world IPv6 launch participating sites don't support v6. Essentially at all. In a couple of cases you do have the Nav bar comes up but you can't click on anything. So we look at some other random websites, mostly US and Canada, just sites that we frequent and surprisingly, 25% of those, of these other sites had some level of v6 support. Surprisingly there are more sites than we expected, that is pretty good. IPv6 only though isn't looking like a really fun place to live just yet, that is why we are still in the time machine. What other set of websites can I look for that I can test whether they have IPv6 connectivity? There are some fabulous companies who have very generously provided service and support for this meeting. Go ahead. Make your guess in your mind for which support v6 and which don't. I have given this presentation at a NANOG and APNIC and Wes George did the original groundwork, the legwork and he present at an IETF meeting. This is about par for the course. This may be even a little bit high because it's almost half. So, I suggest to sponsors of network based organisations that perhaps you want to add IPv6 support. In fact, let me go on, the point here is, we need IPv6 to work. We need to to work everywhere because we are still stuck doing power for as long as there is significant that doesn't support IPv6. We need to work on v6 only, not with v4 crutch. We found we also need to monitor the v6 instance of the website, just because you are monitoring by name doesn't mean you are monitoring both address families. That is kind of important. So there is something else important here I need to say which is we are actually working on some language in our contracts that v4 is becoming more and more expensive, either because you have to do CGN or by addresses or peers or partners or vendors are having to do the same thing and it's becoming more expensive for them, and they are going to raise their prices. As v4 becomes more ex pen circumstances of I don't want to buy from vendors who are causing my customers to still need v4, that is causing me money. If anything you as a vendor do doesn't work in v6 you are costing me money and that is not a good way to win business. If you think you are a network provider of some kind you should be eating your own dog food, don't just pre tinned look v6 and don't support it, turn on v6 an all of your external services. Thank you.
(Applause)
WILL HARGRAVE: Thank you very much, Lee. Do we have any questions at all? We have time for a couple of questions. Wow. Negative time. Thank you very much. Next up, we have Jan Zorz talking about best current operational practices and they work in that area around the world. Jan Zorz and Benno. And if you are rating their talks, bear in mind we did change the order of a few of those today, don't stress, hit refresh.
JAN ZORZ: From Internet society. I would like to give a brief update on what is going on around the world with the BCOP, I had presentation at the previous RIPE meeting, we had the BoF about the BCOP and I think that things are going pretty well.
So, just a refreshment, what is the idea:
We believe that the neutral and regionally organised repositories where the operators would document the arguably the best current operational practices would be a good idea. And these documents should be written by you guys, by the operators, and the process should be owned by the operators.
So, we went around the world with this idea, this is the list of the meetings that we spoke to the operational community and we got back some huge feedback.
So, briefly, where is already started. The main idea is coming from NANOG. It was IP BCOP and now turned into a regional BCOP effort from North America and we are also working on the other areas. I heard yesterday from ARIN that last week, at a NANOG meeting, there is a huge amount of people subscribing to the BCOP Working Group or track there and there is some new material coming in, new documents are being produced so they are doing a great job.
We got enormous feedback from the RIPE community in Dublin and as I try to listen to the community as much as possible, I proposed that we turn upside down everything. So what we heard was, it's top heavy, it is rigid structure and we decided to cut the head off. We are directing our efforts now and resources to help regional BCOP efforts to start and to do their work. I am really happy to announce that Benno Overeinder volunteered to actually run and work on this effort here in the RIPE region. I pretty much like that.
So, what is the idea now? Let's start the work regionally, produce something useful and publish it. This is also what Rob suggested back in Dublin.
Coordination in a global repository might emerge out of the natural need but it should be purely bottom up process that is owned by the operators. This was the proposal for the last time. We decide to screw it.
This is the new idea. More flat, bottom up process, regionally organised and owned by you guys. What we are doing now is we are just putting this little web page on where we just point what is being done in a different regions but there is no secretariat and rigid top heavy structure, just pointing to what is being done around the world.
This is the website and Benno will talk to you what was already identified in RIPE BCOP as potential topics.
BENNO OVEREINDER: Still five minutes to go. Thanks. Topics we have discussed last RIPE meeting and I understand also collected from other meetings and NANOG meetings. So, this is a list of topics people suggest. We discussed it at the meetings, also via e?mail list. And this is just a selection. And probably for more and looking for people to work on this. Inhibiting address spoofing, this is something that is close to my heart. We organised a panel discussion last RIPE, BGP 3884 it's different to implement, even impossible. What can we do? There are ways to implement this to get a better routing hygiene. Call this document good operational practices and we need something, of course, to write it down, but also feedback of the people, it's workable document: And not only just statements, you should do this and you may do this, but really get down to the details, what kind of policies you can implement, what kind of tools are available so really going to really practical hands?on directives so you can more or less easily realise and deploy this technology and these ideas in your network.
Another one of our BGP policies, I grabbed this one for ?? get some more examples, people are asking for policies, example of policies for different network equipment, for the Cisco, Juniper, but also for OpenSource routing implementations like BIRD and Quagga. We can think of DNS policies, how do we run a DNS server, authoritative recursive, how do we select secondary DNS servers. There are a lot of documentation around either, either in RFC, BGP in the IETF, have to bring together on one document so people can pull out one document and read that and more or less that is certain that everything has been documented that is being done to run a DNS server and can include pointers to other authoritative references, etc..
Other things, e?mail policy, spam, spam filtering, for example. ICMP filtering, important, people brought forward, especially with IPv6 doesn't work if you block your ICP packets. Also important for path MTU discovery, of course.
Network performance, how to test. There are a number of documentations going around, in IETF there are a lot of people working in Working Group about measurements and we want to bring it down to what kind of tools can we use, what to measure. The visibility from global Internet. Other ways to measure your visibility on the network, you can go to right stats, for example. So we need people to work on this, document this, write document, review documents, there is also of course for the BCOP this evening at 6 o'clock. We can talk about these topics again. We really look also for people to spend sometime to either write or review documents. You have an example from NANOG?
JAN ZORZ: Yes. So thank you, Benno, for this. You are all invited to come to the BCOP BoF later on this evening at six. I hope it's here. I will just go quickly through this because I need to put a stop to myself. So, what I heard from the operators is that the next speed bump in deploying v6 is their lack of knowledge at their help desk, so they say if you are rolling out the IPv6 then our desk ?? help desk will just burn?down and die. It's not true. We are putting together the document, it's generic troubleshooting for help desks around the world and this should be very, very simple and generic document written by the operators. We already started the document, we will discuss it at the BCOP BoF. We have some good contributors, Lee Howard, you met him before, we have John ?? David freed.man, Sander Steffann and myself contributing, we need more operators to come in and read the documents and start contributing. So we will talk also about this at the BCOP BoF.
In Latin America, we have two volunteers from the region. In Africa region we identified the Mr. Douglas and we are starting the initiative this year in Ivory Coast meeting. In Europe, we have Benno here and we also need more volunteers to run the effort. And in Asia, we are going next year and talk to the Asian operators, the seed was planted at APNIC meeting but we need to go there and talk to operators communities about what we are trying to do.
And this comes to an end. So the BCOP BoF will be in ballroom 1, if that makes any sense. So are there any questions? Or comments?
WILL HARGRAVE: No questions. Thank you very much, you two.
JAN ZORZ: Thank you.
(Applause)
RANDY BUSH: IIJ. You might want to look at the RIPE website, there is some sort of strange thing that documents named RIPE? followed by some numbers.
WILL HARGRAVE: That concludes the presentations for today. Thank you, everyone. So later on, we have an opportunity to meet the RIPE NCC Executive Board, that will be at 6:30 upstairs, and that will be followed by some welcome drinks where you can meet the other attendees. Thanks.
