NFV Tutorial: Managing a Virtual Network Function using SDN and Control Theory

Overview:

This is a basic tutorial that guides you through an experiment using OpenFlow and Network Function Virtualization (NFV). It is recommended that you first read about OpenFlow (http://archive.openflow.org/) and go over a basic OpenFlow tutorial on GENI (http://groups.geni.net/geni/wiki/GENIExperimenter/Tutorials/OpenFlowOVS). The goal of this tutorial is to give you a hands-on experience with OpenFlow, and how it can be used for NFV deployment. Using a basic topology, which contains two sources, a destination, two virtual network functions (IDS), an OVS and a controller, we will show how different OpenFlow rules can be used for NFV management. Once resources are reserved for this topology, the tutorial should take about 45-60 minutes to complete.

Figure 1: The image above depicts the network topology used in this tutorial.

Pre-requisites:

- A GENI account, if you don't have one sign up!

- Familiarity with how to reserve GENI resources (we will be using the GENI Experimenter Portal as our tool).

- Familiarity with logging into GENI compute resources.

- Basic understanding of OpenFlow. An OpenFlow tutorial is here!

- Familiarity with the Unix command line.

- Familiarity with the Python programming language. We will use a controller (POX controller) written in Python for this tutorial.

- You will need to be a member of a project.

Tools:

- Open vSwitch.

- POX controller.

Both of the tools are already installed on the machines where the resources are provided. If it happens not to be installed, take a look at the following tutorials to learn how to install these tools:

Open vSwitch: click here for tutorial.

POX controller: click here for tutorial.

Where to get help:

- POX Wiki

- Open vSwitch

- GENI Wiki

Tutorial Instructions

1. Design the Experiment

The basic topology of the tutorial is shown above. We have two sources that will communicate with the destination via an OVS switch. We have two VMs (VNF1 and VNF2) running instances of the Snort Intrusion Detection System (IDS), representing the Virtual Network Function (VNF). In this tutorial, we will install forwarding rules in the OVS using a controller such that whenever a source sends packets to the destination, a copy of each packet is also sent to one of the IDS for intrusion detection analysis.

Iperf will be run as an application and we will observe the operation and impact of different strategies for load balancing across the IDS instances.

Though in this tutorial, we will be using Snort IDS as an example VNF, one can consider other network functions, e.g., a firewall, a caching (proxy) server, etc.

2. Establish the Environment

2.1. Pre-work:

Skip this section if you have already established your environment for your project.

- Ensure SSH keys are setup. If your SSH keys are not setup before, do the following steps:

- Generate an SSH Private Key on the Profile page on the GENI portal

- Download Private Key to ~/Downloads

- Open terminal and execute

$ mv ~/Downloads/id_geni_ssh_rsa ~/.ssh/.

$ chmod 0600 ~/.ssh/id_geni_ssh_rsa

$ ssh-add ~/.ssh/id_geni_ssh_rsa

- Ensure you are part of a project.

3. Obtain Resources

For this tutorial, you can use resources from any aggregate in GENI, but preferably an instaGENI aggregate (e.g., CENIC IG or Cornell IG). The experiment will need the following:

- 1 Xen VM for the OpenFlow controller with a public IP (controller)

- 1 Xen VM for the OpenFlow virtual switch (OVS)

- 3 Xen VMs, two Sources and one Destination (S1, S2 and destination)

- 2 Xen VMs for two Virtual Network Function instances (VNF1 and VNF2)

We will need two slices for this tutorial:

1. A Slice for the OpenFlow controller.

2. A Slice for the Network consisting of sources, destination, IDSes and OVS.

To reserve resources, use the GENI portal.

3.1. Controller:

- Open the slice that will run the OpenFlow controller. You may call it 'Controller-xx', with xx replaced with your initials or last name. Select a VM running the controller using the RSpec that is available in the portal called XEN VM POX Ctrl. This is shown in the picture below.

- Choose any InstaGENI aggregate (e.g. CENIC or Cornell IG) and reserve resources for the controller. You will see the controller VM as shown below.

3.2. Network:

Create a new slice 'Network-xx', with xx replaced by your initials or last name, for a network whose topology will consist of two sources, a destination, an OVS and two VNFs. Use the Rspec given below to reserve resources for this Network slice.

http://csr.bu.edu/rina/grw-bu2016/nfv/Network_with_NFVimage.xml

Although you can choose any InstaGENI aggregate to reserve resources for Network, for this tutorial, it is simpler (and more effective!) to just use the same IG aggregate as the one used for your Controller slice. You will have a network of 6 VMs as shown below. This may take a while!

3.3. Configure and Initialize:

Although OVS is installed on the host and is meant to act as a software switch, it still needs to be configured. There are two things that need to be configured:

(1) configure your software switch with the interfaces as ports and

(2) point the switch to the OpenFlow controller.

3.3.1. Configure the Software Switch (OVS Window):

1. Login to the OVS host (on the Network slice)

2. Create an Ethernet bridge that will act as our software switch:

sudo ovs-vsctl add-br br0

3. Prepare the interfaces to be added as ports to the OVS switch

Your OVS bridge will be a Layer 2 switch and your ports do not need IP addresses. Before we remove them let's keep some information

- Run ifconfig on ovs node

- Write down the interface names that correspond to the connections to your VNF1 and VNF2 hosts. The correspondence is

o Interface with IP 10.10.1.13 to VNF1 - ethX

o Interface with IP 10.10.1.14 to VNF2 – ethY

Make sure you have noted the names of the interfaces before you proceed. We will need the interface names to run experiments. (Otherwise, you have to figure out later which OVS interface is connected to each host by pinging from, say, one source to each VNF, after running "sudo tcpdump -i ethX" on each OVS interface.)

4. Prepare the interfaces to be added as ports to the OVS switch.

- Your OVS bridge will be a Layer 2 switch and your ports do not need IP addresses. Remove the IP from your data interfaces.

sudo ifconfig eth1 0

sudo ifconfig eth2 0

sudo ifconfig eth3 0

sudo ifconfig eth4 0

sudo ifconfig eth5 0

Be careful not to bring down eth0. This is the control interface, if you bring that interface down you won't be able to login to your host. For all interfaces other than eth0 and l0, remove the IP from the interfaces (your interface names may vary).

Add all the data interfaces to your switch (bridge): Be careful not to add interface eth0. This is the control interface. The other four interfaces are your data interfaces. (Use the same interfaces as you used in the previous step).

sudo ovs-vsctl add-port br0 eth1

sudo ovs-vsctl add-port br0 eth2

sudo ovs-vsctl add-port br0 eth3

sudo ovs-vsctl add-port br0 eth4

sudo ovs-vsctl add-port br0 eth5

Now the software switch is configured! To verify that the five ports are configured, run the following command to see all five ports listed:

sudo ovs-vsctl list-ports br0

3.3.2. Point your switch to a controller

An OpenFlow switch will not forward any packet unless instructed by a controller. Basically the forwarding table is empty, until an external controller inserts forwarding rules. The OpenFlow controller communicates with the switch over the control network and it can be anywhere in the Internet as long as it is reachable by the OVS host.

1. Login to your controller

2. Find the control interface IP of your controller, use ifconfig and note down the IP address of eth0.

3. In order to point our software OpenFlow switch to the controller, in the ovs terminal window, run:

sudo ovs-vsctl set-controller br0 tcp:<controller_ip>:6633

4. Set your switch to fail-safe-mode. For more info read the standalone vs secure mode section. Run:

sudo ovs-vsctl set-fail-mode br0 secure

5. Trust but verify. You can verify your OVS settings by issuing the following:

sudo ovs-vsctl show

Standalone vs Secure mode

The OpenFlow controller is responsible for setting up all flows on the switch, which means that when the controller is not running there should be no packet switching at all. Depending on the setup of your network, such a behavior might not be desired. It might be best that when the controller is down, the switch should default back to being a learning layer 2 switch. In other circumstances however this might be undesirable. In OVS this is a tunable parameter, called fail-safe-mode which can be set to the following parameters:

• standalone [default]: in this case OVS will take responsibility for forwarding the packets if the controller fails
• secure: in this case only the controller is responsible for forwarding packets, and if the controller is down all packets are dropped.

In OVS when the parameter is not set it falls back to the standalone mode. For the purpose of this tutorial we will set the fail-safe-mode to secure, since we want to be the ones controlling the forwarding.

First thing we are doing in the experiment is pinging the VMs. By now, our switch is already configured, so we start working on the controller. As mentioned earlier, we are using a POX controller and it is already installed on the controller host.

1. Login to the hosts

We need to ssh to all our hosts. Again, if you don't know how to login into a node, click here to learn.

Open the following windows:

- one window with ssh into the controller

- one window with ssh into OVS

- one window with ssh into s1

- one window with ssh into VNF1

- one window with ssh into VNF2

- one window with ssh into destination

Note: when you need to open more windows on the same host, it may be convenient to open a new tab in the same window/terminal, e.g., by using Command + "T" on a Mac, and ssh there.

2. Configure and Initialize services

2.1. Use A Learning Switch Controller

This is a very simple example where we are going to run a learning switch control to forward traffic from s1 to VNF1.

1. First start a ping from s1 to VNF1, which should timeout, since there is no controller running.

ping vnf1 -c 10

2. The POX controller is installed under /tmp/pox on the controller host. POX comes with a set of example modules that you can use. One of the modules is a learning switch. Start the learning switch controller which is already available by running the following two commands:

cd /tmp/pox

python pox.py --verbose forwarding.l2_learning

'l2' above uses the letter 'l' as in level and is not the number one. In addition, you should wait for the "INFO ... connected" line to ensure that the switch and the controller are communicating.

The output should look like this:

3. In the terminal of s1, ping VNF1:

Now the ping should work and the output should look like this:

4. Go to your controller host and take a look at the printouts. You should see that your controller installed flows based on the mac addresses of your packets.

In case the controller and OVS are not communicating, you may try changing the port of your controller, this is the command:

sudo ./pox.py --verbose openflow.of_01 --port=443 forwarding.l2_learning

Then tell the ovs switch that the controller will be listening on this new port, i.e. change 6633 to 443: sudo ovs-vsctl set-controller br0 tcp:<controller_ip>:443

2.2 NFV OVS controller

Now we are going to run a different controller that will install OpenFlow rules to support NFV load balancing. In this controller, the traffic shall go from a source to destination, and duplicate packets are sent to one of the IDS nodes (VNF1 or VNF2) for Intrusion detection. The picture below shows a red line representing traffic going from source1 to destination, and the green line represents the duplicate traffic that is sent to VNF1 for intrusion detection.

3. Execute Experiments:

First we need to load the configuration files for the NFV OpenFlow controller.

- We will first remove the default files for the controller and replace them with our controller files. Execute the following:

cd /tmp/pox/ext

sudo chmod 777 ../ext/

sudo rm *

wget http://csr.bu.edu/rina/grw-bu2016/nfv/OVS_files.tar.gz

tar -xvf OVS_files.tar.gz

- Now you should have different files for the controller. Open port.config file to configure the system parameters. You can use any editor to edit the file. We will use nano here as an example.

nano port.config

- You will see a window as shown below. Change the values of vnf1_interface and vnf2_interface to the values that you noted down in Section 3.3.1 in the Design/Setup section of this tutorial. These values will tell the controller which interfaces are connected to VNF1 and VNF2.

Experiment 1: Load Balancing using Round Robin Control

In the first experiment, we will look at a Round Robin load balancer for a VNF Snort application.

Snort will be running as IDS on VNF1 and VNF2 and we will try to balance the load across the two VNF instances by directing each new flow request to one of the two VNF instances in a round robin fashion. 


We will use the Netcat application to generate traffic between a source and destination.

Netcat is a useful application for network debugging and investigation. It is used for reading from and writing to network connections using TCP or UDP. In other words, it is like talking on the phone. One (a node) talks and another one, on the other side, listens and talks back. Using netcat, you can see the communication between a source and destination.

1. Open the controller window and execute the following:

cd /tmp/pox/ext

nano port.config

Here change the controller_type to RR as shown below:

2. Run the NFV_controller in the controller window:

cd /tmp/pox/

python pox.py --verbose NFV_controller

3. Open the VNF1 and VNF2 windows and run Snort IDS on each of the VNFs as shown below:

sudo /usr/local/bin/snort -A console -v -c / -i eth1

You should see the window shown below on the VNF1 and VNF2 windows.

4. In the destination window, run the netcat server:

nc -u -l 5000

5. In the s1 window, run the netcat client to send traffic to the destination node:

nc -u destination 5000

6. Type something on the s1 window and you should see it on the destination window. Now if you look at the VNF1 and VNF2 windows running Snort IDS, you could see packets arriving at the VNF1 and VNF2 nodes in a Round Robin fashion. An example of such an output is shown below on the VNF window running Snort IDS.

In the controller window, you can see that the controller chooses VNF1 and VNF2 in a Round Robin fashion as shown below:

OPTIONAL: Review the RR Controller Code:

The Round Robin code is based on the following algorithm:

If you want to see the code for the Round Robin load balancer, you can see it by executing:

cd /tmp/pox/ext

cat NFV_controller.py

The code that corresponds to Round Robin control is shown below:

Here if the VNF selected for the previous flow was VNF1, VNF2 is selected for the next flow, and vice versa. Once a VNF is selected, packets are sent to the selected VNF:

Experiment 2: Load Balancing using Proportional Integral (PI) Control

In this experiment, we will use a control theoretic approach to do load balancing. An overview of the system is shown in Figure 2.

Figure 2: Overview of the system.

In this control theoretic approach, the load on VNF1 and VNF2 is monitored, and flow-forwarding decisions are made based on the load information retrieved from the hosting VMs.

We will run a RINA distributed application to get the state (load) of the VNFs to the controller VM. Once the controller has the IDS load information, it will use the Proportional Integral (PI) load balancer to balance the load across the VNF instances based on the load information. This load balancing information is then provided to the OVS controller, which updates the OpenFlow rules on the OVS switch to balance the load.

RINA Distributed Application:

First we will run a RINA distributed application to collect the VNF load information on the controller node.

We need Java installed on the nodes to run the RINA application. Install java on VNF1, VNF2 and controller nodes (Type Ctrl-C to exit netcat). To install Java, execute: sudo apt-get install openjdk-7-jdk (If the install fails, you may first run: sudo apt-get update).

1. In the controller window, download the RINA controller code:

cd ~

wget http://csr.bu.edu/rina/grw-bu2016/nfv/Control.tar.gz

tar -xvf Control.tar.gz

2. Type ifconfig to get the IP address of the controller. Save this address as we will need this address to direct the RINA processes on the VNFs to the RINA process on the controller.

3. In a new VNF1 window, download the RINA VNF1 code:

cd ~

wget http://csr.bu.edu/rina/grw-bu2016/nfv/VNF1.tar.gz

tar -xvf VNF1.tar.gz

4. In a new VNF2 window, download RINA code using following commands to get RINA VNF2 code.

cd ~

wget http://csr.bu.edu/rina/grw-bu2016/nfv/VNF2.tar.gz

tar -xvf VNF2.tar.gz

5. Now we will change the IP address in the RINA configuration files for VNF1, VNF2 and controller, so these RINA processes can talk to each other.

In the VNF1 window, execute:

cd ~/VNF1

nano ipcVNF1.properties

At the bottom of the file, change the rina.dns.name and rina.idd.name to the IP address of the controller. The following screenshot shows an example.

In the VNF2 window, execute:

cd ~/VNF2

nano ipcVNF2.properties

At the bottom of the file, again change the rina.dns.name and rina.idd.name to the IP address of the controller.

In the controller window, execute:

cd ~/Control/RINA/

nano ipcControl.properties

At the bottom of the file, again change the rina.dns.name and rina.idd.name to the IP address of the controller.

6. To run the RINA application, follow these steps (make sure you installed Java as noted above):

o In the controller window, execute the following commands:

cd ~/Control/RINA/

./run_controller.sh

o In the VNF1 window, execute the following commands:

cd ~/VNF1/

./run_VNF1.sh

o In the VNF2 window, execute the following commands:

cd ~/VNF2/

./run_VNF2.sh

You should see output on the controller window as shown below:

The RINA application on VNF1 and VNF2 should be run as soon as possible after the RINA application on the controller is started. If you wait for too long, you will get null values for CPU usage, as the controller`s RINA app is not able to subscribe to the CPU load of the VNFs. If this is the case, you should restart all RINA processes. The sample output below shows a case where the controller RINA app is not able to subscribe to the CPU load of VNF2 .

To stop all RINA processes running on a VM, run killall -v java

OPTIONAL: Review the PI Controller Code:

PI controller:

The PI-controller gets the load information of VNF1 and VNF2 using RINA`s distributed application and makes the load balancing decision.

The block diagram of the Proportional Integral (PI) controlled NFV system is shown in Figure 3.

Figure 3: Block diagram of the PI-controller NFV system. System load L and target load T(s)=T/s of VNF1 is used to compute X, i.e. ratio of traffic diverted to VNF2. K` = K/T.

The RINA-based distributed monitoring application provides the VNF1 state (average CPU load) information L(t) to the PI controller. The maximum capacity of a VNF instance is T. If the load on VNF1 exceeds T, new traffic flows are forwarded to a second VNF instance, VNF2. Assuming instantaneous feedback / measured load L(t), the PI control equation is given by:

The code for the PI controller is based on following algorithm. Input IDS_load.txt is the file generated by the RINA distributed application. This file has load information of the VNFs.

___________________________________________________

1. To run the PI-controller, open a new controller window and execute:

cd ~/Control/PI_controller

python PI_controller.py ~/Control/RINA/NFV1.txt

Note that here we are directing PI_controller.py to the NFV1.txt file that is constantly updated by the RINA distributed application with the load information of VNFs.

2. You should see the VNF state information printed on the screen. A sample output is shown below.

Here the target load on VNF1 is 30.0% of CPU usage, i.e. if the CPU load on VNF1 is more than 30.0%, traffic flows will be diverted to VNF2. The `current CPU load` shows the load on VNF1. The next line of the output shows the percentage of flows that will be directed to VNF2 and the last line shows the flows that were being directed to VNF2 before the current control update.

Do not close this window; leave the PI controller running.

PI-based OVS controller:

Now we will run the OVS controller that will get the load balancing decision from the PI-controller and direct the flows accordingly.

1. First we will update the port.config file to direct the controller to the NFV_ratio_PI.txt file generated by the PI-controller, which has the load balancing decision information. In a new controller window, execute:

cd /tmp/pox/ext

nano port.config

o Change the value of controller_type to PI

o Change the value of file_path_pi to the text file that has the PI controller`s output.

/users/<UserName>/Control/PI_controller /NFV_ratio_PI.txt

Change the <UserName> to your user name.

Sample values are shown below.

2. Now we can run the OVS controller. Execute

cd /tmp/pox

python pox.py --verbose NFV_controller

Run Snort and Generate Traffic:

1. First we will run Snort IDS on VNF1 and VNF2. In separate windows for VNF1 and VNF2, execute the following command:

sudo /usr/local/bin/snort -A console -v -c / -i eth1

You should see the window as shown below in the VNF1 and VNF2 windows:

2. We will use the iperf application to generate flows between a source and destination. If iperf is not installed on your nodes, execute:

sudo apt-get install iperf

3. Run iperf server on the destination node:

iperf -s

4. Now we will generate traffic from the sources (s1 and s2) to the destination node using iperf and see how it effects the CPU utilization at VNF1 and VNF2 running Snort IDS. Note that if we run multiple instances of iperf, we can generate significant load on the VNF instances. To run iperf client on a source, execute:

iperf -c destination -t 500 &

Note that you can run multiple instances of iperf by running iperf -c destination -t 500 & multiple time in s1 and s2 nodes. This flow lasts for 500 seconds. To kill all the flows generated at a node, run killall –v iperf

5. Now if you look at the controller window, which is running the PI-controller, you can see the load on VNF1 has significantly increased. If the load is more than 30%, some percentage of the traffic flows will be diverted to VNF2. A sample output is shown below:

Optional: Code Review: The OVS controller gets the load balancing information from the output text file generated by the PI-controller. Based on the value of the control variable (variable X), it sends each new flow to either VNF1 or VNF2. The algorithm for the PI-based OVS controller is shown below:

The corresponding code can be found in /tmp/pox/ext/NFV_controller.py as shown below:

Real Time Graphs:

You can draw real-time CPU usage graphs for both VNF1 and VNF2 nodes. A Python script is provided to produce these real-time graphs. The script periodically retrieves the CPU usage file from the Controller node and plots the graph for it.

1. Download the python script RealTimeGraph.py to your laptop using the following link.

http://csr.bu.edu/rina/grw-bu2016/nfv/RealTimeGraph.py

2. Run the script and direct it to the CPU usage files (NFV1.txt and NFV2.txt) present at the controller. To run the Python script, type the following in the folder where you saved the RealTimeGraph.py file:

python RealTimeGraph.py -n <username>@<controller IP address>

Change <username> to your user name and <controller IP address> to the IP address of the controller.

3. You should see real-time CPU graphs as shown below:

To run the Python script to plot graphs, you need the python plotting library matplotlib. If you do not have this library on your laptop, you can use the following link to download it to your computer:

http://matplotlib.org/users/installing.html

Tear down Experiment and Release Resources:

After you are done with this experiment, close all your open windows and release your resources. In the GENI Portal, select the slice and click on the `Delete` button.

Now you can start designing and running your own experiments!

Sources:

Some parts of this tutorial are taken from GENI OpenFlow Tutorial:

http://groups.geni.net/geni/wiki/GENIExperimenter/Tutorials/OpenFlowOVS

Created by: Nabeel Akhtar

Supervised by: Ibrahim Matta