• Login to Azure Cloud Portal https://portal.azure.com/ with the provided login/password

  

• Click on Cloud Shell icon on the Top Right side of the portal

  

• Select Bash

  

• Click on Show advanced settings

  

• Select

  • Use existing Resource Group - it should auto populate with USERXX-workshop-sdwan (USERXX is your Username)
  • Use existing Storage account - it should auto populate with USERXX##########workshopsdwa (########## is a random string)
  • Use existing File Share - type cloudshell

• Click "Attach Storage"

  

• You should now have access to Azure Cloud Shell console

  

Perform the following step in your Cloudshell console to create your environment.

1. Clone the Github repo
2. Change directory to the se-conf-sdwan-workshop/se-summit folder
3. Initialize Terraform
4. Create Terraform Plan
5. Apply Terraform Plan

Copy and paste these commands into your Cloudshell console. The terraform variable username will be populated with the value of the environment variable USER

git clone https://github.com/FortinetSecDevOps/se-conf-sdwan-workshop.git
cd ./se-conf-sdwan-workshop/se-summit/
terraform init
terraform plan -var="username=${USER}"
terraform apply -var="username=${USER}"

• At the end of this step you should have the following architecture

  

1. Using the Terraform output, verify that you have Web and SSH access to the FortiGates.

   

   • Terraform output can be redisplayed at any point as long as you are in the ./se-conf-sdwan-workshop/se-summit/ directory, by using the command terraform output

     cd
     cd se-conf-sdwan-workshop/se-summit/
     terraform output

2. Connect to the Branch sites FortiGates and check the VPN status.

3. Connect to the Hub FortiGates and check the WAN IP

1. FortiGates in the Hub do not have public IPs, how are they accessible via the Web UI?

2. Why are the VPN connections down?

1. Select the Hub External Load Balancer sdwan-USERXX-workshop-hub1-elb1
2. Click on Backend pools
3. Add FortiGate1 and FortiGate2 port1 interfaces and then click on Save

• 10.10.0.4

• 10.10.0.5

  

1. Select the Hub External Load Balancer sdwan-USERXX-workshop-hub1-elb1

2. Click on Load balancing rules

3. Create Load balancing rules for UDP 500 and UDP 4500 - one rule for each

   

Verify that the FortiGates are responding to Azure Load Balancer Health Checks

1. Select the Hub External Load Balancer sdwan-USERXX-workshop-hub1-elb1

2. Click on Insights - Click the "Refresh" button a few times, eventually (~30 seconds) the FortiGate reachability will be indicated.

   

3. Verify that the VPN connections from the Branch to the Hub are UP

   

4. Verify that the BGP peering with the Hub is UP and that the Branch FortiGate learned the Hub and other Branches' CIDRs. Run the Command get router info routing-table all on all the Branch FortiGates.

• At the end of this step you should have the following architecture.

  

1. Why is one FortiGate depicted as unhealthy by the Azure Hub External Load Balancer?

2. Why is NAT used to access the FortiGates, but for IPSEC VPN traffic Load balancing rules are used?

3. Do FortiGates in the Branches learn Spoke11 and Spoke12 CIDRs?

Create a VNET peering between the Spoke11 VNET and the Hub VNET

1. Select the Spoke VNET, USERXX-workshop-sdwan-spoke11 - (replace USERXX with your username)

2. Click on Peerings

3. Add peering to Hub VNET, USERXX-workshop-sdwan-hub1

4. Repeat the above between Spoke12 VNET, USERXX-workshop-sdwan-spoke12 and the Hub VNET

You will see errors until all the correct selections are made.

1. Verify that the Branch FortiGates have learned the Spoke11 VNET and Spoke12 VNET CIDRs. Run the Command get router info routing-table all on all the Branch FortiGates.

1. Select the Azure Route Server USERXX-workshop-sdwan-RouteServer contained within your Resource Group.

   

2. Click on Peers on the left side of the menu, verify the connection to the Hub FortiGates

3. List the routes learned by Azure Route Server, run the commands below from your Azure Cloud Shell

• The variable ${USER} in the commands reads your username from the environment

az network routeserver peering list-learned-routes -g ${USER}-workshop-sdwan --routeserver ${USER}-workshop-sdwan-RouteServer --name sdwan-fgt1
az network routeserver peering list-learned-routes -g ${USER}-workshop-sdwan --routeserver ${USER}-workshop-sdwan-RouteServer --name sdwan-fgt2

The passive FortiGate will produce empty output

{
  "RouteServiceRole_IN_0": [],
  "RouteServiceRole_IN_1": [],
  "value": null
}

• Can the Hub FortiGate Azure SDN Connector read the Azure environment?

  • Troubleshoot and Make the required changes to allow the FortiGate to retrieve the SDN filters.

    • Hub FortiGate debug the Azure SDN Connector

      diagnose debug application azd -1
      diagnose debug enable

  • Hints:

    ---

    • FGT Branch3 is able to retrieve the filters, why that is not the case for the FortiGates behind Load Balancers?
    • FGT Branch3 is standalone, all other FortiGates are in A-P HA, how does that affect traffic to retrieve SDN information?
    • Hub External Load Balancer needs a management nic backend pool and a TCP rule, any port suffices. This rule is about letting TCP traffic out. The External Load Balancer will let the response traffic back in because the traffic originated internally.

    

1. Create a backend pool on the Hub load balancer using the Hub FortiGate Management Interfaces

   • 10.10.4.4
   • 10.10.4.5

   

   

2. Create a TCP Load Balancer Rule, any port will do, e.g. 13000. This rule will allow TCP response traffic back through the load balancer, when to a TCP request originated from a device in a backend pool associated to the load balancer.

   

3. Create a dynamic address object on the Hub FortiGate, named Spoke_VNETs that resolves to the Spoke VNETs VMs

   

4. Use the object in an existing Policy named Branch to Cloud to restrict traffic coming from the Branches to only VMs in the Spoke VNETs.

   

• Generate Traffic from Branch1 Primary FortiGate:

  1. Connect to the Branch1 Primary FortiGate
  2. Configure ping-options to initiate traffic from FortiGate's private nic (port2).
     • execute ping-options source 172.16.2.5 - source IP depends on which Branch1 FortiGate is primary br1fgt1 or br1fgt2
     • execute ping-options repeat-count 100
  3. Initiate a ping to Spoke11 and Spoke12 Linux VMs (10.11.1.4 and 10.12.1.4)
     • execute ping 10.11.1.4
     • execute ping 10.12.1.4

  

  

• Generate Traffic from Branch1 Linux VM:

  1. Enable serial console access on Branch1 Linux VM

     • Click on the VM sdwan-USERXX-workshop-br1lnx1

     • Go to Boot diagnostics -> Settings -> Select Enable with managed storage account (recommended)

     • Click Save

       

  2. Go to the VM Serial Console

  3. Initiate a ping to Spoke11 and Spoke12 Linux VMs

     ping 10.11.1.4
     ping 10.12.1.4 

     

  4. Does it work?

• At the end of this step you should have the following architecture

  

1. What was missing to allow the FortiGates to retrieve SDN connector filters?

2. Why is the FortiGate only able to retrieve the SDN connector filters in its own Resource Group?

3. Why is the Branch FortiGate able to reach the remote Spoke VNETs VMs (10.11.1.4 and 10.12.1.4) but the Linux VM behind the Branch1 FortiGate cannot?

4. FortiGates at Branch1 and Branch2 site are both behind Azure Load Balancers (behind NAT). Will Branch1 to Branch2 traffic successfully establish an ADVPN shortcut?

1. Click on the Branch1 private route table sdwan-USERXX-workshop-branch1_rt

2. Click Routes

3. Add a default route for 0.0.0.0/0 that points to the Branch1 Internal Load balancer listener IP

4. Repeat the previous step for the Branch2 and Branch3 Route Tables

   • Be sure to use the correct IP as the next hop, that is the correct Internal Load balancer listener IP or FortiGate internal interface.

   Hint: The next hop is a load balancer or a stand-alone FortiGate.

   

   

1. Connect to the Branch1 Linux Host via the serial console

2. Generate traffic to Hub

    ping 10.11.1.4
    ping 10.12.1.4 

3. Does it work now?

1. Click on Spoke11 Linux VM sdwan-USERXX-workshop-spoke11-subnet1-lnx

2. Click on Networking in the Navigation Menu

   

3. Click on the VM nic

   

4. Click on Effective routes

   

5. Check that Azure Route Server has injected the Branch sites CIDRs learnt from the FGT

6. Click on the Hub FGT VM sdwan-USERXX-workshop-hub1-fgt1

7. Click on Networking in the Navigation Menu

   

8. Click on the VM port2 nic

9. Click on Effective routes

• Has Azure Route Server injected the Branch sites CIDRs learnt from the FGT? Why ?

1. Connect to the Branch1 Linux Host via the serial console - sdwan-USERXX-workshop-br1lnx1

2. Generate traffic to Branch2 Linux Host

     ping 172.17.5.4

3. Check if an ADVPN shortcut has been created. Run the command get router info routing-table bgp

   

1. Why has the Azure Route Server (ARS) injected Branch site CIDRs to the Spoke VNET protected subnet but not the FortiGate private subnet?

2. The Branch external load balancer has two front end public IP. How do we ensure that traffic egressing Branch1 on port1 (isp1) always has the same public IP applied? Same for traffic egressing Branch1 on port3 (isp2)

• Connect to the Branch1 Linux Host via the serial console - sdwan-USERXX-workshop-br1lnx1
• Ping a resource in a remote branch site - sdwan-USERXX-workshop-spoke11-subnet1-lnx
  • ping 10.11.1.4
  • Let the ping run

• Connect to the Branch1 Primary FortiGate . Initiate a failover by rebooting the primary FortiGate or by forcing a failover via the CLI

  execute ha failover set 1

• Monitor the number of lost Pings and the failover time

• How long did it take?

• Have the VPNs to the Hub been renegotiated upon failover or maintained?

• Ensure that both Branch1 FortiGates in the cluster are up and running

• Connect to the Branch1 Linux Host via the serial console - sdwan-USERXX-workshop-br1lnx1

• Generate an SSH session to the Spoke Linux VM - sdwan-USERXX-workshop-spoke11-subnet1-lnx

  ssh [email protected]

• From Spoke Linux VM SSH session generate a continuous stream of connections to track the failover event

  while true; date; do curl -I -sw '%{http_code}'  https://www.lemonde.fr/ ; echo -e "\n================="; sleep 1 ; done

• Connect to the Branch1 Primary FortiGate . Initiate a failover by rebooting the primary FortiGate or by forcing a failover via the CLI

• Monitor the SSH connection

• Did you lose the TCP connection?

1. How long was your failover time?

2. Why did we lose the SSH (TCP) session with a "short" failover time?

• Create your vWAN and the vWAN Hub using the CLI commands below. Use the Hub FortiGate location for the VWAN location.

  • You can find the location of your Hub FortiGates with this Azure CLI Command

  az vm show -g ${USER}-workshop-sdwan -n sdwan-${USER}-workshop-hub1-fgt1 -o table

• The variable ${USER} in the commands reads your username from the environment

  az network vwan create --name sdwan-${USER}-workshop-vwan --resource-group  ${USER}-workshop-sdwan --location your-hub-location --type Standard

  If you are prompted to install the extension virtual-wan answer Y

  az network vhub create --address-prefix 10.14.0.0/16 --name ${USER}-vwanhub --resource-group ${USER}-workshop-sdwan --vwan sdwan-${USER}-workshop-vwan --location your-hub-location --sku Standard

  The second command can take several minutes to run, do not Ctrl-C to break out or stop the command. If your Cloudshell session disconnects, reconnects and run ps -ef to determine if az network vhub create... command is still running. Once the command is no longer seen in the ps output the VWAN should be created. Use the command az network vhub list to view your VWAN hub.

  

• Navigate to your Resource Group and verify that you see your vWAN

  

• Click on your vWAN and verify that you see the virtual Hub you just deployed

  

• Click on the vWAN Hub and verify that the deployment and routing status complete

  

• Go to your resource Group and then click on the Hub VNET - USERXX-workshop-sdwan-hub1

• Delete the Hub to Spoke VNET peerings, delete both Spoke11 and Spoke12 peerings

  

• Create Virtual WAN Route Tables

  • Click on your virtual Hub and then click on Route Tables

    

  • Create a Route Table Called Spoke-VNETS. Keep all other settings unchanged

    

  • Repeat the same for FGT vWAN Route Table: FGT-VNET

    

• Create Virtual WAN VNET Connections

  • Go to the vWAN, Click on Virtual Network Connection

    

    • Create a VNET connection for Spoke11, attach it to the Spoke-VNETS Route Table and propagate it to FGT-VNET Route Table - select your resource group and VNET - USERXX

      

    • Repeat the same for Spoke12

    • Repeat the same for FGT VNET connection, attach it to the FGT-VNET Route Table but do not propagate to other Route Tables.

      

      • Is the connection for FGT VNET created?
      • Why not?

    • Locate your own Azure Route Server and delete it

      

    • Try now to connect the FGT VNET to the vWAN Hub, attach it to the FGT-VNET Route Table.

      • Is the connection for FGT VNET created, this try?

      • Why did it work?

        

• Configure Spoke-VNETS Route Table

  • Go your vWAN Hub, click on Routing and then click on Spoke-VNETS Route Table

    

    • Add a default route that points to the FortiGate VNET connection. The next hop ip is the Primary FGT port2 ip

    • Verify that this default route has been propagated to the Spokes VNETs

      • Go to the Spoke11 Linux VM -> Networking -> Click on nic and then click on Effective Routes

      

• At the end of this step you should have the following architecture

  

• Connect to the Branch1 Linux Host via the serial console

• Generate traffic to Hub

   ping 10.11.1.4

• Does it work ?

  

• Troubleshoot and make all the required changes to make it work

  • Hints:

    ---

    • FGT Branch1 does it learn routes to spokes from the Hub?
    • Configure the Hub FGT to advertise Spoke11 and Spoke12 CIDRs to the Branches

      • On the Hub FGT, add Static Routes to Spoke11 and Spoke12. What would be the next-hop?

      • Add Spoke11 and Spoke12 to the list of networks under BGP configuration

        config router bgp
          config network
              edit 1
                  set prefix 10.10.255.1 255.255.255.255
              next
              edit 2
                  set prefix 10.11.0.0 255.255.0.0
              next
              edit 3
                  set prefix 10.12.0.0 255.255.0.0
              next
          end
        end

      • Verify that Branches are now receiving Spoke11 and Spoke12 CIDRs. Use the command get router info routing-table all

      • Does it work now or not yet ?

      • Take a packet capture on the Hub, Do you see echo-requests arriving?
        diagnose sniffer packet any 'net 10.11.0.0/16' 4 0 a

      • Traffic is egressing the Hub FGT on port2, but you don't see any reply?... What is missing?

        • Check FGT Hub port2 effective routes?
        • Do you see spoke11 and spoke12 CIDRs? Why is the vWAN not propagating them to the Route Table attached to the FGT private subnet, sdwan-USERXX-workshop-hub1_fgt-priv_rt?
        • Check the Route Table configuration settings

        

1. Why were we not able to attach the Hub FortiGate VNET to vWAN until we deleted Azure Route Server?

2. Why was the vWAN not able to inject Spoke11 and Spoke12 VNETs CIDRs to FortiGate Private UDR?

3. The above setting is normally set to "yes", why did we set it to "no" ? Hint: We had Azure Route Server before

4. In the Spoke-VNETS vWAN Route Table, the next-hop is the Primary FortiGate IP. What should we add/do to handle failover?