• Login to Azure Cloud Portal https://portal.azure.com/ with the provided login/password

  

• Click on Cloud Shell icon on the Top Right side of the portal

  

• Select Bash

  

• Click on show advanced settings

  

• Select

  • Use existing Resource Group - it should auto populate with studentXX-workshop-sdwan

  • Use existing Storage account - it should auto populate with setrainstudentXX######## (######## is a random string)

  • Use existing File Share - type cloudshell

    

• You should now have access to Azure Cloud Shell console

  

• Clone the Github repo git clone https://github.com/FortinetSecDevOps/se-conf-sdwan-workshop.git

  

• Change to the se-summit folder cd ./se-conf-sdwan-workshop/se-summit/

• Initialize Terraform

  • Run terraform init

• Create Terraform Plan

  • Run terraform plan -var="username=${USER}"

• Apply Terraform Plan

  • bash run terraform apply -var="username=${USER}" and then answer yes
    • This command can also be run as terraform apply -var="username=${USER}" -auto-approve which removes the need to type yes

• At the end of this step you should have the following architecture

  

• Using the Terraform output, verify that you have Web and SSH access to the FortiGates.

  

• Terraform output can be redisplayed at any point as long as you are in the ./se-conf-sdwan-workshop/se-summit/ directory, by using the command

  • terraform output

• Connect to the Branch sites FortiGates and check the VPN status. If they are down try to bring them UP.

• FortiGates in the Hub do not have public IPs, how are they accessible via the Web UI?
• Why the VPN connections are still down?

• Go to the Hub External Load Balancer sdwan-studentXX-workshop-hub1-elb1

• Click on Backend pools

• Add FortiGate1 and FortiGate2 port1 interfaces and then click on Save

  

• Click on the Hub External Load balancer sdwan-studentXX-workshop-hub1-elb1

• Click on Load balancing rules

• Create Load balancing rules for UDP 500 and UDP 4500 - one rule for each

  

• Verify that the FortiGates are responding to Azure Load Balancer Health Checks

  • Click on the Hub External Load balancer

  • Click on Insights - Click the "Refresh" button a few times, eventually (~30 seconds) the FortiGate reachability will be indicated.

    

• Verify that the VPN to the Hub are UP (You may have to reboot the Branch FortiGate one time if the VPN does not come up)

  

• Verify that the BGP peering with the hub is UP and that the Branch FortiGate learned the Hub and other Branches' CIDRs get router info routing-table all

• At the end of this step you should have the following architecture

  

• Why is one FortiGate depicted as unhealthy by the Azure Hub External Load Balancer?
• Why is NAT used to access the FortiGates, but for IPSEC VPN traffic Load balancing rules are used?
• Do FortiGates in the Branches learn Spoke11 and Spoke12 CIDRs?

• Create a VNET peering between the Spoke11 VNET and the Hub VNET

  • Go to the Spoke VNET, studentXX-workshop-sdwan-spoke11 - (replace studentXX with your username)

    • Click on Peerings
    • Add peering to Hub VNET, studentXX-workshop-sdwan-hub1

  • Repeat the above between Spoke12 VNET, studentXX-workshop-sdwan-spoke12 and the Hub VNET

    

• Verify that the Branch FortiGates have learned the Spoke11 VNET and Spoke12 VNET CIDRs

• Go to Azure Route Server

  • studentXX-workshop-sdwan-RouteServer contained within your Resource Group.

    

• Click on Peers on the left side of the menu, verify the connection to the Hub FortiGates

• List the routes learned by Azure Route Server, run the commands below from your Azure Cloud Shell

• The variable ${USER} in the commands reads your username from the environment

az network routeserver peering list-learned-routes -g ${USER}-workshop-sdwan --routeserver ${USER}-workshop-sdwan-RouteServer --name sdwan-fgt1
az network routeserver peering list-learned-routes -g ${USER}-workshop-sdwan --routeserver ${USER}-workshop-sdwan-RouteServer --name sdwan-fgt2

The passive FortiGate will produce empty output

{
  "RouteServiceRole_IN_0": [],
  "RouteServiceRole_IN_1": [],
  "value": null
}

• Is your Hub FortiGate able to see the Dynamic filters ?

  • Troubleshoot and Make the required changes to allow the FortiGate to retrieve the SDN filters.

    • Hub FortiGate debug the Azure SDN Connector

      diagnose debug application azd -1
      diagnose debug enable

  • Hints:

    ---

    • FGT Branch3 is able to retrieve the filters, why that is not the case for the FortiGates behind Load Balancers?
    • FGT Branch3 is standalone, all other FortiGates are in A-P HA, how does that affect traffic to retrieve SDN filters?
    • Hub External Load Balancer needs a management nic backend pool and a TCP rule any port suffices. This rule is about letting TCP traffic out. The External Load Balancer will let the response traffic back in because the traffic originated internally.

    

    

    

    

• On the Hub FortiGate, create a dynamic address object named Spoke_VNETs that resolves to the Spoke VNETs VMs

  

• On the Hub FortiGate, use the object created above on the Branch to Cloud policy to restrict traffic coming from the Branches

  

• Generate Traffic from Branch1 Primary FortiGate:

  1. Connect to the Branch1 Primary FortiGate
  2. Configure ping-options to initiate traffic from FortiGate's private nic (port2).
     • execute ping-options source 172.16.2.5 - source IP depends on which Branch1 FortiGate is primary br1fgt1 or br1fgt2
     • execute ping-options repeat-count 100
  3. Initiate a ping to Spoke11 and Spoke12 Linux VMs (10.11.1.4 and 10.12.1.4)
     • execute ping 10.11.1.4
     • execute ping 10.12.1.4

  

  

• Generate Traffic from Branch1 Linux VM:

  1. Enable serial console access on Branch1 Linux VM

     • Click on the VM studentXX-sdwan-workshop-br1lnx1

     • Go to Boot diagnostics -> Settings -> Select Enable with custom storage account

     • From the dropdown list, select the storage account that is assigned to your username - setrainstudentXX#######

     • Click Save

       

  2. Go to the VM Serial Console

  3. Initiate a ping to Spoke11 and Spoke12 Linux VMs

     ping 10.11.1.4
     ping 10.12.1.4 

     

  4. Does it work?

• At the end of this step you should have the following architecture

  

• What was missing to allow the FortiGates to retrieve SDN connector filters?
• Why is the FortiGate only able to retrieve the SDN connector filters in its own Resource Group?
• Why is the Branch FortiGate able to reach the remote Spoke VNETs VMs (10.11.1.4 and 10.12.1.4) but the Linux VM behind the Branch1 FortiGate cannot?
• FortiGates at Branch1 and Branch2 site are both behind Azure Load Balancers (behind NAT). Will Branch1 to Branch2 traffic successfully establish an ADVPN shortcut?

• Click on the Branch1 private route table studentXX-sdwan-workshop-branch1_rt
• Add a default route for 0.0.0.0/0 that points to the Branch1 Internal Load balancer listener IP
• Repeat the previous step for the Branch2 and Branch3 Route Tables

  • Be sure to use the correct IP as the next hop, that is the correct Internal Load balancer listener IP or FortiGate internal interface. Hint: is the next hop a load balancer or a stand-alone FortiGate

    

    

• Connect to the Branch1 Linux Host via the serial console

• Generate traffic to Hub

   ping 10.11.1.4
   ping 10.12.1.4 

• Does it work now ?

• Go to your resource group and click on Spoke11 Linux VM

• Click on Networking in the Navigation Menu

  

• Click on the VM nic

• Click on Effective routes

• Check that Azure Route Server has injected the Branch sites CIDRs learnt from the FGT

• Go to your resource group and click on the Hub FGT VM

• Click on Networking in the Navigation Menu

  

• Click on the VM port2 nic

• Click on Effective routes

• Has Azure Route Server injected the Branch sites CIDRs learnt from the FGT? Why ?

• Connect to the Branch1 Linux Host via the serial console - studentXX-sdwan-workshop-br1lnx1

• Generate traffic to Branch2 Linux Host

    ping 172.17.5.4

• Check if an ADVPN shortcut has been created

• Why has the Azure Route Server (ARS) injected Branch site CIDRs to the Spoke VNET protected subnet but not the FortiGate private subnet?
• The Branch external Load balancer has two front end public ip. How do we ensure that traffic egressing Branch1 on port1 (isp1) has always the same public ip applied? Same for traffic egressing Branch1 on port3 (isp2)

• Connect to the Branch1 Linux Host via the serial console - studentXX-sdwan-workshop-br1lnx1
• Ping a resource in a remote branch site - sdwan-studentXX-workshop-spoke11-subnet1-lnx
  • ping 10.11.1.4
  • Let the ping run

• Connect to the Branch1 Primary FortiGate . Initiate a failover by rebooting the primary FortiGate or by forcing a failover via the CLI

  execute ha failover set 1

• Monitor the number of lost Pings and the failover time

• How long did it take?

• Have the VPNs to the Hub been renegotiated upon failover or maintained?

• Ensure that both Branch1 FortiGates in the cluster are up and running

• Connect to the Branch1 Linux Host via the serial console - studentXX-sdwan-workshop-br1lnx1

• Generate an SSH session to the Spoke Linux VM - sdwan-studentXX-workshop-spoke11-subnet1-lnx

  ssh [email protected]

• From Spoke Linux VM SSH session generate a continuous stream of connections to track the failover event

  while true; date; do curl -I -sw '%{http_code}'  https://www.lemonde.fr/ ; echo -e "\n================="; sleep 1 ; done

• Connect to the Branch1 Primary FortiGate . Initiate a failover by rebooting the primary FortiGate or by forcing a failover via the CLI

• Monitor the SSH connection

• Did you lose the TCP connection?

• How long was your failover time?
• Why did we lose the SSH (TCP) session with a "short" failover time?

• Create your vWAN and the vWAN Hub using the CLI commands below. Use the Hub FortiGate location for the VWAN location.

  • You can find the location of your Hub FortiGates with this Azure CLI Command

  az vm show -g ${USER}-workshop-sdwan -n sdwan-${USER}-workshop-hub1-fgt1 -o table

• The variable ${USER} in the commands reads your username from the environment

  az network vwan create --name sdwan-${USER}-workshop-vwan --resource-group  ${USER}-workshop-sdwan --location your-hub-location --type Standard

  If you are prompted to install the extension virtual-wan answer Y

  az network vhub create --address-prefix 10.14.0.0/16 --name ${USER}-vwanhub --resource-group ${USER}-workshop-sdwan --vwan sdwan-${USER}-workshop-vwan --location your-hub-location --sku Standard

  

• Navigate to your Resource Group and verify that you see your vWAN

  

• Click on your vWAN and verify that you see the virtual Hub you just deployed

  

• Click on the vWAN Hub and verify that the deployment and routing status complete

  

• Go to your resource Group and then click on the Hub VNET - studentXX-workshop-sdwan-hub1

• Delete the Hub to Spoke VNET peerings, delete both Spoke11 and Spoke12 peerings

  

• Create Virtual WAN Route Tables

  • Click on your virtual Hub and then click on Routing

    

  • Create a Route Table Called Spoke-VNETS. Keep all other settings unchanged

    

  • Repeat the same for FGT vWAN Route Table: FGT-VNET

    

• Create Virtual WAN VNET Connections

  • Go to the vWAN, Click on Virtual Network Connection

    

    • Create a VNET connection for Spoke11, attach it to the Spoke-VNETS Route Table and propagate it to FGT-VNET Route Table - select your resource group and VNET - studentXX

      

    • Repeat the same for Spoke12

    • Repeat the same for FGT VNET connection, attach it to the FGT-VNET Route Table but do not propagate to other Route Tables.

      

      • Is the connection for FGT VNET created?
      • Why not?

    • Locate your own Azure Route Server and delete it

      

    • Try now to connect the FGT VNET to the vWAN Hub, attach it to the FGT-VNET Route Table.

      • Is the connection for FGT VNET created, this try?

      • Why did it work?

        

• Configure Spoke-VNETS Route Table

  • Go your vWAN Hub, click on Routing and then click on Spoke-VNETS Route Table

    

    • Add a default route that points to the FortiGate VNET connection. The next hop ip is the Primary FGT port2 ip

    • Verify that this default route has been propagated to the Spokes VNETs

      • Go to the Spoke11 Linux VM -> Networking -> Click on nic and then click on Effective Routes

      

• At the end of this step you should have the following architecture

  

• Connect to the Branch1 Linux Host via the serial console

• Generate traffic to Hub

   ping 10.11.1.4

• Does it work ?

  

• Troubleshoot and make all the required changes to make it work

  • Hints:

    ---

    • FGT Branch1 does it learn routes to spokes from the Hub?
    • Configure the Hub FGT to advertise Spoke11 and Spoke12 CIDRs to the Branches

      • On the Hub FGT, add Static Routes to Spoke11 and Spoke12. What would be the next-hop ?

      • Add Spoke11 and Spoke12 to the list of networks under BGP configuration

        config router bgp
          config network
              edit 1
                  set prefix 10.10.255.1 255.255.255.255
              next
              edit 2
                  set prefix 10.11.0.0 255.255.0.0
              next
              edit 3
                  set prefix 10.12.0.0 255.255.0.0
              next
          end
        end

      • Verify that Branches are now receiving Spoke11 and Spoke12 CIDRs. Use the command get router info routing-table all

      • Does it work now or not yet ?

      • Take a packet capture on the Hub, Do you see echo-requests arriving?
        diagnose sniffer packet any 'net 10.11.0.0/16' 4 0 a

      • Traffic is egressing the Hub FGT on port2, but you don't see any reply?... What is missing?

        • Check FGT Hub port2 effective routes?
        • Do you see spoke11 and spoke12 CIDRs? Why the vWAN is not propagating them to the Route Table attached to the FGT private subnet, sdwan-studentXX-workshop-hub1_fgt-priv_rt?
        • Check the Route Table configuration settings

        

• Why were we not able to attach the Hub FortiGate VNET to vWAN until we deleted Azure Route Server?

• Why was the vWAN not able to inject Spoke11 and Spoke12 VNETs CIDRs to FortiGate Private UDR?

• The above setting is normally set to "yes", why did we set it to "no" ? Hint: We had Azure Route Server before

• In the Spokes-VNET vWAN Route Table, the next-hop is the Primary FortiGate IP. What should we add/do to handle failover ?