This lab will show you how to bring advanced ML vision capabilities to your applications with the Custom Vision Service. The service makes it easy to build custom image classifiers and provides APIs and tools to help you improve your classifier over time. The lab is split in two parts: first, we'll show you how to use the Custom Vision portal to build a classifier and then we'll show you how to do the same thing using the SDK. In both cases, we'll be building a classifier that can distinguish between a Surface Pro and a Surface Studio.

The Custom Vision portal requires either a work or school account (e.g. Office 365 accounts) or a personal Microsoft account (e.g. @outlook.com). You can create a new personal Microsoft account here (signup.live.com) with an existing email address (or create a new one).

The first part of the lab, where the portal is used to build a classifier, only requires a modern web browser.

The second part of the lab, where the SDK is used to build a classifier, requires Visual Studio 2017 on Windows. Get it here. While the SDK in preview is Windows-only, the underlying RESTful APIs can be accessed from any platform. Learn more about the training API here and the prediction API here.

Before we can start training our classifier, we need to create a project in the Custom Vision portal. The Custom Vision Service is currently in preview so we don't need to provision a subscription in Azure (as is the case with most of the other Cognitive Services).

1. Open a browser and navigate to the Custom Vision (customvision.ai) portal.
2. Log in using a valid account (learn more above).
3. If this is your first time visiting the portal, it will request some permissions. Click the Yes button to agree (you can revoke these permissions later if necessary).
4. If this is your first time visiting the portal, it will also prompt you to agree with the terms and conditions. Check the box to indicate consent and then click the I agree button.
5. Click New Project.
6. Provide the required values:
   • Name: customvisionlab
   • Description: Custom Vision Lab
   • Domains: General (i.e. the default)
7. Click Create Project.

Training an image classifier requires example images. While our data set is relatively small and can be uploaded via a web browser, it's likely that production scenarios will require hundreds or thousands of images and thus are best uploaded via the API (covered later in the lab).

1. Click the Training Images tab at the top of the page.
2. Click the Add images button.
3. Click the Browse local files button.
4. Browse to $/Training/Images/surface-pro and select all of the images.
5. Click Open.
6. In the Add some tags to this batch of images field, enter the tag surface-pro.
7. Click on the + button next to the field.
8. Click the Upload 15 files button.
9. Wait for the upload to complete. This can take a few moments.
10. Click the Done button.
11. Click the Add images button.
12. Click the Browse local files button.
13. Browse to $/Training/Images/surface-studio and select all of the images.
14. Click Open.
15. In the Add some tags to this batch of images field, enter the tag surface-studio.
16. Click on the + button next to the field.
17. Click the Upload 15 files button.
18. Wait for the upload to complete. This can take a few moments.
19. Click Done.

Now that our data set is available, and each image correctly tagged, the training process can begin. This is a quick operation given the small size of our data set. The output is a classifier that can be used to make predictions. As more training data is added, we can create a new iteration of the classifier and analyze its performance.

1. Click on the green Train button in the top nav bar.
2. Wait for the training to complete.
3. Review the new classifier's precision and recall. Mouse over the tooltips to learn more about these concepts.

It's recommended that at least 30 images of each class or category are included for a prototype classifier. In the interests of time, we're only providing 15 images for each class so we should expect slightly lower accuracy compared to a fully trained version.

The portal provides a test page for us to do a basic smoke test of our model with some new, unseen data. In a production scenario, the RESTful API is a more appropriate mechanism for doing bulk predictions (covered later in the lab).

1. Click on the Quick Test button in the top nav bar.
2. Click the Browse local files button.
3. Browse to $/Prediction/Images/surface-pro and select the pro_test.jpg image.
4. Click Open.
5. Wait for the classification to run and ensure it is correctly recognized as a surface-pro.
6. Click the Browse local files button.
7. Browse to $/Prediction/Images/surface-studio and select the studio_test.jpg image.
8. Click Open.
9. Wait for the classification to run and ensure it is correctly recognized as a surface-studio.
10. Close the Quick Test side bar by clicking on the X icon in the top right.
11. Click on the Predictions tag in the top nav bar.
12. Here we can see the new test data. If we click on an image, we can manually tag it and add it to our data set. This image will then be included next time we re-train our classifier, potentially leading to improved accuracy.

Now that we've seen the Custom Vision Service in action, next we'll examine how we can use the API to complete the same process. Using the API helps us train and predict more efficiently. In order to accelerate the lab, we've prepared a basic console application as a starting point. In this lab, we'll be using the SDK - rather than the RESTful API - but the pattern of interaction is very similar.

1. Open Visual Studio 2017 from the Start Menu.
2. Click Open Project/Solution from the File > Open menu.
3. Select the solution file $/CustomVision.sln and wait for it to load.
4. Right-click the Training project and select Manage NuGet Packages....
5. Click on the Browse tab.
6. Enter Microsoft.Cognitive.CustomVision.Training in the search box and press Enter.
7. Click on the matching result and then click Install in the right panel. When prompted, click OK and then I Accept.

The first step is to upload training data. In our case, the images are local so will be uploaded directly to the service; however, it's also possible to submit public URLs to training images and the service will automatically retrieve them. Once the images are uploaded, we can start the training process. This is an asynchronous operation so we need to monitor it.

1. Open the Program.cs file in the Training project.
2. Add the UploadTrainingData method to upload the new training images to our project:

   private static void UploadTrainingData(Guid projectId, TrainingApi trainingApi)
   {
       Console.WriteLine("Uploading images");
       foreach (var tagName in new[] {"surface-pro", "surface-studio"})
       {
           var tag = trainingApi.CreateTag(projectId, tagName);
   
           var images = Directory.GetFiles($@"..\..\Images\{tagName}")
             .Select(f => new ImageFileCreateEntry(Path.GetFileName(f), File.ReadAllBytes(f)))
             .ToList();
   
           var batch = new ImageFileCreateBatch(images, new List<Guid> { tag.Id });
           trainingApi.CreateImagesFromFiles(projectId, batch);
       }
   }

3. Add the TrainClassifier method to invoke the training operation and wait for it to complete:

   private static void TrainClassifier(Guid projectId, TrainingApi trainingApi)
   {
       Console.WriteLine("Training classifier");
   
       var iteration = trainingApi.TrainProject(projectId);
       while (iteration.Status == "Training")
       {
           Thread.Sleep(1000);
           iteration = trainingApi.GetIteration(projectId, iteration.Id);
       }
   
       // Make the newly trained version the default for RESTful API requests
       iteration.IsDefault = true;
       trainingApi.UpdateIteration(projectId, iteration.Id, iteration);
   
       Console.WriteLine("Training complete. Press any key to exit.");
   }

4. Add the following code to the Main method to retrieve credentials and invoke our new methods:

   var trainingApi = new TrainingApi();
   trainingApi.ApiKey = ConfigurationManager.AppSettings["trainingKey"];
   
   var projectName = ConfigurationManager.AppSettings["projectName"];
   var project = trainingApi.GetProjects().Single(p => p.Name == projectName);
   
   UploadTrainingData(project.Id, trainingApi);
   TrainClassifier(project.Id, trainingApi);
   
   Console.ReadKey();

In order to interact with the training API, we need the key. This gives us access to all projects under our Custom Vision Service account. To identify the correct project to work with, we need the project name as well.

1. Return to the browser and the Custom Vision (customvision.ai) portal.
2. Click the home icon in the top left corner of the portal.
3. Click New Project.
4. Provide the required values:
   • Name: customvisionlab-api
   • Description: Custom Vision Lab - API
   • Domains: General (i.e. the default)
5. Click Create Project.
6. Click on the settings icon in the top right corner of portal.
7. Copy the Training Key located under the Subscription Keys section.
8. Return to Visual Studio.
9. Open the App.config file in the Training project.
10. Find the trainingKey app setting and replace <your_key> with the key you copied to clipboard.

    <add key="trainingKey" value="<your_key>" />

11. Find the projectName app setting and update it with your project name (customvisionlab-api).

    <add key="projectName" value="<your_project>" />

The application will use the images under $\Training\Images\{tag} in the training process. The entire process should only take a few moments given the small size of our data set.

1. In Visual Studio, right-click the Training project.
2. Under Debug, click Start new instance.
3. Wait for the upload and training process to complete.
4. Once you see the Training complete message, press any key to exit.
5. Return to Microsoft Edge and the Custom Vision (customvision.ai) portal.
6. Open your project (customvisionlab-api) and click the Performance tab.
7. Here we can see the classifier's precision and recall.

Now that the classifier is in place, we can use the prediction endpoint to classify new, unseen data. This can be accomplished with the prediction API.

1. Return to Visual Studio.
2. Right-click the Prediction project and select Manage NuGet Packages....
3. Click on the Browse tab.
4. Enter Microsoft.Cognitive.CustomVision.Training in the search box and press Enter.
5. Click on the matching result and then click Install in the right panel.

   In our app, the training SDK is also required in order to resolve the project ID from the project name. The alternative would be to add the project ID to configuration and then this dependency can be eliminated.

6. Enter Microsoft.Cognitive.CustomVision.Prediction in the search box and press Enter.
7. Click on the matching result and then click Install in the right panel. When prompted, click OK and then I Accept.
8. Open Program.cs in the Prediction project.
9. Add GetProjectId method to resolve the project ID from the project name in our config file:

   private static Guid GetProjectId()
   {
       var trainingApi = new TrainingApi();
       trainingApi.ApiKey = ConfigurationManager.AppSettings["trainingKey"];
   
       var projectName = ConfigurationManager.AppSettings["projectName"];
       var project = trainingApi.GetProjects().Single(p => p.Name == projectName);
   
       return project.Id;
   }

10. Add GetPredictionEndpoint method to retrieve credentials and prepare the SDK to make predictions:

    private static PredictionEndpoint GetPredictionEndpoint()
    {
        var predictionEndpoint = new PredictionEndpoint();
        predictionEndpoint.ApiKey = ConfigurationManager.AppSettings["predictionKey"];
        return predictionEndpoint;
    }

11. Add the following code to the Main method to invoke the prediction endpoint:

    var projectId = GetProjectId();
    var endpoint = GetPredictionEndpoint();
    
    Console.WriteLine("Testing images");
    foreach (var tagName in new[] {"surface-pro", "surface-studio"})
    {
        var images = Directory.GetFiles($@"..\..\Images\{tagName}");
        foreach (var image in images)
        {
            var memoryStream = new MemoryStream(File.ReadAllBytes(image));
            var prediction = endpoint.PredictImage(projectId, memoryStream);
            
            var topPrediction = prediction.Predictions.First();
            Console.WriteLine($"{image} predicted to be {topPrediction.Tag} ({topPrediction.Probability:P1})");
        }
    }
    
    Console.WriteLine("Press any key to exit");
    Console.ReadKey();

In addition to the information required by the training API to resolve the project ID, a separate prediction API key is required to access the prediction functionality.

1. Open the App.config file in the Prediction project.
2. Find the trainingKey app setting and replace <your_key> with the key from the Training project App.config.

   <add key="trainingKey" value="<your_key>" />

3. Find the projectName app setting and replace <your_project> with the project name from the Training project App.config.

   <add key="projectName" value="<your_project>" />

4. Return to the browser and the Custom Vision (customvision.ai) portal.
5. Click on the settings icon in the top right corner of portal.
6. Copy the Prediction Key located under the Subscription Keys section.
7. Return to Visual Studio.
8. Find the predictionKey app setting and replace <your_key> with the key you copied to clipboard.

   <add key="predictionKey" value="<your_key>" />

The application will use the images under $/Prediction/Images/{tag} and return the most likely predictions for each.

1. In Visual Studio, right-click the Prediction project.
2. Under Debug, click Start new instance.
3. Review the output and ensure the images are correctly classified. The pro_test.jpg should be classified as a surface-pro and the studio_test.jpg should be classified as a surface-studio.
4. Once prompted, press any key to exit.
5. Return to the browser and the Custom Vision (customvision.ai) portal.
6. Open your project (customvisionlab-api) and click the Predictions tab.
7. As before, we can see the new data collected by the prediction endpoint. If we click on an image, we can manually tag it and add it to our data set.

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.