Introduction to the Drone Journey

This post is the seventh post in documenting the steps I went through on my journey to build an autonomous, voice-controlled, face recognizing drone. There are 6 other posts building up to this one which you can find at the end of this post.

Focus of this post

In this post I am going to pick-up where we left off and look at :

• How to use the Microsoft Cognitive Services Face API to recognize a specific face. Specifically we will explore the face.identify approach.
• How to build the identify approach into your DroneWebServer.js file such that the drone will land when it sees a named person.

My issues

I have to admit to facing quite some challenges getting face.identify to work. Using the face.identify capability requires a large number of things to be completed, in a very specific order, before it will work.

Node.js does not make doing things in a specific order easy without nested functions, callbacks and other things.  The truth is that I wasted a large number of hours before I realized that the asynchronous nature of node.js was a major source of many of  the issues I was  running into rather than me misusing the Cognitive Services APIs o any issue with the APIs themselves.

I also found that the online examples sometimes do not really give you the step by step help you might need. this looked to mostly be the case in the Node.JS world as other programming languages  seemed much more comprehensive. In the node.js world you essentially get the code to look at and try to understand which does not help with “ordering” issues.

When you add to that it is entirely possible to create things, people for example, with the same name multiple times (without realizing it) you can see where my issues came from.

I did speak with Lukas again at this stage. He told me that he set up his target faces using the SDK console rather than through code so I hit a dead end there as I was determined to try avoid that (I must admit I thought about it ;)..).

This is where my first outreach to someone at Microsoft came. I contacted Chris Thrasher who very kindly sent me some code samples he had worked on. He too found the challenge of things not being called in order which he solved using the “bluebird” package.

Thanks to his pointer and his sample code I identified a spot I was not doing something right which allowed me to move forwards.  I want to be sure to say thanks to Chris because without his help I would have stopped at face.similar.

A warning: This is a monster blog post. I decided to put it all in one rather than break it up. Grab a coffee, give yourself sometime and enjoy the ride!

Breaking it into two blocks of work

As with everything in coding the best approach is to split things into smaller problems and then solve them at that level. Ultimately you can build things up to get to where you want to go. The use of face.identify certainly calls for that approach. There are many steps needed to setup before you can start to use what you have to identify faces.

Much like in the face.similar case there are some setup tasks. As in that case I defaulted to using functions I could comment out selectively to let me execute things in the right ode.

One additional thing I did, due to the amount of setup, was split the process of setting up people and faces from the process of actually running the identification.

I created 2 files. The first was called setupFaceIdentify.js which I  put all the functions in which I needed to setup my target people and faces. The second was called FaceIdentify.js which I would use to actually perform the task of identification.

setupFaceIdentify.js

By now you know the first thing we need to do is include the project-oxford package, define our API key and create the client object we will use.

var oxford = require('project-oxford');
var apikey ="INSERT YOUR API KEY HERE"; 
var client = new oxford.Client(apikey); 

To use face.identify we need to:

1. Create a personGroup
2. Create, for each person we want to identify, a person within that personGroup and a series images that capture them
3. Run Training on the personGroup so that it can be used for identification

I also found that along the way it made sense to have a few other functions. These are ones to “Delete a personGroup” and to “Get a list of the people in a PersonGroup”. The first one I found important as I sometimes had a person in the group twice with different person ids and it helps to wipe out the group as part of a clean up and restart. The second is useful for debugging if something does not seem to be working which is how I found I had multiple copies of the same person at one stage.

Creating a personGroup

Below is the function I created to create a personGroup. The first thing it does is make a called to face.personGroup,get passing in a specific groupId. The call to get returns an object, if it exists, and if not it returns an error which we can use to then drive the creation.

If the error from the get call results in ‘PersonGroupNotFound’ being returned then we can call the function that actually creates the personGroup. The function we call is face.personGroup.create which takes two parameters. The first is the group ID and the other is the group description. For simplicity I use the one thing for both.

 function GroupCreate(groupId) {
   client.face.personGroup.get(groupId)
      .then(function(response) {
          console.log ("Group Exists: " + groupId);
      })
      .catch(function(error) {
          if (error.code == 'PersonGroupNotFound') {
             client.face.personGroup.create(groupId,groupId)
            console.log ("Created Group:" + groupId);
          } else {
              console.log ("Ran into an error");
          }
      })
 }

We can then add the call to this function at the end of our setupFaces.js as shown below.

var myGroup = "drone_demo_group";
GroupCreate(myGroup);

Executing this using node setupFaces.js gets the following output.

The corresponding delete function is shown below.

 function GroupDelete(groupId) {
   client.face.personGroup.delete(groupId)
      .then(function(response) {
          console.log ("Deleting Group " + groupId);
      })
      .catch(function(error) {
            console.log ("Group not found or other error")
       })
 }

We can then comment out the call to the GroupCreate function and instead make a call to the GroupDelete function as shown below

var myGroup = "drone_demo_group";
GroupDelete(myGroup);
//GroupCreate(myGroup);

This will give you the following output when you execute node setupFaces.js.

Because of the way node.js works if you have both lines uncommented, and therefore asynchronously executing, the first one to complete will report its answer and do its deed which then causes issues sometimes for the second one. This race condition caused me pain so I stuck to the simple process of stepping one by one commenting and uncommenting while setting up.

Now that we have the personGroup the next step I took was to create a helper function that could show me the people contained in the personGroup using the list function.

Listing people in a personGroup

This is pretty simple to do. There is a function face.person.list to which you can pass a group ID and it will display all the people.  After first creating the group it will be empty. You can see the code to do that below.

function displayPersonList(groupId) {
   client.face.person.list(groupId)
   .catch(function(e) {
      console.log(e); // "something went wrong"
   }).then(function (response){
      console.log("Person List for Group:" + groupId); 
      console.log(response); 
   });
}

We can now call that function on our existing group (if your last call had deletion you need to make the call to create it again).

var myGroup = "drone_demo_group";
 //GroupDelete(myGroup);
 //GroupCreate(myGroup);
 displayPersonList(myGroup);

Provided the list exists you will get this response back when you execute node setupFaces.js. You will see an empty array [] being passed back. This is normal as we have not yet added any people 🙂

The next step is to add people and their associated images to our personGroup.  There are actually two steps to this. The first is to add the person themselves using face.person.create and the second is to associate images of that person with the person you have just created using face.person.addFace. I again created a function to do this which allowed me to pass in a single person and multiple images at once to save on calls. You can see it below (note two lines are wrapped that need to be on one line – you will spot them).

Essentially we are calling face.person.create and passing it the groupId we want to create the person in plus a name. This in turn returns a personId which we then use when using the  face.person.addFace  function along with the groupId.  Essentially the output of one function is driving the execution of the second one and then we are using the callback capabilities to make sure this runs in the right order.

 function addPerson(groupId, person) {
        console.log('adding person(' + person.name + ')');
        client.face.person.create(groupId, person.name)
            .then(function(response) {
                var personId = response.personId;
                var i;
                console.log(response);
                for(i = 0; i < person.images.length; i++) {
                    console.log('adding person(' + person.name + ').face(' + person.images[i] + ')');
                    client.face.person.addFace(groupId, personId, {url: person.images[i]});
                }
            })
           .catch(function(error) {
             console.log(error);
           })
 }

It is important to note that in this example I am passing in URLs. You will see this in the JSON being formed as part of addFace. if you want to pass in a local file you need to change url: to path: , in the relevant places is the code, and then simply provide a local file.  I have also left a lot of the logging I used in this sample so you will see the things it is doing in your console window when you execute it. You may want to delete or comment those out later.

At this stage you could have multiple lines calling addPerson with different parameters. I went one step further and created additional functions for specific people. I did this mostly as I thought it made commenting and uncommenting easier plus when testing it was easier to see who I had loaded and who I had not loaded into my training faces.

Two examples of these calls are below. One adds Donald Trump and the second adds me :).

Again be wary of the wrapping text in the sample code below.  The format is clear but if in doubt click the thumbnail  image, on the right, of the code which will show you it laid out in my code editor.

function addTrump(groupId){
      addPerson(groupId, {name: 'Donald Trump',
          images: [                   
'https://upload.wikimedia.org/wikipedia/commons/thumb/d/d2/Donald_Trump_August_19%2C_2015_%28cropped%29.jpg/450px-Donald_Trump_August_19%2C_2015_%28cropped%29.jpg',
'http://static6.businessinsider.com/image/55918b77ecad04a3465a0a63/nbc-fires-donald-trump-after-he-calls-mexicans-rapists-and-drug-runners.jpg',
                'http://cdn.bgr.com/2016/01/donald-trump.png',
                'http://i2.cdn.cnn.com/cnnnext/dam/assets/151029092255-donald-trump-pout-large-169.jpg'           
              ]})       
}

function addMark(groupId){
      addPerson(MyGroup, {name: 'Mark Torr',
            images: [   
          'http://www.iweek.co.za/images/stories/2010/MARCH_2013/mark_torr.jpg'
            ]})
}

With those functions we can now amend our code and first make a call to addTrump using my now famous patented approach by editing the setupFaces.js to include the functions above and to comment out and add the relevant function.

var myGroup = "drone_demo_group";
 //GroupDelete(myGroup);
 //GroupCreate(myGroup);
 //displayPersonList(myGroup);
addTrump(myGroup);

Once you have done that you can go ahead and include me as well 🙂

var myGroup = "drone_demo_group";
//GroupDelete(myGroup);
//GroupCreate(myGroup);
//displayPersonList(myGroup);
//addTrump(myGroup);
addMark(myGroup);

If all has gone well then the two successive executions of setupFaces should have seen you get this output.

Now running displayPersonList(myGroup); produces this output.

Here we can see our group. We can see two people in it. We can see 4 faces associated with one person with persistedFaceIds and we can see one with the second person. All is good.  You should always check this before you do the next step as if this is wrong other things take a funny turn too and you essentially need to delete the group and start building it all back up again from the beginning (easiest way!).

So just when you think you are all setup there is one more thing you need to do. Once you have everything in place you need to train your personGroup using the face.personGroup.trainingStart function.  My code to do that is below.

It is not too complicated. You pass a groupId to the face.personGroup.trainingStart function and the training happens. Once that is done I am making a called to face.personGroup,trainingStatus to check that everything has gone well.

function GroupTrain(groupId) {
    client.face.personGroup.trainingStart(groupId)
    .catch(function(e) {
       console.log(e); // "All gone pete tong"
    }).then(function (response){     
       client.face.personGroup.trainingStatus(groupId)
       .catch(function(e) {
          console.log(e); // "All gone pete tong"
       }).then(function (response){
          console.log("Training Status:"); 
          console.log(response);
       });     
   });
}

Lets go ahead and call this on our nicely formed personGroup once you have moved the function over into setupFaces.js. You can see everything is commented out except the group ID we are using and the training function.

var myGroup = "drone_demo_group"; 
//GroupDelete(myGroup); 
//GroupCreate(myGroup); 
//displayPersonList(myGroup); 
//addTrump(myGroup); 
//addMark(myGroup); 
GroupTrain(myGroup);

The output that we see is pretty simple if all has gone well.

Congratulations. You stuck with it and you have now got most things ready so we can move onto actually using the face.identify capability. Now you can see why this was a lot harder than face.similar which required far less setup.

Setup FaceIdentify.js

For the next part of this we will now actually use all that setup work and see if we can identify people in images. Since images often have more than one face we will also design this version to handle that (my find.similar example sort of assumed one face per image).

We have 2 major steps:

1. We need to create a mapping so that if we find someone we can discover who they are.
2. We need to write a function that will first use face.detect on an image to get a faceId and then for each face found call our to face.identify

I am not going to sugar coat this. It is not the simplest of code but if you look at it for a while you will get the understanding and I will try to explain as we go along.

Firstly we need to ensure we setup as always with the project oxford package, define our API Key and create the client object. From there we will create 2 new arrays. I will explain what we use them for as we go along.  So we start out with this code

var oxford = require('project-oxford');
var apikey ="INSERT YOUR API KEY HERE"; 
var client = new oxford.Client(apikey);
var idMap = [];
var people = [];

The next thing we will do is a mapping exercise to populate the idMap array. To do that there is a function involved which will call the face.person.list function (we saw that used earlier) to get a listing of “person” in the personGroup back.

For each person that is returned we are going to create a specific entry in the idMap array that is referenced using their personId (which is unique to them) and assign the returned person to them. This will become important later as this is how we will do some lookup.

The function looks as follows.

function mapPersonList(groupId) {
   client.face.person.list(groupId)
   .catch(function(e) {
      console.log(e); // "oh, no!"
   }).then(function (response){
      idMap = [];
      response.forEach(function(person) {
         idMap[person.personId] = person;
      });  
   });
}

You need to add these lines to the end of FaceIdentify.js so that when you call it the function will run. These two lines need to be to run everytime you want to do an identification.

var myGroup = "drone_demo_group"; 
mapPersonList(myGroup);

Executing this, at this stage, with node FaceIdentify.js gets you no output if all goes well. With that task done we move onto the final code which actually takes an image and does all the magic using face.detect and face.identify. The function to do this is pasted below and in the next few lines I will explain what is going off.  Take note of line wraps again.

If we start to look at the function the first thing that is happening is we are making a call to face.detect. In that call to face.detect we are passing in a URL (note you can adjust this to a path on your computer by using path instead of url in the JSON)  and we are telling the function that we want to get a faceId back (returnFaceId:true).

Using call backs (in other worse once that function has responded) we are looking at what comes back. The first check on the reponse.length is ensuring that there were faces found in the image. If there were no faces found then we need to stop what we are doing in this function.

If we have a response then we start some work. When reponse comes back it is actually a series of face faces that have been detected. For each face found we are going to grab them and do some mapping work.

Once we have those faces as an array we will then pass their Ids into the face.identify function along with the groupId and then that face.identify is going to try identify them.

face.identify returns and ordered list of those people that it has found and then there is some mapping work done so that we can match them back to identified people or simply label them as unknown.

Finally we write out the findings.

   function identifyURL(testUrl,groupId) {
       client.face.detect({url:testUrl, returnFaceId:true})
            .then(function (response) {
                if (response.length == 0) {
                    console.log("Response Blank");
                } else {
                     var faceIds = [];
                     var faceMap = [];
                     response.forEach(function(face) {
                         //console.log("Counting Faces In Image");
                         faceIds.push(face.faceId);
                         faceMap[face.faceId] = face;
                     });
                     //console.log(faceMap);
                     client.face.identify(faceIds, groupId)
                          .then(function(response) {
                              //console.log("Identifying Faces In Image");
                              //console.log(response);
                              response.forEach(function(face) {
                           if (face.candidates && face.candidates.length > 0) {
                                    var topCandidate = face.candidates[0];
               faceMap[face.faceId]['person']  = idMap[topCandidate.personId];
              faceMap[face.faceId]['confidence']  = topCandidate.confidence;
                                }
                           });
                     
                           for (var faceId in faceMap) {
                                //console.log("Mapping Faces 3");
                                 people.push(faceMap[faceId]);
                                 //console.log (faceMap[faceId]);
var name = (faceMap[faceId].person && faceMap[faceId].person.name) || '<unknown>';
 
     console.log(name + ' @ ' + JSON.stringify(faceMap[faceId].faceRectangle));
                           }
                     });   
                }
          })   
    }

Once you have got this far you can make a call to the function passing in a URL from the Web (note I found some URLs simply do not comply. The only way I could check is by trying it in the test area in the documentation).

Once such sample call might be this one.

identifyURL('http://cdn.bgr.com/2016/01/donald-trump.png',myGroup);

The image we are analyzing is this one

When we run the code we now get this output in our console.

This should look familiar. See that information about the location of the rectangle we can draw around the face. It has also correctly identified Donald Trump.

I know what you are thinking though. You are thinking that of course it found him as this image was one we used in our training (can you remember that far back?).

So lets give it a go with a different image featuring Donald Trump along with some other people.

identifyURL('http://magazinespain.com/wp-content/uploads/donald-trump-4.jpg',myGroup);

This is using this image as input

… when we execute we see the output below where we find 5 unknown faces in the image (if you care to look at location you can find out which one is missed) and of course we find Donald Trump.

You will have to experiment yourself now to really trust me but this works pretty well once setup at finding specific people.

Using the Identification code to make the Drone Smart

I know right. After a large number of blogs not touching the Drone you have seen the magic word again. The excitement is coming back :).

First things first. The code we have used here trained the drone to find me or to find Donald Trump. I do not have Donald Trump but I do have me :).

You will need to adjust what you have done to include an image of yourself if you want it to find you. Ideally you will include multiple images in various levels of quality and poses. It means you need to step through the setupFaces steps again and also ensure you do the training. Once you have got that far you will be good to come back to this place :).

Move code into DroneWebServer.js

The first thing I am going to do is move over the code from FaceIdentify.js into my DroneWebServer.js file.  I need to move over the variable definitions for idMap and for people. I also need to move over the complete functions mapPersonList and identifyURL.  Lastly, do not forget to move over the myGroup variable.

We also need to do a little editing as we now have two variables called client (I know.. I only worked this out later but going back and adjusting all along would have taken too long).  Lets change the one related to project oxford (so Microsoft Cognitive Services to be Oxclient. This also means you need to edit it in your mapPersonList and identifyUL functions. It is only in 3 or 4 places so it should be quick. You do not need to adjust the setupFaces.js, unless you really want to, as executing that is always separate so there is no clash.

Lastly. As we will be working mostly with local images (you cannot pass a localhost URL to cognitive services… found that out way too late ;)…) you need to make some other adjustments.

1. Rename identifyURL to be called identifyPATH
2. Change testURL as a paremeter to read testPath
3. Adjust the  JSON passed into face.detect to say path:testPath instead of url:testURL

The top of that function should now look as shown below

function identifyPATH(testPath,groupId) {
      Oxclient.face.detect({path:testPath, returnFaceId:true})

With all of the functions copied over and with the correct variables defined, and packages included, the start of your DroneWebServer.js file should now look as follows (with the right IP address for your drone of course). In addition the two functions should be included in your file right after the last line shown below.

var arDrone = require('ar-drone');
var path = require('path');
var express = require('express');
var fs = require('fs');
var gm = require('gm');
var client = arDrone.createClient({ip: '192.168.2.170'});

var oxford = require('project-oxford');
var apikey ="INSERT YOUR API KEY HERE"; 
var Oxclient = new oxford.Client(apikey); 
var idMap = [];
var people = [];
var myGroup = "drone_demo_group";

var app = express();
app.use(express.static('public'));

mapPersonList(myGroup);identifyPATH(__dirname +  '/Public/DroneImage.png',myGroup); 

for (var faceId in faceMap) {
   people.push(faceMap[faceId]);
   var name = (faceMap[faceId].person && faceMap[faceId].person.name) || '<unknown>';
   
   console.log(name + ' @ ' + JSON.stringify(faceMap[faceId].faceRectangle));
   var topy = faceMap[faceId].faceRectangle.top;
   var topx = faceMap[faceId].faceRectangle.left;
   var bottomx = faceMap[faceId].faceRectangle.left + faceMap[faceId].faceRectangle.width;
   var bottomy = faceMap[faceId].faceRectangle.top + faceMap[faceId].faceRectangle.height;
   var textx = topx ;
   var texty = topy - 10; 
   var TextOut = name;
   gm('public/DroneImage.png')
       .fill('none')
       .stroke("red",4)
       .drawRectangle(topx,topy,bottomx,bottomy)
       .fontSize("20px")
       .stroke("red",2)
       .font('/Windows/Fonts/trebuc.ttf')
       .drawText(textx,texty,TextOut)
       .write('public/DroneImage.png', function (err) {
             if( err ) throw err;
             console.log(response);  
        }
    )        
 }

var found = false;

if (name == "Mark Torr") {
       client.land();
       found = true;       client.stop();}

Where are we and next steps

We have now come a LONG way. We have got to know the Face API of Microsoft Cognitive Services and especially, in t his post, we have used face.identify to be able to look for specific people in images and identify them.  From there we updated our drone code so that if a known person is found the drone will land.

That means we have dealt with the face-recognizing part of this blog. It has taken us a while but I hope the step by step, deep, way I have done it helps you understand my journey and the API better.

In the next blog we will move on to start working with the Text To Speech part of the Microsoft Cognitive Services API.  What we will do is make a call to the relevant API such that when the drone takes off it will say who it is looking for and also so that when the person is found it greets them before landing.

I hope you are still on this journey and enjoying the ride!

Earlier Posts in the series

• Building an autonomous, voice-controlled, face recognizing drone
• The Voice-Controlled, Face Recognizing, Drone Journey – Part 1
• The Voice-Controlled, Face Recognizing, Drone Journey – Part 2
• The Voice-Controlled, Face Recognizing, Drone Journey – Part 3
• The Voice-Controlled, Face Recognizing, Drone Journey – Part 4
• The Voice-Controlled, Face Recognizing, Drone Journey – Part 5