Drone, Internet Of Things

The Voice-Controlled, Face Recognizing, Drone Journey – Part 3

Leave a comment
Total Shares

Introduction to the Drone Journey

Application Framework

This post is the fourth post in documenting the steps I went through on my journey to build an autonomous, voice-controlled, face recognizing drone. There are 3 other posts building up to this one which you can find at the end of this post.

Focus of this post

In this post I am going to pick-up where we left off and cover how you can:

  • Develop a better looking web page to control the drone and add  a few additional control capabilities
  • Grab PNG images from the drone camera and feed them back to a web page for real-time viewing

Improving the Web Application

Making the web application look a bit better than the initial “onclick” text is a labor of love many of you will carry out far better than I.  At this stage of the development I knew I wanted to use clickable images control the drone and I also knew I was going to be streaming a picture from the drone to the web page somehow.

The first step was simply to add more controls for the drone and then to make them accessible via a better looking front end.  The commands I decided to surface were:

  • takeoff
  • land
  • calibrate
  • hover
  • photos
  • clockwise

To do that you need 6 suitable images. I looked around and settled on 6 which I then saved to my local machine in a new directory called public in c:\Development\Drone. I will explain why I created and stored them in the public directory later.

I then edited my index.html file we created previously to include all the images and call the appropriate actions.

<html>
<script language='javascript'>
function call(name) {
  var xhr = new XMLHttpRequest();
  xhr.open('GET', name, true);
  xhr.send();
}
</script>
<body>
<table border=1>
<tr>
    <td>
        <table cellpadding="4">
        <tr>
          <td align="center">
              <a onclick="call('takeoff');"><img height=70 Width=70 src="http://localhost:3000/takeoff.png"></a>
          </td>
        </tr>
        <tr>
          <td align="center">
              <a onclick="call('land');"><img height=70 Width=70 src="http://localhost:3000/land.png"></a>
          </td>
        </tr>
        <tr>
          <td align="center">
              <a onclick="call('photos');"><img height=70 Width=70 src="http://localhost:3000/camera.png"></a>
          </td>
        </tr>
        <tr>
          <td align="center">
              <a onclick="call('calibrate');"><img height=60 Width=70 src="http://localhost:3000/calibrate.png"></a>
          </td>
        </tr>
        <tr>
          <td align="center">
              <a onclick="call('hover');"><img height=70 Width=70 src="http://localhost:3000/hover.png"></a>
          </td>
        </tr>
        <tr>
          <td align="center">
              <a onclick="call('clockwise');"><img height=70 Width=70 src="http://localhost:3000/clockwise.png"></a>
          </td>
        </tr>
        </table>
     </td>
</tr>
</table>
</div>
</body>
</html>

If you save this file, fire up DroneWebServer.js and then head to http://localhost:3000 then you will not see the images. In addition, even when that issue is fixed (I will explain how below in a moment) then without further updates to many will not do anything.

This is because your DroneWebServer.js needs some adjustment to:

  1. Serve static files from a specific directory
  2. Add in the extra actions we need to support to control our Drone
Serve up your images

So now you need to open up and start to edit your DroneWebServer.js. The first thing we need to do is tell express that it can serve up files in a static directory called public.  The one additional line of code to do this is shown below. You can inser this right after you have the code var app = express();

app.use(express.static('public'));

If you were now to stop DroneWebServer from executing and start it again you will find that refreshing http://localhost:3000 your images should now show up as shown below (of course you will have different images).

Nicer Web Page

Add in your actions

Having got the images in place we now need to modify DroneWebServer.js to handle the actions we are going to send so that the commands can be relayed to the drone itself. Below is the code. You can replace your current “app.get” calls with the set of 6 below.

// This router is sending a command to the drone 
// to take off
app.get('/takeoff', function(req, res) {
 client.takeoff();
 console.log("Drone Taking Off");
 });

// This router is sending a command to the drone
// to land
app.get('/land', function(req, res) {
 client.stop(0);
 client.land();
 console.log("Drone Landing");
});

// This router is sending a command to the drone
// to calibrate. Causes the drone to fully
// rotate and balance
app.get('/calibrate', function(req, res) {
 client.calibrate(0);
 console.log("Drone Calibrating");
 });

// This router is sending a command to the drone 
// to cancel all existing commands. Important if
// turning clockwise and you want to stop for
// example
app.get('/hover', function(req, res) {
 client.stop(0);
 console.log("Hover");
 });

// Placeholder function that will later capture 
// the photos
app.get('/photos', function(req, res) {
 console.log("Drone Taking Pictures");
});

// This router is sending a command to the drone 
// to turn clockwise
app.get('/clockwise', function(req, res) {
 client.clockwise(0.5);
 console.log("Drone Turning Clockwise");
});

Now you can save the DroneWebServer.js file. Next up fire it up again “Node DroneWebServer.js” and you can fly your drone by simply clicking on the various images.

Grabbing the drone PNG and displaying in your web page

If you have been studying your drone you will, by now, know that it has a couple of cameras on it. The ar-drone library includes the capability to grab a PNG image from the video.

What we will do is edit the app.get(‘photos’) part of the DroneWebServer.js. The full code that you will need is shown below. I want to step through and explain what is going off.

  1. var pngStream = client.getPngStream(); is essentially initializing the object that will grab the image from the video stream of the drone camera.
  2. We then initialize a couple of variables. period is used to set how often we want to grab an image (in this case every 2 seconds) and lastFrameTime which will be used to count and see if those 2 seconds have elapsed
  3. We are then testing the pngStream object.
    • If there is an error we are writing it out to the console which means you will see it in the node.js window you started the application from
    • If there is data then we will  grab the current time and set a variable now which we can check to see if it is more and the elapsed period we want between images, in this case 2000ms.
    • If more than 2000ms has passed then we set the variable lastFrameTime to be now
    • Once we have done that we can try to write a file into the directory where the DroneWebServer.js was started from.
    • We do a quick check if that file write succeeded or not. If it did not then we throw an error in the console otherwise we just write “Saved Frame” in the console so that we can see a picture was captured.
app.get('/photos', function(req, res) {
   console.log("Drone Taking Pictures");    
   var pngStream = client.getPngStream();
   var period = 2000; // Save a frame every 2000 ms.
   var lastFrameTime = 0;
   pngStream
     .on('error', console.log)
     .on('data', function(pngBuffer) {
        var now = (new Date()).getTime();
        if (now - lastFrameTime > period) {
           lastFrameTime = now;
           fs.writeFile(__dirname + '/public/DroneImage.png', pngBuffer, function(err) {
           if (err) {
             console.log("Error saving PNG: " + err);
           } else {
             console.log("Saved Frame");  
          }
      });
     }
  });
 });

Because we will be writing a file we also need to make use of the node package fs. There is nothing to install as it is there by default but we do need to add one more line into our DroneWebServer.js file to make it available. We do that by adding  the code:

var fs = require('fs');
Install ffmpeg to render the PNG

So we are nearly there but not quite. The next thing you need is a package to actually render the PNG. The one we need is called ffmpeg.

To install you need to go to c:\Development\Drone and run the command npm install ffmpeg. Once installed you should see the message as shown below

ffmpeg

This specific package allows you to call command line ffmpeg which means you also need to install that. To get the ffmpeg package for Windows you need to head to http://ffmpeg.zeranoe.com/builds and download the appropriate version for windows. For me that is the 64-bit version and I also take the latest release version which in my case is 3.2.2 setting options as outlined below. From there I hit the blue download FFmpeg button which starts the download of the ffmpeg zip archive which you should save into the c:\Development\Drone directory

Next up you need to extract the zip file to the c:\Development\Drone folder at which point you shuld have a directory c:\Development\Drone\ffmpeg-3.2.2-win64-static (numbers may change based on your version).

The last thing you need to do is add the ffmpeg executable to your path. You do that by:

  1. Right click on the windows start Icon in windows 10
  2. Select System and Security
  3. Select System
  4. Select, on the left hand side, Advanced System Settings
  5. In the resulting dialogue box click the Environment Variables button towards the bottom right
  6. Click on path in the system variables at the bottom and then click the edit button
  7. Click on New
  8. Add the path “c:\Development\Drone\ffmpeg-3.2.2-win64-static\bin” and click ok.
  9. You now need to close the command window and reopen it
  10. If you have the path right typing ffmpeg will result in you getting a version number. If it is wrong you will get a message that it is not recognized and you need to check the path

With that done you now have the relevant system files ready to rock.

Fly the drone…  remember to hit your “picture” image on the web page to trigger the function

If you go ahead and run this you should start to see the file “DroneImage.png” start to show up in c:\Development\Drone\public. Note that when the drone lands it keeps taking pictures until you stop the application execution by killing the DroneWebServer. I had lots of pictures of my feet 🙂

If you want to see the images in the browser I found the simplest way was to create a separate html file that rendered the image I was creating and then to place that in an iframe which refreshed regularly on the main index.html.

The DroneImage.html file which I loaded into an iFrame is pretty simple as shown below. It basically just contains the image we are creating and we are setting headers so the page does not get cached. If you create this and place it into your “public” directory then it will be returned when requested.

<head>
    <meta http-Equiv="Cache-Control" Content="no-cache" />
    <meta http-Equiv="Pragma" Content="no-cache" />
    <meta http-Equiv="Expires" Content="0" />
    <meta http-Equiv="refresh" Content="1" />
</head>
<body>
<img src="http://localhost:3000/DroneImage.png"/>
<body>

Next we need to edit our index.html file so that we include the iFrame as shown below. The core things here are shown in bold. I am not going to go into IFrames here but wanted to let you see how to edit your index.html file.

<html>
<script language='javascript'>
function call(name) {
  var xhr = new XMLHttpRequest();
  xhr.open('GET', name, true);
  xhr.send();
}

window.setInterval("reloadIFrame();", 4000);
function reloadIFrame() {
 document.getElementById("iframe_DroneImage").src="http://localhost:3000/DroneImage.html";
}

</script>
<body>
<table border=1>
<tr>
   <td align ="center">
       <iframe src="http://localhost:3000/DroneImage.html" height="380" width="660" name="iframe_DroneImage"></iframe>
   </td>
    <td>
        <table cellpadding="4">
        <tr>
          <td align="center">
              <a onclick="call('takeoff');"><img height=70 Width=70 src="http://localhost:3000/takeoff.png"></a>
          </td>
        </tr>
        <tr>
          <td align="center">
              <a onclick="call('land');"><img height=70 Width=70 src="http://localhost:3000/land.png"></a>
          </td>
        </tr>
        <tr>
          <td align="center">
              <a onclick="call('photos');"><img height=70 Width=70 src="http://localhost:3000/camera.png"></a>
          </td>
        </tr>
        <tr>
          <td align="center">
              <a onclick="call('calibrate');"><img height=60 Width=70 src="http://localhost:3000/calibrate.png"></a>
          </td>
        </tr>
        <tr>
          <td align="center">
              <a onclick="call('hover');"><img height=70 Width=70 src="http://localhost:3000/hover.png"></a>
          </td>
        </tr>
        <tr>
          <td align="center">
              <a onclick="call('clockwise');"><img height=70 Width=70 src="http://localhost:3000/clockwise.png"></a>
          </td>
        </tr>
        </table>
     </td>
</tr>
</table>
</div>
</body>
</html>

If you have got this far then you should now be able to start up your DroneWebServer using “node DroneWebServer.js” and navigate to http://localhost:3000 where you should see the last image you captured and be ready to take off.

Once you have taken off you can click the pictures icon and the page (doing it before will cause errors as it does not start to stream until it has taken off) will refresh showing you the latest images it is capturing. It should look a little like what is shown below with the appropriate picture captured from your drone.

Image in Page

Congratulations.. you now are grabbing photos from the drone and constantly updating them on a local web page! It took a lot of steps but it is not really that hard when you know all of those steps 🙂

Where are we and next steps

Up until now we have not really added any intelligence to our drone. We have essentially decoupled it from the applications it comes, and got it able to be controlled from our computer and grabbed an image from it on a continuous basis.

The next step of the project is to make the drone intelligent and so in the next blog we will:

  1. Install the packages you need to access Microsoft Cognitive Services from node.js
  2. Understand some of the capabilities of Microsoft Cognitive Services and those which will be used for the next steps of the project.
  3. Start to see how the face API can be used from node.js
  4. Understand how to use the ImageMagick library to annotate the images that come back with the faces in node.js

So enjoy the milestone we have reached. There is about to be a lot of detail on APIs 🙂

Previous posts in the series

 

Appendix 1 – DroneWebServer.js Code.

DroneWebServer.js code if you have followed along so far:

var arDrone = require('ar-drone');
var path = require('path');
var express = require('express');
var fs = require('fs');

var client = arDrone.createClient({ip: '192.168.2.170'});
var app = express();

app.use(express.static('public'));

app.get('/', function (req, res) {
  res.sendFile(path.join(__dirname + '/index.html'));
});

// This router is sending a command to the drone
// to take off
app.get('/takeoff', function(req, res) {
  client.takeoff();
  console.log("Drone Taking Off");
});

// This router is sending a command to the drone
// to land
app.get('/land', function(req, res) {
  client.stop(0);
  client.land();
  console.log("Drone Landing");
});

// This router is sending a command to the drone
// to calibrate. Causes the drone to fully
// rotate and balance
app.get('/calibrate', function(req, res) {
  client.calibrate(0);
  console.log("Drone Calibrating");
});

// This router is sending a command to the drone
// to cancel all existing commands. Important if
// turning clockwise and you want to stop for
// example
app.get('/hover', function(req, res) {
   client.stop(0);
   console.log("Hover");
});

// function to capture the photos
app.get('/photos', function(req, res) {
  console.log("Drone Taking Pictures");
  var pngStream = client.getPngStream();
  var period = 2000; // Save a frame every 2000 ms.
  var lastFrameTime = 0;

  pngStream
    .on('error', console.log)
    .on('data', function(pngBuffer) {
      var now = (new Date()).getTime();
      if (now - lastFrameTime > period) {
        lastFrameTime = now;
          fs.writeFile(__dirname + '/public/DroneImage.png', pngBuffer, function(err) {
          if (err) {
            console.log("Error saving PNG: " + err);
          } else {
            console.log("Saved Frame");
          }
        });
      }
   });
});

// This router is sending a command to the drone
// to turn clockwise
app.get('/clockwise', function(req, res) {
   client.clockwise(0.5);
   console.log("Drone Turning Clockwise");
});

app.listen(3000, function () {
});

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.