The Voice-Controlled, Face Recognizing, Drone Journey – Part 9

Total Shares

Introduction to the Drone Journey

Telemetry Data

This post is the tenth post in documenting the steps I went through on my journey to build an autonomous, voice-controlled, face recognizing drone. There are 9 other posts building up to this one which you can find at the end of this post.

Focus of this post

Up until now we have been mostly working on controlling the drone, using the Microsoft Cognitive Services Face API to identify people and lastly making use of the Microsoft Cognitive Services API to convert text to speech and speech to text.

In this blog the focus will move to:

Ultimately we will have built an intelligent end-to-end IoT solution featuring analytics and visualization. The instructions here can be used to also understand how to get data in from other devices as well!

Assumptions

This post makes a few assumptions.

  1. You have signed up for a free Microsoft Azure Account. https://azure.microsoft.com/en-us/free/ and you can log in.  You will be asked for a  credit card for identity verification but the trial is completely free. If you have an MSDN Subscription you may be eligible for free credits to Microsoft Azure every month. Check your MSDN account page for details.
  2. You have signed up for a free PowerBI account https://app.powerbi.com/signupredirect?pbi_source=web and you can sign in.

The Azure Account will come with $200 of credits and this will be needed when we start to work with Azure Stream Analytics.

Setting up your IoT Hub

In your browser navigate to the Azure Portal and log into your account.  From there you will follow these steps.

  • The first step is to click on the green “+” icon in the top left. That will expand the marketplace pane. In the marketplace pan select IoT Hub.

Azure IoT Hub

  • Once you have selected the IoT Hub you will be presented with a pane to provide various configuration information. Here is what you should do:
    • Set the name to be whatever you want to use. I would recommend your name followed by “dronedemo”. In mine I have called it “markdronedemo”.
    • The next step is to set the pricing and scale tier. For our demo we will limit the data coming into the IoT Hub and so we can modify the tier from S1 Standard to Free. To do that you need to click on the option and select “F1 Free” from the options that are presented. It is VITAL at this point that you press the “Select” button at the bottom of this page to take this pricing tier.  An image is below to show you the option you need.
      Azure Hub 1
    • With the pricing tier set the Device-to-cloud partitions and IoT Hub Units options will be greyed out as they have set limitations at the free tier.
    • Next up you need to select your subscription. I have blanked mine out on purpose. If you are using a trial you need to select it here. If you already have an Azure account then select the appropriate subscription.
    • The final two options let you set the name for a Resource Group. This is nothing more than metadata to let you group things together. Give it the name DroneDemo to help you find things related to this easily later.
    • Set the location. I am going to keep it at West US which is the default.
    • Lastly click the pin to Dashboard option and press the Create button.

Congratulations – Your IoT Hub is now being provisioned. You will be returned to the dashboard. Because you clicked pin Dashboard you will initially see a message that it is being deployed on your dashboard. You will need to need to wait a few minutes now until it is finished and you can start to use it.

Getting IoT Hub Connection Strings

The Azure IoT Hub is a very secure service.  To connect to it a unique connection string containing a unique token is needed. To get this code for your IoT Hub you need to do the following:

  1. From the Azure dashboard, once it is deployed, click on the tile for “markdronedemo” (or whatever name you gave it).
  2. On the left hand side click on the “Shared access policies” option under General.
  3. In the pane that opens on the right click on iothubowner
  4. You will now see 4 shared access keys displayed
    • Primary Key
    • Secondary Key
    • Connection String – Primary Key
    • Connection String – Secondary Key
  5. Click on the option to the right of the “Connection String – Primary Key” to copy the connection string to the keyboard. I would save this somewhere you can copy it for now otherwise you will need to come back here when we need it. Be careful with this. Anyone who has this string can connect to your IoT Hub.

Registering Your Drone As A Valid Device

As I mentioned previously the Azure IoT Hub is a very secure service. Besides needing the connection string it only allows connections from known devices that have proper credentials.

There are a few ways to generate the credentials a device needs. In this post we will make use of the DeviceExplorer utility. There are also many ways to authenticate to the IoT Hub (again – to support the security you want). This post focuses on the simple use of pre-shared keys.

Before you can use the Device Explorer you need to download it.  If you are using Windows, download and run Device Explorer. After running the installation the installed DeviceExplorer.exe can be found at C:\Program Files (x86)\Microsoft\DeviceExplorer.

Next up you need to run the utility. The first thing you need to do is to provide the utility with the iothubowner connection string, which we copied previously, in the IoT Hub Connection String field found in the configuration tab as shown below.  Be sure to click the Update button and the Generate SAS buttons.

Device 1

Now that you have configured the utility to connect to your IoT Hub you can use it to generate the keys your devices need. We only need one key for our drone.

To create the key you need to:

  1. Open the management tab and click on the create button.
  2. Provide a name for the device. I will use parrotdrone.
  3. Click the Create button in the Create Device Dialoge
  4. Click Done in the resulting dialogue box

Drone Registration

Once you press Done you will be redirected to the device list and you should see your device in the list.  Once the device has been created grab the device-specific connection string by selecting it in the Devices list, right-clicking and selecting Copy connection string for selected device. This is going to give you the information you MUST use such that your device will be authenticated.

Device 2

At this stage you are all set on the IoT Hub side of the house. You have create an instance of a hub and you have created a representation of your physical device. The next steps are to get the telemetry data sent from the Drone.

Getting Telemetry Data from the Drone to the IoT Hub

To get telemetry from the drone to the IoT hub we need to:

  1. Install some specific node packages to help with out communication to the IoT Hub from node.js. These are azure-iot-device and azure-iot-device-http.
  2. Capture the Telemetry Data in our DroneWebServer.js file
  3. Build up a JSON message and send that to the IoT Hub
Install packages

By now you are an expert in this. Use the two commands below in your c:\Development\Drone directory to get the packages.

npm install azure-iot-hub-device
npm install azure.iot-hub-device-http

We are going to use HTTP to send out information to the IoT Hub. This is being used because it is the simplest to demo as it uses well defined ports. You can use other protocols such as AMQP and MQTT but then you might need specific ports open.

Edit DroneWebServer.js to initialize everything

Include into your DroneWebServer.js the following lines to include the packages and prepare to send the data to the IoT Hub you just created. The code is essentially doing some setup of the connection to the IoT Hub and creating an object we can use later.

var device = require('azure-iot-device');

var clientFromConnectionString = require('azure-iot-device-http').clientFromConnectionString;

var location = process.env.DEVICE_LOCATION || 'Drone at Home';

var connectionString = process.env.IOTHUB_CONN || 'Your IoT Hub Drone Connection String Here';

// Create droneclient that will manage the connection to your IoT Hub
// Created in the context of an Azure IoT device, which is why
// you use a device-specific connection string.
var droneclient = clientFromConnectionString(connectionString);
var deviceId = device.ConnectionString.parse(connectionString).DeviceId;
Grabbing telemetry information

This part can be very interesting if you want it to be. The drone emits a massive amount of sensor data.  To get a look at it you can simply add the following line of code:

client.on(‘navdata’, console.log);

For now go ahead and add this into your app.get(‘/hover … router function such that clicking the hover button (which does nothing if the drone is on the ground) will start to display the data into the console window. If you do that, save DroneWebServer.js and start it up using node DroneWebServer.js then clicking the hover button will show you the mass of sensor data you have available (click the image, as with all images, to view full size).

Sensor Data

To keep this simple we will grab two specific pieces of information:

  1. The Altitude
  2. The Battery Percentage

You can literally grab anything from this long list to visualize. It is also useful, when you have a problem with the drone, to look at some of this data to diagnose if one of the engines is not performing as it should be. Think, if you monitored that over time, you will likely be able to predict a failure of the engine before it happens.

Anyway… to grab the two pieces of information we will modify the code we inserted to be more specific.

 client.on('navdata', function(sensordata) {
 console.log("Drone Altitude: " + sensordata.demo.altitude);
 console.log("Drone Battery Percentage: " + sensordata.demo.batteryPercentage);
 });

If you then re-start your DroneWebServer.js application, hit the takeoff button, then hover button and finally the land button (from your web page) you will see the data appearing in the console as shown in the video below (I took off and landed a few times in the video – battery is newly charged so not yet declining).

As you can see the drone is generating a lot of data very quickly. Think about commercial vehicles generating this sort of data many times over with thousands of sensors.

Sending the Telemetry information to the IoT Hub

The last thing we need to do on the server/drone side is to grab the Telemtry information we now have, build a JSON string and send it to the IoT Hub. To do that we will add the following code.

The first part is building the payload which is simply a JSON formatted string we will send. We then create a message object and so we can see what is being sent we write it out.

The last part actually sends the message. There is also a small helper function which will tell you if the send succeeded or not which needs to be included.

Be careful of word wrap if you copy this code .

var payload = JSON.stringify({
  deviceId: deviceId,
  location: location,
  altitude: sensordata.demo.altitude,
  batterypercentage: sensordata.demo.batteryPercentage});
        
// Create the message based on the payload JSON
var message = new device.Message(payload);
      
// For debugging purposes, write out the message payload to the 
console.log("Sending message: " + message.getData());

// Send the message to Azure IoT Hub
droneclient.sendEvent(message, printResultFor('send'));
   
// Helper function to print results in the console
function printResultFor(op) {
   return function printResult(err, res) {
   if (err) console.log(op + ' error: ' + err.toString());
   };

If you have copied this in then you should have code looking a little like shown below.

Telemetry Data

Then you should be able to go ahead and fire up your DroneWebServer.js again by using node DroneWebServer.js from your c:\Development\Drone directory.

You should then see output a little like below if you hit the hover button, take  off and land. I have included a screen shot below this time. This is the continuous stream of data heading to the IoT Hub. Please ensure you stop your application otherwise it will keep sending data forever and you will max out your free messages 🙂

IoT Hub Data Stream

Congratulations. You are now finished with the sending of the data to the IoT Hub. If you go back to the Azure Portal you will see that all those messages just arrived.

Azure Messages

In fact we can see that we sent 251 messages from an allowed 8k a day (on the free plan) which constituted 3% of the allowed total.

Be careful with your testing. Once you hit your limits on the free plan you need to wait for the next day! By my calculations the drone was flying for literally 25 seconds sending data and we used 3%. This means you will use up almost all your messages with about 13 minutes of  flight unless you do something such as only sending every 5th message.

Using stream analytics to capture data for visualization

In this part we will look at the use of Azure Stream Analytics to capture the data we have coming into the Hub and share it with PowerBI in real-time. To do this we need to:

  1. Create a new stream analytics job
  2. Define an input stream for the created job
  3. Define an output stream for the created job
  4. Write a query which selects from the input stream and places what we want to see in the output stream.
Create stream analytics job

To create this we need to return to the Azure Portal. From there again select the + icon in the top left and in the marketplace pane again select Internet of Things. Once that is done you will see Azure Stream analytics which you can select.

Stream Analytics

To create the service you need to give it a name. I called my job dronetelemetry, select the appropriate subscription, choose the existing resource group you previously created and take the same location you placed your IoT Hub at (for latency purposes).

Again – I would recommend pinning it to the dashboard as that makes it easier to find.

Stream Analytics 1

Now just hit the create button at the top and sit back while Azure provisions that magically for you.

Define input and output streams.

Once the job is created you can go into the main screen which shows you all the options. We will be working right now in the job topology section to define things.

Firstly click on inputs. In the resulting pane that appears click the “+ Add” option at the top. Most of the options will be set. You need to provide an input alias (I used droneinputstream). Then you need to make sure to the source type is Data Stream (which is the default) and modify change the Source to be IoT Hub.

The rest of the options will be updated automatically if you only have one IoT Hub in the subscription. If you have multiple then you need to set the right options for the Hub you want this stream to be working with.

The screenshot below shows what you are aiming for. Most of this happens automatically as you set the option to use IoT Hub from the current subscription.  If you have something similar click the create button.

Input Stream

Next we need to define the output stream.  From the dronetelemetry stream analytic job select outputs from the Job Topology.  In the resulting pane, just like for inputs, select the “+ Add” option at the top.

We now need to customize this just like we did for the input job:

  • Set the sink to be PowerBI.  Note that you will NOT be able to go further unless you have a valid PowerBI Account you can authorize.
  • Set the output alias to something sensible. I used droneoutputstream.
  • Once you are authorized you need to then set the DataSetname. I used DroneDataset.
  • Lastly set the Table Name (this is what you will see in PowerBI). I used DroneInfo,

You should have something looking like this now. When you do, and are happy, go ahead and click the create button at the bottom of the pane.

Output Setup

Write the query

The last thing we need to do now is create the query that takes the data from the input stream and writes it to the output stream.  From the dronetelemetry stream analytic job select query from the Job Topology.

Azure makes it pretty easy to set this up.  When you enter into the query screen you will be sown the defined inputs and outputs on the left.  This is very useful to help you remember names :).

Next you need to go edit the query on the right.

  • First up edit between the brackets that say [YourOutputAlias] and put in the name of your outputstream which in my case is droneoutputstream.
  • Then edit between the brackets that say [YourInputAlias] and put in the name of your input stream which in my case is droneinputstream.
  • Next we need to update the query which currently as a * to select the fields we want and add a timestamp field. Thequery should look something like shown below.Stream Query
Start the job

You can now go back to the  dronetelemetry stream analytic job and press the “Start” button at the top.

Congratulations. You now have your streaming analytics functioning.

Using PowerBI to visualize your captured telemetry data

The last step on this epic journey is to use PowerBI to visualize your captured telemetry data. I will show you how to do this via a series of images. It is important that once you have the stream analytics and PowerBI setup that you take off, collect some data, land and stop the data feed into the IoT Hub. This will help when you need to find the streaming dataset in PowerBI. For the purposes of the demo I am not going to keep historical data although that is entirely possible.

Once you have captured some data you can go into PowerBI. Find the right workspace (in my case it was Demo Projects). From there you need to locate the Streaming datasets option, on the left hand pane, and click on it. An image is below with an arrow pointing the way.

Power BI Streaming Dataset

When you click on the Streaming datasets option you will be taken to a new screen where you can hopefully see a dataset called DroneDataset. Note that you will NOT see this unless your streaming analytics job is running and in some cases unless you have already sent some data once. An image is below. You cannot proceed further until you see this.

Streaming Dataset View

Next click the + sign next to the Dashboards option so you create a new Dashboard. Give it the name of “Drone Telemetry”. Once created select it.

Dashboard Plus

Now we need to add some tiles to the dashboard.  To do that we need to click the + sign at the top of the canvas.

Plus Sign

Clicking the + will open up the Add Tile dialogue shown below. Click on the CUSTOM STREAMING DATA box and then click the Next Button

New Tile

The next screen asks you for your streaming data set. Hopefully you can see your DroneDataset there. If so select it and then click the Next button.

Select Data

On the next screen we define the type of visualization we want. First up we will add a Line Chart. You will need to adjust the Visualization type. Next you need to set the variables to use for the Axis and also for the values. You can see I have used Timestamp and dronealtitude.

Lastly – you can define the time window you want to visualize. I used 1 minute because the drone takes off and lands quickly. If we knew things would happen for longer you can change this setting. Click Next when you are happy with your choices.

Line Chart

Next up you need to set the titles for the Line Chart. Set that as you like then hit the Apply button. This will create the visualization on the canvas.

Set Titles

 

The Line chart visualization is now all set to go. It will show data as it flows.  Next lets add a Card Visualization tile that shows the current value of altitude.

You need to go ahead and follow the same steps of hitting the plus at the top. Then select custom streaming data and lastly select the Drone Dataset. On the next screen you can then simply choose a visualization type of card and set the field to be the dronealtitude before hitting next.

Card Visual

As with the Line Chart you can set the titles. Once you have done that hit the Apply button.

Card Titles

Congratulations. You now have the tiles all set to let you see the Drone Altitude telemetry data as it flows. You can resize those how you want on the canvas.

Layout

Next up follow the same stapes for the battery information we are sending via our stream.  If you do that, and tidy up the layout, you will end up with something looking as shown below.

Power BI Dashboard Final

You can see the dashboard on display by clicking on Drone Telemetry and resizing, moving around the tiles. These tiles will now update in real-time, using the last 60 seconds of data, so as you take off and land with telemetry on you will see the real time data flow as shown in the video below.

And there we have it… your data is now displayed, in real-time, that you sent from the drone.

Where are we and next steps

Phew. If you made it this far then congratulations. We just pulled together 3 separate Azure services together with our Drone project using Microsoft Cognitive Services.  That made this a longer post but I hope you found it useful.

This post really concludes the series I put together in general from a coding perspective. There are some other things I have subsequently added into my project such as:

  1. Better controls so that the Drone will not take off if the person you tell it to find is not known
  2. Some handling of responses back to the browser to prevent the browser thinking the request had not being handled and re-tried (you will have found out about this as the drone randomly took off sometimes ;)..)
  3. I also built a bit more conversation into the Drone so I could converse with it.
  4. On top of this I played a little with the rendering so unknown faces in images would have a red box, known but not the target  yellow and known and our target a green. This just makes it nicer when you are seeing multiple people in an image.

In my next post I will share the completed project video with you as well as showing a diagram of the whole architecture.

Previous Posts In This Series

Leave a Reply

Your email address will not be published. Required fields are marked *