Control Smart Home With Magic Wand – Part 3: Software (PyPotter)

This is the third and final detailed post about controlling your smart home with a magic wand. This post will cover the software (which I’m calling PyPotter) that analyzes the streamed IR video feed to determine if a wand is present, and if so tracks the pattern the wand is making to see if it matches any of the pre-defined patterns. If the wand pattern does match a pre-defined pattern the software will make a call to your smart home controller to perform the desired action. With the right configuration you can make it do almost anything. I’ve personally turned on and off lights, changed the color of lights, played music, cooked popcorn, revealed a hidden message, and turned on a fountain all with different wand motions.

If you missed them, it would be worthwhile to be aware of part 1 and part 2 of this series before continuing.

Home Assistant

In order to be able to work with different smart devices and be as flexible as possible, PyPotter is designed to work with the smart home software Home Assistant. Home Assistant is a fantastic open source software package for home automation. It is how I use and manage my house’s smart devices. PyPotter uses Home Assistant’s REST API to give your wand the ability to control any device configured to work with your Home Assistant. A full Home Assistant tutorial is outside the scope of this post. If you don’t use Home Assistant and don’t want to, you can always modify the source code to control your devices in a different way.

How Does It Work?

PyPotter processes data input from the WiFi Streaming IR Camera we made in part 2. An example input frame is shown below.

Original Frame

Each frame of video has its background removed using Open CV’s MOG2 background subtractor. This removes portions of the image that are not moving to make it easier to pick out the wand tip. This will help if there is something particularly reflective in your background that could be confused for a wand tip.

Background Removed

After the background is removed, each pixel of the frame is checked to see if it is above a given brightness threshold. A new frame containing only pixels over the brightness threshold is created.

Threshold Output

This frame, with its background removed and containing only pixels over the brightness threshold, is now ready for analysis. If no wands tip has been identified, the frame is fed through Open CV’s “goodFeaturesToTrack” function to find the most prominent corners in the image, which should be the tip of our wand (if present in the frame).

Once a wand has been found, the following frames are sent through Open CV’s “calcOpticalFlowPyrLk” to calculate the optical flow of the wand tip. If all goes well the wand’s current position is calculated based on its previous position , previous frame, and the current frame. PyPotter keeps track of the x and y coordinates of the wand tip as it moves throughout the frames and stores them in a list. Using these points it calculates the total distance traveled and the distance traveled recently. A frame containing the full path of the wand is created.

Full Wand Path

Once the distance traveled recently drops below a set value (showing that the wand has stopped moving) the full wand path frame is checked against a pre-defined set of symbols using Open CV’s k-Nearest Neighbor algorithm. This algorithm will return the name of the set of data that corresponds most with the given input. PyPotter is trained for several different spell patterns, as well as a special training set (called “mistakes”) meant to catch patterns picked up that do not correspond to spells. This bucket is required for our usage as k-Nearest Neighbor always returns the closest match, so you need at least one bucket with data that you don’t want to help prevent false positive results.

When the name of the spell is determined via the kNN algorithm, PyPotter will make a call to Home Assistant’s Rest API based on the spell to perform. The code has built in support for many of Home Assistant’s services including automation, light, media_player, and switch.

Available Spells

Aguamenti
Alohomora
Incendio
Locomotor
Reparo
Revelio
Silencio
Specialis Revelio
Tarantallegra

Usage

PyPotter has three required arguments:

  • Video Source URL – This is the mjpeg streaming URL of our WiFi Streaming IR Camera. Example: http://192.168.1.107:8080/?action=stream
  • Home Assistant URL – This is the URL of the Home Assistant that we will use to control our smart devices. Example: http://192.168.1.100:8123 or https://mydomain.duckdns.org:8123
  • API Token: Your Home Assistant REST API token. Follow instructions here to get create and get a token for your profile.

PyPotter has three optional arguments:

  • Remove Background – Defaults to True. Defines whether PyPotter should remove the background from frames. Not removing the background can save significant CPU usage. Must be 4th argument.
  • Show Output Windows – Defaults to True. Defines whether PyPotter shows the output video windows. Must be 5th argument.
  • Training – Default to False. If set to “True” PyPotter will save the image of every computed full track so you can add it to the training folder for new or improved spell matching. Must be 6th argument.

PyPotter is not meant to actually be ran on a Raspberry Pi (the “Py” in its name is for Python). PyPotter can (and has) run on a Pi, but you will get better performance running it on a standard PC. I originally developed it on a Windows PC and I currently run it on an Ubuntu server. By using a PC instead of a Pi we get access to higher frame rates, better background removal algorithms, higher resolution pattern recognition, and faster throughput. The downside is that it is not running on its own stand-alone device, so if you don’t have a system that is always on it can be a bit of a pain to make sure it is running. If you do end up using it on a Pi, I highly recommend turning off the Remove Background feature.

Acknowledgements

PyPotter owes some of its code and inspiration to two projects doing similar things. The first of those is Sean O’Brien’s Raspberry Potter project. In this project Sean uses a similar method (IR Lights / IR Camera / wand) to control “Ollivander’s Lamp”. The second project is mamacker’s “pi_to_potter” which builds on Sean’s work. In pi_to_potter mamacker significantly improves the original Raspberry Potter software by using pattern recognition for detecting wand patterns. PyPotter builds on both of these projects, adding its own improvements and integrating with Home Assistant to allow for controlling almost any smart device.

Where is the code?

The code is available on GitHub here.

5 Replies to “Control Smart Home With Magic Wand – Part 3: Software (PyPotter)”

  1. Great work Adam. I am working on a Raspberry Pi 3 Model B with
    a version of mamaker’s trainedwpins.py script, and I am wanting to add in a bit of background subtraction to take away pesky background lights or reflections and a starting point; so the tracking will not start until the wand tip stays still under a threshhold of distance over a threshold of time. Any pointers would be appreciated.

    1. Hi David, thank you for the kind words. The source code provided on my GitHub should take care of both of those issues for you. It has background subtraction by default.

      The tracking in my code resets after a period of minimal wand movement. This allows you to bring the wand up to the camera, pause it for a moment to reset the tracking, make your shape, then pause again for a moment to end that track.

      If you need the functionality above combined with makmacker’s usage of pins, you should be able to replace the Spell function in my code with his.

      Let me know if you have any trouble.

  2. Hi Adam,

    I am using a Picamera with a Raspberry Pi.

    I am importing picamera and imutils

    and from imutils.video.pivideostream import PiVideoStream
    and then videosource = PiVideoStream().start()

    In your code, what are the changes necessary to use the picamera straight off the Raspberry Pi?

  3. Hi Adam,

    Just checking in again. I saw the article that you wrote there’s a version of this that you make for the raspberry Pi. Could you leave a software link for it.

    Thank you,

    David

    1. Hi David,
      The code published in the GitHub can be ran on a Pi in its current form. You don’t need to use PiVideoStream, you can use mjpgstreamer as described in the camera blog post. In my testing I had mjpgstreamer and PyPotter running on the same Pi. Running them both on the same Pi works. The framerate was lower than I preferred, which is why I run PyPotter on a more powerful machine. I hope that helps!

Leave a Reply

Your email address will not be published. Required fields are marked *