This is the third and final detailed post about controlling your smart home with a magic wand. This post will cover the software (which I’m calling PyPotter) that analyzes the streamed IR video feed to determine if a wand is present, and if so tracks the pattern the wand is making to see if it matches any of the pre-defined patterns. If the wand pattern does match a pre-defined pattern the software will make a call to your smart home controller to perform the desired action. With the right configuration you can make it do almost anything. I’ve personally turned on and off lights, changed the color of lights, played music, cooked popcorn, revealed a hidden message, and turned on a fountain all with different wand motions.
In order to be able to work with different smart devices and be as flexible as possible, PyPotter is designed to work with the smart home software Home Assistant. Home Assistant is a fantastic open source software package for home automation. It is how I use and manage my house’s smart devices. PyPotter uses Home Assistant’s REST API to give your wand the ability to control any device configured to work with your Home Assistant. A full Home Assistant tutorial is outside the scope of this post. If you don’t use Home Assistant and don’t want to, you can always modify the source code to control your devices in a different way.
How Does It Work?
PyPotter processes data input from the WiFi Streaming IR Camera we made in part 2. An example input frame is shown below.
Each frame of video has its background removed using Open CV’s MOG2 background subtractor. This removes portions of the image that are not moving to make it easier to pick out the wand tip. This will help if there is something particularly reflective in your background that could be confused for a wand tip.
After the background is removed, each pixel of the frame is checked to see if it is above a given brightness threshold. A new frame containing only pixels over the brightness threshold is created.
This frame, with its background removed and containing only pixels over the brightness threshold, is now ready for analysis. If no wands tip has been identified, the frame is fed through Open CV’s “goodFeaturesToTrack” function to find the most prominent corners in the image, which should be the tip of our wand (if present in the frame).
Once a wand has been found, the following frames are sent through Open CV’s “calcOpticalFlowPyrLk” to calculate the optical flow of the wand tip. If all goes well the wand’s current position is calculated based on its previous position , previous frame, and the current frame. PyPotter keeps track of the x and y coordinates of the wand tip as it moves throughout the frames and stores them in a list. Using these points it calculates the total distance traveled and the distance traveled recently. A frame containing the full path of the wand is created.
Once the distance traveled recently drops below a set value (showing that the wand has stopped moving) the full wand path frame is checked against a pre-defined set of symbols using Open CV’s k-Nearest Neighbor algorithm. This algorithm will return the name of the set of data that corresponds most with the given input. PyPotter is trained for several different spell patterns, as well as a special training set (called “mistakes”) meant to catch patterns picked up that do not correspond to spells. This bucket is required for our usage as k-Nearest Neighbor always returns the closest match, so you need at least one bucket with data that you don’t want to help prevent false positive results.
When the name of the spell is determined via the kNN algorithm, PyPotter will make a call to Home Assistant’s Rest API based on the spell to perform. The code has built in support for many of Home Assistant’s services including automation, light, media_player, and switch.
PyPotter has three required arguments:
- Video Source URL – This is the mjpeg streaming URL of our WiFi Streaming IR Camera. Example: http://192.168.1.107:8080/?action=stream
- Home Assistant URL – This is the URL of the Home Assistant that we will use to control our smart devices. Example: http://192.168.1.100:8123 or https://mydomain.duckdns.org:8123
- API Token: Your Home Assistant REST API token. Follow instructions here to get create and get a token for your profile.
PyPotter has three optional arguments:
- Remove Background – Defaults to True. Defines whether PyPotter should remove the background from frames. Not removing the background can save significant CPU usage. Must be 4th argument.
- Show Output Windows – Defaults to True. Defines whether PyPotter shows the output video windows. Must be 5th argument.
- Training – Default to False. If set to “True” PyPotter will save the image of every computed full track so you can add it to the training folder for new or improved spell matching. Must be 6th argument.
PyPotter is not meant to actually be ran on a Raspberry Pi (the “Py” in its name is for Python). PyPotter can (and has) run on a Pi, but you will get better performance running it on a standard PC. I originally developed it on a Windows PC and I currently run it on an Ubuntu server. By using a PC instead of a Pi we get access to higher frame rates, better background removal algorithms, higher resolution pattern recognition, and faster throughput. The downside is that it is not running on its own stand-alone device, so if you don’t have a system that is always on it can be a bit of a pain to make sure it is running. If you do end up using it on a Pi, I highly recommend turning off the Remove Background feature.
PyPotter owes some of its code and inspiration to two projects doing similar things. The first of those is Sean O’Brien’s Raspberry Potter project. In this project Sean uses a similar method (IR Lights / IR Camera / wand) to control “Ollivander’s Lamp”. The second project is mamacker’s “pi_to_potter” which builds on Sean’s work. In pi_to_potter mamacker significantly improves the original Raspberry Potter software by using pattern recognition for detecting wand patterns. PyPotter builds on both of these projects, adding its own improvements and integrating with Home Assistant to allow for controlling almost any smart device.
Where is the code?
The code is available on GitHub here.