Fully Local AI Vehicle Detection

Intro

You know those old school driveway alarms? The ones you can put at the end of your driveway, which make a chime inside your house go off when a vehicle crosses them? You know: the ones that end up going off when a person walks by, a squirrel runs nearby, or the wind somehow sets it off for seemingly no reason? Well this is my take at a modern, high-tech version of that.

Let me start with the fun part: a demo of the finished product (make sure to turn sound on):

What you're seeing here is a TP-Link Tapo security camera halfway down the driveway detecting a vehicle has turned into the driveway using AI. The camera sends a push notification to Home Assistant, which then sends out a message to a Home Assistant Voice and a Google Nest Speaker (cloud dependent, for comparison).

Hardware

• TP-Link Tapo C325WB - Tapo is fast becoming my favorite camera line. It has local 24/7 recording to a microSD card, supports the open ONVIF camera protocol, implements the RTSP streaming protocol, has a local AI model with person, pet, and vehicle detection, and has an optional subscription service for cloud clip upload/backup.
• Beelink EQ14 Mini PC - Intel N150, 16GB DDR4, 500GB M.2 SSD - this is running Home Assistant. It's overkill for what I need here, but these N100/N150 boxes are really great!
• Home Assistant Voice Preview Edition - the recently released smart speaker from Nabu Casa, the company behind Home Assistant. It supports fully local control, which is why you hear the message on this speaker first.
• Google Nest Mini - the second (lower) speaker that announces the message after a several second delay, since it's the only cloud-dependent (optional) portion of this automation.
• Deco X50-Outdoor - the Wi-Fi AP mounted to the outside wall of my garage that the camera connects to. Since the camera is powered from a lamp post halfway down the driveway, I don't have an easy way to run a hard wired data connection. I've found the Deco APs to have great range, and the C325WB's Wi-Fi radio seems to have a fairly stable connection.
• Vertical Pole Mount - to mount the Tapo camera to the lamp post.
• 3D Printed Stand for the HA Voice - special shout out to RuddO for making a great 3d model for the speaker stand.

Here's a view of the camera mounted on the lamp post and the AP mounted in my garage:

Software

• Home Assistant is the core of this automation, receiving the events from the camera and sending events out to the speakers. Home Assistant has become incredibly popular and has an integration for almost everything at this point. I highly recommend it as the best home automation platform.
• Home Assistant ONVIF Integration - Open Network Video Interface Forum (ONVIF for short) is an open industry standard for controlling IP-based cameras. One of its features is event detection, where it has both a "pull" protocol and a "push" protocol for getting events from a camera. In this case, vehicle detection is one of the events sent out by the Tapo cameras.
• HomeAssistant - Tapo: Cameras Control / pytapo - An integration that adds much more functionality for controlling Tapo devices over the base ONVIF library built in to Home Assistant.
• Piper - the text to speech (TTS) system developed by the Open Home Foundation. This is a very fast, fully local TTS engine which I'm using to convert "vehicle in driveway" to speech that can be sent to a speaker.
• Chime TTS - An integration that can combine sound effects and TTS audio to create a combined output. I'm using it to add the little "chord" sound that plays just before the "vehicle in driveway" message on the Home Assistant Voice speaker.
• Google Assistant SDK - An integration for Home Assistant that allows connecting to Google Nest speaker devices. With this integration, you can send a broadcast to one or more speakers. Unfortunately, this is the one component that requires the cloud, which introduces a delay. It also adds an "Incoming broadcast: it says:" prefix before the "vehicle in driveway" message, which can't be removed. It also doesn't allow specifying a volume for the message. All of that combined makes this pretty unfortunate, because I have several of these around my house.

Coding

It wouldn't have been as fun of a project if it all worked out of the box, right? When I pulled in my Tapo cameras to Home Assistant, I noticed that they had a binary sensor for motion detection but not the more advanced person, vehicle, and pet detectors. The motion detector also didn't seem to be working properly.

I started looking into home assistant logs and noticed a strange 500 error communicating with the camera using ONVIF:

illegal status line: bytearray(b'POST /onvif/service\x00HTTP/1.1\x00\x00Host\x00 192.168.56.109:2020\x00\x00Accept\x00 */*\x00\x00Accept-Encoding\x00 gzip, deflate, br\x00\x00Connection\x00 keep-alive\x00\x00User-Agent\x00 ZHTTP/1.1 500 Internal Server Error'

What's going on here? If you remember above I said that ONVIF has two delivery mechanisms in the events spec: pull and push. The pull method is similar to HTTP long polling, where you send a request to the camera asking for recent events along with a timeout (e.g. 60 seconds). The camera responds when it has an event to deliver or times out after the 60 seconds with nothing. With push, you give the camera a URL to send HTTP messages to when an event occurs. Push is faster and more efficient, but requires more set up on the client.

The Home Assistant ONVIF integration tries both but prefers push. If it can't set up a push notification, it falls back to pull, and this 500 error was happening while trying to set up a push notification. The camera was responding with a 500 error that wasn't even valid HTTP, containing null bytes as a separator instead of newlines.

To debug this, I installed an independent ONVIF client from Happytimesoft to see if it worked, and it did! I used Wireshark to capture XML payloads from the two clients to compare them and see what might be different that triggered the 500 error. Eventually, I figured out the camera's XML parser was not happy with inline namespaces such as <ns0:Subscribe xmlns:ns0="http://docs.oasis-open.org/wsn/b-2"> vs. defining the namespaces at the top of the document and using shorthand in each element, e.g. <wsnt:Subscribe>.

I sent a pull request to set a namespace prefix in python-onvif-zeep-async, the library used by the Home Assistant ONVIF integration, then a pull request to Home Assistant to pull in the updated version of the library.

I'm not sure why, but the Tapo cameras don't seem to send events reliably (or at all) when using a pull subscription. Push subscriptions got me up and running, receiving motion events reliably in Home Assistant. Next up, though, the ONVIF integration has a parser library that parses events to turn them into sensors. The ONVIF events spec is intentionally wide open in terms of the event payloads. I found this message in the log telling me that the event coming from the Tapo camera wasn't recognized:

clipped for brevity

Driveway: No registered handler for event from tapo_controltapo.driveway.c325wb.lan: { ...  'Topic': { '_value_1': 'tns1:RuleEngine/TPSmartEventDetector/TPSmartEvent', ... 'Data': { 'SimpleItem': [ { 'Name': 'IsVehicle', 'Value': 'true' } ... }

Okay, easy enough, I thought. I found the parser module that looked easily extensible to add more types. Unfortunately, it had no unit tests, which made it very difficult to test new parser types. A debugging workflow that includes getting into my car to drive down the driveway clearly wasn't going to work. So I added a new unit test for event types (which took a while to get working!) and sent a pull request to Home Assistant.

Conclusion

Yes, it's a little silly, but I think it's a great demonstration of the power of open protocols, open source code, and local processing. Like many in the home automation community, I value having local control of my smart devices. We often hear that cloud services are required to enable powerful features, but by carefully selecting the hardware components you buy and the software components you run, you can unlock impressive building blocks! Most importantly, I had a blast working on this and learned a lot along the way.

A Home Office Status Light

I'm a proud member of the post-covid work from home crew. I love working from home because of two main aspects: first, I have an office with a door that closes, a large desk with three monitors, and the peace and quiet I need to do focus work. Second, it gives me enormous flexibility to choose my working hours, interleaving them with things like walking the dog, meeting with a contractor, picking up the kids from school, etc. And of course, it allows me choose where I live without having to worry about wasting time commuting.

The entrance to my office is a pair of french doors with glass panels so you can peek in at what I'm doing. But my monitors face an opposite wall, so you can't see what's on my screens. Because of that, there wasn't an easy way for my wife and kids to know if I'm in a meeting where I don't want to be interrupted. They'd knock or wave at me through the window until I responded with a thumbs up or thumbs down.

Here's my solution! A small light on my desk that turns red when I'm in a meeting, green when I'm not, and turns off when I leave my desk.

To do this, I leveraged my favorite home automation platform, Home Assistant. I split up the project into two parts: the detection sensors and the signaling. The signaling part was easy: I connected a Sengled Zigbee RGBW A19 bulb to my network and named it "Jeff's Office Status".

The detection part was more difficult. The easiest approach I could find would be installing System Bridge. However, I had some security concerns with installing it, and I had a tough time even trying to run it. Eventually, I realized it doesn't support getting the camera status on Linux anyway.

I decided to write my own little script to do the sensor detection and send the results using the Home Assistant REST API. I did some research on how to detect if the camera is turned on in Linux. Unfortunately, I couldn't find any event-driven way to do it, so the best approach I came up with was to poll the sysfs file /sys/module/uvcvideo/refcnt. The contents of the file changes from 0 to 1 when the camera is turned on.

I similarly didn't find an event-driven way to detect if my monitor is on, so I went with a polling approach by running the xset command with the -q parameter. It spits out a bunch of human readable text, which includes the string "Monitor is On" if the monitor is on.

I wrapped it up in a Python script that polls both the camera and screen every 5 seconds, posting to Home Assistant whenever the value changes from off to on or vice versa. I added the script to run under systemd so it's always running on my work machine.

Lastly, I created a Home Assistant automation that links up the two binary sensors with the color bulb. You can view both my Python script and the Home Assistant automation yaml file at my Office Light GitHub repo.

Google Calendar E Ink Display

I've always wanted to display our family calendar in a central location, like in the kitchen. Various options exist, but a powered display really limits where you can place something, and I never liked the way it would look.

When I saw the OpenEPaperLink project, I had to try it out. Here's the end result:

The hardware I purchased for this:

• OpenEPaperLink Mini-AP v3 as a base station
• 4.2" epaper tag for the screen

These things are surprisingly affordable for what they offer. The rest of the project is all software, primarily Home Assistant. The more I've used Home Assistant, the more I'm impressed with the amazing community surrounding it.

The things I had to install in Home Assistant:

• Google Calendar home assistant integration: this allows you to pull information from your Google Calendar, which imports it to home assistant as Calendar entities.
• OpenEPaperLink home assistant integration: this detects the OpenEPaperLink hub on your network and adds devices for each of the displays. It then exposes a service call you can use to send data.
• Home Assistant Node-RED: this integrates Node-RED into home assistant, which is a flow-based programming interface. It's much more flexible than Home Assistant's built-in yaml-based routines. In particular, it has the ability to execute JavaScript code during a flow, which is necessary to transform the data (more on this later).

Once this was all set up and configured, I could create the Node-RED flow. Here's what the overall flow looks like:

The first step in the flow is an Inject node, which is configured to trigger the flow once an hour.

The next step is a call service node which calls calendar.get_events. This is a new method, which replaces calendar.list_events, which doesn't seem to be documented yet. The parameters I pass get all calendar events in the next 4 days, starting from 3 hours ago:

{

   /* 3 hours ago */

   "start_date_time": $fromMillis(

      $millis() - 1000 * 60 * 60 * 3),

   "duration": {"days": 4}

}

The output gets piped into a function node. This is really the only code I had to write for this. Instead of inlining it here, I placed it in a gist here.

The script takes the message payload output from the calendar service call, parses the dates and times, has some special handling for multi-day events, formats the dates and events in a custom format, and then transforms it into the output payload format expected by the open_epaper_link.drawcustom service call.

Next, I wanted to prevent the display from refreshing unnecessarily. I piped the output of my function into a Filter node. This conveniently has a mode where you can stop the flow if the input doesn't change. E Ink devices only consume a lot of power when the display changes. However, the way the open_epaper_link.drawcustom service call works is it renders into an image, which then gets sent to the display. If you send the same image twice, it still consumes power. To prevent refreshing the display when not necessary, the whole flow gets stopped by this filter node if what's being displayed doesn't change.

The next step is a call service node again, this time with open_epaper_link.drawcustom as the target. The parameter this time is very simple, {"payload": payload}, since my function outputs in the payload format expected by the service call.

That's it! Now I have an auto updating E Ink display that shows my calendar. I bought a few more of these E Ink displays to play around with, so I'll hopefully be adding more!

Vaurien: The Chaos Proxy

I had a need to test out a program's behavior when its backend web server returned errors. Unit testing would have been difficult in this case, because I specifically wanted to test the program's macro result when encountering, say, 10% server response errors.

After searching around, I came across Vaurien, the Chaos TCP Proxy. It's seriously cool. As its name suggests, it's a TCP proxy server that you can route a local connection through to a backend server. In TCP mode, it can delay packets, insert bad data, or drop packets, testing the socks (pun intended) off your application.

It also supports the application-level protocols: HTTP, Memcache, MySQL, Redis, and SMTP, and its architecture is very modular, so it's easy to plug in new protocols.

For example, here's how I can run it in HTTP mode:

$ vaurien --protocol http --proxy 127.0.0.1:8888 \
          --backend www.google.com:80 --behavior 50:error

This says to run an HTTP proxy that connects to google.com on the backend and return a 5xx HTTP error code 50% of the time. Testing it out with curl:

$ curl --head -H "Host: www.google.com" http://127.0.0.1:8888/
HTTP/1.1 200 OK
...

$ curl --head -H "Host: www.google.com" http://127.0.0.1:8888/
HTTP/1.1 500 Internal Server Error
Content-Type: text/html; charset=UTF-8

$ curl --head -H "Host: www.google.com" http://127.0.0.1:8888/
HTTP/1.1 502 Bad Gateway
Content-Type: text/html; charset=UTF-8

Filed this away as a super useful tool.

Git Branches by Date

I tend to create a huge number of branches in my git repositories, but I have a bad habit of not cleaning them up once I'm finished with them.

I found a nice command from an answer on StackOverflow that allows you to sort branches by date. I modified it slightly to also show the date when printing the branch information:

$ git for-each-ref --sort=-committerdate refs/heads/ --format='%(committerdate) %09 %(refname:short)'
Mon Jan 14 23:46:15 2013 +0000   keyfile-seek-rebase
Sat Jan 12 02:09:04 2013 +0000   perfdiag-mbit-fix
Fri Jan 11 17:38:13 2013 -0800   keyfile-seek
Thu Jan 10 01:05:43 2013 +0000   master

You can also set the date format to be relative (or other possibilities, see man git-for-each-ref):

$ git for-each-ref --sort=-committerdate refs/heads/ --format='%(committerdate:relative) %09 %(refname:short)'
3 days ago   keyfile-seek-rebase
6 days ago   perfdiag-mbit-fix
6 days ago   keyfile-seek
8 days ago   master

I added it to my .gitconfig file as an alias:

[alias]
    branchdates = for-each-ref --sort=-committerdate refs/heads/ --format='%(committerdate:relative) %09 %(refname:short)'

That allows me to just type git branchdates and get a nice listing of my local branches by date.

Google Cloud Storage Signed URLs

Google Cloud Storage has a feature called Signed URLs, which allows you to use your private key file to authorize access to a specific operation to a third-party.

Putting all the bits together to create a properly signed URL can be a bit tricky, so I wrote a Python example that we just open-sourced in a repository called storage-signedurls-python. It demonstrates signing a PUT, GET, and DELETE request to Cloud Storage.

The example uses the awesome requests module for its HTTP operations and PyCrypto for its RSA signing methods.

Google Cloud Storage JSON Example

I recently started working at Google on the Google Cloud Storage platform.

I wanted to try out the experimental Cloud Storage JSON API, and I was extremely excited to see that they support CORS for all of their methods, and you can even enable CORS on your own buckets and files.

To show the power of Cloud Storage + CORS, I released an example on GitHub that uses HTML, JavaScript and CSS to display the contents of a bucket.

The demo uses Bootstrap, jQuery, jsTree, and only about 200 lines of JavaScript that I wrote myself. The project is called Metabucket, because it's very "meta" - the demo is hosted in the same bucket that it displays!