Working around Bluetooth HCI problems in Home Assistant

Are you using Bluetooth / Bluetooth Low Energy devices in your home automation?  I'm using the Xiaomi Mijia ones.

If you are, you might have encountered undecipherable kernel errors like these:

Bluetooth: hci0: hardware error
Bluetooth: hci0: unexpected event
Bluetooth: hci0: start background scanning failed
Bluetooth: hci0: Opcode 0x200c failed: -16
Bluetooth: hci0: Unable to disable scanning: -16
Bluetooth: hci0: disable scanning failed: -16
Bluetooth: hci0: start background scanning failed: -16

I know I do, with both the TP-Link UB500 and the TP-Link UB400 adapters.  When these errors appear on my machine's kernel log, I know everything Bluetooth is toast — effectively, my Home Assistant instance stops receiving data from the various Bluetooth Low Energy devices set up around the house.  There's some software defect (or perhaps a firmware issue with the devices) that basically makes them worthless after roughly two hours of polling Bluetooth LE sensors; the HCI device becomes sort of "jammed" and no bluetoothctl command can reset it, stop scanning, start scanning, or anything else.

Short of buying another adapter, however, there's a workaround you can use: you can simply tell the kernel to reset the HCI device.  And you're about to get the necessary program code to do exactly that, fully automatically, when necessary.

The fix — a watchdog program and a service to run it

To reset the HCI device, I wrote an extremely short program which you can use by deploying it to file /usr/local/bin/hci-resetter and making the file executable (chmod +x /usr/local/bin/hci-resetter).

#!/usr/bin/python3

import glob
import os
import re
import subprocess
import sys
import time


regexes = [
    re.compile(x) for x in [
        "Bluetooth: (hci[0-9]*): hardware error",
        "Bluetooth: (hci[0-9]*): unexpected event",
        "Bluetooth: (hci[0-9]*): start background scanning failed",
        "Bluetooth: (hci[0-9]*): CSR:.*failed",
        "Bluetooth: (hci[0-9]*): command.*timeout",
        "Bluetooth: (hci[0-9]*): unexpected event.*length",
    ]
]


def get_device_and_driver_path(hci):
    base_folder = f"/sys/class/bluetooth/{hci}"
    device_folder = os.path.join(base_folder, "device")
    driver_folder = os.path.join(device_folder, "driver")
    device_path = os.path.realpath(device_folder)
    driver_path = os.path.realpath(driver_folder)
    device = os.path.basename(device_path)
    return device, driver_path


def shut_off_and_on(device, driver_path):
    unbind, bind = (
        os.path.join(driver_path, "unbind"),
        os.path.join(driver_path, "bind"),
    )
    with open(unbind, "w") as f:
        f.write(device + "\n")
    time.sleep(0.5)
    with open(bind, "w") as f:
        f.write(device + "\n")
    time.sleep(0.5)


def start_monitor():
    return subprocess.Popen(["dmesg", "-W"],
        universal_newlines=True,
        stdout=subprocess.PIPE
    )


def reset_hci(hci):
    device, driver_path = get_device_and_driver_path(hci)
    print(f"Resetting HCI {hci}", file=sys.stderr)
    shut_off_and_on(device, driver_path)
    print(f"HCI {hci} has been reset", file=sys.stderr)


if sys.argv[1:] and sys.argv[1] == "--now":
    retval = 0
    for hci in glob.glob("/sys/class/bluetooth/hci*"):
        if not os.path.isdir(hci):
            continue
        try:
            reset_hci(os.path.basename(hci))
        except Exception as e:
            print("Error resetting HCI: %s" % e)
            retval = 1
    sys.exit(retval)


p = start_monitor()
while True:
    line = p.stdout.readline().rstrip()
    if not line:
        break
    else:
        hci = None
        for r in regexes:
            match = r.search(line)
            if match is not None:
                hci = match.group(1)
        if hci is not None:
            print(f"HCI {hci} has failed", file=sys.stderr)
            reset_hci(hci)
            p.stdout.close()
            p.kill()
            p.wait()
            p = start_monitor()

p.stdout.close()
sys.exit(p.wait())

The program is fairly simple — it looks for the kernel messages that indicate the HCI device is toast, then resets the device by unbinding it from the driver then rebinding it, and then goes back to dutifully monitoring kernel messages.

To get it to start on boot, you'll need a service unit /etc/systemd/system/hci-resetter.service:

[Unit]
Description=Bluetooth HCI resetter service
After=bluetoothd.service

[Service]
Type=simple
ExecStart=/usr/local/bin/hci-resetter
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=bluetooth.target

After saving this file, simply enable it by running systemctl enable --now hci-resetter.service.

If you want to reset the HCI devices on your system now, you can run hci-resetter --now.

You'll note that this program isn't exactly useful in all scenarios.  Resetting the HCI device is inherently an operation that will tear down any ongoing Bluetooth connection, which is fine if what you're doing is polling thermometers.  So this workaround works well in the Home Assistant case, but it won't help you if what you need is to sustain a long steady connection between your Bluetooth dongle and other hardware, like headphones.