↓ Skip to main content
fricklr
fricklr
Smarthome 7 mins

AI-powered surveillance cameras with Home Assistant, Frigate and Google Coral

With Frigate, surveillance camera software has moved into Home Assistant that can intelligently evaluate image content.

Home-Assistant Smarthome
AI-powered surveillance cameras with Home Assistant, Frigate and Google Coral
Table of Contents

We have 4 cameras in use around the house. Over time, I’ve tried different systems, including Motion Eye, iSpy, Zoneminder and Shinobi. None of the solutions really convinced me, especially since they all worked with the same “dumb” surveillance functions and thus detected a lot of unimportant movements. In addition, they were quite CPU-hungry. This is not a problem with my system, as all my NAS and smart home functions run on an HP Microserver with an Intel XEON E3 1230 CPU and it has enough performance reserves, but for the functionality offered it was simply too much. The motion detection also hardly went beyond that of my HIKVision cameras .

⚠ [affiliate] Keine Produkte mit Cache für home-assistant. Bitte affiliate-sync --lang en ausführen.

Frigate surveillance software with AI function

With Frigate , there has been surveillance software for cameras for some time that is deeply integrated into Home Assistant and analyzes the image content via AI (via Tensor Flow). Frigate FrigateAI-based surveillance camera software with object detection (people, animals, vehicles). Runs completely locally and can be used as a Home Assistant integration can, for example, distinguish between people, cats, cars, trucks and many other objects. At the same time, the events can be used as sensors in Home Assistant and thus control corresponding functions such as outdoor lighting, etc. Frigate can be installed very simply via the Supervisor SupervisorManagement layer of Home Assistant OS/Supervised. The supervisor installs and monitors core, add-ons and backups – the system’s control center and is thus seamlessly integrated into Home Assistant.

Setup is done via a YAML YAMLData format (YAML Ain’t Markup Language). Used in Home Assistant for configurations, automations and scenes. Easy to read but error-prone with incorrect indentation configuration file in which the cameras and their functions are configured. Ideally, the objects to be recognized should fit into a square of 300x300 pixels in size. A very high camera resolution is therefore not necessarily better. A frame rate of 5 fps is also sufficient for exact image recognition.

An example configuration looks like this:

yaml 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

  einfahrt:
    ffmpeg:
      inputs:
        - path: rtsp://user:password@MYCAMERAIP/Streaming/Channels/101
          roles:
            - detect
            - clips
            - rtmp

    width: 1280
    height: 720
    fps: 5
    objects:
      track:
        - person
        - cat
        - dog
      filters:
        person:
          min_score: 0.6
          threshold: 0.5
        cat:
          min_score: 0.4
          threshold: 0.5
    motion:
      mask:
        - 0,679,167,543,467,205,751,158,1280,308,1280,0,0,0
      contour_area: 80

    clips:
      enabled: true
      pre_capture: 5

In the “Input” area, you define the address of the surveillance camera and the desired functions. In this case, objects are to be detected and video clips of events are to be recorded. At the same time, the camera image is to be passed on as an RTMP stream. This makes the camera stream accessible and integrable via the standardized address rtmp://HA-IP/live/cameraname.

Under “Objects”, you define which objects are to be recognized and with “Filters” the detection threshold is set. With the “Mask”, areas can be excluded from detection and with “pre_capture” you define how many seconds before the event should be included in the recording. Retention periods can be defined for each camera and for each object type, after which the clips and images are automatically deleted. For example, you can keep clips of people for 10 days, but clips of vehicles only for 3 days.

Frigate’s events and settings can be accessed via MQTT MQTTLightweight publish/subscribe messaging protocol. Used in smart homes to exchange sensor data and control commands between devices . However, there is also an extension frigate-hass-integration that you can simply install via HACS HACSHome Assistant Community Store – unofficial marketplace for integrations, add-ons and custom dashboards from the community and that provides convenient sensor entities, switches and binary sensors (“motion detected”). Via the switch entities, you can, for example, deactivate object detection when the terrace door is open (assuming a corresponding door sensor), so that you are not constantly detected yourself and unnecessarily recorded. Or you stick an NFC tag on the house wall, which you scan with your smartphone and which then deactivates the detection for this area via a corresponding automation for, say, 1 hour.

Speaking of areas: In the Frigate configuration, you can also define “Zones”. If you have, for example, a camera area in which the garage entrance, the house entrance and the mailbox are located, you can define zones for each of these areas. These then selectively report when motion has been detected in this zone.

You can also integrate cameras that deliver an MPEG stream instead of an RTSP stream, which is the case with my WANTEC Monolith C IP/VoIP door intercom . For this, the MPEG stream must be converted via FFMPG. This looks like this in my frigate.yml:

yaml 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

  eingang:

    ffmpeg:

      inputs:

        - path: http://user:password@IP-WANTEC-CAMERA:8080/?action=stream
            roles:
            - detect
            - clips

      input_args:
              - -avoid_negative_ts
              - make_zero
              - -fflags
              - nobuffer
              - -flags
              - low_delay
              - -strict
              - experimental
              - -fflags
              - +genpts+discardcorrupt
              - -r
              - "5" # <---- adjust depending on your desired frame rate from the mjpeg image
              - -use_wallclock_as_timestamps
              - "1"

      output_args:
        detect: -f rawvideo -pix_fmt yuv420p
        clips: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v libx264 -an

    width: 1280
    height: 720
    fps: 15
    clips:
      enabled: true

    objects:
      track:
        - person
        - cat
        - dog
      filters:
        person:
          min_score: 0.6
          threshold: 0.6

    motion:
      mask:
        - 0,406,0,380,0,306,0,0,545,0,544,277
      contour_area: 80

This also turned the camera of the door intercom into a fully-fledged surveillance camera including object detection, and all without any cloud services.

Less CPU load and better detection with Google Coral

What’s special about Frigate is also that, in addition to GPUs (with supported graphics cards/chips), you can also integrate the Google Coral USB Accelerator. This is a TPU (Tensor Processing Unit) in the form of a small USB3 dongle. This enormously relieves the CPU and the small Google Coral analyzes over 100 frames per second, which is very helpful especially with multiple cameras or when using a Raspberry Pi.

The Google Coral is available, for example, from electronics retailer Pollin for 69.99 € plus shipping. If you sign up for the Pollin newsletter , you get a 5 euro voucher and thus save the shipping costs!

The Google Coral is integrated in the frigate.yml with these parameters:

yaml 
1
2
3
4

detectors:
  coral:
    type: edgetpu
    device: usb

In practice

For setting up Frigate, you need to study the documentation a bit, which still has some gaps in some areas. As always, the HA forum and the HA Sub-Reddit help here. However, my 4 cameras were quickly integrated and worked flawlessly. In general, real object detection (and differentiation) is a blessing! Moving branches or birds no longer lead to unwanted recordings and notifications. The detection performance is very good, even if some cats are recognized as dogs :-)

The latency for object detection is under 1 second for me (which is probably also due to the Google Coral) and thus Frigate and the corresponding switch are also suitable for replacing a dumb motion detector and not turning on the outdoor lighting for every cat. The detection works with my HIKVision cameras with their IR lighting and good image quality even at night when it’s raining.

Conclusion

In my opinion, Frigate is currently the best free software for surveillance cameras. Development is still active and some things have to be worked out quite laboriously, but the result is already completely convincing. I would definitely recommend the Google Coral , as you can also run Frigate on the Raspberry Pi without any problems with it.

Although many objects are supported by Frigate by default, including giraffe, zebra, bananas and toothbrushes, Frigate becomes really interesting when it can also distinguish people. I’m currently collecting images of us to train custom models for Tensor Flow and face recognition. However, I still have to deal with this in detail, as the topic is currently not yet on Frigate’s feature list.

Hi! I'm Markus – the Fricklr. ⚡ Electrical engineer, 🎵 musician (bass, guitar, keys) and professionally 'something with the internet' 🌐 for over 30 years – nowadays with AI too 🤖 – a 55-year-old generalist from Bavaria, Germany.

Your feedback helps me write even more interesting posts. After "Yes" there's also a small option to support my work.

Explore related topics

Content Map →
Show related content as a list
💡 Drag, hover & click to explore
Bigger dots are more relevant
Explore all Smarthome →

What do you think? Leave a comment!

Share your thoughts, questions or experiences with the community.

Links marked with this symbol are affiliate links. As an Amazon Associate I earn from qualifying purchases. There are no additional costs for you.