dummylabs / thewatchman Goto Github PK

Suggestion- how about an ignore list?

For example, in an ESPHome device I have scripts, and to call one I use script.execute
Problem is, Watchman thinks that this is an entity.

After I upgraded to HA 2022.5, HACS is showing available updates falsely. Please look at this screenshot.

For example - it is showing to update from v0.5.2 to v0.5.1, however v0.5.2 is the latest version which is already installed.

The github also shows the latest release as v0.5.2. https://github.com/dummylabs/thewatchman/releases/

Watchman sensors are not linked to the config entry, they do not support entity registry like other integrations.

My report generates some false positives for missing entities belonging to template variables inside automation and scripts.

See below for the example on light.attributes which is a variable in a script:

Relevant lines from the script:

variables:
  lights_auto: >
    {% set lights = namespace(color=[]) %} {% for light in states.light |
    selectattr('attributes.supported_color_modes') %}
      {% if ('rgb' in light.attributes.supported_color_modes) or ('hs' in light.attributes.supported_color_modes) %}
        {% set lights.color = lights.color + [ light ] %}
      {% endif %}
    {% endfor %} {{ lights.color | map(attribute="entity_id") | list }}

I suspect parsing the scripts correctly to identify the correct fields on each object would be out of scope. So would it be possible to detect dynamic variables and exclude them from watchtower?

I have some template sensor https://github.com/clempat/home-assistant/blob/main/configurations/heating/heating.yaml#L16

Like

        {%- set data = namespace(radiator_ids=[]) -%}
        {%- for switch in states.switch -%}
          {%- if switch.entity_id is match('switch.heating') and switch.state in ['unavailable', 'unknown'] -%}
            {%- set data.radiator_ids = data.radiator_ids + [switch.entity_id] -%}
          {%- endif -%}
        {%- endfor -%}
        {{ data.radiator_ids | list | count }}

or

        {%- macro getRoomTemperature(room) -%}
            {%- set data = namespace(temperatures=[]) -%}
            {%- set temperature_entities =  expand(area_entities(room)) | selectattr('attributes.device_class', 'eq', 'temperature') | rejectattr('state', 'in', ['unavailable', 'unknown']) -%}
            {%- for entity in temperature_entities -%}
            {%- set device_id = device_id(entity.entity_id) -%}
              {%- if device_attr(device_id, 'model') is search('weather', ignorecase=True) or device_attr(device_id, 'model') is search('sensor.ht', ignorecase=True) %}
                  {%- set data.temperatures = data.temperatures + [entity.state] -%}
              {%- endif -%}
            {%- endfor -%}
            {{ data.temperatures | map('float') | average | round(1) }}
        {%- endmacro -%}
        {{ getRoomTemperature("Salon") }}

I get from watchman:

| switch.heating                 | missing | configurations/heating/heating.yaml:16,25                    |
| sensor.ht                      | missing | configurations/heating/heating.yaml:63,79,95,111,127,143,159 |
|                                |         | ,175,191,207,223,239  

I am uncertain if it makes sense to consider those as entity as they are in those templates function which allow having partial name.

Since watchman catches up every state change of every entity from configuration files, we can fire up a custom event with entity name, which was changed, and its context. The event can also be reflected in the HA event log. This feature should be optional due to enormous amount of possible events.

Inspired by #29 and #50.

Hey, thanks for this great integration! I'm setting it up for the first time and narrowing down some missing entities. I've noticed that many false positives are being generated from files like service.yaml inside custom_components.

For example:

-== Missing 4 entity(ies) from 17 found in your config:
+--------------------------------+---------+--------------------------------------------------------------+
| Entity ID                      | State   | Location                                                     |
+--------------------------------+---------+--------------------------------------------------------------+
| sensor.station_name            | missing | custom_components/opensprinkler/services.yaml:6,18           |
| fan.office_purifier            | missing | custom_components/coway/services.yaml:7,19                   |
| sensor.general_waste           | missing | custom_components/garbage_collection/services.yaml:9,21,33,4 |
|                                |         | 8,60                                                         |

The garbage collection file is a good example: https://github.com/bruxy70/Garbage-Collection/blob/master/custom_components/garbage_collection/services.yaml. On closer investigation, the false positive appears to be an example like so:

  fields:
    entity_id:
      description: The garbage_collection sensor entity_id
      example: sensor.general_waste

It's easy enough to add each of these directories to the ignore_files config, and that's what I'm doing now. I tried excluding entities like example.* but of course that didn't work. Is there an easier to way to bulk ignore these examples? Thank you for your time!

Love this project! I would like to use it a little differently, if this would be possible. If there were the option to use an "Included Entities" list to manually select and add to a list, that would be amazing. I'd like to add wildcards of "camera." and "binary_sensor." for instance.

I did try creating a separate folder for it to monitor, but wildcards would not work in the yaml file I created for it to scan, only singular entities.

Would it be possible to add this functionality? It would work a lot better for me as opposed to it listing the hundreds and hundreds of entities in my config, many of which have a default state of unavailable or unknown, and then having to go in and manually exclude all the ones I don't want.

Thanks!

sensor.watchman_missing_entities and sensor.watchman_missing_services should automatically be updated once an entity or service changed their state.

Special report mode to detect entities which are not used by the configuration. It's aimed to detect and remove orphaned/unused entities from the HA database.

Is there a way to get Watchman to send a notification only when there are entities that are unavailable?

There are many spelling errors or just beginners mistakes, that could be checked for.

• if a state for covers is 'closed' instead of ‘off’
• common spelling errors such as 'off ’ or ‘Off’
• and probably more

This would be extremely helpful and I think it could be done with the way the plugin operates.

Current implementation assumes send_notification and create_file service parameters are true unless they were explicitly set. send_notification:true, in turn, requires service parameter to be defined either in configuration.yaml or as a part of service data. This forces watchman to complain about missing service parameter. Such behavior may confuse users instead of presenting things simple and clear. Alternative solutions could be:

1. Make both service parameters optional and do nothing if both were not set (issue a persistent notification warning)
2. Require at least one parameter to be specified in the service data (throw an error upon service execution)

I'm getting reports of a missing ewelink device which is not missing

| switch.10009d2045 | missing | lovelace/tiles/david.yaml:24 |
| switch.10009d1d84 | missing | lovelace/tiles/david.yaml:31

Prior to upgrading to 0.6, I was able to email a report with the service call below.

service: watchman.report
data:
  create_file: false
  send_notification: true
  parse_config: true
  service: notify.smtp_matt
  data:
    title: "Watchman Report"

Now I receive the following error

And the following logs

2022-06-17 13:41:14 ERROR (MainThread) [homeassistant.components.automation.watchman_email_alert] While executing automation automation.watchman_email_alert
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 521, in async_trigger
await self.action_script.async_run(
File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 1513, in async_run
await asyncio.shield(run.async_run())
File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 405, in async_run
await self._async_step(log_exceptions=False)
File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 449, in _async_step
self._handle_exception(
File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 472, in _handle_exception
raise exception
File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 447, in _async_step
await getattr(self, handler)()
File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 680, in _async_call_service_step
await service_task
File "/usr/src/homeassistant/homeassistant/core.py", line 1627, in async_call
task.result()
File "/usr/src/homeassistant/homeassistant/core.py", line 1664, in _execute_service
await cast(Callable[[ServiceCall], Awaitable[None]], handler.job.target)(
File "/config/custom_components/watchman/__init__.py", line 234, in async_handle_report
await async_report_to_notification(
File "/config/custom_components/watchman/__init__.py", line 416, in async_report_to_notification
data = {} if service_data is None else json.loads(service_data)
File "/usr/local/lib/python3.9/json/__init__.py", line 339, in loads
raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not dict

The only thing that changed in HA was updating watchman and I have verified the email notification service still works properly.

Let me know if more info is needed.
Thank you

When looking at the logbook changes in 2022.6.0 I noticed sensor.watchman_last_updated is updated roughly every minute without the service being called to generate a new file.

Thank you for this integration!

After executing the following service:

service: watchman.report
data:
  send_notification: true
  data:
    parse_mode: html

The report file is generated, but the telegram notification doesn't work:

Logger: homeassistant.components.telegram_bot
Source: components/telegram_bot/__init__.py:587
Integration: Telegram bot (documentation, issues)
First occurred: 10:26:02 (1 occurrences)
Last logged: 10:26:02

Error sending message: Can't parse entities: can't find end of the entity starting at byte offset 283. Args: (10464662, '-== Reporte Watchman ==- \n\n-== Congratulations, all 53 services from your config are available!\n\n-== Missing 3 entity(ies) from 258 found in your config:\ndevice_tracker.nmap_joaquina [missing] in: automations.yaml:58\ndevice_tracker.nmap_pedro [missing] in: automations.yaml:59\nbinary_sensor.lluvia [missing] in: automations.yaml:998,1006,1043\n\n\n-== Report created on 23 Feb 2022 10:26:02\n-== Parsed 180 files in 1.86s., ignored 25 files \n-== Generated in: 0.00s. Validated in: 0.01s.'), kwargs: {'parse_mode': 'Markdown', 'disable_web_page_preview': None, 'disable_notification': False, 'reply_to_message_id': None, 'reply_markup': None, 'timeout': None}

I have installed v0.4.0. As per the release notes, it said that default report file name was changed to watchman.txt. However, the file is still being created as thewatchman_report.txt

I noticed in the const.py file that the DEFAULT_REPORT_FILENAME parameter is "thewatchman_report.txt"

The report is telling me zone.split is a missing entity when, it's not an entity. Only the 1st zone.split is throwing the error, the other is fine.

action:
  - variables:
      zone: "{{ trigger.zone.attributes.friendly_name }}"
  - service: notify.alexa_media
    data:
      target:
        - group.all_alexa
      message: >
        {{ 'XXX is approaching home - ' + zone.split('Km Radius')[0].split('Home')[1] + 'k' }}
      data:
        type: tts
        method: all
  - service: notify.XXX
    data:
      message: >
        {{ 'XXX is approaching ' + zone.split('Km Radius')[0] + 'Km'}}

I spent last 30 minutes trying to open PR, but I have to concede with the realization I'm too old and stupid as github kept forcing me to compare branches or some mumbo jumbo, I just wanted to submit PR edit for 1 bloody text file, but I digress.

Anyways, you need to add qutoes to the example @ https://github.com/dummylabs/thewatchman#send-report-via-telegram-bot
so instead of Notification service data: {parse_mode:html} it says Notification service data: {"parse_mode":"html"} otherwise you'll get

when using the example as it is.

sensor.watchman_last_updated should not update when an entity from the ignored_items changes its state.

Hi,
you can create scenes "on the fly" (https://www.home-assistant.io/integrations/scene/#applying-a-scene-without-defining-it) during an automation and and they do not exist outside the context of the automation, however watchman will keep reporting them as missing.

Hello,
I keeps seeing in hacs that I need to "upgrade" from v.0.5.2 to V0.5.1.
I think this is because of the extra period between V and 0 in the V.0.5.2 version. See the attached image.

Logger: homeassistant.config_entries
Source: config_entries.py:316
First occurred: 12:21:45 PM (1 occurrences)
Last logged: 12:21:45 PM

Error importing platform config_flow from integration watchman to set up watchman configuration entry: No module named 'custom_components.watchman.config_flow'

Hello,
Thanks for the great integration! Hopefully, the core developers see its utility and work with you to add this functionality natively!

I've set up thewatchman to send an email nightly if there are any missing entities or services. Would also be handy to be able to receive an email for entities added or removed since the last notification as well as entities that are unknown or unavailable.

Thanks again!

Can you add an option to output the report as markdown? makes it nice and readable in vscode as a preview and also solves issue #3 as you can just render the file using a markdown card :). Love the integration btw

As I understand, ignored_states and ignored_items work independently. Thus, if there is something in ignored_items -- let's say "button.*" for example, then all buttons are always ignored -- likewise, if there is something in ignored_states -- let's say "unknown", then all items with unknown state are always ignore.

Is there a way to use a combination of both? So for example, I only want to ignore button entities, whose state is unknown and not those whose state is e.g. missing. Is this currently possible, and if not, would you consider adding such functionality?

I could envision something like:

{"button.*": ["unknown"]}

In theory, this could substitute/combine both ignored_states and ignored_items, with

{"*": ["unknown"]}

being like ignored_states = "unknown", and

{"button.*": ["*"]}

(or "all", instead of "*" for states?) being like ignored_items = "button.*"

Thanks for your consideration! (and the nice tool :-) )

Add l10n support for notifications and the report.

Could you add the ability to output either a separate report or a configurable option to output the complete list of known entities, preferably as a csv, with a column alongside each containing the watchman output, i.e. OK, Missing, Unknown, Unavailable

I have installed via HACS, rebooted, emptied the buffer and tried to find the integration with no luck. I have tried to roll back one version, but still no cigar. I didn't use it between some updates, so I'm not sure where things got wrong. The earlier versions worked perfect.

k

The ignored files doesn't seem to work. I've tried leading slash, no leading slash, leading '/config' no leading '/config', wrap in "", no wrap in "", path/to/file, no/path/to/file, wildcards (in "" of course, I RTFM'd). I could not get either of my files to be excluded. Fell back to individual entities excluded for the win.

v0.5.0
Configured through UI, not in yaml.
Great tool, thanks.

First THANKS for this addon. I waited ages for such a scanning tool to easily fix my dozens automations etc and now here it is. Fixed about 20 errors that were hidden deep in my yamls. Now for one thing that it complains about, but that is no error:

+--------------------------------+---------+--------------------------------------------------------------+
| Entity                         | State   | Location                                                     |
+--------------------------------+---------+--------------------------------------------------------------+
| timer.finished                 | missing | automations.yaml:269,333,758,907                             |
+--------------------------------+---------+--------------------------------------------------------------+

The corresponding automations look like this:

- id: zigbee-join-timer-deaktivieren
  alias: Zigbee Join nach Timer deaktivieren
  trigger:
  - platform: event
    event_type: timer.finished
    event_data:
      entity_id: timer.zigbee_permit_join
  action:
  - service: mqtt.publish
    data:
      topic: zigbee2mqtt/bridge/config/permit_join
      payload: 'false'
  - service: input_boolean.turn_off
    entity_id: input_boolean.zigbee_permit_join
  mode: single

This is a quite existing event type and no missing entity. So it's a false positive.

something is wrong with the release numbering.
A pic says more then 1000 words:

I was browsing my home assistant error logs and I found this:

This error originated from a custom integration.

Logger: homeassistant
Source: custom_components/watchman/utils.py:255
Integration: Watchman (documentation, issues)
First occurred: 20:35:04 (1 occurrences)
Last logged: 20:35:04

Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/config/custom_components/watchman/__init__.py", line 257, in async_on_home_assistant_started
    parse_config(hass, reason="HA restart")
  File "/config/custom_components/watchman/__init__.py", line 330, in parse_config
    entity_list, service_list, files_parsed, files_ignored = parse(
  File "/config/custom_components/watchman/utils.py", line 255, in parse
    for i, line in enumerate(open(yaml_file, encoding="utf-8")):
FileNotFoundError: [Errno 2] No such file or directory: '/config/ext/crowdsec/parsers/http-logs.yaml'

I'm mounting some shared volumes into home assistant so I can modify their configurations within automations, one of which belongs to CrowdSec. Sadly this places a bunch of symlinks into that directory which, when not inside the CrowdSec image, result into inaccesible symlinks.

For my config I'll just add anything external to the ignore list, but properly handling this exception would improve this plugin.
By the way; Love the insight this gives me! Takes a lot of the gueswork out of cleaning up old deprecated stuff from my configuration.

Hi,

I noticed the states on:

sensor.watchman_missing_entities
and
sensor.watchman_missing_services

are in str format instead of int. It causes problems, for example, when creating automations, we need to convert the state to int.

So, could the state be already in int format?

Thank you.

I am not sure what's happening here. I created a report and removed some entities highlighted in the report.

I recreated the report and those entities are still showing up in the file. I can see that the Report created timestamp in the footer of the file is being updated but do not see the contents being updated.

Hi,

I did see this plugin on youtube, and i wanted to try it aswell.
Well i downloaded this last version on hacs, and after rebooting hass i went to devices and wanted to add the Watchmen.
After that it starts spinning endless while adding it, causing my home assistant to get unresponsive and i can't connect to it.
So i need to manual reboot the host.

I'm running Home Assistant OS 7.5 on virtualbox, last Homeassistant 2022.4.7

regards

I have a door contact that is currently unavailable because of low battery, however, Watchman is not reporting this device to be unavailable. Is Watchman limited to certain types of devices?

HI
This is a great utility and helped me get my system down to zero after finding some 35 errors.
But that all changed after upgrading Dwains Dashboard to V3
Now Watchman reports 100's of issues but i am not sure if it a DD issue or do i need to do something in WM?
I guess i could ignore but i don’t want to ignore something that might be slowing down my setup.
Please see here for output

PS How do you track down the .storage/lovelace:1146 entries ?

I'm not sure if this is possible. I would like watchman to have an option or be able to detect scenes created in a script or automation and not flag those scene as missing.

My example is it creates a new scene service: scene.create based on the current entity states in the snapshot_entities parameter.

I have added the scene entity to the ignore list which also works.

My full script:

lounge_flash_lounge_lights_green:
    alias: "Lounge: Flash Lounge Lights Green"
    sequence:
      - service: scene.create
        data:
          scene_id: current_lounge_lights
          snapshot_entities:
            - light.lounge_left_lamp
            - light.lounge_right_lamp
      - service: scene.turn_on
        target:
          entity_id: scene.lounge_lights_green
        data:
          transition: 0
      - delay:
          hours: 0
          minutes: 0
          seconds: 0
          milliseconds: 500
      - service: scene.turn_on
        target:
          entity_id: scene.current_lounge_lights
    mode: single
    icon: mdi:lamp

Watchman report extract:

+--------------------------------+---------+--------------------------------+
| Entity ID                      | State   | Location                                                     |
+--------------------------------+---------+--------------------------------+
| scene.current_lounge_lights    | missing | scripts.yaml:217

Would be kind of cool to have the report as exploreable dynamic dashboard listing the elements which have issues :)

I have noticed a small bug with the timestamp formatting of the sensor.watchman_last_updated entity when using the iOS / iPadOS Home Assistant apps.

When viewing any lovelace dashboards with the watchman_last_updated entity displayed, the entity would show as Invalid timestamp and I would get the following error in the HA logs:
ERROR (MainThread) [frontend.js.latest.202203010] https://[my-HA-URL]/frontend_latest/92deadc4.js:258:77216 RangeError: number argument must be finite

The errors do not occur on Windows with Edge.

The error did not occur for any of my other device_class: timestamp entities.
Looking at the formatting of my working entity it is using (what I have just learnt is) the ISO 8601 format of YYYY-MM-DDTHH:MM:SS+UTCOFFSET, e.g. 2022-03-03T14:12:17+00:00

The watchman_last_updated sensor is substituting the T with a space character, e.g. 2022-03-10 17:33:25.133995+11:00

I found that the HA templating documentation reference this issue:

If your template is returning a timestamp that should be displayed in the frontend (e.g., as a sensor entity with device_class: timestamp), you have to ensure that it is the ISO 8601 format (meaning it has the “T” separator between the date and time portion). Otherwise, frontend rendering on macOS and iOS devices will show an error.

I'm running:
Watchman: 0.5.0 (through HACS)
Home Assistant Core: 2022.3.2
Supervisor: 2022.03.2
Home Assistant OS: 7.4
Hardware: Raspberry Pi 4b

Would the date format of the sensor be something you would be happy to change? Let me know if there is any other information or testing I can help with, and thanks very much for a great integration!

Edit: The above documentation suggests enforcing iso conversion with the function isoformat(). That didn't work for me with the watchman sensor, but for testing, I created the below template sensor which did not generates the errors:
- name: "Watcher Test"
state: "{{ as_timestamp(states('sensor.watchman_last_updated')) | timestamp_custom('%Y-%m-%dT%H:%M:%S%z') }}"
device_class: timestamp

Add support for entitites like states.binary_sensor.motion.last_updated.

The readme example on how to create markdown for missing services is no longer correct. It iterates over a services attribute, which is now called entities. (That or the attribute got renamed by mistake)

In HACS there's a new version announced, only it's a downgrade?

On Github only releases till 2.5.0 are shown? Something wrong?