zoe-codez / digital-alchemy-nestjs Goto Github PK

Hard coded as "Sat 10:49:00.001" format right now

Description

It should be possible to define portions of an application configuration in external files.

User level common library definitions

In ~/.config/digital-alchemy/{library_name}(""|".yml"|".yaml"|".json"|".ini"), definitions can be placed as updated defaults.
These definitions should be inserted into the configuration with the following priority:

• ...
• normal file based
• >>common library<<
• application level defaults
• ...

Inline references to external files

A post-processing step for the configuration should be added. It should check the top level of each module configuration, then each of their properties (NOT RECURSIVELY, that logic must be implemented at a library level if it's desired).

These references should take the form of a $ref property. These are file locations (potentially optional object path also) to an external file which contains the information. For property level $refs, some coersion will be needed for the value. The file may contain the plan word "redis", if used on libs.boilerplate.CACHE_PROVIDER, for example. It doesn't make sense to run that through any ini / etc parser logic

Library import example

[libs.tty]
  $ref=./file

{
  "libs": {
    "tty": {
      "$ref": "./file"
    }
  }
}

Property import example

[libs.tty.THEME]
  $ref=./file

{
  "libs": {
    "tty": {
      "THEME": {
        "$ref": "./file"
      }
    }
  }
}

Pagedown should always pick the last visible item in the list. For some reason, when there are hidden items present the end of the list (possibly in the middle too), the cursor is allowed to pick list length + 1 as a position. There is likely a missing ARRAY_OFFSET somewhere

The previous publish pipeline relied on the nx workspace file. Needs use a bash find to locate projects instead now

https://www.home-assistant.io/integrations/mqtt/#mqtt-discovery

Set up a new service for managing interactions via the home assistant mqtt integration. Creating new entities, updating states, and other related functionality.

Should be set up as a second module inside the home assistant library to create a soft dependency on the mqtt library.

Current Situation

The master set of definitions for all the home assistant entities is just the current entity state exported as JSON: https://github.com/zoe-codez/digital-alchemy/blob/main/apps/hass-type-generate/src/main.ts#L57

Typescript helpfully turns this into more useful definitions, but a lot of information is lost in translation.

New Solution

Use comment-json to inject tsdoc definitions into the object. A new provider should be created that can perform the lookups, and merge in the data.

Below is a starting list of content that can be placed inside various structures.

domains

• simple listing of all entity ids
• related services that can be called
• related integrations
  • sanity check if this makes sense, some integrations like tp-link appear to set up as individual integrations for each device. That would be less than ideal

entities

• provided by integration
• related devices
• unique status values

entity attributes + status

• unique values observed from history lookup

Notes on rebuilding

Right now, the process is a pretty lightweight "just the current state" dump. The above goals call for some historical information, which requires many lookups against home assistant, making this a much more expensive operation to perform. Since the rebuild operation is tied to the postinstall hook, this will likely end up dramatically increasing dependency update times.

The hook should be left in, but the hook should add a new switch to identify itself. A new configuration property should be introduced using that switch, as well as a second one (boolean) to disable rebuilds via hooks. The hass-type-generate script (without the switch) will still be able to rebuild on demand.

Additional configuration items

History look back range

Control how much history to use in looking for unique status / attribute values

Quality of life improvement -

When ctrl is pressed, actions that interact with the cursor should operate with words, instead of characters.

• ctrl+backspace: erase previous word
• ctrl+delete: erase next word
• ctrl+left/right: cursor movement related to the current position

Description

Add the ability to work with external editors as a method of gathering a string value. The configuration variable for this should utilize env.EDITOR as the default value.

The prompt should write the default value to a text temp file, then pass it as the first argument to the configured editor. execa should be able to connect things up so the editor works properly. When the editor completes, read the contents of the text file back, remove it, and return the value.

Extra investigation: env.VISUAL

The workflow is definitely different from above, but unclear exactly how. Needs investigation.

Overview

Instead of providing definitions to the builder with the elements array, it should be possible to provide an object definition as a json schema.

This would most cleanly be implemented as a standalone component, similar to the array builder.

New Component

The component should utilize the array + object builder for rendering, adding any extra glue if needed. The object builder will need to add some sort of support for recursion to properly handle this.

The new component will need to keep track of the current object path, the state of the schema it is building against, and the value. Since json schema is intended to be a validation format, not a form rendering format, some information (like labels) will need to be fudged to help things make sense.

TODO: invent a name for this that can be use with prompt service

Reference docs for NestJS: https://docs.nestjs.com/fundamentals/custom-providers#export-custom-provider

Current Syntax

Default value must be provided as ready to go value. Value can be calculated prior to calling bootstrap.

@LibraryModule({
  configuration: {
    [INSTANCE_PRIORITY]: {
      default: 20,
      type: "number",
    },
  },
})
export class ExampleModule {}

Proposed update

NestJS custom should be allowed in the default property, in place of a plain value.
These will only be partial definitions, since useExisting / useFactory don't make sense in this context.

Example in action

@LibraryModule({
  configuration: {
    [INSTANCE_PRIORITY]: {
      default: { 
        // omit `provide`, since it gets generated internally
        inject: [SpecialSymbol],
        useFactory(item: SpecialSymbol) {
          return item.getPriority();
        }
      },
      type: "number",
    },
  },
})
export class TestModule {}

Required workflow updates

Generate values (early bootstrap)

1. scan config object, generate list of providers to have nest resolve
2. give provide values to each item to make it a valid provider, then inject into a scope AutoConfigService can access
3. tag each config item with a symbol to use for lookups later

• requires some type updates, make sure they only affect inwards facing code

Value resolution

AutoConfigService.get

Currently, this method just accepts the .default property as part of defaulting. This logic should be updated to check for the existence of additional metadata (provided above), and to do a lookup for the resolved value.

The lookup will require access to the nestjs application object (defaultValue = await app.resolve(LOOKUP_SYMBOL))

🤦

These libraries should be capable of generating dashboards lovelace on demand based off the entities they consume / generate.

Open ended question on how that looks. On the technical side, it should be an opt in feature that will automatically manage part of lovelace in some way

Getting some deprecated warnings

It should be possible to update the menu component to support launching a child editor inside of the keymap async callbacks. This idea is already used inside of the object builder component

This feature is currently intended for delete / confirm workflows, but should be built to support the full set of editors that object builder does.

When toggling items between source / current, the cursor doesn't always react naturally. Callouts

• gets stuck when removing final item from a side

Configuration properties like SCAN_CONFIG, HELP, CONFIG are all intended to be consumed as command line switches, and do not make sense to be consumed in any other context. Boilerplate still reports them as being consumable via any source, including file based configurations, which is both wrong and confusing as a potential use case.

The issue is mostly in the reporting of available configuration properties. --help should continue to work as is, reporting everything. Some method of flagging these properties should be made available so that config builder knows to not display these as configurable.

There should be some sort of challenge & response system integrated into the entity package setup that can be used to explicitly tell:

• if a given package was included by home assistant
  • vs just having it's config written, and not !included
• what version of that package is currently loaded in memory
  • having updates written, but not loaded

Extra requirements

• This system should be callable from anywhere. It cannot have a "replies back to ip:port" type of automation to announce data
• Must rely on 100% vanilla yaml (no mqtt or other integrations)
• A standard webhook style "send config request, get json back" is ideal
• No storing values in entity states / attributes

Issue

When large strings are input using the object builder, the value column can incorrectly exceed the maximum width of the terminal window. This causes the text to wrap, and the bottom bar to also wrap in an attempt to match the width. Ends up looking very unusual.

There is currently some string shortening logic:

• Hard coded value
• Logic using

It is not directly related to the table rendering process, so it's mostly a coincidence that it's being sorta helpful.

Fix

Update the interface to the type printer to take in an object as the 2nd argument. That argument is intended to be for internal use only, so there shouldn't be any breaking changes to deal with.

The configuration object should be able to take in a max string length. -1 (default) means unlimited.

TableService will need it's calls to textRendering.type to be adjusted to pass along a max width calculated using this logic

screen width - (external margin * 2) - (max width of label column) - (internal padding * 4) - 3 (from border lines) + c

ellipsis should be a gray.bold

Current situation

Currently, there are two methods of providing object definitions to the configuration system

internal

This type is intended to act as a pass through for other libraries. The usage is intended to be a bit sloppy, preferably referencing where to find the correct definition in the description.

Example usage in MQTT: https://github.com/zoe-codez/digital-alchemy/blob/main/libs/mqtt/src/modules/mqtt.module.ts#L15

Example config ini form

[libs.mqtt.CLIENT_OPTIONS]
  host=192.168.1.100

record

This type is intended to create simple key/value pairs. Both sides should be a string for this.

Example usage in ini form

[application.PICO_MAPPINGS]
  2=sensor.office_pico
  8=sensor.games_pico
  9=sensor.loft_pico
  10=sensor.bedroom_pico
  11=sensor.bed_pico
  12=sensor.living_pico
  13=sensor.office_pico_2
  14=sensor.office_pico_1

New object type

The object type is intended to allow a more free form object definition inside the configuration system. Validation can optionally be done against the value, preventing the app from booting + printing errors. Validation is performed once at boot, with one of the following mechanisms.

no validation mechanism

Configuration will ensure the value is an object. Beyond that, it doesn't care.

• if null/undefined: use blank object
• if boolean/number/string: fatal error

json schema

Value will be checked with AJV at boot.

class-validator

A properly annotated class can be provided to check against the value.

Investigate: can class-transformer do anything productive here?

Extra notes

Data merging

By default, the configuration system should replace the entire object every time it finds one from a higher priority source. The configuration definition should provide an optional boolean flag that will tell the configuration system to merge objects together instead of replacing.

changed mind, handle at application layer

Update the builds across all libraries to have a consistent usage of peerDependencies. At minimum, all @digital-alchemy dependencies should always appear as peerDependencies

The current format is prettier, but the output of util inspect is faster to understand. Update the text rendering type method to wrap util inspect instead of it's current functionality

Check into if the object printing for pretty logger can be similarly updated

https://docs.nestjs.com/devtools/ci-cd-integration

The custom keymap should only be available in normal find mode. The rendering reflects this, but search mode seems to be incorrectly using them anyways.

Support is very basic, in a "here is the websocket, go do it yourself" capacity with no real type support right now.

@digital-alchemy/home-assistant

• Build out basic internal type definitions
• Build provider capable of actively managing the registry, and consuming the event stream from home assistant
  • Use EntityManagerService as reference
• Attach to SocketManagerService

@digital-alchemy/hass-cli

Expose functionality as a demo

@digital-alchemy/hass-type-generate

New types to generate

• area list
• icon list

@digital-alchemy/home-assistant needs support for number domain template entities

https://www.home-assistant.io/integrations/template/#number

The weird dual rendering situation that's going on in table builder is causing confusing type definitions and internal feature gaps as single object rendering gets more attention.

Goals

Table builder

This should be converted to exclusively be a single object builder. The "single" rendering mode

Workflow: list builder

There isn't a really sane way to hide / show columns when each row can have different rules. Not aiming to make excel. The current "multi" rendering mode would be better expressed as a combination of menu and object builder in a canned workflow.

This appears as a socket frame that appears immediately after Home Assistant replies with auth_ok.

{"type":"supported_features","id":1,"features":{"coalesce_messages":1}}

It appears alter the way Home Assistant sends updates over the websocket. With this enabled, messages can appear as both single objects, or an array of objects. Objects appear to be patch changes, rather than full size entity states being sent with every update.

Example update

{"id":3,"type":"event","event":{"c":{"sensor.adguard_home_dns_queries":{"+":{"s":"1171995","lc":1669568008.42719,"c":"01GJX0447BF912599CZBXNY4AX"}}}}}

Motivation

Many "dump data to the screen -> return to component" workflows would be conveniently handled by an internal binding. The lack of a convenient way to do this is causing excessing quantities of information to be dumped into help text

Changes

API

In addition to the current format:

{
  entry: ["label", "value"],
  highlight: "auto"
}

There should be a secondary object format available which provides some internal bindings and customized arguments

POC: Data inspector mode

Submenu pickmany:

• dump data to file (temp / default some persistent spot / default last base dir)
• print to screen

POC: Expanded help text

Button to hide widget, dump full current item help text to console.
Any key to hide

Every time the nested enum prompt closes, it adds another line to the top

Description

Currently, the configuration is a bit of a black box on how it arrives at some of the values. Some additional trace logs should be provided to describe what's going on as it sets values. Since the configuration flow already works in a priority order, this should result in logs that can be read backwards to identify where a given value came from.

When the postinstall step of hass-type-generate fails, it does so in a way that causes yarn roll back the install. On a lower powered device, this can be a very frustrating rollback to experience after waiting 5 minutes to download and link everything.

The code should be updated so that it quietly bails with a success code + error log, instead of exiting with an error status. This should keep the new modules around, but with the unmodified dynamic types. The npx hass-type-generate command should be able to build them again later to correct the issue

Blocked by #1

Description

Most of these elements are currently hard coded values within tty

• colors
• padding offsets
• font selections
• specific font awesome icons
• ui icons (ex: using | as separators)

These need to be identified, and altered to work based off configurations. A new theme service should be added as a generic handler for this work.

Configuration format

A new THEME configuration should be attached to the module. This must be an object, which will contain the specific theme information. Everything should be optional in the theme, defaulting to current hard coded values if not explicitly provided.

Specific organization of properties TBD after some discovery

Potential future expansion

Implementation should not require significant changes for these goals.
They will not necessarily be implemented as part of this issue though.

• passing some sort of theme info as config args
• callback style coloring (ex: for rainbow bars)
• theme definitions in external files (creation of canned named themes)
• altering theme information without restarting the app

Anticipated breaking changes

Existing module configurations controlling any of these attributes as one offs will be dropped.

Issues

Rendering

Help text takes priority over the useful part of the component when there are insufficient rows to render everything with.

API

• Limiting to only strings forces extra code

Changes

Render priority

Height calculations will prioritize:

1. component
2. callback output
3. item help text
4. header
5. keybindings
6. component help

mental note: some toggles, and low hanging fruit configurability may be interesting

Help text as string array

Simple join back together on new line, one less command

Help text extends object

Run through text rendering debug

https://www.home-assistant.io/integrations/light.template/

Intent would to be to use in combination with rgb matrix eventually

Waiting on updates to NX to better support building. The rest of the repo so far seems pretty close to where it needs to be

https://togithub.com/nrwl/nx/pull/10414

When Digital-Alchemy-TS/core#1 & Digital-Alchemy-TS/core#20 are complete, the configuration builder needs to be upgraded to support the newly available definitions.

Add commands for interacting with home assistant backups. The UI seems to be interacting with the commands through the websocket. These are some observed socket frames from interactions

Commands

Generate backup

{"type":"backup/generate","id":32}

Backup Info

Request

{"type":"backup/info","id":33}

Result

{"id":33,"type":"result","success":true,"result":{"backups":[{"slug":"4d8b857e","name":"Core 2022.10.5","date":"2022-11-03T16:39:57.425304-05:00","path":"/config/backups/4d8b857e.tar","size":73.08}],"backing_up":false}}

Remove

{"type":"backup/remove","slug":"4d8b857e","id":35}

Download

Request

{"type":"auth/sign_path","path":"/api/backup/download/65f819e4","id":37}

Response

{"id":37,"type":"result","success":true,"result":{"path":"/api/backup/download/65f819e4?authSig=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiI0Njc1MmUzZmUzM2Y0YTJlOThmZTlmOWFhZWMxY2QzYyIsInBhdGgiOiIvYXBpL2JhY2t1cC9kb3dubG9hZC82NWY4MTllNCIsInBhcmFtcyI6e30sImlhdCI6MTY2OTMwOTMxNSwiZXhwIjoxNjY5MzA5MzQ1fQ.1EDVKt6nXWs0GbTo46kXK2ufOH3hXWJDZNCD_9N3PTg"}}

• Download URL: {BASE_DOMAIN}{path}

TTY

Add as flag to string editor, for when multiline content is expected

Squash all the errors arising from the typescript rule: noImplicitAny, and enable in root tsconfig

Currently, component keymaps are hard coded at the top of their relevant files. It would be nice to set things up so that those entries could be substituted via configuration variables / parameters passed in on a per-call basis.

@digital-alchemy/home-assistant needs support for select push entities

https://www.home-assistant.io/integrations/template/#select

Issues

• Menu fuzzy search currently pulls from all available options, merging them into a single list. This can lead to losing context as a user

• The search box acts a lot less sophisticated than the one from the text box component

Solutions

Maintain the dual column layout & type groupings

Title, it is jarring to lose this

Keyboard refinements

Cursor is now shared between filters and filtered list

• Cursor in data
  • interactions should generally stay the same
  • regular key presses will continue to append/subtract text to the filter box
  • if at the top of the item list, up / page up will move back up into filter box
• Cursor in search box
  • text will respond according to cursor position
  • left / right / home / end react as expected
  • pressing down / page down will move to the top of the most recently used side
• everywhere
  • escape clears the search text

Search refinements

• Search text is preserved for duration of component

  • Currently resets every time search is opened

• Search only within single column (left / right)

Columns frequently get used for different purposes. Ex: left is dynamic list from database, right is static list of actions.
Controllable via component options

• Configurable highlighting (boolean)

  • Type grouping
  • Help text
  • Label

• Configurable search target object

Not all data may necessarily be rendered, but it may be desirable to search against it anyway

Add optional callback to menu component that will allow for the external cancelling of the menu. Ex: timeout

Cursor will incorrectly attempt to select hidden properties

The way the sensor was set up never had a chance of working. The sensor needs to be updated to work off of yaml based automations.