[Feature Request] Hope to get the token usage report per AOAI connection deployment about promptflow HOT 4 OPEN

@Wix97 From pf perspective, it would be complicated to define such a contract to summarize tokens according to the tokens.

I have a proposal: what if we expose the ability to get the tokens in other nodes?
For example, we provide such an experession: {llm_node_1.system_metrics.usage} to get the usage dictionary of a node, then you can have a summary node to get the dictionary for yourself.

from promptflow.

Hi @Wix97 . Thanks for reaching us, could you please elaborate more about the 'usage of each AOAI connection deployment'? Is that related to any run, for example a batch run with different deployment, to get the usage of each deployment, or just overall token usage of a deployment? For the latter one, seems that should be the capability of AzureOpenAI instead of promptflow.

from promptflow.

Sorry for the confusion, the first case is what I wanted to mention here.
For example, there is a flow consuming 2 AOAI connections, each connection has 2 deployments. It would be helpful if promptflow can generate token usage report such like:

  "token_usage": {
    "connection_1": {
      "connection_1_deployment_1": {
        "prompt_tokens": 123,
        "completion_tokens": 456
      },
      "connection_1_deployment_2": {
        "prompt_tokens": 123,
        "completion_tokens": 456
      }
    },
    "connection_2": {
      "connection_2_deployment_1": {
        "prompt_tokens": 123,
        "completion_tokens": 456
      },
      "connection_2_deployment_2": {
        "prompt_tokens": 123,
        "completion_tokens": 456
      }
    }
  }
}

from promptflow.

LGTM👍

from promptflow.