openxaiproject / pnpxai Goto Github PK

We are not providing target layer as a kwarg, by automatically locating candidate layer. So, I think we have two choices for the next step

• keep this policy and remove unnecessary kwargs such as attribute_to_layer_input, relu_attributions and attr_dim_summation
• modify the explainers allowing user to manually input target layer as well as automation, keeping kwargs

I think the second one is more desirable.

The robustness test for RAP fails with memory issue on the gpu server. This is not the matter of robustness in computing time, but I guess the testing should cover this issue. One of my idea is to make separate testing code to check robustness of explainer about memory. What do you think about this @enver1323 ?

Evaluation Specification

I've tried to implement an experiment on quality of metrics over parameter settings. For example, I wanted to check whether MuFidelity(n_perturbations=20) returns similar value with MuFidelity(n_perturbations=200). To do this, I tried replacing auto-evaluator to my own evaluator (This is not elegant way. I think we need to talk about this as a separate issue) and running it.

proj = Project("test proj")
expr = proj.create_auto_experiment(model=model, data=data)

# replace to my own
expr.evaluator = XaiEvaluator(metrics=[
    MuFidelity(n_perturbations=20),
    MuFidelity(n_perturbations=200),
])

expr.run()

As a result, run.evaluations returned the last metric values only, even though the first metric values were calculated. The reason is that the current version of evaluator is collecting the values by name of metric, not the index of it. There are several ways to implement such experiment in the current version i.e. creating multiple experiments over different param settings, However, it is not efficient because it runs explainer twice.

Currently,

from pnpxai.evaluator.mu_fidelity import MuFidelity
from pnpxai.evaluator.sensitivity import Sensitivity
from pnpxia.evaluator.complexity import Complexity

I think the following is more convenient, flexible and intuitive

from pnpxai.evaluator.metrics import MuFidelity, Sensitivity, Complexity

Currently, experiments inputs request is initialized with page loading, which causes a delay in page loading. Let's bind this request to the modal window openning condition. This would allow quick initial page load

Draft

UI Design

아래의 사항을 참고하여 Figma로 UI Design

• 주요 페이지 구성
  • Model Detection Info
    • Model Name
    • Model Layer
  • Algorithm Visualization
    • 주어진 Sample들에 대해서 아래의 정보를 보여줌
    • Sample Image, Predicted Label, True Label, Overlapped Heatmap, Evaluation Score

UI Design in Figma following details:

Main Page Structure

• Model Detection Info
  • Model Name
  • Model Layer
• Algorithm Visualization
  • Display the following information for the given samples
  • Sample Image, Predicted Label, True Label, Overlapped Heatmap, Evaluation Score"

How can we address the Plotly Object?

Suggesttion

• Request to save the heatmap image to server with API Request
• Request evaluation score data to server with API Request.
• Visualize the data using plotly.js