Comments (2)
Hi @sergeichukd,
We provide a number of classification and information retrieval metrics that can be used during training via the EvalCallback
object in tensorflow_similarity.callbacks
. I general we tend to use the binary_accuracy metric as this can be thought of as the within threshold precision * the recall or total number of elements in the database. There are more details in the doc string under the classification metrics dir.
Regarding the comment. The issue here is that we train the model using a subset of the classes but include all classes in the validation dataset. This gives us a sense of how well the model will generalize to new unseen classes, but also means the validation metrics will lag behind the train metrics.
Ket me know if you have any questions about the EvalCallback object or any of the metrics. Here is a more detailed example of running some of the evaluations after the model is finished training.
# Assumes you have trained a model and that index data != query data...
# Add all examples to the index
brute_force_search = NMSLibSearch(
distance="cosine",
dim=model.output.shape[1],
method='brute_force',
)
# Create or clear the index
try:
model.reset_index() # clear the index
except AttributeError:
model.create_index(brute_force_search) # or create it.
model.index(index_x, y=index_y, data=index_human_readable_data)
calibrate_metrics = model.calibrate(
query_x,
y=query_y,
thresholds_targets = {"0.99": 0.99, "0.95": 0.95, "0.90": 0.90, "0.85": 0.85, "0.80": 0.80},
calibration_metric="binary_accuracy",
)
eval_cal = model.evaluate_classification(
query_x,
y=query_y,
extra_metrics=['precision', 'binary_accuracy', 'recall', 'npv', 'fpr'],
)
def make_recall_at_k(k: int) -> RecallAtK:
return RecallAtK(k=k, average="macro")
def make_precision_at_k(k: int) -> PrecisionAtK:
return PrecisionAtK(k=k, average="macro")
def make_map_at_r(targets_y: np.ndarray, max_class_count: int) -> MapAtK:
class_counts = Counter(targets_y)
max_class_count = min(max(class_counts.values()), max_class_count)
return MapAtK(
r=class_counts,
clip_at_r=True,
k=max_class_count,
name="map@R",
)
def make_r_precision(
targets_y: np.ndarray, max_class_count: int
) -> PrecisionAtK:
class_counts = Counter(targets_y)
max_class_count = min(max(class_counts.values()), max_class_count)
return PrecisionAtK(
r=class_counts,
clip_at_r=True,
k=max_class_count,
name="R_Precision",
)
recall_at_k = [make_recall_at_k(k) for k in [1, 2, 4, 8, 16, 32]]
precision_at_k = [make_precision_at_k(k) for k in [1, 2, 4, 8, 16, 32]]
metrics = [
make_map_at_r(df["label"].cat.codes.values, 300),
make_r_precision(df["label"].cat.codes.values, 300),
]
metrics.extend(recall_at_k + precision_at_k)
eval_cal = model.evaluate_retrieval(
query_x,
y=query_y,
retrieval_metrics=metrics,
)
from similarity.
Thank you, @owenvallis!
This callback is extremely helpful for me
from similarity.
Related Issues (20)
- [Typo] Lables instead of labels HOT 1
- self-supervised segmentation and detection
- VicReg implementation gives "nan" loss HOT 3
- Problem using the "brute_force" search method HOT 3
- README.md typo
- Out of memory error: Multi-GPUs not used when training semi-supervised model
- Error with multishotfilesampler HOT 4
- Race condition during EvalCallback HOT 1
- Add Colab example demonstrating multiple metrics in Callback.
- Replace all examples using MemorySamplers with the TFData_Samplers HOT 1
- Replace all examples using NMSLIB with KNNSearch HOT 1
- Add ConvNeXtSmall to architectures HOT 1
- Add warning / update to TF addon to help people transition
- Custom Architecture for Backbone model
- Update readme with v0.18 info
- TripletLoss will produce only NaN when using mixed bfloat16 precision
- tfa.losses.TripletSemiHardLoss() replacement? HOT 2
- Calculation of mean Averge Precision (mAP)
- Tensorflow > 2.11 ? HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from similarity.