Comments (12)
Hey guys, I guess it's a problem with evaluate==0.4.2. I replaced the cached exact_match.py script (the file where the error occured) with a previous version (sry not sure which version it is exactly), and everything works fine from my side. Please feel free to use this as a temperory solution, but I think some bug need to be solved within version 0.4.2.
The exact_match.py I used: (I obtained it from my partner, credit to him :)
# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Exact Match metric."""
import re
import string
import datasets
import numpy as np
import evaluate
_DESCRIPTION = """
Returns the rate at which the input predicted strings exactly match their references, ignoring any strings input as part of the regexes_to_ignore list.
"""
_KWARGS_DESCRIPTION = """
Args:
predictions: List of predicted texts.
references: List of reference texts.
regexes_to_ignore: List, defaults to None. Regex expressions of characters to
ignore when calculating the exact matches. Note: these regexes are removed
from the input data before the changes based on the options below (e.g. ignore_case,
ignore_punctuation, ignore_numbers) are applied.
ignore_case: Boolean, defaults to False. If true, turns everything
to lowercase so that capitalization differences are ignored.
ignore_punctuation: Boolean, defaults to False. If true, removes all punctuation before
comparing predictions and references.
ignore_numbers: Boolean, defaults to False. If true, removes all punctuation before
comparing predictions and references.
Returns:
exact_match: Dictionary containing exact_match rate. Possible values are between 0.0 and 1.0, inclusive.
Examples:
>>> exact_match = evaluate.load("exact_match")
>>> refs = ["the cat", "theater", "YELLING", "agent007"]
>>> preds = ["cat?", "theater", "yelling", "agent"]
>>> results = exact_match.compute(references=refs, predictions=preds)
>>> print(round(results["exact_match"], 2))
0.25
>>> exact_match = evaluate.load("exact_match")
>>> refs = ["the cat", "theater", "YELLING", "agent007"]
>>> preds = ["cat?", "theater", "yelling", "agent"]
>>> results = exact_match.compute(references=refs, predictions=preds, regexes_to_ignore=["the ", "yell"], ignore_case=True, ignore_punctuation=True)
>>> print(round(results["exact_match"], 2))
0.5
>>> exact_match = evaluate.load("exact_match")
>>> refs = ["the cat", "theater", "YELLING", "agent007"]
>>> preds = ["cat?", "theater", "yelling", "agent"]
>>> results = exact_match.compute(references=refs, predictions=preds, regexes_to_ignore=["the ", "yell", "YELL"], ignore_case=True, ignore_punctuation=True)
>>> print(round(results["exact_match"], 2))
0.75
>>> exact_match = evaluate.load("exact_match")
>>> refs = ["the cat", "theater", "YELLING", "agent007"]
>>> preds = ["cat?", "theater", "yelling", "agent"]
>>> results = exact_match.compute(references=refs, predictions=preds, regexes_to_ignore=["the ", "yell", "YELL"], ignore_case=True, ignore_punctuation=True, ignore_numbers=True)
>>> print(round(results["exact_match"], 2))
1.0
>>> exact_match = evaluate.load("exact_match")
>>> refs = ["The cat sat on the mat.", "Theaters are great.", "It's like comparing oranges and apples."]
>>> preds = ["The cat sat on the mat?", "Theaters are great.", "It's like comparing apples and oranges."]
>>> results = exact_match.compute(references=refs, predictions=preds)
>>> print(round(results["exact_match"], 2))
0.33
"""
_CITATION = """
"""
@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
class ExactMatch(evaluate.Metric):
def _info(self):
return evaluate.MetricInfo(
description=_DESCRIPTION,
citation=_CITATION,
inputs_description=_KWARGS_DESCRIPTION,
features=datasets.Features(
{
"predictions": datasets.Value("string", id="sequence"),
"references": datasets.Value("string", id="sequence"),
}
),
reference_urls=[],
)
def _compute(
self,
predictions,
references,
regexes_to_ignore=None,
ignore_case=False,
ignore_punctuation=False,
ignore_numbers=False,
):
if regexes_to_ignore is not None:
for s in regexes_to_ignore:
predictions = np.array([re.sub(s, "", x) for x in predictions])
references = np.array([re.sub(s, "", x) for x in references])
else:
predictions = np.asarray(predictions)
references = np.asarray(references)
if ignore_case:
predictions = np.char.lower(predictions)
references = np.char.lower(references)
if ignore_punctuation:
repl_table = string.punctuation.maketrans("", "", string.punctuation)
predictions = np.char.translate(predictions, table=repl_table)
references = np.char.translate(references, table=repl_table)
if ignore_numbers:
repl_table = string.digits.maketrans("", "", string.digits)
predictions = np.char.translate(predictions, table=repl_table)
references = np.char.translate(references, table=repl_table)
score_list = predictions == references
return {"exact_match": np.mean(score_list)}
from evaluate.
@sxluo Thanks for your advice!
I originally have this line:
export HF_ENDPOINT=https://hf-mirror.com
However I cannot remove this line, or change this to https://huggingface.co
. Otherwise i wouldn't be able to connect to HF on my server. (mainland China).
So I clone 'evaluate' to my local path, and change this line:
metric = evaluate.load("accuracy")
to:
metric = evaluate.load(MY_PATH+"/evaluate/accuracy/accuracy.py")
It turns out to work well :>. Hope this could help others.
from evaluate.
I can see this in blue.py
由于流量过大,本站暂不支持访问 Spaces 空间,可前往 Hugging Face 官网查看。
返回上一页 前往官网入口<script>
function goBack() {
window.history.back();
}
function redirectToOfficialApplicationPage() {
const url = new URL(window.location.href);
const path = url.pathname;
const regex = /^\/spaces\/(.*)$/;
const match = path.match(regex);
if (match) {
const officialPageUrl = `https://huggingface.co${path}`;
window.location.href = officialPageUrl;
} else {
const officialPageUrl = `https://huggingface.co/spaces`;
window.location.href = officialPageUrl;
}
}
// Dynamically update the button text based on the 'next' URL parameter
window.onload = () => {
const url = new URL(window.location.href);
const path = url.pathname;
const regex = /^\/spaces\/(.*)$/;
const match = path.match(regex);
if (match) {
const projectName = match[1];
document.getElementById('applyButton').textContent = `前往官网 ${projectName} 入口`;
}
};
</script>
I guess the problem may be from the https://hf-mirror.com
from evaluate.
Have you solved this problem? I also encountered
from evaluate.
Same here
from evaluate.
Same here
from evaluate.
Same here with version 0.4.2
from evaluate.
Same problem
from evaluate.
Same problem. I tried the solution proposed by @ErikaaWang and found it worked.
from evaluate.
I fixed the issue by changing HF_ENDPOINT to https://huggingface.co/.
from evaluate.
Same problem, I tried the solution from @ZeguanXiaoand, but it didn't work out
from evaluate.
Same problem, I tried the solution from @ZeguanXiaoand, but it didn't work out
Try: export HF_ENDPOINT="https://huggingface.co"
from evaluate.
Related Issues (20)
- Cannot use it offline! HOT 1
- Allow for specify coda device in perplexity evaluation
- [Question] How to have no preset values sent into `.compute()`
- METEOR has no option to return unaggregated results
- Perplexity metric does not apply batching correctly to tokenization HOT 1
- Module 'glue' doesn't exist on the Hugging Face Hub either. HOT 1
- Unable to run pip install evaluate[template] HOT 1
- [FR] Confidence intervals for metrics
- How to pass generation_kwargs to the TextGeneration evaluator ?
- Metrics for multilabel problems don't match the expected format. HOT 2
- [Question]Shall we adding a faster BLEU score calculator?
- Can't load exist dataset for evaluation HOT 1
- SyntaxError: closing parenthesis '}' HOT 3
- Evaluation of empty strings with MAUVE results in error
- Execution of example from the Using the evaluator docs fails due to unspecified tokenizer
- LocalModuleTest.test_load_metric_code_eval fails with "The "code_eval" metric executes untrusted model-generated code in Python."
- Evaluation of form feed symbol with BLEU results in error
- Gradio dependency issue
- AttributeError: 'CombinedEvaluations' object has no attribute 'evaluation_modules' HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from evaluate.