requests.exceptions.HTTPError: SequenceLabelingPreprocessor: 404 Client Error: Not Found for url: http://www.modelscope.cn/api/v1/models/xlm-roberta-large/revisions?EndTime=1688313600 about pgim HOT 4 CLOSED

刚刚查到一个解决方案：https://developer.aliyun.com/ask/626669，解决了这个问题，2024-06-13 22:33:27,979 - ERROR - modelscope - Authentication token does not exist, failed to access model xlm-roberta-large which may not exist or may be private. Please login first.
之后确实出现了huggingface的问题，然后看你之前说的加上了环境变量就可以了。感谢感谢！！
export HF_ENDPOINT=https://hf-mirror.com/

from pgim.

我们团队成员两天前在新机器上顺利复现了该模型，所以应该不是模型地址的原因，可以尝试排查一下网络？

from pgim.

你好哇，我直接搜索也是报这个错呢，我看前面获取数据貌似也是可以的。
然后我的整个报错信息是这样的：

2024-06-13 22:32:41,955 - modelscope - INFO - PyTorch version 1.13.1 Found.
2024-06-13 22:32:41,955 - modelscope - INFO - Loading ast index from /data2/hhy/.cache/modelscope/ast_indexer
2024-06-13 22:32:41,955 - modelscope - INFO - No valid ast index found from /data2/hhy/.cache/modelscope/ast_indexer, generating ast index from prebuilt!
2024-06-13 22:32:42,040 - modelscope - INFO - Loading done! Current index file version is 1.7.1, with md5 22229ed48df4c136e5165c03566fd2b7 and a total number of 861 components indexed
2024-06-13 22:32:44,536 - modelscope - WARNING - The reference has been Deprecated in modelscope v1.4.0+, please use from modelscope.msdatasets.dataset_cls.custom_datasets import TorchCustomDataset
2024-06-13 22:32:44,747 - INFO - adaseq.data.dataset_manager - Will use a custom loading script: /data2/hhy/MNER/adaseq/adaseq/data/dataset_builders/named_entity_recognition_dataset_builder.py
Downloading data: 525kB [00:00, 7.11MB/s]
Generating train split: 4000 examples [00:00, 7265.71 examples/s]
Generating valid split: 1000 examples [00:00, 13259.06 examples/s]
Generating test split: 3257 examples [00:00, 13454.56 examples/s]
2024-06-13 22:32:47,306 - INFO - adaseq.data.dataset_manager - First sample in train set: {'id': '0', 'tokens': ['RT', '@JayKenMinaj', '_', ':', 'Me', 'outside', 'of', 'where', 'George', 'Zimmerman', 'got', 'shot', 'at', '.', 'You', 'know', 'God', 'is', 'so', 'good', '.', 'http://t.co/Z3neVBQ7vF'], 'spans': [{'start': 8, 'end': 10, 'type': 'PER'}], 'mask': [True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]}
Counting labels by count_span_labels: 100%|████████████████████████████████████| 4000/4000 [00:00<00:00, 10238.49 examples/s]
Counting labels by count_span_labels: 100%|████████████████████████████████████| 1000/1000 [00:00<00:00, 10096.76 examples/s]
2024-06-13 22:32:47,798 - INFO - adaseq.data.preprocessors.sequence_labeling_preprocessor - label_to_id: {'O': 0, 'B-LOC': 1, 'I-LOC': 2, 'E-LOC': 3, 'S-LOC': 4, 'B-ORG': 5, 'I-ORG': 6, 'E-ORG': 7, 'S-ORG': 8, 'B-OTHER': 9, 'I-OTHER': 10, 'E-OTHER': 11, 'S-OTHER': 12, 'B-PER': 13, 'I-PER': 14, 'E-PER': 15, 'S-PER': 16}
2024-06-13 22:33:27,979 - ERROR - modelscope - Authentication token does not exist, failed to access model xlm-roberta-large which may not exist or may be private. Please login first.
2024-06-13 22:33:27,979 - ERROR - modelscope - Response details: {'Code': 10010205001, 'Message': '获取模型信息失败，信息：record not found', 'RequestId': '5cf2843a-ad38-4792-b054-4815881ce4d8', 'Success': False}

Traceback (most recent call last):
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/urllib3/connection.py", line 198, in _new_conn
sock = connection.create_connection(
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
raise err
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
socket.timeout: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/urllib3/connectionpool.py", line 793, in urlopen
response = self._make_request(
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/urllib3/connectionpool.py", line 491, in _make_request
raise new_e
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/urllib3/connectionpool.py", line 467, in _make_request
self._validate_conn(conn)
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
conn.connect()
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/urllib3/connection.py", line 616, in connect
self.sock = sock = self._new_conn()
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/urllib3/connection.py", line 207, in _new_conn
raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x7f5b1a8a8af0>, 'Connection to huggingface.co timed out. (connect timeout=10)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/requests/adapters.py", line 589, in send
resp = conn.urlopen(
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/urllib3/connectionpool.py", line 847, in urlopen
retries = retries.increment(
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/urllib3/util/retry.py", line 515, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /xlm-roberta-large/resolve/main/config.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f5b1a8a8af0>, 'Connection to huggingface.co timed out. (connect timeout=10)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/huggingface_hub/file_download.py", line 1722, in _get_metadata_or_catch_error
metadata = get_hf_file_metadata(url=url, proxies=proxies, timeout=etag_timeout, headers=headers)
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/huggingface_hub/file_download.py", line 1645, in get_hf_file_metadata
r = _request_wrapper(
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/huggingface_hub/file_download.py", line 372, in _request_wrapper
response = _request_wrapper(
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/huggingface_hub/file_download.py", line 395, in _request_wrapper
response = get_session().request(method=method, url=url, **params)
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/huggingface_hub/utils/_http.py", line 66, in send
return super().send(request, *args, **kwargs)
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/requests/adapters.py", line 610, in send
raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /xlm-roberta-large/resolve/main/config.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f5b1a8a8af0>, 'Connection to huggingface.co timed out. (connect timeout=10)'))"), '(Request ID: 95b7d315-4d56-4dcc-9d8f-a9968d366891)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/transformers/utils/hub.py", line 417, in cached_file
resolved_file = hf_hub_download(
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/huggingface_hub/file_download.py", line 1221, in hf_hub_download
return _hf_hub_download_to_cache_dir(
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/huggingface_hub/file_download.py", line 1325, in _hf_hub_download_to_cache_dir
_raise_on_head_call_error(head_call_error, force_download, local_files_only)
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/huggingface_hub/file_download.py", line 1826, in _raise_on_head_call_error
raise LocalEntryNotFoundError(
huggingface_hub.utils._errors.LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/data2/hhy/MNER/adaseq/adaseq/data/tokenizer.py", line 19, in build_tokenizer
return AutoTokenizer.from_pretrained(model_name_or_path, use_fast=use_fast, **kwargs)
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/transformers/models/auto/tokenization_auto.py", line 658, in from_pretrained
config = AutoConfig.from_pretrained(
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/transformers/models/auto/configuration_auto.py", line 944, in from_pretrained
config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/transformers/configuration_utils.py", line 574, in get_config_dict
config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/transformers/configuration_utils.py", line 629, in _get_config_dict
resolved_config_file = cached_file(
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/transformers/utils/hub.py", line 452, in cached_file
raise EnvironmentError(
OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like xlm-roberta-large is not the path to a directory containing a file named config.json.
Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/modelscope/utils/registry.py", line 212, in build_from_cfg
return obj_cls(**args)
File "/data2/hhy/MNER/adaseq/adaseq/data/preprocessors/sequence_labeling_preprocessor.py", line 38, in init
super().init(model_dir, label_to_id=label_to_id, return_offsets=True, **kwargs)
File "/data2/hhy/MNER/adaseq/adaseq/data/preprocessors/nlp_preprocessor.py", line 71, in init
self.tokenizer = build_tokenizer(
File "/data2/hhy/MNER/adaseq/adaseq/data/tokenizer.py", line 22, in build_tokenizer
model_name_or_path = get_or_download_model_dir(model_name_or_path)
File "/data2/hhy/MNER/adaseq/adaseq/utils/hub_utils.py", line 14, in get_or_download_model_dir
model_cache_dir = snapshot_download(
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/modelscope/hub/snapshot_download.py", line 96, in snapshot_download
revision = _api.get_valid_revision(
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/modelscope/hub/api.py", line 464, in get_valid_revision
revisions = self.list_model_revisions(
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/modelscope/hub/api.py", line 433, in list_model_revisions
handle_http_response(r, logger, cookies, model_id)
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/modelscope/hub/errors.py", line 90, in handle_http_response
raise error
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/modelscope/hub/errors.py", line 82, in handle_http_response
response.raise_for_status()
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/requests/models.py", line 1024, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://www.modelscope.cn/api/v1/models/xlm-roberta-large/revisions?EndTime=1688313600

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/data2/hhy/MNER/adaseq/scripts/train.py", line 38, in
train_model_from_args(args)
File "/data2/hhy/MNER/adaseq/adaseq/commands/train.py", line 85, in train_model_from_args
train_model(
File "/data2/hhy/MNER/adaseq/adaseq/commands/train.py", line 159, in train_model
trainer = build_trainer_from_partial_objects(
File "/data2/hhy/MNER/adaseq/adaseq/commands/train.py", line 199, in build_trainer_from_partial_objects
preprocessor = build_preprocessor(config.preprocessor, labels=dm.labels)
File "/data2/hhy/MNER/adaseq/adaseq/data/preprocessors/nlp_preprocessor.py", line 251, in build_preprocessor
return ms_build_preprocessor(config, field_name, kwargs) # type: ignore
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/modelscope/preprocessors/builder.py", line 21, in build_preprocessor
return build_from_cfg(
File "/data2/hhy/anaconda3/envs/mner/lib/python3.8/site-packages/modelscope/utils/registry.py", line 215, in build_from_cfg
raise type(e)(f'{obj_cls.name}: {e}')
requests.exceptions.HTTPError: SequenceLabelingPreprocessor: 404 Client Error: Not Found for url: http://www.modelscope.cn/api/v1/models/xlm-roberta-large/revisions?EndTime=1688313600

from pgim.

是否可能是因为连不上huggingface所导致的？

from pgim.