Comments (2)
haven-jeon
commented on August 24, 2024
The performance of the tokenization algorithm is difficult to judge. With wiki documents, it was trained with the sentencepiece algorithm with a vocab size of 8k. Tokenizer is trained by considering word frequency and LM score. First, it seems appropriate to raise an issue along with concerns about how to measure the tokenizer's performance. What is the performance of the KoBERT tokenizer numerically?
from kobert.
haven-jeon
commented on August 24, 2024
Since there is no answer, I am closing the issue.
from kobert.
Related Issues (20)
- 파이썬 설치 오류 2가지 문의(fastcache, tensorflow)
- [BUG] onnxruntime error HOT 3
- [BUG] colab환경에서 라이브러리 설치 오류, import 오류 HOT 11
- [BUG] PIP설치 오류 HOT 2
- [BUG] HOT 11
- BERT Dataset 'not a string' error HOT 1
- 제발 도와주세요 개발환경 문제 [BUG] HOT 4
- [BUG] HOT 2
- type object 'BERTVocab' has no attribute 'from_sentencepiece' HOT 1
- [BUG] 코렙에서 Kobert 설치가 원활히 되지 않습니다.
- AttributeError 해결 도와주세요ㅠㅠ[BUG]
- [BUG] pip install error
- 오프라인 환경에서 vocab 접근할 수 있는 방법
- Used number of token for pretraining
- [BUG] 코랩에서 안뜨던 오류가 윈도우로 옮기니 typeError가 뜹니다..ㅠㅠ
- [BUG] M1 MAC ModuleNotFoundError: No module named 'kobert.utils'; 'kobert' is not a package HOT 1
- [BUG] Jupyter 가상환경 내 KOBERT 설치 불가 오류
- [FEATURE] optuna와 앙상블 적용방법에 대한 질의
- [Installation Error] Having an issue with install the package. HOT 3
- [FEATURE] kobert로 단어 임베딩을 진행할 수 있을까요?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kobert.