naxda / takos-alpha Goto Github PK

Simply create your own Korean spacing model with your sentence text file!

This is an alpha version. It will be installed by pip install in the future.

git clone https://github.com/Taekyoon/takos-alpha.git
pip install -r requirements
python setup.py install

• Python 3.6.2

## versions will be update soon
pip install -r requirements

Currently, I designed this library configs using JSON file, so you need to input configs.json path to agent object as a parameter.

JSON sample configs are in scripts folder, and you can apply these files to create WordSegmentAgent object. To test your spacing model, I added 1,100 sample sentences which are sampled from Korean Wiki dataset. The model will not perform well, but you will understand how this agent works.

from takos.agents.word_segment import WordSegmentAgent
spacing_agent = WordSegmentAgent('your_config.json')

Once you want to input your sentence file to spacing model, you have to configure JSON parameter "dataset" on your configs.json. Here are the variables of dataset configs.

{
  ......
    "dataset": {
        "name": "your_dataset",
        "train": {
          "vocab_min_freq": 10,
          "input": "./samples/train.txt"
        },
        "test": {
          "limit_len": 150,
          "input": "./samples/test.txt"
        }
      }
  ......
}

When you tune those configuration as you like, your model is ready to train and evaluate and run!

The train method is performed by configs file, and those are the variables you need to consider.

{
  ......
    
    "train": {
        "epochs": 100,
        "eval_steps": -1,
        "learning_rate": 3e-4,
        "eval_batch_size": 10,
        "batch_size": 64,
        "sequence_length": 50
      }
}

spacing_agent.train()

spacing_agent.eval()

''' Results
+-----------+--------+
|    Name   | Score  |
+-----------+--------+
| WER score | 0.4896 |
| SER score | 0.8200 |
|  F1 score | 0.8925 |
| ACC score | 0.8950 |
+-----------+--------+
'''

spacing_agent('학교종이땡땡땡')

''' Results

{'input': '학교종이땡땡떙',
 'label': ['B', 'I', 'I', 'E', 'B', 'I', 'E'],
 'sequence_score': array([12.990488], dtype=float32),
 'output': '학교종이 땡땡떙',
 'segment_pos': [0, 4]}
 
'''

Still, this is an alpha version library. So if you have any issues while using this, feel free to contact by writing issues.