PyNLPIR defines a function called open. This should be renamed so as to not clash with the standard Python function.

This will make PyNLPIR's API backwards-incompatible, so it would require a major version bump to indicate its difference with the previous versions.

When the get_key_words function is called with weights enabled, certain sentences will cause errors.
OS: Ubuntu server 14.10, but the error is probably non os-specific.

import pynlpir
pynlpir.open()
pynlpir.get_key_words('我们很好,你呢') #['我们']
pynlpir.get_key_words('我们很好,你呢', weighted=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.4/dist-packages/pynlpir/__init__.py", line 220, in get_key_words
    word, pos, weight, count = w.split('/')
ValueError: need more than 3 values to unpack

pynlpir.open lacks a license_code argument. Users are forced to use the ctypes interface if they want to specify a license. We should add a license_code keyword argument to pynlpir.open that defaults to None.

RuntimeError: NLPIR function 'NLPIR_Init' failed.

They released a new version on Janurary 29 2016.
The previous license is invalid.
I tried to update the license file, but failed to make it work.
Will you update the pynlpir these days ?

Thanks!

Best regards,
YY

Hi,

I was using the get_key_words function, just found a bug. Sometimes, the text was short, then there were no keywords, so the value "fresult" is empty, then we can't split an empty string.

add a judgement may solve this problem.

anyway, THX for your tool~

Hello.

there is something wong when i install this package.

woule you mind help me fix it ?

system: window 64bit
python 3.4.1

error info:

OSERROR: [WinError126]

I didn't find functions about New Word Indentification in the API page http://pynlpir.readthedocs.org/en/latest/api.html#pynlpir

In the NLPIR system, the functions are like
NLPIR_NWI_Start();
NLPIR_NWI_AddFile(sInputFile);
NLPIR_NWI_Complete();

How do I use these functions in python ?

Hi there,

Happy new year :)

Here is the problem I met:

After 2015-01-01, all invoke to pynlpir.open() will raise following error:

RuntimeError: NLPIR function NLPIR_Init failed.

I tried to install the latest code and then tried to install the code after replacing *.so files, the same error still been raised.

My env is CentOS 6.4 64bits. Python 2.6.

Update as solved:
Data directory should also be updated.

In docs/api.rst, ValueError is incorrectly associated with an API initialization error. This should read RuntimeError.

if the text has double-slash(//), program will throw "UnboundLocalError: local variable 'pos_entry' referenced before assignment".

An example: 转发微博 //@张明明:霸气全露

Suggestion:
change line 175 in init.py to:
token = tuple(t.rsplit('/', 1))

pynlpir.segment(text, pos_names='all') fails for certain texts when pynlpir.segment(text) doesn't, which is not an expected behaviour I think.

For example,

When text = u'其中，新增了甲卡西酮、曲马多、安钠咖等12种新类型毒品的定罪量刑数量标准，并下调了在我国危害较为严重的毒品氯胺酮的定罪量刑数量标 准。'

pynlpir.segment(text, pos_names='all') causes the following Error:

No handlers could be found for logger "pynlpir.pos_map"

TypeError Traceback (most recent call last)
in ()
----> 1 pynlpir.segment(text, pos_names='all')

/Users/kensk8er/anaconda/envs/env2/lib/python2.7/site-packages/pynlpir/init.pyc in segment(s, pos_tagging, pos_names, pos_english)
203 token = (token[0], None)
204 if pos_names is not None and token[1] is not None:
--> 205 pos_name = _get_pos_name(token[1], pos_names, pos_english)
206 token = (token[0], pos_name)
207 tokens[i] = token

/Users/kensk8er/anaconda/envs/env2/lib/python2.7/site-packages/pynlpir/init.pyc in _get_pos_name(code, name, english, delimiter)
148
149 """
--> 150 pos_name = pos_map.get_pos_name(code, name, english)
151 return delimiter.join(pos_name) if name == 'all' else pos_name
152

/Users/kensk8er/anaconda/envs/env2/lib/python2.7/site-packages/pynlpir/pos_map.pyc in get_pos_name(code, name, english)
182
183 """
--> 184 return _get_pos_name(code, name, english)

/Users/kensk8er/anaconda/envs/env2/lib/python2.7/site-packages/pynlpir/pos_map.pyc in _get_pos_name(pos_code, names, english, pos_map)
158 "look for child name for '%s'" % (pos_entry[1], pos_code))
159 sub_pos = _get_pos_name(pos_code, names, english, sub_map)
--> 160 pos = pos + sub_pos if names == 'all' else (sub_pos, )
161 name = pos if names == 'all' else pos[-1]
162 logger.debug("Part of speech name found: '%s'" % repr(name)

TypeError: can only concatenate tuple (not "NoneType") to tuple

I don't fully understand the code, but it looks like it's just the case that you need to add the check wheter sub_pos is None and do appropriate processing.

The issue was found in "Ubuntu 14.04 and OSX 10.11.04".
I've installed pynlpir by pip install pynlpir.

I install the package by pip install pynlpir, It reports pynlpir__init__.py line97.in open
raise RuntimeError("NLPIR function 'NLPRI_Init' failed.")
I installed 32-bit python 2.7.11 on 64-bit Windows 10.
Is there anything I could do to fix this?thanks for help.

I'm trying to segment a somewhat large file (~700MB), but the program exits prematurely after processing about 20% of the file, without any error message.
Here's the script I use:
cat input.zh | sudo python3 tokenizeAndPOS-zh-pynlpir.py > output.zh
And my program tokenizeAndPOS-zh-pynlpir.py is as simple as this:

import sys
import pynlpir
pynlpir.open(encoding_errors='ignore')
for line in sys.stdin:
        line = line.strip()
        words = pynlpir.segment(line, pos_names=None)
        print(' '.join(word for word, flag in words))
pynlpir.close()

I even tried locating the line where the program stopped, and feeding in the lines around, then the program worked fine. It appears that the problem is not caused by special character sequences.

for this text:
text = "电影里阿斯顿·马丁DB5最终的结局是被反派席尔瓦炸毁，令我扼腕叹息，庆幸的是，这部DB5并不是“货真价实”的真家伙，不然DB5的存世名单里又将少了一位。"
pynlpir.segment(text)

got the error: ValueError: part of speech not recognized: 'Rg'. that means: no key "Rg" in POS_MAP dictionary?

thanks!

My python version is 2.7
PyNLPIR version is 0.2.2 and I uses "pip install pynlpir" to install it.
OS is Ubuntu 12.04, 64bit
Thank you!

pynlpir.open's inline documentation is missing any note about what a ValueError means when it is raised.

The Sphinx documentation incorrectly describes the ValueError (what it describes is RuntimeError). Its 'ValueError' should be renamed 'RuntimeError' and the correct ValueError description should be added, just like in the inline documentation.

when nlpir.GetKeyWords return a empty string, get_key_words will be crash,like this.

import pynlpir
pynlpir.open()
s = '欢迎科研人员、技术工程师、企事业单位与个人参与NLPIR平台的建设工作。'
pynlpir.get_key_words(s)
[u'']
pynlpir.get_key_words(s, weighted=True)
----> 1 pynlpir.get_key_words(s, weighted=True)

/usr/lib/python2.7/site-packages/pynlpir/init.pyc in get_key_words(s, max_words, weighted)
217 weights, words = [], []
218 for w in fresult:
--> 219 word, pos, weight = w.split('/')
220 weight = _to_float(weight)
221 weights.append(weight or 0.0)
ValueError: need more than 1 value to unpack

python2.7
linux64
pynlpir 0.1.3

我想问一下有没有情感分析的模块？

Hello, i'm using mac os x, and i've just successfully installed pynlpir-0.4 on my mac,
however, when i tried the simple example in your tutorial, code is like below:

coding=utf-8

import pynlpir
pynlpir.open()
s = '欢迎科研人员、技术工程师、企事业单位与个人参与NLPIR平台的建设工作。'
pynlpir.segment(s)
pynlpir.close()

Some error message showed like below:
/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 /Users/titansonia/PycharmProjects/spider_v1.1/word_seg.py

[2015-12-28 15:45:36] Cannot open file
Cannot write log file /Library/Python/2.7/site-packages/pynlpir/Data/20151228.err!
[2015-12-28 15:45:36] /Library/Python/2.7/site-packages/pynlpir/Data/KeyBlackList.pdat
Cannot write log file /Library/Python/2.7/site-packages/pynlpir/Data/20151228.err!
[2015-12-28 15:45:36] Cannot open file /Library/Python/2.7/site-packages/pynlpir/Data/NewWord.lst
Cannot write log file /Library/Python/2.7/site-packages/pynlpir/Data/20151228.err!

Process finished with exit code 0

I don't know why, cause the installation process went on well also.
Please help me..

I installed it by downloading the zip file.

pip install from the pypi lack of nlpir library.

It run right after that.

However, without any change, I got the message today:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/lomiz/Downloads/pynlpir/pynlpir/__init__.py", line 99, in open
    raise RuntimeError("NLPIR function 'NLPIR_Init' failed.")
RuntimeError: NLPIR function 'NLPIR_Init' failed.

# And the output of data_dir, encoding_constant 
data_dir:/home/lomiz/Downloads/pynlpir/pynlpir
encoding_constant:1

There is no change in the source code.
And I check the directory,
libraries here as they were.

>>> import os
>>> data_dir = '/home/lomiz/Downloads/pynlpir/pynlpir/' 
>>> os.listdir(data_dir + './lib/')
['NLPIR64.lib', 'NLPIR32.dll', 'NLPIR32.lib', 'libNLPIR32.so', 'NLPIR64.dll', 'libNLPIR64.so']

Is there any contraints in those libraries? Like time limits?

PS:
The result I got in 2.7.10, 3.4.3, 3.5.1 with clean install.

I use nlpir.AddUserWord to add "北京师范大学" to the dictionary, the return value is 2
But when I call nlpir.ParagraphProcessA("北京师范大学",ctypes.byref(size),True)
The result is only "北京" without the last four characters.

我测试了一下，全局变量ENCODING确实改变了，但是_decode(s, encoding=ENCODING)
中传入的encoding的值仍然为原本的值，python版本为2.7

On the docs welcome page, "Chinese" is misspelled:

PyNLPIR allows you to easily segment Chinee text using NLPIR, one of the most widely-regarded Chinese text analyzers:

Thanks for your work, help me quickly focus on my work with python and NLPIR.
All works well, but when I try with user_dict, problems occurs. Actually, I am not very familiar with python ctypes, so If you can offer a full example, really appreciate.

1. nlpir.Init() should take 3 arguments, the last None for license can not be missed.
2. I add my dict file under Dict directory, and nlpir.ImportUserDict(dict_file), the content is
   双卡双待 np

3. ss = bytes("我想要一个双卡双待的手机", encoding='utf-8')
   s = c_char_p(ss)
   size = ctypes.c_int()
   result = nlpir.ParagraphProcessA(s, ctypes.byref(size), True)
   print(repr(result))
   result_t_vector = ctypes.cast(result, ctypes.POINTER(nlpir.ResultT))
   words = []
   for i in range(0, size.value):
       r = result_t_vector[i]
       word = s[r.start:r.start+r.length]
       words.append((word, r.sPOS)) 
   
   print(words)

The code can not run...

Thanks and hope for your help.

The current part of speech map in pos_map.py needs to be updated for the 2015 NLPIR release that is now bundled with PyNLPIR.

Traceback (most recent call last):
File "main.py", line 29, in
words = pynlpir.segment(text)
File "D:\Program Files\Python27\lib\site-packages\pynlpir-0.1.1-py2.7.egg\pynlpir__init__.py", line 165, in segment
_get_pos_name(token[1], pos_names, pos_english))
IndexError: tuple index out of range

PyNLPIR currently provides an API that developers can use to create software that will segment text. It would be nice to have a simple command-line interface to the segment() function.

A possible example is: pynlpir "FOOBAR" and pynlpir -i foo.txt

thanks for your work. what's the format of the user dict? I can't import it right

>>> import pynlpir
>>> pynlpir.open()
[2015-08-06 11:05:07] Cannot open file /usr/local/lib/python2.7/dist-packages/PyNLPIR-0.3.2-py2.7.egg/pynlpir/Data/NewWord.lst
Cannot write log file /usr/local/lib/python2.7/dist-packages/PyNLPIR-0.3.2-py2.7.egg/pynlpir/Data/20150806.err!

but seems it doesn't affect segment function...

Travis CI supports testing on Linux and OS X. Right now, PyNLPIR is setup to only test on Linux. We should add test support for OS X.

NLPIR's license is frequently updated. Whenever this happens, it is difficult to use PyNLPIR. A workaround is to manually download a new license file.

Instead, I propose that PyNLPIR not include a license at all. A command-line utility that's packaged with PyNLPIR can be used to download/update the license file.

I'm thinking something like pynlpir-license-update or pynlpir --update.

Hi, tsroten, another question may bother you ;-)

➜  correct_me ./wds.py 
Traceback (most recent call last):
  File "./wds.py", line 201, in <module>
    pynlpir.open()
  File "/store/user_homes/Study/MachLearn/sogou_txt/correct_me/pynlpir/pynlpir/__init__.py", line 97, in open
    raise RuntimeError("NLPIR function 'NLPIR_Init' failed.")
RuntimeError: NLPIR function 'NLPIR_Init' failed.
➜  correct_me cat pynlpir/pynlpir/Data/20150801.err 
[2015-08-01 00:23:31]   /store/user_homes/Study/MachLearn/sogou_txt/correct_me/pynlpir/pynlpir/Data/NLPIR.user Not valid license or your license expired! Please feel free to contact pipy_zhang@msn.com! 
[2015-08-01 00:35:22]   /store/user_homes/Study/MachLearn/sogou_txt/correct_me/pynlpir/pynlpir/Data/NLPIR.user Not valid license or your license expired! Please feel free to contact pipy_zhang@msn.com! 
[2015-08-01 00:37:36]   /store/user_homes/Study/MachLearn/sogou_txt/correct_me/pynlpir/pynlpir/Data/NLPIR.user Not valid license or your license expired! Please feel free to contact pipy_zhang@msn.com! 
➜  correct_me 

import pynlpir
pynlpir.open()

Traceback (most recent call last):
File "<pyshell#1>", line 1, in
pynlpir.open()
File "D:\Python27\lib\site-packages\pynlpir__init__.py", line 80, in open
raise RuntimeError("NLPIR function 'NLPIR_Init' failed.")
RuntimeError: NLPIR function 'NLPIR_Init' failed.

When I was trying to test this package with using the example you provide, I got error as following:
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-1: invalid continuation byte

I was working with the package in both windows8 and Ubuntu 15 (but in a VM). Both gave me the same error.

The performance is relative worse even than jieba.
Here is an experiment:
The string is: s='我是歌手神截图上演宫心计 邓紫棋茜拉笔笔有杀气'
Here is the result of PyNLPIR:
我, 是, 歌手, 神, 截, 图, 上演, 宫, 心计, , 邓紫棋, 茜, 拉, 笔, 笔, 有, 杀气

And here is the result of jieba:
我, 是, 歌手, 神, 截图, 上演, 宫, 心计, , 邓紫棋茜, 拉, 笔笔, 有, 杀气

As we can see, PyNLPIR divides the combination " 截图".
Is there any advanced or payed version of NLPIR that can be used under Python?

你好！很高兴你能提供支持Python3的NLPIR。你在doc中给出的例子，执行结果为：
[('欢迎', 'verb'), ('科研', 'noun'), ('人员', 'noun'), ('、', 'punctuation mark:enumeration comma'), ('技术', 'noun'), ('工程师', 'noun'), ('、', 'punctuation mark:enumeration comma'), ('企事业', 'noun'), ('单位', 'noun'), ('与', 'conjunction:coordinating conjunction'), ('个人', 'noun'), ('参与', 'verb'), ('NLPIR', 'noun'), ('平台', 'noun'), ('的', 'particle:particle 的/底'), ('建设', 'verb:noun-verb'), ('工作', 'verb:noun-verb'), ('。', 'punctuation mark:period')]
请问程序中是否可以设置词性标注集？都有哪些词性标注集可选呢？ 谢谢

If you run pynlpir.segment('E\n'), then the program is stuck -- if there's 'n'` and an English word. It is easy to get this bug, please fix this, thanks.

The following should be added to the travis config file:

sudo: false

That will allow the builds to be run much faster.

In The Rest of the NLPIR Functions section of the tutorial, the link to the NLPIR/ICTCLAS download page is outdated. It should be updated to the most recent NLPIR version.

Help.I install the newest vertion 0.4,but still RuntimeError: NLPIR function 'NLPIR_Init' failed.
Environment: Windows-64bit + Python 2.7-32bit.
Any one have an idea?

$ git clone https://github.com/tsroten/pynlpir.git
$ sudo pip install -e pynlpir
Obtaining file:///home/jingjin/pynlpir
Installing collected packages: PyNLPIR
Running setup.py develop for PyNLPIR
Successfully installed PyNLPIR

$ python setup.py test
running test
running egg_info
writing PyNLPIR.egg-info/PKG-INFO
writing top-level names to PyNLPIR.egg-info/top_level.txt
writing dependency_links to PyNLPIR.egg-info/dependency_links.txt
reading manifest file 'PyNLPIR.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching 'docs/_build'
writing manifest file 'PyNLPIR.egg-info/SOURCES.txt'
running build_ext
Traceback (most recent call last):
File "setup.py", line 66, in
test_suite='pynlpir.tests'
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib/python2.7/dist-packages/setuptools/command/test.py", line 135, in run
self.with_project_on_sys_path(self.run_tests)
File "/usr/lib/python2.7/dist-packages/setuptools/command/test.py", line 116, in with_project_on_sys_path
func()
File "/usr/lib/python2.7/dist-packages/setuptools/command/test.py", line 160, in run_tests
testLoader = cks
File "/usr/lib/python2.7/unittest/main.py", line 94, in init
self.parseArgs(argv)
File "/usr/lib/python2.7/unittest/main.py", line 149, in parseArgs
self.createTests()
File "/usr/lib/python2.7/unittest/main.py", line 158, in createTests
self.module)
File "/usr/lib/python2.7/unittest/loader.py", line 130, in loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "/usr/lib/python2.7/unittest/loader.py", line 103, in loadTestsFromName
return self.loadTestsFromModule(obj)
File "/usr/lib/python2.7/dist-packages/setuptools/command/test.py", line 36, in loadTestsFromModule
tests.append(self.loadTestsFromName(submodule))
File "/usr/lib/python2.7/unittest/loader.py", line 91, in loadTestsFromName
module = import('.'.join(parts_copy))
File "/home/jingjin/pynlpir/pynlpir/tests/test_init.py", line 8, in
pynlpir.open()
File "/home/jingjin/pynlpir/pynlpir/init.py", line 80, in open
raise RuntimeError("NLPIR function 'NLPIR_Init' failed.")
RuntimeError: NLPIR function 'NLPIR_Init' failed.

In [1]: import pynlpir

In [2]: str=u"传闻称，陈学冬情系郭敬明，两人关系是同性关系，难道这是真的？空穴来风必有因，既然传闻有了，也就有可能，不排除陈学冬就是喜欢郭敬明呢。板直一个人，还挺难的，如果宋佳真的喜欢陈学冬，就看看能不能板直他了。"

In [3]: pynlpir.segment(str)
段错误 (core dumped)
[root@namenode lucosax]#

:(...

Notice:this time,the OS is CentOS ,not my MAC,:)
Thanks!

Downloading/unpacking pynlpir
Downloading PyNLPIR-0.1.3.tar.gz (12.7MB): 12.7MB downloaded
Running setup.py (path:/private/tmp/pip_build_root/pynlpir/setup.py) egg_info for package pynlpir
Traceback (most recent call last):
File "", line 17, in
File "/private/tmp/pip_build_root/pynlpir/setup.py", line 12, in
import pynlpir
File "pynlpir/init.py", line 24, in
from . import nlpir, pos_map
File "pynlpir/nlpir.py", line 120, in
libNLPIR = load_library(sys.platform, is_64bit)
File "pynlpir/nlpir.py", line 111, in load_library
platform)
RuntimeError: Platform 'darwin' is not supported by NLPIR.
Complete output from command python setup.py egg_info:
Traceback (most recent call last):

File "", line 17, in

File "/private/tmp/pip_build_root/pynlpir/setup.py", line 12, in

import pynlpir

File "pynlpir/init.py", line 24, in

from . import nlpir, pos_map

File "pynlpir/nlpir.py", line 120, in

libNLPIR = load_library(sys.platform, is_64bit)

File "pynlpir/nlpir.py", line 111, in load_library

platform)

RuntimeError: Platform 'darwin' is not supported by NLPIR.

Cleaning up...
Command python setup.py egg_info failed with error code 1 in /private/tmp/pip_build_root/pynlpir
Storing debug log for failure in /Users/Lucosax/Library/Logs/pip.log

thanks

u'\u7ea2\u70e7\u8089'
Traceback (most recent call last):
File "C:\Users\asus\workspace\py2\src\NLP\np\�ִ�.py", line 23, in
b=pynlpir.segment(s)
File "C:\Python27\lib\site-packages\pynlpir__init__.py", line 183, in segment
pos_name = get_pos_name(token[1], pos_names, pos_english)
File "C:\Python27\lib\site-packages\pynlpir__init_.py", line 129, in _get_pos_name
pos_name = pos_map.get_pos_name(code, name, english)
File "C:\Python27\lib\site-packages\pynlpir\pos_map.py", line 183, in get_pos_name
return _get_pos_name(code, name, english)
File "C:\Python27\lib\site-packages\pynlpir\pos_map.py", line 149, in _get_pos_name
pos_code)
ValueError: part of speech not recognized: 'gms'
i do not know why that happen.
if i divide the words into three pieces there is not error but i get them into a sentence the error occur
i try to go to this page http://202.38.128.96:96/nlpir/ to segment the words there is nothing wrong
i guess maybe there are some thing wrong with the decode

It will throw an exception from init.py when import pynlpir.
Line 76: raise RuntimeError("NLPIR function 'NLPIR_Init' failed.")

The problem seems due to the NLPIR library which the package wraps. I have viewed the offical notice of nlpir, the author said the license of version 20140324 is expired, the new version will soon release, and non-commercial user could get a temporary license of three months before the new release. (the author's mail: [email protected])

The notice is in the 8th floor of this thread: http://www.bigdatabbs.com/forum.php?mod=redirect&goto=findpost&ptid=5331&pid=13882

@hunkyguy pointed out that pos_map.py is missing a comma in one of the tuple pairs.