标签归档:分词

自然语言处理工具包spaCy介绍

spaCy 是一个Python自然语言处理工具包,诞生于2014年年中,号称“Industrial-Strength Natural Language Processing in Python”,是具有工业级强度的Python NLP工具包。spaCy里大量使用了 Cython 来提高相关模块的性能,这个区别于学术性质更浓的Python NLTK,因此具有了业界应用的实际价值。

安装和编译 spaCy 比较方便,在ubuntu环境下,直接用pip安装即可:

sudo apt-get install build-essential python-dev git
sudo pip install -U spacy

不过安装完毕之后,需要下载相关的模型数据,以英文模型数据为例,可以用"all"参数下载所有的数据:

sudo python -m spacy.en.download all

或者可以分别下载相关的模型和用glove训练好的词向量数据:

# 这个过程下载英文tokenizer,词性标注,句法分析,命名实体识别相关的模型
python -m spacy.en.download parser

# 这个过程下载glove训练好的词向量数据
python -m spacy.en.download glove

下载好的数据放在spacy安装目录下的data里,以我的ubuntu为例:

textminer@textminer:/usr/local/lib/python2.7/dist-packages/spacy/data$ du -sh *
776M    en-1.1.0
774M    en_glove_cc_300_1m_vectors-1.0.0

进入到英文数据模型下:

textminer@textminer:/usr/local/lib/python2.7/dist-packages/spacy/data/en-1.1.0$ du -sh *
424M    deps
8.0K    meta.json
35M ner
12M pos
84K tokenizer
300M    vocab
6.3M    wordnet

可以用如下命令检查模型数据是否安装成功:

textminer@textminer:~$ python -c "import spacy; spacy.load('en'); print('OK')"
OK

也可以用pytest进行测试:

# 首先找到spacy的安装路径:
python -c "import os; import spacy; print(os.path.dirname(spacy.__file__))"
/usr/local/lib/python2.7/dist-packages/spacy

# 再安装pytest:
sudo python -m pip install -U pytest

# 最后进行测试:
python -m pytest /usr/local/lib/python2.7/dist-packages/spacy --vectors --model --slow
============================= test session starts ==============================
platform linux2 -- Python 2.7.12, pytest-3.0.4, py-1.4.31, pluggy-0.4.0
rootdir: /usr/local/lib/python2.7/dist-packages/spacy, inifile:
collected 318 items

../../usr/local/lib/python2.7/dist-packages/spacy/tests/test_matcher.py ........
../../usr/local/lib/python2.7/dist-packages/spacy/tests/matcher/test_entity_id.py ....
../../usr/local/lib/python2.7/dist-packages/spacy/tests/matcher/test_matcher_bugfixes.py .....
......
../../usr/local/lib/python2.7/dist-packages/spacy/tests/vocab/test_vocab.py .......Xx
../../usr/local/lib/python2.7/dist-packages/spacy/tests/website/test_api.py x...............
../../usr/local/lib/python2.7/dist-packages/spacy/tests/website/test_home.py ............

============== 310 passed, 5 xfailed, 3 xpassed in 53.95 seconds ===============

现在可以快速测试一下spaCy的相关功能,我们以英文数据为例,spaCy目前主要支持英文和德文,对其他语言的支持正在陆续加入:

textminer@textminer:~$ ipython
Python 2.7.12 (default, Jul  1 2016, 15:12:24)
Type "copyright", "credits" or "license" for more information.

IPython 2.4.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python'
s own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import spacy          

# 加载英文模型数据,稍许等待
In [2]: nlp = spacy.load('en')

Word tokenize功能,spaCy 1.2版本加了中文tokenize接口,基于Jieba中文分词:

In [3]: test_doc = nlp(u"it's word tokenize test for spacy")            

In [4]: print(test_doc)
it's word tokenize test for spacy

In [5]: for token in test_doc:                                          
    print(token)
   ...:    
it
'
s
word
tokenize
test
for
spacy

英文断句:

In [6]: test_doc = nlp(u'Natural language processing (NLP) deals with the application of computational models to text or speech data. Application areas within NLP include automatic (machine) translation between languages; dialogue systems, which allow a human to interact with a machine using natural language; and information extraction, where the goal is to transform unstructured text into structured (database) representations that can be searched and browsed in flexible ways. NLP technologies are having a dramatic impact on the way people interact with computers, on the way people interact with each other through the use of language, and on the way people access the vast amount of linguistic data now in electronic form. From a scientific viewpoint, NLP involves fundamental questions of how to structure formal models (for example statistical models) of natural language phenomena, and of how to design algorithms that implement these models.')

In [7]: for sent in test_doc.sents:
    print(sent)
   ...:    
Natural language processing (NLP) deals with the application of computational models to text or speech data.
Application areas within NLP include automatic (machine) translation between languages; dialogue systems, which allow a human to interact with a machine using natural language; and information extraction, where the goal is to transform unstructured text into structured (database) representations that can be searched and browsed in flexible ways.
NLP technologies are having a dramatic impact on the way people interact with computers, on the way people interact with each other through the use of language, and on the way people access the vast amount of linguistic data now in electronic form.
From a scientific viewpoint, NLP involves fundamental questions of how to structure formal models (for example statistical models) of natural language phenomena, and of how to design algorithms that implement these models.


词干化(Lemmatize):

In [8]: test_doc = nlp(u"you are best. it is lemmatize test for spacy. I love these books")

In [9]: for token in test_doc:                                                      
    print(token, token.lemma_, token.lemma)
   ...:    
(you, u'you', 472)
(are, u'be', 488)
(best, u'good', 556)
(., u'.', 419)
(it, u'it', 473)
(is, u'be', 488)
(lemmatize, u'lemmatize', 1510296)
(test, u'test', 1351)
(for, u'for', 480)
(spacy, u'spacy', 173783)
(., u'.', 419)
(I, u'i', 570)
(love, u'love', 644)
(these, u'these', 642)
(books, u'book', 1011)

词性标注(POS Tagging):

In [10]: for token in test_doc:                                                    
    print(token, token.pos_, token.pos)
   ....:    
(you, u'PRON', 92)
(are, u'VERB', 97)
(best, u'ADJ', 82)
(., u'PUNCT', 94)
(it, u'PRON', 92)
(is, u'VERB', 97)
(lemmatize, u'ADJ', 82)
(test, u'NOUN', 89)
(for, u'ADP', 83)
(spacy, u'NOUN', 89)
(., u'PUNCT', 94)
(I, u'PRON', 92)
(love, u'VERB', 97)
(these, u'DET', 87)
(books, u'NOUN', 89)

命名实体识别(NER):

In [11]: test_doc = nlp(u"Rami Eid is studying at Stony Brook University in New York")

In [12]: for ent in test_doc.ents:
    print(ent, ent.label_, ent.label)
   ....:    
(Rami Eid, u'PERSON', 346)
(Stony Brook University, u'ORG', 349)
(New York, u'GPE', 350)

名词短语提取:

In [13]: test_doc = nlp(u'Natural language processing (NLP) deals with the application of computational models to text or speech data. Application areas within NLP include automatic (machine) translation between languages; dialogue systems, which allow a human to interact with a machine using natural language; and information extraction, where the goal is to transform unstructured text into structured (database) representations that can be searched and browsed in flexible ways. NLP technologies are having a dramatic impact on the way people interact with computers, on the way people interact with each other through the use of language, and on the way people access the vast amount of linguistic data now in electronic form. From a scientific viewpoint, NLP involves fundamental questions of how to structure formal models (for example statistical models) of natural language phenomena, and of how to design algorithms that implement these models.')


In [14]: for np in test_doc.noun_chunks:
    print(np)
   ....:    
Natural language processing
Natural language processing (NLP) deals
the application
computational models
text
speech
data
Application areas
NLP
automatic (machine) translation
languages
dialogue systems
a human
a machine
natural language
information extraction
the goal
unstructured text
structured (database) representations
flexible ways
NLP technologies
a dramatic impact
the way
people
computers
the way
people
the use
language
the way
people
the vast amount
linguistic data
electronic form
a scientific viewpoint
NLP
fundamental questions
formal models
example
natural language phenomena
algorithms
these models

基于词向量计算两个单词的相似度:

In [15]: test_doc = nlp(u"Apples and oranges are similar. Boots and hippos aren't.")

In [16]: apples = test_doc[0]

In [17]: print(apples)
Apples

In [18]: oranges = test_doc[2]

In [19]: print(oranges)
oranges

In [20]: boots = test_doc[6]

In [21]: print(boots)
Boots

In [22]: hippos = test_doc[8]

In [23]: print(hippos)
hippos

In [24]: apples.similarity(oranges)
Out[24]: 0.77809414836023805

In [25]: boots.similarity(hippos)
Out[25]: 0.038474555379008429

当然,spaCy还包括句法分析的相关功能等。另外值得关注的是 spaCy 从1.0版本起,加入了对深度学习工具的支持,例如 Tensorflow 和 Keras 等,这方面具体可以参考官方文档给出的一个对情感分析(Sentiment Analysis)模型进行分析的例子:Hooking a deep learning model into spaCy.

参考:
spaCy官方文档
Getting Started with spaCy

注:原创文章,转载请注明出处及保留链接“我爱自然语言处理”:http://www.52nlp.cn

本文链接地址:自然语言处理工具包spaCy介绍 http://www.52nlp.cn/?p=9386

中文分词入门之字标注法4

上一节主要介绍的是利用最大熵工具包来做字标注中文分词,这一节我们直奔主题,借用条件随机场工具“CRF++: Yet Another CRF toolkit”来完成字标注中文分词的全过程。

关于条件随机场(CRF)的背景知识,推荐参考阅读一些经典的文献:《条件随机场文献阅读指南》,另外再额外推荐一个tutorial:《Classical Probabilistic Models and Conditional Random Fields》, 这份关于CRF的文档分别从概率模型(NB,HMM,ME, CRF)之间的关系以及概率图模型背景来介绍条件随机场,比较清晰:

While a Hidden Markov Model is a sequential extension to the Nave Bayes Model, Conditional Random Fields can be understood as a sequential extension to the Maximum Entropy Model.

如果这些还不够过瘾,推荐课程图谱上收录的Coursera创始人之一Daphne Koller的“概率图模型公开课”,相信拿下这门课之后,对于上述概率模型,会有一种“一览众山小”的感觉。
继续阅读

初学者报到: 实现了一个最大匹配的分词算法

看了一段时间了的自然语言,不过还是很初级。

今天下载了一个分词的字典,自己用python写了一个分词的函数。

放上来,给大家踩,不吝赐教!

用的是最大匹配算法。

# -*- coding: cp936 -*-

import string

def loaddict():
filename=raw_input('Enter file name:')
f = open(filename,'r')

DICT={}
for eachLine in f:
dictStr = eachLine.decode('cp936')
strList=dictStr.split("\t",2)
DICT[strList[0]]=strList[1].split("\n",1)[0]
global DIC_MAXL
if(DIC_MAXL<len(strList[0])):
DIC_MAXL = len(strList[0])
f.close()
print("max length:")
print(DIC_MAXL)
return DICT;

def segmentation(dic):
sentence = unicode(raw_input('请输入中文句子:'),'cp936')
print sentence
length=len(sentence)
print('length:')
print length
global DIC_MAXL
if(length<DIC_MAXL):
wlen=length
else:
wlen=DIC_MAXL
testS=sentence
wordList=[]
while(len(testS)>0):
word=testS[0:wlen]
meet=False;
while((not meet )and (len(word)>0) ):
if(dic.has_key(word)):
wordList.append(word)
testS=testS[len(word):len(testS)]
meet=True;
else:
if(len(word)==1):
wordList.append(word)
testS=testS[len(word):len(testS)]
meet=True;
else:
word=word[0:len(word)-1]
return wordList

DIC_MAXL=0
dictionary=loaddict()
print DIC_MAXL
while(True):
wordl=segmentation(dictionary)
for eachChar in wordl:
print eachChar

真的很初级,大家轻踩!

也别不好意思踩,踩了我就能进步了!

多谢

Beautiful Data-统计语言模型的应用三:分词8

  对于一个包含n个字符的单词来说,利用语言模型进行分词的前提是首先枚举出所有的候选切分,而segment函数中:
  candidates = ( [first] + segment( rem ) for first, rem in splits( text ) )
的作用正是如此,它包含了递归调用,因此能枚举出所有的候选切分。那么,这个函数的时间复杂度是多少呢?一个包含n个字符的字符串有2^(n-1)种不同的分词方案(在字符之间有n-1个位置,每一个位置既可以作为单词边界也可以不作为边界),因此segment函数的时间复杂度为O(2^n),难怪之前的测试当字符串比较长时就跑不出结果了! 继续阅读

Beautiful Data-统计语言模型的应用三:分词5

  上一节我们已经解读了Pdist类中的__init__函数,这一节重点关注一下Python类中的__call__函数。首先简单的回顾一下上一节提到的一段话:缺省的情况,对于未知的单词,其概率均为1/N,但是对于每一个实例,Pdist均提供一个函数重载这个缺省值。为了避免过长 的单词拥有过高的概率,我们从概率10/N出发,对于候选单词的每一个字母都除以10。 继续阅读

Beautiful Data-统计语言模型的应用三:分词4

  对于Pw函数,这里稍微多做一点说。首先我们从一元语言模型的文件里读取单词及其计数,如果一个单词在语料库中出现,它的概率就是Count(word)/N,这里N是语料库的单词数目的规模。事实上,相对于使用完整的1千3百万单词(词型)的一元语言模型,Peter Norvig大牛对这个一元语言模型进行了简化:(a) 创建了一个更通用的词汇表,并且其中的单词是大小写不敏感(不区分)的,故“the”,”The”以及“THE”的计数是加在一起作为“the”的计数的;(b)只有由字母(letter)组合的单词才被计入其中,而对于其他包含数字或者标点的“单词”则被过滤,故“+170.002”以及“can’t”都不会被计入;(c)只列出其中最常用的1百万单词中的前1/3,也就是333333个单词。 继续阅读

Beautiful Data-统计语言模型的应用三:分词3

  本节我们进入《Beautiful Data》中分词的编码阶段,完整的程序及数据大家可以在“Natural Language Corpus Data: Beautiful Data”上下载ngrams.zip,我这里主要做一些解读。程序由python实现,无论在Linux或者Windows平台下,只要安装了相应的python版本,程序均可以通过测试,不过我所使用的是python2.6,注意,在python3.0上会有一些问题。 继续阅读

Beautiful Data-统计语言模型的应用三:分词2

  现在,我们就可以应用这个方法来进行分词了。首先定义一个函数:segment,其输入是一串没有空格的字符串,而输出则是一个单词列表,既最好的分词结果: 继续阅读