Question : python nltk tokenize
Answered by : tame-trout-t4sj3d17cx7b
>>> import nltk
>>> sentence = """At eight o'clock on Thursday morning
... Arthur didn't feel very good."""
>>> tokens = nltk.word_tokenize(sentence)
>>> tokens
['At', 'eight', "o'clock", 'on', 'Thursday', 'morning',
'Arthur', 'did', "n't", 'feel', 'very', 'good', '.']Source : https://stackoverflow.com/questions/15057945/how-do-i-tokenize-a-string-sentence-in-nltk | Last Update : Fri, 05 Nov 21
Question : import word_tokenize
Answered by : itchy-impala-bq5sjjonzyz7
import nltk
from nltk import word_tokenizeSource : | Last Update : Mon, 13 Jul 20
Question : nltk python how to tokenize text
Answered by : jedadiah-mcfarland
>>> tokens = word_tokenize(raw)
>>> type(tokens)
<class 'list'>
>>> len(tokens)
254354
>>> tokens[:10]
['The', 'Project', 'Gutenberg', 'EBook', 'of', 'Crime', 'and', 'Punishment', ',', 'by']Source : https://www.nltk.org/book/ch03.html | Last Update : Mon, 28 Feb 22