python 2.7 - When I am creating a dictionary for German Language , I am facing some problems in making tokens -

-

i using sublime text editor coding.

code: 

# coding: utf-8 import nltk line = "frau präsidentin, zu recht befaßt sich das parlament regelmäßig mit der verkehrssicherheit." print nltk.word_tokenize(line.decode('utf8'))

result:

[u'frau', u'pr', u'\xe4', u'sidentin', u',', u'zu', u'recht', u'befa', u'\xdf', u't', u'sich', u'das', u'parlament', u'regelm', u'\xe4', u'\xdf', u'ig', u'mit', u'der', u'verkehrssicherheit', u'.'] [finished in 0.4s]

still tokens not correct. because breaking präsidentin sub token dont want.

according docs:

this particular tokenizer requires punkt sentence tokenization models installed.

i'm guessing need these, include german model. instructions installing these can found @ http://www.nltk.org/data.html, or models can directly downloaded here


Comments

Post a Comment

Popular posts from this blog

php - Using str_replace to translate a MySQL Table in html -

-
i have website contains pricelist of cs:go items. now, how did this: mysql db w/ prices --> select db x ( item name ) --> echo table ( works fine ) now want translation of item names. how planned it: mysql db --> $result= ( signature character ) --> $translate = str_replace (signature character $result) however, there's issue code or str_replace think. $translate returns $result no filtering. <?php require 'lang.php'; $servername = "localhost"; $username = ""; $dbname = ""; $password = ""; // create connection $conn = new mysqli($servername, $username, $password, $dbname); // check connection if ($conn->connect_error) { die("connection failed: " . $conn->connect_error); } // filter name $sql = "select name, price utf name 'k%'"; $result = $conn->query($sql); // translate $tr = str_replace('k','karambit','$result'); // ech...
Read more

SQL Server - MyCTE query based on 24 hour period (next day) -

-
i have bit of code: ;with mycte ( select *, row_number() over(partition carduser order cardtableid) newvariation cardchecker ) update mycte set status = newvariation which updates status column, want happen on 24 hour period, status starts again next day @ 1, , counts again based on carduser specified above: current data , happens: 2 aaa 1 2015-06-25 08:00:00.000 123 1 null 3 ccc 1 2015-06-25 00:00:00.000 124 1 null 4 aaa 1 2015-06-25 17:30:00.000 125 2 null 5 aaa 1 2015-06-26 17:30:00.000 125 *3* null what want happen: 2 aaa 1 2015-06-25 08:00:00.000 123 1 null 3 ccc 1 2015-06-25 00:00:00.000 124 1 null 4 aaa 1 2015-06-25 17:30:00.000 125 2 null 5 aaa 1 2015-06-26 17:30:00.000 125 *1* null im not quite sure how add above query possible point me in right direction? the main problem eventtime field contains both date , time, adding partition means status 1 based on time parameter of field thanks ...
Read more

asp.net mvc - Cannot display error message on Editor or EditorFor -

-
in mvc application, can display error messages inputs i.e. textboxfor , dropdownlist , etc. however, using same approach, cannot show validation error kendo().editor() or kendo().editorfor() . tried validationmessage beside validationmessagefor , not make sense. might problem be? view: <script type="text/javascript"> $(function () { //for using kendo validation: $("form").kendovalidator(); }); </script> //validation works textboxfor, dropdownlist, etc.: @html.labelfor(m => m.summary) @html.textboxfor(m => m.summary, new { maxlength = 50, @class = "k-textbox"}) @html.validationmessagefor(m => m.summary) //validation not work editor or editorfor: @html.labelfor(m => m.description, new { maxlength = 1000, @class = "editor-label" }) @(html.kendo().editor() //@(html.kendo().editorfor(m=>m.description) //also tried .name("description") .htmlattributes(new { @class...
Read more