python - k-fold Cross Validation for determining k in k-means? -



python - k-fold Cross Validation for determining k in k-means? -

in document clustering process, info pre-processing step, first applied singular vector decomposition obtain u, s , vt , choosing suitable number of eigen values truncated vt, gives me document-document correlation read here. performing clustering on columns of matrix vt cluster similar documents , chose k-means , initial results looked acceptable me (with k = 10 clusters) wanted dig bit deeper on choosing k value itself. determine number of clusters k in k-means, suggested @ cross-validation.

before implementing wanted figure out if there built-in way accomplish using numpy or scipy. currently, way performing kmeans utilize function scipy.

import numpy, scipy # preprocess info , compute svd u, s, vt = svd(a) # tfidf representation of original term-document matrix # obtain document-document correlations vt # 50 threshold obtained after examining scree plot of s docvectors = numpy.transpose(self.vt[0:50, 0:]) # prepare info run k-means whitened = whiten(docvectors) res, idx = kmeans2(whitened, 10, iter=20)

assuming methodology right far (please right me if missing step), @ stage, standard way of using output perform cross-validation? reference/implementations/suggestions on how applied k-means appreciated.

to run k-fold cross validation, you'd need measure of quality optimize for. either classification measure such accuracy or f1, or specialized 1 such v-measure.

even clustering quality measures know of need labeled dataset ("ground truth") work; difference classification need part of info labeled evaluation, while k-means algorithm can create utilize info determine centroids , clusters.

v-measure , several other scores implemented in scikit-learn, generic cross validation code , "grid search" module optimizes according specified measure of evaluation using k-fold cv. disclaimer: i'm involved in scikit-learn development, though didn't write of code mentioned.

python statistics numpy nlp machine-learning

Comments

Post a Comment

Popular posts from this blog

iphone - Dismissing a UIAlertView -

iphone - Dismissing a UIAlertView - i set in app purchase app, , when user taps button, purchase started. basically, tap button, , depending on speed on net connection, waiting 10 seconds until new alert view comes asking if purchase product. user tap button multiple times since nil came up, , multiple purchase alert views come up. additionally, maybe seen user app bug. in end, problem. i want alert view come spinning wheel says "loading..." when users taps purchase button. problem is, how dismiss when new alert view comes asking user if want purchase product? if ([uialertview alloc] says: @"whatever apple's alert view says") { //dismiss "loading..." alert view here } i uncertainty work, input appreciated. thanks! you need have access alertview. can this. create alertview instance var in app delegate , when want show loading initialize instance var assign property , when want dismiss phone call [alertviewinstance d...
Read more

intellij idea - Update external libraries with intelij and java -

intellij idea - Update external libraries with intelij and java - simple plenty really. i'm using apache jars, i'll create changes jar's every often. these jars listed external libs in intelij (i.e classpath looking install dir of jar's). when create changes intelij doesn't seem know new implementation. have remove jar external library , re-ad it. does know have intelij picks changes automatically ? i've done clean , rebuild project had little effect. you can go preferences -> build, execution, deployment -> build tools -> maven -> importing , check box says import maven projects automatically . in mac, can command + shift + a , come in action reimport , click on reimport maven projects . intellij-idea
Read more

javascript - send data from a new window to previous window in php -

javascript - send data from a new window to previous window in php - i have image in main.php file : <img src="..\images\image.gif" class="cur" onclick="imgwin();"> when click on image below function has called : function imgwin() { imagewin=window.open("../pages/img.php","imagewin","toolbar=no,width=500,height=200,directories=no,menubar=no,scrollbars=yes"); } this function opens img.php in new window img.php file : <html> <head> <title>upload image</title> </head> <body> <form name="imgform" method="get" action="#"> address : <input type="file" name="imgurl" size="50"> description : <input type="text" name="description" size="50"> <input type="submit" value="sumbit...
Read more