scala - Is it possible to correctly calculate SVD on IndexedRowMatrix in Spark? -

-

i've got indexedrowmatrix [m x n], contains x non-zero rows. i'm setting k = 3.

when try calculate svd on object computeu set true, dimensions of u matrix [m x n], when correct dimensions [m x k].

why happen?

i've tried converting indexedrowmatrix rowmatrix , calculating svd. result dimensions [x x k], calculates result non-zero rows (matrix dropping indices, in documentation).

is possible convert matrix, keeping rows indices?

    val csv = sc.textfile("hdfs://spark/nlp/merged_sparse.csv").cache()  // original file      val data = csv.mappartitions(lines => {         val parser = new csvparser(' ')         lines.map(line => {           parser.parseline(line)         })       }).map(line => {         matrixentry(line(0).tolong - 1, line(1).tolong - 1 , line(2).toint)        }     )      val coordinatematrix: coordinatematrix = new coordinatematrix(data)     val indexedrowmatrix: indexedrowmatrix = coordinatematrix.toindexedrowmatrix()     val rowmatrix: rowmatrix = indexedrowmatrix.torowmatrix()       val svd: singularvaluedecomposition[rowmatrix, matrix] = rowmatrix.computesvd(3, computeu = true, 1e-9)      val u: rowmatrix = svd.u // u factor rowmatrix.     val s: vector = svd.s // singular values stored in local dense vector.     val v: matrix = svd.v // v factor local dense matrix.      val indexedsvd: singularvaluedecomposition[indexedrowmatrix, matrix] = indexedrowmatrix.computesvd(3, computeu = true, 1e-9)      val indexedu: indexedrowmatrix = indexedsvd.u // u factor rowmatrix.     val indexeds: vector = indexedsvd.s // singular values stored in local dense vector.     val indexedv: matrix = indexedsvd.v // v factor local dense matrix.

it looks bug in spark mllib. if you size of row vector in indexed matrix correctly return 3 columns:

indexedu.rows.first().vector.size

i looked @ source , looks they're incorrectly copying current number of columns indexed matrix:

val u = if (computeu) {   val indexedrows = indices.zip(svd.u.rows).map { case (i, v) =>     indexedrow(i, v)   }   new indexedrowmatrix(indexedrows, nrows, ncols) //ncols incorrect here } else {   null }

looks prime candidate bugfix/pull request.


Comments

Post a Comment

Popular posts from this blog

python - No exponential form of the z-axis in matplotlib-3D-plots -

-
Image
i have similar problem described in how prevent numbers being changed exponential form in python matplotlib figure : i don't want (in special case) weird scientific formatting of axis. problem different have problem @ z -axis. 2-d plots can use ax.get_yaxis().get_major_formatter().set_useoffset(false) . , there no function ax.get_zaxis() what use format z -axis same way? edit: example: from mpl_toolkits.mplot3d import axes3d import numpy np import sys import matplotlib import matplotlib.pyplot pyplot def func(xi, ti): res = 10e3 + np.cos(ti) * np.sin(xi) return res if __name__ == '__main__': timespacing = 20 timestart = 0 timeend = 1 time = np.linspace(timestart, timeend, timespacing) widthspacing = 50 widthstart = 0 widthend = 3 width = np.linspace(widthstart, widthend, widthspacing) reslist = [] matplotlib.rcparams['legend.fontsize'] = 10 fig = pyplot.figure() ax = fig.gca(projection = '3d...
Read more

php - Best Light server (Linux + Web server + Database) for Raspberry Pi -

-
i install web server database on raspberry pi (little computer). computer has 1gb ram. i want know best combination: linux distribution , web server , dbms run local server multiple users minimal latency, use php on server. , best settings performance , not have bugs (memory usage, disable plugin, disable service, etc)? i thought light debian , lighttpd server , sqlite database. solution? i think lighttpd + sqlite great choice. linux distro, debian @ centos or tiny core linux , although i'm not sure of compatibility pi. obviously, can't go wrong raspbian if want use in production , more stable performance, few more pi's , set them in cluster .
Read more

c# - "Newtonsoft.Json.JsonSerializationException unable to find constructor to use for types" error when deserializing class -

-
i developing app in xamarin android. have splash screen serialize class , pass mainactivity using intent . when try deserialize in mainactivity error message : "serializationexception unable find constructor use types" serialization : void loaddata() { currency currency = new currency(this); intent = new intent(this, typeof(mainactivity)); intent.putextra("currency", newtonsoft.json.jsonconvert.serializeobject(currency)); startactivity(intent); } deserialization : cur = newtonsoft.json.jsonconvert.deserializeobject<currency> (intent.getstringextra("currency")); // error here class constructor : public currency(context context) { thiscontext = context; } i began experiencing problem after added parameter context context constructor. has caused this? how can solve it? possible serialize/deserialize class has parameters? you need have empty constructor in currency class in order deserialize it. ...
Read more