apache spark - Are cached RDDs resilient to graceful worker shutdown? -

-

i have (very) small spark cluster used 'sandpit' environment several people. occasionally, need restart worker nodes in course of maintaining cluster.

if running job working off rdd has been .cache()'d, , worker stopped gracefully (by running ./stop-slave.sh on node), happens portion of cached rdd?

the 2 scenarios can think of (assuming storage level rdd memory_only, no replication) that:

  1. the worker distributes portion of rdd across other workers;
  2. the portion of rdd held worker lost, , must recomputed.

the documentation suggests partition recomputed, it's unclear whether covers 'graceful' worker shutdown.


Comments

Post a Comment

Popular posts from this blog

python - No exponential form of the z-axis in matplotlib-3D-plots -

-
Image
i have similar problem described in how prevent numbers being changed exponential form in python matplotlib figure : i don't want (in special case) weird scientific formatting of axis. problem different have problem @ z -axis. 2-d plots can use ax.get_yaxis().get_major_formatter().set_useoffset(false) . , there no function ax.get_zaxis() what use format z -axis same way? edit: example: from mpl_toolkits.mplot3d import axes3d import numpy np import sys import matplotlib import matplotlib.pyplot pyplot def func(xi, ti): res = 10e3 + np.cos(ti) * np.sin(xi) return res if __name__ == '__main__': timespacing = 20 timestart = 0 timeend = 1 time = np.linspace(timestart, timeend, timespacing) widthspacing = 50 widthstart = 0 widthend = 3 width = np.linspace(widthstart, widthend, widthspacing) reslist = [] matplotlib.rcparams['legend.fontsize'] = 10 fig = pyplot.figure() ax = fig.gca(projection = '3d...
Read more

c# - "Newtonsoft.Json.JsonSerializationException unable to find constructor to use for types" error when deserializing class -

-
i developing app in xamarin android. have splash screen serialize class , pass mainactivity using intent . when try deserialize in mainactivity error message : "serializationexception unable find constructor use types" serialization : void loaddata() { currency currency = new currency(this); intent = new intent(this, typeof(mainactivity)); intent.putextra("currency", newtonsoft.json.jsonconvert.serializeobject(currency)); startactivity(intent); } deserialization : cur = newtonsoft.json.jsonconvert.deserializeobject<currency> (intent.getstringextra("currency")); // error here class constructor : public currency(context context) { thiscontext = context; } i began experiencing problem after added parameter context context constructor. has caused this? how can solve it? possible serialize/deserialize class has parameters? you need have empty constructor in currency class in order deserialize it. ...
Read more

Why does a .NET 4.0 program produce a system.unauthorizedAccess error on a Windows Server 2012 machine with .NET 4.5 installed? -

-
Image
i'm getting error seems occur on machines running windows server 2012 . application written .net 4.0 environment. hypothesis in .net 4.5 isn't playing nicely application. have no other ideas. the application designed create pdf files. calling multiple instances of itself. main application call exe file set of parameters indicate pdf files create. in way can call multiple exe files, speeding process. testing purposes, can disable calls outside programs changing setting create pdf files same program, more slowly. the pdf files saved onto local hard drive. user should have access local drives , administrator privileges. there no hidden folders. as main application goes call new instance of itself, crash. see picture below. either can't use exe file, or can't write local drive. i'm not sure which; if need create log will, can determine if application can't call exe file or if application can't write local drive. however, if run same application adm...
Read more