oracle - Taking too long to insert records in Elasticsearch in c# -

-

the following code pulls data 2 tables table1 , table2, performs join on them, on field 3 , indexes elasticsearch. total number or rows need indexing around 500 million. code inserts 5 million records in 1 hour, way take 100 hours complete. there way can make faster?

        public static void selection()         {             uri node = new uri("http://localhost:9200");             connectionsettings settings = new connectionsettings(node);             elasticclient client = new elasticclient(settings);              int batchsize = 100;             string query = "select table1.field1, table2.field2 table1 join table2 on table1.field3=table2.field3";              try             {                 oraclecommand command = new oraclecommand(query, con);                 oracledatareader reader = command.executereader();                  list<record> l = new list<record>(batchsize);                 string[] str = new string[2];                 int currentrow = 0;                  while (reader.read())                 {                     (int = 0; < 2; i++)                         str[i] = reader[i].tostring();                     l.add(new record(str[0], str[1]));                      if (++currentrow == batchsize)                     {                         commit(l, client);                         l.clear();                         currentrow = 0;                     }                 }                 commit(l, client);             }             catch(exception er)             {                 console.writeline(er.message);             }          }          public static void commit(list<record> l, elasticclient client)         {             bulkdescriptor = new bulkdescriptor();             foreach (var x in l)                 a.index<record>(op => op.object(x).index("index").type("type"));             var res = client.bulk(d => a);             console.writeline("100 records more inserted.");         }

any appreciated! :)

can try using lower level client i.e. elasticsearchclient ?

here sample example - 

//fill data in elasticdatarows  stringbuilder elasticdatarows = new stringbuilder() elasticdatarows.appendline("{ \"index\":  { \"_index\": \"testindex\", \"_type\": \"accounts\" }}"); elasticdatarows.appendline(jsonconvert.serializexmlnode(objxml, newtonsoft.json.formatting.none, true));  var node = new uri(objextsetting.selectsinglenode("settings/elasticsearchurl").innertext); var config = new connectionconfiguration(node); elasticsearchclient objelasticclient = new elasticsearchclient(config);  //insert data elasticsearch var response = extractioncontext.objelasticclient.bulk(message.elasticdatarows.tostring());

elasticsearchclient not typed nest. can convert class object data json using newtonsoft.json.

as per testing more faster nest api.

thanks, sameer


Comments

Post a Comment

Popular posts from this blog

python - No exponential form of the z-axis in matplotlib-3D-plots -

-
Image
i have similar problem described in how prevent numbers being changed exponential form in python matplotlib figure : i don't want (in special case) weird scientific formatting of axis. problem different have problem @ z -axis. 2-d plots can use ax.get_yaxis().get_major_formatter().set_useoffset(false) . , there no function ax.get_zaxis() what use format z -axis same way? edit: example: from mpl_toolkits.mplot3d import axes3d import numpy np import sys import matplotlib import matplotlib.pyplot pyplot def func(xi, ti): res = 10e3 + np.cos(ti) * np.sin(xi) return res if __name__ == '__main__': timespacing = 20 timestart = 0 timeend = 1 time = np.linspace(timestart, timeend, timespacing) widthspacing = 50 widthstart = 0 widthend = 3 width = np.linspace(widthstart, widthend, widthspacing) reslist = [] matplotlib.rcparams['legend.fontsize'] = 10 fig = pyplot.figure() ax = fig.gca(projection = '3d...
Read more

php - Best Light server (Linux + Web server + Database) for Raspberry Pi -

-
i install web server database on raspberry pi (little computer). computer has 1gb ram. i want know best combination: linux distribution , web server , dbms run local server multiple users minimal latency, use php on server. , best settings performance , not have bugs (memory usage, disable plugin, disable service, etc)? i thought light debian , lighttpd server , sqlite database. solution? i think lighttpd + sqlite great choice. linux distro, debian @ centos or tiny core linux , although i'm not sure of compatibility pi. obviously, can't go wrong raspbian if want use in production , more stable performance, few more pi's , set them in cluster .
Read more

c# - "Newtonsoft.Json.JsonSerializationException unable to find constructor to use for types" error when deserializing class -

-
i developing app in xamarin android. have splash screen serialize class , pass mainactivity using intent . when try deserialize in mainactivity error message : "serializationexception unable find constructor use types" serialization : void loaddata() { currency currency = new currency(this); intent = new intent(this, typeof(mainactivity)); intent.putextra("currency", newtonsoft.json.jsonconvert.serializeobject(currency)); startactivity(intent); } deserialization : cur = newtonsoft.json.jsonconvert.deserializeobject<currency> (intent.getstringextra("currency")); // error here class constructor : public currency(context context) { thiscontext = context; } i began experiencing problem after added parameter context context constructor. has caused this? how can solve it? possible serialize/deserialize class has parameters? you need have empty constructor in currency class in order deserialize it. ...
Read more