amazon s3 - Logstash S3 output plugin codec issue -

-

i experiencing strange behavior logstash, using combination of codecs on input/file , output/s3. there seems issue output/s3 logstash plugin, cannot part files upload s3 unless specify codec in output/s3 plugin.

i tailing java application log files, ideally using input/file plugin watch log files in directory , make sure stack traces encountered (and new lines) wrapped same logstash event. this:

input {   file {     path => "c:/some/directory/logs/*"           codec => multiline {         pattern => "^%{datestamp}"         negate => true         => "previous"     }   } }

this append stack traces parent events. want perform 2 different output/s3 operations (essentially recreating raw log line line, , uploading event json) :

output {      s3{      access_key_id => "mykey"      secret_access_key => "myseckey"      region => "us-east-1"      bucket => "somebucket"      size_file => 10000      upload_workers_count => 2      restore => true      prefix => "rawlogs/"      temporary_directory => "c:/temp/logstash/raw"      time_file => 5    }  s3{      access_key_id => "mykey"      secret_access_key => "myseckey"      region => "us-east-1"      bucket => "somebucket"      size_file => 10000      upload_workers_count => 2      restore => true      prefix => "jsoneventlogs/"      temporary_directory => "c:/temp/logstash/json"      time_file => 5      codec => "json_lines"  } }

the s3 upload uses "json_lines" codec works fine, raw log upload uses default "plain" codec not work @ all. files sit in temp directory , never pushed s3. have tried use "line" codec, still same behavior. if remove "multiline" codec input/file plugin , use in output/s3 raw plugin, upload s3 fine, each newline in stacktrace coming in own event, codec not seem job.

any idea why output/s3 plugin seems work json_lines , multiline?


Comments

Post a Comment

Popular posts from this blog

python - No exponential form of the z-axis in matplotlib-3D-plots -

-
Image
i have similar problem described in how prevent numbers being changed exponential form in python matplotlib figure : i don't want (in special case) weird scientific formatting of axis. problem different have problem @ z -axis. 2-d plots can use ax.get_yaxis().get_major_formatter().set_useoffset(false) . , there no function ax.get_zaxis() what use format z -axis same way? edit: example: from mpl_toolkits.mplot3d import axes3d import numpy np import sys import matplotlib import matplotlib.pyplot pyplot def func(xi, ti): res = 10e3 + np.cos(ti) * np.sin(xi) return res if __name__ == '__main__': timespacing = 20 timestart = 0 timeend = 1 time = np.linspace(timestart, timeend, timespacing) widthspacing = 50 widthstart = 0 widthend = 3 width = np.linspace(widthstart, widthend, widthspacing) reslist = [] matplotlib.rcparams['legend.fontsize'] = 10 fig = pyplot.figure() ax = fig.gca(projection = '3d...
Read more

php - Best Light server (Linux + Web server + Database) for Raspberry Pi -

-
i install web server database on raspberry pi (little computer). computer has 1gb ram. i want know best combination: linux distribution , web server , dbms run local server multiple users minimal latency, use php on server. , best settings performance , not have bugs (memory usage, disable plugin, disable service, etc)? i thought light debian , lighttpd server , sqlite database. solution? i think lighttpd + sqlite great choice. linux distro, debian @ centos or tiny core linux , although i'm not sure of compatibility pi. obviously, can't go wrong raspbian if want use in production , more stable performance, few more pi's , set them in cluster .
Read more

c# - "Newtonsoft.Json.JsonSerializationException unable to find constructor to use for types" error when deserializing class -

-
i developing app in xamarin android. have splash screen serialize class , pass mainactivity using intent . when try deserialize in mainactivity error message : "serializationexception unable find constructor use types" serialization : void loaddata() { currency currency = new currency(this); intent = new intent(this, typeof(mainactivity)); intent.putextra("currency", newtonsoft.json.jsonconvert.serializeobject(currency)); startactivity(intent); } deserialization : cur = newtonsoft.json.jsonconvert.deserializeobject<currency> (intent.getstringextra("currency")); // error here class constructor : public currency(context context) { thiscontext = context; } i began experiencing problem after added parameter context context constructor. has caused this? how can solve it? possible serialize/deserialize class has parameters? you need have empty constructor in currency class in order deserialize it. ...
Read more