java - Unresponsive actor system: ThreadPoolExecutor dispatcher only creates core thread pool, apparently ignores max thread pool size -

update: i've found program remains responsive if set threadpoolexecutor's core pool size same max pool size (29 threads). however, if set core pool size 11 , max pool size 29 actor system ever creates 11 threads. how can configure actorsystem / threadpoolexecutor continue create threads exceed core thread count , stay within max thread count? prefer not set core thread count max thread count, need threads job cancellation (which should rare event).

---

i have batch program running against oracle database, implemented using java/akka typed actors following actors:

1. batchmanager in charge of talking rest controller. manages queue of uninitialized batch jobs; when uninitialized batch job polled queue turned jobmanager actor , executed.
2. jobmanager maintains queue of stored procedures , pool of workers; initializes each worker stored procedure, , when worker finishes sends procedure's result jobmanager, , jobmanager sends stored procedure worker. batch terminates when job queue empty , workers idle, @ point jobmanager reports results batchmanager, shuts down workers (via typedactor.context().stop()), , shuts down. jobmanager has promise<status> completion completed when job finishes or else when job terminated due cancellation or fatal exception.
3. worker executes stored procedure. creates oracleconnection , callablestatement used execute stored procedures, , registers onfailure callback jobmanager.completion abort connection , cancel statement. callback doesn't use actor system's execution context, instead uses execution context created cached executor service created in batchmanager.

my config is

{"akka" : { "actor" : { "default-dispatcher" : {     "type" : "dispatcher",     "executor" : "default-executor",     "throughput" : "1",     "default-executor" : { "fallback" : "thread-pool-executor" }     "thread-pool-executor" : {         "keep-alive-time" : "60s",         "core-pool-size-min" : coreactorcount,         "core-pool-size-max" : coreactorcount,         "max-pool-size-min" : maxactorcount,         "max-pool-size-max" : maxactorcount,         "task-queue-size" : "-1",         "task-queue-type" : "linked",         "allow-core-timeout" : "on" }}}}} 

the number of workers configured elsewhere, workercount = 8; coreactorcount workercount + 3 while maxactorcount workercount * 3 + 5. i'm testing on macbook pro 10 2 cores , 8gb of memory; production server considerably larger. database i'm talking behind extremely slow vpn. i'm running of using oracle's javase 1.8 jvm. local server tomcat 7. oracle jdbc drivers version 10.2 (i might able convince powers use newer version). methods either return void or future<> , ought non-blocking.

when 1 batch terminates there no issue - next batch starts full complement of workers. however, if terminate current batch via jobmanager#completion.tryfailure(new cancellationexception("batch cancelled")) onfailure callbacks registered workers fire off, , system becomes unresponsive. debug printlns indicate new batch starts 3 out of 8 functioning workers, , batchmanager becomes unresponsive (i added future<string> ping command returns futures.successful("ping") , times out). onfailure callbacks executing on separate thread pool, , if on actor system's thread pool should have high enough max-pool-size accommodate original jobmanager, workers, onfailure callbacks, , second jobmanager , workers. instead seem accommodating original jobmanager , workers, new jobmanager , less half of workers, , nothing left on batchmanager. computer i'm running on short on resources, seems ought able run more dozen threads.

is configuration issue? due jvm-imposed limit and/or tomcat-imposed limit? due problem how i'm handling blocking io? there several other things doing wrong, these came mind.

gist of cancellablestatement callablestatement , oracleconnection cancelled

gist of immutable cancellablestatements created

gist of jobmanager's cleanup code

config dump obtained via system.out.println(mergedconfig.tostring());

---

edit: believe i've narrowed down problem actor system (either configuration or interaction blocking database calls). eliminated worker actors , moved workload runnables execute on fixed-size threadpoolexecutor, each jobmanager creates own threadpoolexecutor , shuts down when batch completes (shutdown on normal termination, shutdownnow on exceptional termination). cancellation runs on cached thread pool instantiated in batchmanager. actor system's dispatcher still threadpoolexecutor half dozen threads allocated it. using alternate setup, cancellation executes expected - workers terminate when database connections aborted, , new jobmanager executes full complement of worker threads. indicates me not hardware/jvm/tomcat issue.

---

update: did thread dump using eclipse's memory analyzer. discovered cancellation threads hanging on callablestatement.close(), reordered cancellation oracleconnection.abort() preceded callablestatement.cancel() , cleared problem - cancellations (apparently) executed correctly. worker threads continued hang on statements, though - suspect vpn may partially or totally blame this.

performanceasync-akka.actor.default-dispatcher-19   @ java.net.socketinputstream.socketread0(ljava/io/filedescriptor;[biii)i (native method)   @ java.net.socketinputstream.read([biii)i (socketinputstream.java:150)   @ java.net.socketinputstream.read([bii)i (socketinputstream.java:121)   @ oracle.net.ns.packet.receive()v (unknown source)   @ oracle.net.ns.datapacket.receive()v (unknown source)   @ oracle.net.ns.netinputstream.getnextpacket()v (unknown source)   @ oracle.net.ns.netinputstream.read([bii)i (unknown source)   @ oracle.net.ns.netinputstream.read([b)i (unknown source)   @ oracle.net.ns.netinputstream.read()i (unknown source)   @ oracle.jdbc.driver.t4cmarengine.unmarshalub1()s (t4cmarengine.java:1109)   @ oracle.jdbc.driver.t4cmarengine.unmarshalsb1()b (t4cmarengine.java:1080)   @ oracle.jdbc.driver.t4c8oall.receive()v (t4c8oall.java:485)   @ oracle.jdbc.driver.t4ccallablestatement.dooall8(zzzz)v (t4ccallablestatement.java:218)   @ oracle.jdbc.driver.t4ccallablestatement.executeforrows(z)v (t4ccallablestatement.java:971)   @ oracle.jdbc.driver.oraclestatement.doexecutewithtimeout()v (oraclestatement.java:1192)   @ oracle.jdbc.driver.oraclepreparedstatement.executeinternal()i (oraclepreparedstatement.java:3415)   @ oracle.jdbc.driver.oraclepreparedstatement.execute()z (oraclepreparedstatement.java:3521)   @ oracle.jdbc.driver.oraclecallablestatement.execute()z (oraclecallablestatement.java:4612)   @ com.util.cpprocexecutor.execute(loracle/jdbc/oracleconnection;ljava/sql/callablestatement;lcom/controller/basejobrequest;)v (cpprocexecutor.java:57) 

however, after fixing cancellation order still have problem actor system isn't creating enough threads: i'm still getting 3 out of 8 workers in new batch, new workers being added cancelled workers have network connections time out. in total i've got 11 threads - core pool size, out of 29 threads - max pool size. apparently actor system ignoring max pool size parameter, or i'm not configuring max pool size correctly.

(disclaimer: don't know akka)

by below configuration of queue-size=-1, guess, task queue unbounded.

  "task-queue-size": "-1",   "task-queue-type": "linked" 

threadpoolexecutor not spawn beyond core pool size unless work queue full , not able queue. if task queue full, start spawning upto max threads.

if fewer corepoolsize threads running, executor prefers adding new thread rather queuing. if corepoolsize or more threads running, executor prefers queuing request rather adding new thread. if request cannot queued, new thread created unless exceed maximumpoolsize, in case, task rejected.

please check if can fix limited queue size , see if threads increasing max threads. thanks.