# Launching coroutine from parallel stream is causing deadlock

**URL:** <https://discuss.kotlinlang.org/t/launching-coroutine-from-parallel-stream-is-causing-deadlock/5723>\
**Category:** Libraries\
**Created:** [December 7, 2017, 5:56pm UTC](https://discuss.kotlinlang.org/t/launching-coroutine-from-parallel-stream-is-causing-deadlock/5723 "2017-12-07T17:56:37Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![darksnake](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.kotlinlang.org/darksnake/32/2479_2.png) [@darksnake](https://discuss.kotlinlang.org/u/darksnake)\
**Post date:** [December 7, 2017, 5:56pm UTC](https://discuss.kotlinlang.org/t/launching-coroutine-from-parallel-stream-is-causing-deadlock/5723/1 "2017-12-07T17:56:38Z")

</div>

I’ve got a totally unexpected deadlock in my Java/Kotlin code and finally localaized it to this example:

```kotlin
import kotlinx.coroutines.experimental.*

fun main(args: Array<String>) {
    val stream = (1..10).map{ num->
        async(CommonPool, CoroutineStart.LAZY) {
        	println("I am running $num")
    	}
    }.stream()

    stream.parallel().forEach{ deferred->
        runBlocking { 
            deferred.await() 
        }
    }
}

```

Just tested it in kotlin online and it hangs. If I remove `parallel()`, it works as expected.

My real much more complicated and it is not that simple to exclude this parallel processing without damaging Java part of the program. The problem is possibly is appearing because kotlin blocks all of the threads in the common pool, So it is probably could not be helped. But maybe there is a way around that.

**UPDATE** The problem seems to arise only for lazy coroutines. Removing `CoroutineStart.LAZY` also solves the problem.

---

<div class="post-metadata">

**Author:** ![darksnake](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.kotlinlang.org/darksnake/32/2479_2.png) [@darksnake](https://discuss.kotlinlang.org/u/darksnake)\
**Post date:** [December 7, 2017, 7:14pm UTC](https://discuss.kotlinlang.org/t/launching-coroutine-from-parallel-stream-is-causing-deadlock/5723/2 "2017-12-07T19:14:10Z")

</div>

Found a workaround by creating deferred with `CoroutineStart.DEFAULT` just when it should be started instead of starting previously created lazy deferred. It seems to be a bug though. Also I found that problem could be caused not only by parallel stream, but also by calling coroutine from strange places like groovy script.

---

<div class="post-metadata">

**Author:** ![nanodeath](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.kotlinlang.org/nanodeath/32/1803_2.png) [@nanodeath](https://discuss.kotlinlang.org/u/nanodeath)\
**Post date:** [December 7, 2017, 9:34pm UTC](https://discuss.kotlinlang.org/t/launching-coroutine-from-parallel-stream-is-causing-deadlock/5723/3 "2017-12-07T21:34:27Z")

</div>

Plus running `runBlocking` inside of a for-loop instead of outside seems…unconventional, at best.

---

<div class="post-metadata">

**Author:** ![darksnake](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.kotlinlang.org/darksnake/32/2479_2.png) [@darksnake](https://discuss.kotlinlang.org/u/darksnake)\
**Post date:** [December 8, 2017, 6:31am UTC](https://discuss.kotlinlang.org/t/launching-coroutine-from-parallel-stream-is-causing-deadlock/5723/4 "2017-12-08T06:31:34Z")

</div>

It could not be avoided since coroutine part is originally called from some Java/Groovy code. In order to move `runBlocking` to the top level, one needs to rewrite everything in kotlin.

---

<div class="post-metadata">

**Author:** ![darksnake](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.kotlinlang.org/darksnake/32/2479_2.png) [@darksnake](https://discuss.kotlinlang.org/u/darksnake)\
**Post date:** [December 8, 2017, 12:02pm UTC](https://discuss.kotlinlang.org/t/launching-coroutine-from-parallel-stream-is-causing-deadlock/5723/5 "2017-12-08T12:02:13Z")

</div>

Filed [an issue](https://youtrack.jetbrains.com/issue/KT-21717). I hope it will be fixed. My workaround works, but it forces me to completely avoid lazy coroutines and I really like them.

---

<div class="post-metadata">

**Author:** ![elizarov](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.kotlinlang.org/elizarov/32/2164_2.png) [@elizarov](https://discuss.kotlinlang.org/u/elizarov)\
**Post date:** [December 8, 2017, 6:16pm UTC](https://discuss.kotlinlang.org/t/launching-coroutine-from-parallel-stream-is-causing-deadlock/5723/6 "2017-12-08T18:16:15Z")

</div>

This is not a bug. This code creates a genuine deadlock. This code does `runBlocking` inside `stream.parallel()` which dispatches to `CommonPool` (by default), so all the threads in common pool are blocked, but, at the same time, coroutines in this code are dispatched to the the same `CommonPool`, which is blocked by `runBlocking { deferred.await() }`, so they cannot execute. As a general rule, the code that runs inside parallel streams should never block (`runBlocking` should not be used there). If this cannot be avoided, then coroutines shall be dispatched to a separate thread pool.

---

<div class="post-metadata">

**Author:** ![darksnake](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.kotlinlang.org/darksnake/32/2479_2.png) [@darksnake](https://discuss.kotlinlang.org/u/darksnake)\
**Post date:** [December 8, 2017, 6:33pm UTC](https://discuss.kotlinlang.org/t/launching-coroutine-from-parallel-stream-is-causing-deadlock/5723/7 "2017-12-08T18:33:12Z")

</div>

I figured that much. The problem is that I do not have parallel stream in my real code. I have a call from groovy and it still causes deadlock which is very hard to debug. I did not found what actually causes the dead lock.  
The strange thing is that everything works fine for default coroutine and does not work for a lazy one.

I just tried to debug my code and `runBlocking` is called from main thread and locks from the fist call if debugger is correct.

---

<div class="post-metadata">

**Author:** ![darksnake](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.kotlinlang.org/darksnake/32/2479_2.png) [@darksnake](https://discuss.kotlinlang.org/u/darksnake)\
**Post date:** [December 8, 2017, 7:08pm UTC](https://discuss.kotlinlang.org/t/launching-coroutine-from-parallel-stream-is-causing-deadlock/5723/8 "2017-12-08T19:08:44Z")

</div>

OK, I run some tests and actually tried to move `runBlocking` to the top ensuring it is run only once for the whole executrion. The deadlock still occurs. So it is not connected to the problem I’ve described at the topic. I will try to localize it, but it is not that simple.

---

<div class="post-metadata">

**Author:** ![elizarov](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.kotlinlang.org/elizarov/32/2164_2.png) [@elizarov](https://discuss.kotlinlang.org/u/elizarov)\
**Post date:** [December 8, 2017, 9:38pm UTC](https://discuss.kotlinlang.org/t/launching-coroutine-from-parallel-stream-is-causing-deadlock/5723/9 "2017-12-08T21:38:56Z")

</div>

Parallel stream blocks CommonPool completely. It is so by design. You cannot combine parallel stream and coroutines running in CommonPool. You should dispatch your coroutines to another thread pool.

---

<div class="post-metadata">

**Author:** ![darksnake](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.kotlinlang.org/darksnake/32/2479_2.png) [@darksnake](https://discuss.kotlinlang.org/u/darksnake)\
**Post date:** [December 9, 2017, 5:48am UTC](https://discuss.kotlinlang.org/t/launching-coroutine-from-parallel-stream-is-causing-deadlock/5723/10 "2017-12-09T05:48:54Z")

</div>

I just made sure that I never use parallel stream in this code and switched default `CommonPool` to custom fixed thread pool. The deadlock still appears. And it does not explain, why only lazy coroutines are locking.

I will search further.

---

<div class="post-metadata">

**Author:** ![elizarov](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.kotlinlang.org/elizarov/32/2164_2.png) [@elizarov](https://discuss.kotlinlang.org/u/elizarov)\
**Post date:** [December 9, 2017, 6:03am UTC](https://discuss.kotlinlang.org/t/launching-coroutine-from-parallel-stream-is-causing-deadlock/5723/11 "2017-12-09T06:03:41Z")

</div>

Can you reproduce it with self-contained code?

---

<div class="post-metadata">

**Author:** ![darksnake](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.kotlinlang.org/darksnake/32/2479_2.png) [@darksnake](https://discuss.kotlinlang.org/u/darksnake)\
**Post date:** [December 9, 2017, 6:08am UTC](https://discuss.kotlinlang.org/t/launching-coroutine-from-parallel-stream-is-causing-deadlock/5723/12 "2017-12-09T06:08:07Z")

</div>

I am currently trying to do just that. It works fine with one hand-made coroutine, but does not work in a framework. The framework has internal lazy computation dependencies (I was using `CompletableFuture` to manage them before) and I think it could be somehow the source of the problem, but for now I still can’t localize it.  
I will get back to you as soon as I find something.

---

<div class="post-metadata">

**Author:** ![darksnake](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.kotlinlang.org/darksnake/32/2479_2.png) [@darksnake](https://discuss.kotlinlang.org/u/darksnake)\
**Post date:** [December 9, 2017, 7:02pm UTC](https://discuss.kotlinlang.org/t/launching-coroutine-from-parallel-stream-is-causing-deadlock/5723/13 "2017-12-09T19:02:27Z")

</div>

I’ve spent almost two days trying to find the problem. For now I have two very similar calls to the same coroutine and everything works fine in one case and hangs in the other. I’ve eliminated any stream (or at least parallel streams in the process and ensured that actual call occurs from the main thread).

The only difference I see when debugging is that field `_state` equals `Empty{New}` in working case and `List{New}[InvokeOnCompletion[kotlinx.coroutines.experimental.future.FutureKt$asCompletableFuture$2@7671cb68]]` in hanging case. Searching through my code I found that indeed there is second case there is a `CompletableFuture` that depends on coroutine in question. What I don’t understand is why it affects the coroutine. In my case this dependent future should never be called anyway because it also depends on some coroutines which are never started (using `CompletableFuture.allOf()`).

Tomorrow I will try to buuild a standalone example, but I hardly can imagine this to be intended behavior.

---

<div class="post-metadata">

**Author:** ![elizarov](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.kotlinlang.org/elizarov/32/2164_2.png) [@elizarov](https://discuss.kotlinlang.org/u/elizarov)\
**Post date:** [December 9, 2017, 9:15pm UTC](https://discuss.kotlinlang.org/t/launching-coroutine-from-parallel-stream-is-causing-deadlock/5723/14 "2017-12-09T21:15:19Z")

</div>

Thanks. Please, make sure you are using the latest version of coroutines library (current 0.20). There was a bug in coroutine state-management that was fixed recently and it did affect coroutines in `List{New}` state.

---

<div class="post-metadata">

**Author:** ![darksnake](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.kotlinlang.org/darksnake/32/2479_2.png) [@darksnake](https://discuss.kotlinlang.org/u/darksnake)\
**Post date:** [December 10, 2017, 7:35am UTC](https://discuss.kotlinlang.org/t/launching-coroutine-from-parallel-stream-is-causing-deadlock/5723/15 "2017-12-10T07:35:33Z")

</div>

I wish I knew about it sooner. Seems to be working fine on 0.20.