Streaming a JSON list in akka/pekko(-http)?

What do you mean?

Some time ago, I had a problem: given a list of objects in scala, and with circe as a JSON library (but it’s the same everywhere), how can I make a streaming HTTP response so that nothing explodes and I can save everything as a compliant JSON list? You can’t simply take a list and spit it out, it won’t be recognized as a valid JSON by any parser.

Now it’s been too much time to remember all the approaches and the problems I encountered while to solve this, but I guess it can be useful to recollect what I did, both for me so that I can’t go find the solution again, and who knows, even other people.

Ok, but how did you solve it then?

To be honest, it was fairly simple: you need a Publisher[T] and some Source (akka.stream.scaladsl) magic:

    val beginning = Source.single(ByteString("["))
    val end = Source.single(ByteString("]"))

    val middle = Source
      .fromPublisher(publisher)
      .map { il =>
        ByteString(il.asJson)
      }
      .intersperse(ByteString(","))

    Source.combine(beginning, middle, end)(Concat(_))

The secret here is, of course, to concat the three sources in order, with [ and [ and a nice .intersperse to put the comma at the right places. You’ll get, then, a nice Source output that you can plug wherever you like.