As asked
Given a list of URLs, fetch them all in parallel with a concurrency limit of 10, collect results (including errors per URL), and return a map of url to {:ok, body} or {:error, reason}. Use Task.async_stream.
Sample answer outline
Task.async_stream/3 with max_concurrency: 10 and on_timeout: :kill_task handles parallelism and timeout. The result stream emits {:ok, result} or {:exit, reason} per task. A strong answer wraps the fetch in a try/rescue or relies on on_timeout: :kill_task and collects results with Enum.reduce or Enum.into, handling task failures gracefully rather than letting one error crash the caller. Note: on_exit is not a valid option; the correct option is on_timeout.
Reference implementation (elixir)
defmodule Fetcher do
def fetch_all(urls) do
urls
|> Task.async_stream(
fn url -> {url, HTTPoison.get(url)} end,
max_concurrency: 10,
timeout: 5_000,
on_timeout: :kill_task
)
|> Enum.reduce(%{}, fn
{:ok, {url, {:ok, %{body: body}}}}, acc -> Map.put(acc, url, {:ok, body})
{:ok, {url, {:error, reason}}}, acc -> Map.put(acc, url, {:error, reason})
{:exit, _reason}, acc -> acc # TODO: track which URL timed out
end)
end
endExpect these follow-ups
- What is the difference between Task.async_stream and Task.async + Task.await in a loop?
- How do you set a per-task timeout and a global deadline for the whole batch?