response body is compressed in the retry function I provided

[stefanluptak]

I am doing this

Req.request(base_request(), url: "some url", method: :get)

defp base_request do
  Req.new(retry: &maybe_retry/2, retry_delay: &retry_delay/1, max_retries: 10)
end

and inside my maybe_retry/2 function, when I try to access the response.body, the content is compressed.

After I do this below, it works.

defp maybe_retry(request, %Req.Response{status: 403} = response) do
  {_request, %Req.Response{body: decompressed_body}} = Req.Steps.decompress_body({request, response})

  decoded_body = Jason.decode!(decompressed_body)

  match?(%{"message" => "You have exceeded a secondary rate limit" <> _}, decoded_body)
end

I assumed that the body will be already decompressed as it is in the result of the Req.new() call. Am I mistaken or is this a bug?

[wojtekmach]

The reason it works like this is retry is the very first response step, https://github.com/wojtekmach/req/blob/v0.5.6/lib/req/steps.ex#L85, so it happens before we decompress and decode. I'll need to think some more if we can change the order.

[stefanluptak]

I think just mentioning this in the docs could be enough.

[gmile]

I just stumbled on another case related to decompression: errors raised for uncompressed responses look cryptic. Here's what I am doing:

• I opt-in to http_errors: :raise in most cases in my back-end code,

• I make an HTTP call to a service on Google Cloud, which returns a response with 409 status,

• the exception looks like this:

  ** (RuntimeError) The requested URL returned error: 404
  Response body: <<31, 139, 8, 0, 0, 0, 0, 0, 2, 255, 69, 142, 193, 10, 194, 48, 16, 68, 239, 253, 138, 37, 103, 49, 181, 141, 82, 60, 171, 199, 246, 162, 183, 66, 9, 237, 170, 145, 52, 145, 236, 22, 133, 210, 127, 55, 173, 130, 199, 247, ...>>
      (req 0.5.7) lib/req/steps.ex:1927: Req.Steps.handle_http_errors/1
      (req 0.5.7) lib/req/request.ex:1120: anonymous fn/2 in Req.Request.run_response/2
      (elixir 1.17.2) lib/enum.ex:4858: Enumerable.List.reduce/3
      (elixir 1.17.2) lib/enum.ex:2585: Enum.reduce_while/3
      (req 0.5.7) lib/req/request.ex:1047: Req.Request.run/1
      (req 0.5.7) lib/req.ex:1100: Req.request!/2
      (api 1.0.0) lib/my_app/my_module.ex:112: MyApp.MyModule.delete_user_data/1
      iex:1: (file)
  

I understand the error happens due to how steps are ordered here. And am able to work around it simply by setting decompression as the first step, like this:

Req.new()
|> Req.Request.prepend_response_steps(decompress: &Req.Steps.decompress_body/1)

It would be nice to have this mentioned in the docs somewhere (for example a note for http_errors: :raise option), or have ability to specify that decompression (any other steps, like decode?) right before an exception is being raised.

(my problem seems only mildly related to the topic here, but I wasn't sure if it warrants a GH issue of its own)