-
Notifications
You must be signed in to change notification settings - Fork 119
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to send request with body larger than 64Kb #309
Comments
Thank you for the report, I’ll look into it soon. If you can drop down to using Finch directly and see the problem there too it’s most likely nothing I can do in Req. |
As a workaround you can also try forcing http/1: connect_options: [protocols: [:http1]] |
Thank you, I will try this. As it doesn't seem to be related to |
I confirm that switching to HTTP/1 is working: iex(1)> Req.request(method: :post, url: "https://httpbin.org/post", body: :crypto.strong_rand_bytes(65537), connect_options: [protocols: [:http1]])
{:ok,
%Req.Response{
status: 200,
headers: %{
"access-control-allow-credentials" => ["true"],
"access-control-allow-origin" => ["*"],
"connection" => ["keep-alive"],
"content-type" => ["application/json"],
"date" => ["Tue, 20 Feb 2024 17:18:22 GMT"],
"server" => ["gunicorn/19.9.0"]
},
body: %{
"args" => %{},
"data" => "data:application/octet-stream;base64,TRUNCATE",
"files" => %{},
"form" => %{},
"headers" => %{
"Accept-Encoding" => "gzip",
"Content-Length" => "65537",
"Host" => "httpbin.org",
"User-Agent" => "req/0.4.11",
"X-Amzn-Trace-Id" => "Root=1-65d4dede-6682bc935550847d57d9f18a"
},
"json" => nil,
"origin" => "201.103.95.210",
"url" => "https://httpbin.org/post"
},
trailers: %{},
private: %{}
}} |
I don’t know enough about HTTP/2 but if you can reproduce it in Finch please open up the issue there, it feels like something that should work out of the box there. |
@wojtekmach Hey sorry to reopen the issue, I was doing some testing with Finch in order to open the isuee there but something is wrong with Req. If I'm doing this with the iex(1)> Finch.build(:post, "https://nghttp2.org/httpbin/post", [], :crypto.strong_rand_bytes(65538)) |> Finch.request(Req.Finch)
{:error,
%Mint.HTTPError{
reason: {:exceeds_window_size, :request, 65535},
module: Mint.HTTP2
}} But if I create a Finch instance with the default options provided by Finch, it works: iex(1)> {:ok, pid} = Finch.start_link(name: MyFinch, pools: %{ default: [ count: 1, size: 50 ] })
{:ok, #PID<0.635.0>}
iex(2)> Finch.build(:post, "https://nghttp2.org/httpbin/post", [], :crypto.strong_rand_bytes(65538)) |> Finch.request(MyFinch)
{:ok,
%Finch.Response{
status: 200,
body: "{\n \"args\": {}, \n \"data\": \"data:application/octet-stream;base64,TRUNCATED",
headers: [
{"date", "Tue, 20 Feb 2024 18:01:14 GMT"},
{"content-type", "application/json"},
{"content-length", "87690"},
{"access-control-allow-origin", "*"},
{"access-control-allow-credentials", "true"},
{"x-backend-header-rtt", "0.159641"},
{"strict-transport-security", "max-age=31536000"},
{"connection", "close"},
{"alt-svc", "h3=\":443\"; ma=3600, h3-29=\":443\"; ma=3600"},
{"server", "nghttpx"},
{"via", "1.1 nghttpx"},
{"x-frame-options", "SAMEORIGIN"},
{"x-xss-protection", "1; mode=block"},
{"x-content-type-options", "nosniff"}
],
trailers: []
}}
With HTTP/2 protocol only, it works: iex(1)> {:ok, pid} = Finch.start_link(name: MyFinch, pools: %{ default: [ protocols: [:http2] ] })
{:ok, #PID<0.635.0>}
iex(2)> Finch.build(:post, "https://nghttp2.org/httpbin/post", [], :crypto.strong_rand_bytes(65538)) |> Finch.request(MyFinch)
{:ok,
%Finch.Response{
status: 200,
body: "{\n \"args\": {}, \n \"data\": \"data:application/octet-stream;base64,TRUNCATED" <> ...,
headers: [
{"date", "Tue, 20 Feb 2024 18:03:36 GMT"},
{"content-type", "application/json"},
{"content-length", "87690"},
{"access-control-allow-origin", "*"},
{"access-control-allow-credentials", "true"},
{"x-backend-header-rtt", "0.170131"},
{"strict-transport-security", "max-age=31536000"},
{"connection", "close"},
{"alt-svc", "h3=\":443\"; ma=3600, h3-29=\":443\"; ma=3600"},
{"server", "nghttpx"},
{"via", "1.1 nghttpx"},
{"x-frame-options", "SAMEORIGIN"},
{"x-xss-protection", "1; mode=block"},
{"x-content-type-options", "nosniff"}
],
trailers: []
}}
It seems the default option Quote from Finch.start_link/1:
Would be an option to have an Elixir configuration to override the option passed to the start of Thank you |
Oh so if you pass connect_options: [protocols: :http2]] in Req it works? |
Actually, yes: Req.post(url: "https://nghttp2.org/httpbin/post", body: :crypto.strong_rand_bytes(65538), connect_options: [protocols: [:http2]]) |
@wojtekmach Do you think it should be documented somewhere? |
Ok, great find. Could you open an issue on Finch after all? If the http 1/2 negotiation there is broken we should address it there. |
I created the issue there. Thank you for your assistance. |
@wojtekmach should the default |
@kzemek yup, I released v0.4.13 which uses the old default: |
Hi,
I have faced an issue with the
Req
library when sending a request with a body greather than65_535
bytes. I faced the issue passing the payload to AWS Lambda and I was able to reproduce it with a Mock service on Internet. Also it seems the issue only concerns HTTP/2 adapter.The issue is related to Mint HTTP/2 option:
INITIAL_WINDOW_SIZE
where the default limit is:65,535
. It seems the only options is to pass a body as stream or iodata but the use ofExAws.Lambda
is limiting my options, I've also tried to change the window size settings but without success.FYI: I use
Req
asExAws
HTTP client adapter.Steps to reproduce
It works:
It doens't work:
Trying to change the settings and this (eventhough I feel I'm using it wrong):
Please let me know if you need additional information.
Regards,
The text was updated successfully, but these errors were encountered: