how to support chinese in http request body?

hi, I was wondering if you had a quick tip on how I could go around such an error, in particular if it would be possible to mediate this via an option.

large in this particular case means a content-length greater than 60742854 [so "Content-Length","60742854"]. I'm using lhttpc in a wrapper which retrieves data from elastic search and in some cases where the amount of data I'm retrieving exceeds this particular threshold I get this exception. Any help will be appreciated. Thanks.

More details on the exception:

{error,
{enomem,
[{lhttpc_client,read_length,4,
[{file,"src/lhttpc_client.erl"},{line,639}]},
{lhttpc_client,handle_response_body,4,
[{file,"src/lhttpc_client.erl"},{line,484}]},
{lhttpc_client,read_response,4,
[{file,"src/lhttpc_client.erl"},{line,446}]},
{lhttpc_client,execute,9,
[{file,"src/lhttpc_client.erl"},{line,169}]},
{lhttpc_client,request,9,
[{file,"src/lhttpc_client.erl"},{line,92}]}]}}

Background: Erlang Release 17 uses UTF-8 encoding of source files by default. Certain bit-sequences are not valid in UTF-8.

Some files are not UTF-8 encoded in lhttpc. lhttpc.app.src has the name Oscar Hellström encoded in ISO-8851-1 for instance.

Suggestion: Use the UNIX file tool no all files and make sure they are UTF-8 encoded.

There is self explanatory example of this behaviour below:

7> lhttpc:request("https://money.yandex.ru",head,[],infinity).
{ok,{{200,"OK"},
[{"Server","lighttpd/1.4.26"},
{"Date","Fri, 02 Sep 2011 20:29:02 GMT"},
{"Expires","Fri, 02 Sep 2011 20:29:02 GMT"},
{"Content-Type","text/html; charset=windows-1251"},
{"Cache-Control","max-age=0, proxy-revalidate"}],
<<>>}}

8> lhttpc:request("https://money.yandex.ru",get,[],infinity).
** exception exit: {{case_clause,{ok,<<"\r\n">>}},
[{lhttpc_client,read_trailers,4},
{lhttpc_client,read_chunked_body,4},
{lhttpc_client,handle_response_body,4},
{lhttpc_client,read_response,4},
{lhttpc_client,execute,9},
{lhttpc_client,request,9}]}
in function lhttpc:request/9

This is not a issue, but discussion.

lhttpc_client.erl uses {Host, Port, Ssl} tuple for lookup keep-alive sockets in lhttpc_manager.erl
So:

1. client connects to server {Host, Port, Ssl}
2. server returns response and keep connection alive
3. lhttpc keep open socket on lhttpc_manager under {Host, Port, Ssl} key
4. client makes one more request to that host {Host, Port, Ssl} and can reuse this socket

But in case when proxy enabled, there is one more component that involved in the process:

1. client connects to sever {Host, Port, Ssl} through proxy-server {PHost, PPort, PSsl}, by specifying proxy option
2. server returns response and may keep connection alive; proxy keep connection with client alive
3. lhttpc keep open socket (connected to proxy-server {PHost, PPort, PSsl}) under key {Host, Port, Ssl} !!!
4. client makes one more connection to that host {Host, Port, Ssl}, but provide OTHER proxy option. But lhttpc_manager return cached connection to proxy-server {PHost, PPort, PSsl}

My suggestion: include proxy option in the cache key, eg host {Host, Port, Ssl, ProxyUrl}

Hello dear devs.
Thanks for the wonderful lhttpc library!
Personally I like to specify my dependencies by version, to ensure that I don't get breakage in the future.

The most recent tag (1.2.6) doesn't add Content-Length to PATCH requests. I was working on a pull request when I realized this had already been solved on master.
Would you mind tagging the latest version? For example as 1.2.7?

Appreciate it!

Thank you!

Hi, using lhttpc I ran into a URL parsing issue. Since according to rfc3986 I am allowed to use '@' in query strings I did not urlencode them when sending an lhttpc request. This resulted in lhttp parsing the url wrong in the lhttpc_lib.erl:parse_url/1 function. (It interpreted the domain, the path, and parts of the query string before the '@' sign as username, rest of the query string as the host to connect to.)

I ended up using a workaround locally (but please take the following code with a grain of salt, I just started programming in erlang)

-spec parse_url(string()) -> #lhttpc_url{}.
parse_url(URL) ->
UrlParsingRegexp = "^(?:(?:(([^:\\/#\\?]+:)?(?:(?:\/\/)(?:(?:(?:([^:@\\/#\\?]+)(?:\:([^:@\\/#\\?]))?)@)?(([^:\\/#\\?\\]\[]+|\[[^\\/\\]@#?]+\])(?:\:([0-9]+))?))?)?)?((?:\/?(?:[^\\/\\?#]+\/+))(?:[^\\?#])))?(\?[^#]+)?)(#.)?",
{match, Matches} = re:run(URL, UrlParsingRegexp, [{capture, all, list}]),
{Scheme, User, Passwd, Host, Port, Path} = split_matched_url_to_parts(Matches),
#lhttpc_url{
host = string:to_lower(Host),
port = Port,
path = Path,
user = User,
password = Passwd,
is_ssl = (Scheme =:= https)
}.

split_matched_url_to_parts([_Href, _Origin, Protocol, Username, Password, _Host, HostName, Port, Path, Query | _]) ->
{get_scheme_from_parts(Protocol), Username, Password, HostName, get_port_from_parts(Protocol, Port), Path ++ Query};
split_matched_url_to_parts([_Href, _Origin, Protocol, Username, Password, _Host, HostName, Port, Path | _]) ->
{get_scheme_from_parts(Protocol), Username, Password, HostName, get_port_from_parts(Protocol, Port), Path}.

get_port_from_parts("http:", []) -> 80;
get_port_from_parts("https:", []) -> 443;
get_port_from_parts(Protocol, Port) -> list_to_integer(Port).

get_scheme_from_parts("http:") -> http;
get_scheme_from_parts("https:") -> https.

Or I need to add implementation for it and make pull request?

[Pasted from an email I got from a lhttpc user]

For certain bad input lhttpc will crash with an exception rather than returning {error, Reason} with sensible reason.

While the input might be invalid, I still think it would be nice if it was handled internally i lhttpc, especially since badarg is a pretty vague error. All of these invalid inputs are handled gracefully by httpc on the other hand.

Some examples that I have encountered:

Non ascii characters:

% Latin-1 "å"

lhttpc:request("http://www.dammsugarpåsar.nu", get, [], 1000).
** exception exit: badarg

% Ascii bell

lhttpc:request("http://www." ++ [7] ++ ".com", get, [], 1000).
** exception exit: badarg

% Bigger than byte()

lhttpc:request("http://www." ++ [256] ++ ".com", get, [], 1000).
** exception exit: badarg

% Nothing after "http://"

lhttpc:request("http://", get, [], 1000).
** exception exit: badarg

% Emptry string

lhttpc:request("", get, [], 1000).
** exception error: no function clause matching
lhttpc_lib:split_scheme([]) (src/lhttpc_lib.erl, line 109)
in function lhttpc_lib:parse_url/1 (src/lhttpc_lib.erl, line 104)
in call from lhttpc:request/6 (src/lhttpc.erl, line 177)

% Omit "http://"

lhttpc:request("www.google.com", get, [], 1000).
** exception error: no function clause matching
lhttpc_lib:split_scheme("www.google.com") (src/lhttpc_lib.erl, line 109)
in function lhttpc_lib:parse_url/1 (src/lhttpc_lib.erl, line 104)
in call from lhttpc:request/6 (src/lhttpc.erl, line 177)

% Non-existing domain name

lhttpc:request("http://www.thisurldoesntexist.com", get, [], 1000).
** exception exit: {nxdomain,
[{lhttpc_client,send_request,1,
[{file,"src/lhttpc_client.erl"},{line,175}]},
{lhttpc_client,execute,9,
[{file,"src/lhttpc_client.erl"},{line,132}]},
{lhttpc_client,request,9,
[{file,"src/lhttpc_client.erl"},{line,79}]}]}
in function lhttpc:request/9 (src/lhttpc.erl, line 323)

This would be really neat if it would be possible to connect to unix sockets :) pretty please?

is:open is:issue author:tumikosha D:\erlang_projects\paukan>rebar g-d ERROR: More than one .app.src file in d:/erlang_projects/paukan/src ==> idna (get-deps) ==> ssl_verify_hostname (get-deps) ==> hackney (get-deps) ==> jsx (get-deps) ==> craterl (get-deps) ==> paukan (get-deps) Pulling lhttpc from {git,"https://github.com/esl/lhttpc",{tag,"1.3.0"}} Cloning into 'lhttpc'... ERROR: Failed to extract name from d:/erlang_projects/paukan/deps/lhttpc/src/lhttpc.app.src: {error, {error, {27, file_io_server, invalid_unicode}}} ERROR: 'get-deps' failed while processing D:/erlang_projects/paukan: rebar_abort

Your LASTNAME have non unicode simbols. It fails REBAR dependency embeddings
%%% @author Oscar Hellstr<>m [email protected]

non-unicode character in author's name in src/lhttpc.app.src caused this error:

2 sofa% make 
./rebar compile
ERROR: Failed to extract name from /home/fisher/prj/erl/lhttpc/src/lhttpc.app.src: {27,
                                                                                    file_io_server,
                                                                                    invalid_unicode}
Makefile:13: recipe for target 'compile' failed
make: *** [compile] Error 1

after brutal change in beautiful scandinavian name it compiles as intended:

diff --git a/src/lhttpc.app.src b/src/lhttpc.app.src
index 1c3d93b..52533da 100644
--- a/src/lhttpc.app.src
+++ b/src/lhttpc.app.src
@@ -24,7 +24,7 @@
 %%% ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 %%% ----------------------------------------------------------------------------

-%%% @author Oscar Hellstr<F6>m <[email protected]>
+%%% @author Oscar Hellstrm <[email protected]>
 %%% @doc This is the specification for the lhttpc application.
 %%% @end
 {application, lhttpc,
(END)

but with a lot of warnings and sugestions:

0 sofa% make
./rebar compile
==> lhttpc (compile)
src/lhttpc.erl:28: Warning: Non-UTF-8 character(s) detected, but no encoding declared. Encode the file in UTF-8 or add "%% coding: latin-1" at the beginning of the file. Retrying with latin-1 encoding.
Compiled src/lhttpc.erl
src/lhttpc_lib.erl:29: Warning: Non-UTF-8 character(s) detected, but no encoding declared. Encode the file in UTF-8 or add "%% coding: latin-1" at the beginning of the file. Retrying with latin-1 encoding.
Compiled src/lhttpc_lib.erl
src/lhttpc_manager.erl:28: Warning: Non-UTF-8 character(s) detected, but no encoding declared. Encode the file in UTF-8 or add "%% coding: latin-1" at the beginning of the file. Retrying with latin-1 encoding.
Compiled src/lhttpc_manager.erl
src/lhttpc_sup.erl:28: Warning: Non-UTF-8 character(s) detected, but no encoding declared. Encode the file in UTF-8 or add "%% coding: latin-1" at the beginning of the file. Retrying with latin-1 encoding.
Compiled src/lhttpc_sup.erl
src/lhttpc_sock.erl:29: Warning: Non-UTF-8 character(s) detected, but no encoding declared. Encode the file in UTF-8 or add "%% coding: latin-1" at the beginning of the file. Retrying with latin-1 encoding.
Compiled src/lhttpc_sock.erl
src/lhttpc_client.erl:29: Warning: Non-UTF-8 character(s) detected, but no encoding declared. Encode the file in UTF-8 or add "%% coding: latin-1" at the beginning of the file. Retrying with latin-1 encoding.
Compiled src/lhttpc_client.erl
./rebar doc
==> lhttpc (doc)

Thanks for your time and good luck!