Giter VIP home page Giter VIP logo

eimp's Introduction

Erlang Image Manipulation Process

CI Coverage Status Hex version

eimp is an Erlang/Elixir application for manipulating graphic images using external C libraries. It supports WebP, JPEG, PNG and GIF.

Requirements

  • GNU Make
  • GCC
  • Erlang/OTP 17 and higher
  • libgd
  • libwebp
  • libpng
  • libjpeg

NOTE: It's hard to say which versions of the C libraries are required, but it seems like not too old versions should work well.

Install

$ ./configure
$ make

Note that running the configure script is highly recommended, so you should consider adding it in pre-hooks of your rebar configuration.

If no C libraries are found at compile time, the package will still be compiled. In this case the only usable function would be get_type/1.

Application design

The C code is compiled into external native binary called eimp, which is connected to Erlang VM using an external port. This is done because used C libraries are known not to be extremely stable, thus, if you wrap them into the emulator using a driver or NIF, they can crash the whole emulator.

When being loaded, the application starts a single eimp process per CPU core and uses a round-robin pool to schedule tasks to them. Simple recovery mechanisms are also supported:

  • if a request to eimp process has failed, next eimp process in the pool is picked, until the pool is exhausted
  • if an eimp process is dead, it will be restarted automatically
  • an eimp process is protected against decompression bombs

Usage

Before using the application you should start it with either eimp:start() or application:start(eimp).

API

Current API is simple and supports only a few functions:

convert/2

-spec convert(In :: binary(), Format :: png|jpeg|webp|gif) -> {ok, Out :: binary()} |
                                                              {error, Reason :: error_reason()}.

Shorthand for convert(In, Format, []).

convert/3

-spec convert(In :: binary(), Format :: png|jpeg|webp|gif,
              Options :: [convert_opt() | limit_opt()]) ->
                   {ok, Out :: binary()} |
                   {error, Reason :: error_reason()}.

The function converts incoming data In into format Format. Note that you don't have to pass the format of incoming data, becasue it will be detected automatically using get_type/1 function. In the case of an error you can use Reason to produce a human-readable diagnostic text using format_error/1. The function also accepts a proplist of Options. Currently available options are:

  • {scale, {Width, Height}}: scales image to the new Width and Height. No scaling is applied by default.
  • {rate_limit, N}: limit the number of calls to N per minute, where N > 0. Must be used only in conjunction with limit_by.
  • {limit_by, Term}: apply rate_limit (see above) to the entity associtated with Term. The Term may represent any value, such as an IP address, a username and so on. The Term must not be atom undefined. For example a call to convert(Data, Format, [{limit_by, {192,168,0,1}}, {rate_limit, 10}]) will fail with {error, too_man_requests} if called more than 10 times within a minute.

WARNING: the maximum resolution of an incoming image is hardcoded to be 25Mpx. This is a protection against decompression bombs.

identify/1

-spec identify(Img :: binary()) -> {ok, Info :: info()} | {error, error_reason()}.

Shorthand for identify(Img, []).

identify/2

-spec identify(Img :: binary(), LimitOptions :: [limit_opt()]) ->
                    {ok, Info :: info()} | {error, error_reason()}.

The function returns information about image Img, where Info is represented as:

[{type, Type :: img_type()},
 {width, Width :: non_neg_integer()},
 {height, Height :: non_neg_integer()}]

It is safe to assume that Info always contains all these properties. You can set limiting options in LimitOptions, that is rate_limit and limit_by. The meaning of the limiting options is the same as in convert/3.

NOTE: If you only need to get a type of an image, you're better off using get_type/1 function, because it doesn't involve interaction with eimp process and is, thus, much faster.

format_error/1

-spec format_error(Reason :: error_reason()) -> binary().

Creates diagnostic text from an error generated by convert/2. The Reason can have the following values:

-type error_reason() :: unsupported_format |
                        timeout |
                        disconnected |
                        encode_failure |
                        decode_failure |
			transform_failure |
			too_many_requests |
			image_too_big.

get_type/1

-spec get_type(Data :: binary()) -> png | jpeg | webp | gif | unknown.

Detects image format of Data.

is_supported/1

-spec is_supported(Format :: atom()) -> boolean.

Returns true if Format is known and compiled and false otherwise.

supported_formats/0

-spec supported_formats() -> [png | jpeg | webp | gif].

Returns a list of all known and compiled formats.

eimp's People

Contributors

badlop avatar mremond avatar nosnilmot avatar prefiks avatar triaxx avatar zinid avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

eimp's Issues

How to extend eimp

@zinid
I was trying to extend eimp to scale (resize) and crop for this i found
gdImageScale
gdImageCrop
gdImageCropAuto functions
https://libgd.github.io/manuals/2.2.4/files/gd_interpolation-c.html#gdImageScale
https://libgd.github.io/manuals/2.2.5/files/gd_crop-c.html#Cropping

However i see in eimp.c
main () calls loop (void) and it calls only convert()
How do extend / change so that i can also add other gd function support.

did not quite understand

if (buf) {
	  if (read_buf(buf, size) == size) {
	    pid_size = buf[0];
	    if (size >= pid_size + 3) {
	      result = convert(buf, buf[pid_size + 1], buf[pid_size + 2],
			       buf+pid_size+3, size-pid_size-3);
	    }
	  }
	}

Just want to check the roadmap

@zinid Do we also have plans for

  1. Image cropping / rotate / resize
  2. Extract image from Video
  3. Where will this dep used in ejabberd

I know its just a day old dep but i am super exicited for using.
I have already started using it will report if i find any bugs.

Also why are we using GD any specific reason? from all the tests over the web GrapicsMagick is fastest later ImageMagick and the SLOWEST of all is GD ...

eacces erlang open_port

i failed to start ejabberd server with these following error :

2022-08-05 15:37:12.989810+07:00 [error] <0.178.0>@supervisor:start_children/2:363 SUPERVISOR REPORT:
    supervisor: {local,eimp_sup}
    errorContext: start_error
    reason: {eacces,
                [{erlang,open_port,
                     [{spawn_executable,
                          "/home/xxxxx/ejabberd/lib/eimp-1.0.19/priv/bin/eimp"},
                      [{packet,4},binary]],
                     []},
                 {eimp_worker,start_port,1,
                     [{file,"src/eimp_worker.erl"},{line,179}]},
                 {eimp_worker,init,1,[{file,"src/eimp_worker.erl"},{line,68}]},
                 {gen_server,init_it,2,[{file,"gen_server.erl"},{line,417}]},
                 {gen_server,init_it,6,[{file,"gen_server.erl"},{line,385}]},
                 {proc_lib,init_p_do_apply,3,
                     [{file,"proc_lib.erl"},{line,226}]}]}
    offender: [{pid,undefined},
               {id,eimp_worker_1},
               {mfargs,{eimp_worker,start_link,[eimp_worker_1,1]}},
               {restart_type,permanent},
               {shutdown,5000},
               {child_type,worker}]

NOTE : The same binary source can be started successfully on other VM.. could you plz advice what is the possible issue on that server ?

  • ejabberd 21.01.7
  • Erlang/OTP 23

Thanks.

eimp:is_supported/1 returns false for any type

Hello, I have a trouble with using library: function eimp:is_supported/1 returns false for any type

Here is my setup:
OS: Ubuntu 16.04.3 LTS
libjpeg: libjpeg-dev/xenial,now 8c-2ubuntu8 amd64 [installed]
libgd: libgd-dev/xenial-updates,xenial-security,now 2.1.1-4ubuntu0.16.04.11 amd64 [installed]
libpng: libpng12-dev/xenial-updates,xenial-security,now 1.2.54-1ubuntu1.1 amd64 [installed]
libwebp: libwebp-dev/xenial,now 0.4.4-1 amd64 [installed]
erlang: esl-erlang/unknown,now 1:21.3.2-1 amd64 [installed,upgradable to: 1:21.3.3-1]

Could you please help me to debug this issue?

eunit tests fail

Currently, eunit tests run with eunit fail, after calling rebar get-deps and rebar compile (which both seem to work), I get:

$ rebar eunit
==> erlang-p1-imp (eunit)
Compiled src/eimp_app.erl
Compiled src/eimp_sup.erl
Compiled src/eimp.erl
Compiled src/eimp_worker.erl
Compiled test/eimp_test.erl

=ERROR REPORT==== 1-Oct-2017::12:05:03 ===
Failed to read ./priv/bin/eimp: no such file or directory
=ERROR REPORT==== 1-Oct-2017::12:05:03 ===
Failed to read ./priv/bin/eimp: no such file or directoryeimp_test: png_to_jpeg_test...*failed*
in function eimp_test:convert/2 (test/eimp_test.erl, line 144)
**error:{badmatch,{error,disconnected}}
  output:<<"">>

eimp_test: png_to_webp_test...*failed*
in function eimp_test:convert/2 (test/eimp_test.erl, line 144)
**error:{badmatch,{error,disconnected}}
  output:<<"">>

eimp_test: png_to_gif_test...*failed*
in function eimp_test:convert/2 (test/eimp_test.erl, line 144)
**error:{badmatch,{error,disconnected}}
  output:<<"">>


.... <repeating messages for various tests skipped>


=ERROR REPORT==== 1-Oct-2017::12:05:03 ===
Failed to read ./priv/bin/eimp: no such file or directoryeimp_test: too_big_test...*failed*
in function eimp_test:'-too_big_test/0-fun-0-'/1 (test/eimp_test.erl, line 112)
**error:{assertEqual,[{module,eimp_test},
              {line,112},
              {expression,"eimp : convert ( Data , jpeg )"},
              {expected,{error,image_too_big}},
              {value,{error,disconnected}}]}
  output:<<"">>


=INFO REPORT==== 1-Oct-2017::12:05:03 ===
    application: eimp
    exited: stopped
    type: temporary
eimp_test: disconnected_test...*failed*
in function eimp_test:'-disconnected_test/0-fun-0-'/1 (test/eimp_test.erl, line 138)
in call from eimp_test:disconnected_test/0 (test/eimp_test.erl, line 138)
**error:{assertEqual,[{module,eimp_test},
              {line,138},
              {expression,"file : delete ( Bin )"},
              {expected,ok},
              {value,{error,enoent}}]}
  output:<<"">>

=======================================================
  Failed: 26.  Skipped: 0.  Passed: 6.

Regarding the message (seen in above output) "Failed to read ./priv/bin/eimp: no such file or directory", I can confirm that at this point the binary exists:

ls -lh priv/bin/eimp 
-rwxr-xr-x 1 mati mati 91K Oct  1 12:04 priv/bin/eimp

Building fail on : Centos 6

[root@vps2 eimp]# make
./rebar get-deps compile
==> eimp (get-deps)
Pulling p1_utils from {git,"https://github.com/processone/p1_utils",
                           {tag,"1.0.10"}}
Initialized empty Git repository in /root/eimp/deps/p1_utils/.git/
==> p1_utils (get-deps)
==> p1_utils (compile)
Compiled src/p1_server.erl
Compiled src/p1_time_compat.erl
Compiled src/p1_options.erl
Compiled src/p1_nif_utils.erl
Compiled src/treap.erl
Compiled src/p1_utils_sup.erl
Compiled src/p1_file_queue.erl
Compiled src/p1_queue.erl
Compiled src/p1_utils.erl
Compiled src/p1_http.erl
Compiled src/p1_fsm.erl
Compiled src/p1_edoc_layout.erl
==> eimp (compile)
Compiled src/eimp_sup.erl
Compiled src/eimp_app.erl
Compiled src/eimp.erl
Compiled src/eimp_worker.erl
Compiling c_src/eimp.c
c_src/eimp.c: In function 'check_jpeg_header':
c_src/eimp.c:92: warning: implicit declaration of function 'jpeg_mem_src'
c_src/eimp.c: In function 'transform':
c_src/eimp.c:281: warning: implicit declaration of function 'gdImageScale'
c_src/eimp.c:281: warning: assignment makes pointer from integer without a cast
c_src/eimp.o: In function `transform':
/root/eimp/c_src/eimp.c:281: undefined reference to `gdImageScale'
c_src/eimp.o: In function `check_jpeg_header':
/root/eimp/c_src/eimp.c:92: undefined reference to `jpeg_mem_src'
collect2: ld returned 1 exit status
ERROR: sh(cc c_src/eimp.o   -lwebp -lgd  -lpthread  -L"/usr/local/lib/erlang/lib/erl_interface-3.7.20/lib" -lerl_interface -lei -o priv/bin/eimp)
failed with return code 1 and the following output:
c_src/eimp.o: In function `transform':
/root/eimp/c_src/eimp.c:281: undefined reference to `gdImageScale'
c_src/eimp.o: In function `check_jpeg_header':
/root/eimp/c_src/eimp.c:92: undefined reference to `jpeg_mem_src'
collect2: ld returned 1 exit status

ERROR: compile failed while processing /root/eimp: rebar_abort
make: *** [src] Error 1

Compilation error: 'erl_interface.h' file not found

According to OTP 23 Potential Incompatibilities:

  • erl_interface: Removed the deprecated parts of erl_interface (erl_interface.h and essentially all C functions with prefix erl_)

And I do get the error:

/deps/eimp/c_src/eimp.c:21:10: fatal error: 'erl_interface.h' file not found

Feature Request: Image decoder

Looking at the C file, I see that It might be possible to get pixel data for an image. Is this something that could be a good feature? I'd like to work on it, but I want to know if this is a good first PR, or if this is in the eimp roadmap.

How to add rebar3 pre-hook to run configure script before compilation?

Hi. My rebar3 project uses eimp as dependency. I tried to add
{pre_hooks, [{compile, "sh ${REBAR_DEPS_DIR}/eimp/configure"}]}.
to my rebar.config in order to run configure script before eimp compilation, but rebar3 run this hook after all deps compilation and before main application compilation. How can I add the hook in my rebar.config so that it will be called before eimp compilation?

Build fails with Rebar3

Hi! When building on Ubuntu 16.04 with Rebar3 I get next error:

===> Fetching eimp ({git,"https://github.com/processone/eimp.git",
                                {branch,"master"}})
===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace or consult rebar3.crashdump
===> When submitting a bug report, please include the output of `rebar3 report "your command"`
Makefile:6: recipe for target 'compile' failed
make: *** [compile] Error 1

In ebar3.crashdump I have

Error: {badmatch,
           {error,
               {228,file,
                {error,
                    {badmatch,["3","5","0+build","3974","refb5d7d35"]},
                    [{erl_eval,expr,3,[]}]}}}}
[{rebar_config,consult_file_,1,
               [{file,"/home/andrew/rebar3_test/_build/default/lib/rebar/src/rebar_config.erl"},
                {line,230}]},
 {rebar_config,consult_file,1,
               [{file,"/home/andrew/rebar3_test/_build/default/lib/rebar/src/rebar_config.erl"},
                {line,212}]},
 {rebar_prv_install_deps,handle_dep,6,
                         [{file,"/home/andrew/rebar3_test/_build/default/lib/rebar/src/rebar_prv_install_deps.erl"},
                          {line,280}]},
 {rebar_prv_install_deps,update_unseen_dep,9,
                         [{file,"/home/andrew/rebar3_test/_build/default/lib/rebar/src/rebar_prv_install_deps.erl"},
                          {line,273}]},
 {lists,foldl,3,[{file,"lists.erl"},{line,1263}]},
 {rebar_prv_install_deps,handle_profile_level,7,
                         [{file,"/home/andrew/rebar3_test/_build/default/lib/rebar/src/rebar_prv_install_deps.erl"},
                          {line,183}]},
 {rebar_prv_install_deps,do_,1,
                         [{file,"/home/andrew/rebar3_test/_build/default/lib/rebar/src/rebar_prv_install_deps.erl"},
                          {line,82}]},
 {rebar_core,do,2,
             [{file,"/home/andrew/rebar3_test/_build/default/lib/rebar/src/rebar_core.erl"},
              {line,154}]}]

Rebar3 and Erlang versions

rebar 3.5.0+build.3974.refb5d7d35 on Erlang/OTP 20 Erts 9.1.5

How can I fix it? Thanks in advance.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.