open-dating / adult-image-detector Goto Github PK

View Code? Open in Web Editor NEW

70.0 9.0 21.0 240 KB

Use deep neural networks and other algos for detect nude images

Home Page: https://adult-image-detector.herokuapp.com/

Dockerfile 11.02% Go 88.38% Shell 0.60%

opencv dnn rest-api nsfw

adult-image-detector's Introduction

adult-image-detector

Use deep neural networks and other algos for detect nude images in images and pdf files.

Try detection

Usage

For detect nudity in images exec:

curl -i -X POST -F "image=@Daddy_Lets_Me_Ride_His_Cock_preview_720p.mp4.jpg" http://localhost:9191/api/v1/detect

Result:

{
  "app_version":"0.2.0",
  "open_nsfw_score":0.81577397,
  "an_algorithm_for_nudity_detection": true,
  "image_name":"Daddy_Lets_Me_Ride_His_Cock_preview_720p.mp4.jpg"
}

For detect nudity in pdf exec:

curl -i -X POST -F "pdf=@Daddy_Lets_Me_Ride_His_Cock_preview_720p.mp4.pdf" http://localhost:9191/api/v1/pdf_detect

Result:

{
  "app_version": "0.4.0",
  "result": {
    "2021-06-05T09:01:38Z_ac749c19-8bd1-48fa-88d5-0a448c0d948c_1_Im4.png": {
      "open_nsfw_score": 0.0015096113,
      "an_algorithm_for_nudity_detection": false
    },
    "2021-06-05T09:01:38Z_ac749c19-8bd1-48fa-88d5-0a448c0d948c_1_Im5.png": {
      "open_nsfw_score": 0.00092005456,
      "an_algorithm_for_nudity_detection": false
    }
  },
  "nudity_detection_disabled": false,
  "image_scoring_disabled": false,
  "image_name": "Daddy_Lets_Me_Ride_His_Cock_preview_720p.mp4.pdf",
  "an_algorithm_for_nudity_detection": false,
  "open_nsfw_score": 0.0015096113
}

Docker

Run

docker run -p 9191:9191 opendating/adult-image-detector

Build

git clone https://github.com/open-dating/adult-image-detector --recursive
docker build -t adult-image-detector .

Development

git clone https://github.com/open-dating/adult-image-detector --recursive
cd docker/dev
docker-compose up

Test

cd docker/test
docker-compose up

Install to heroku

Use deploy button or:

fork, create app and change stack to container

heroku stack:set container

Requirements

Go 1.17

opencv 4.5.1

Development without docker

Recursive clone that repo:

git clone https://github.com/open-dating/adult-image-detector --recursive

or manually install submodules:

git submodule init
git submodule update

Install opencv 4.5.1

Run hot reload with fresh:

go install github.com/pilu/fresh@0fa698148017fa2234856bdc881d9cc62517f62b
fresh

adult-image-detector's People

Contributors

Stargazers

Watchers

adult-image-detector's Issues

PR proposal: detecting images in PDF file

hi,
as of now, raw images are supported. I would like to add support for extracting images from PDF file and detect.
If you think this is a good addition, I can send a PR.

Error when build docker

I got this error when build the docker

=> [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                              0.1s
 => => transferring dockerfile: 3.01kB                                                                                                                                                                                                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                 0.1s
 => => transferring context: 45B                                                                                                                                                                                                                                                                                  0.0s
 => [internal] load metadata for docker.io/library/ubuntu:18.04                                                                                                                                                                                                                                                   0.0s
 => [internal] load build context                                                                                                                                                                                                                                                                                 0.2s
 => => transferring context: 24.13MB                                                                                                                                                                                                                                                                              0.2s
 => [opencv 1/3] FROM docker.io/library/ubuntu:18.04                                                                                                                                                                                                                                                              0.0s
 => [opencv 2/3] RUN apt-get update && apt-get install -y --no-install-recommends             git build-essential cmake pkg-config unzip libgtk2.0-dev             curl ca-certificates libcurl4-openssl-dev libssl-dev             libavcodec-dev libavformat-dev libswscale-dev libtbb2 libtbb-dev            155.9s
 => [opencv 3/3] RUN curl -Lo opencv.zip https://github.com/opencv/opencv/archive/4.5.1.zip &&             unzip -q opencv.zip &&             curl -Lo opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/4.5.1.zip &&             unzip -q opencv_contrib.zip &&             rm opencv.zip o  292.2s 
 => [gocv 1/3] RUN apt-get update && apt-get install -y --no-install-recommends             git software-properties-common &&             curl -Lo go1.14.2.linux-amd64.tar.gz https://dl.google.com/go/go1.14.2.linux-amd64.tar.gz &&             tar -C /usr/local -xzf go1.14.2.linux-amd64.tar.gz &&         39.8s 
 => [gocv 2/3] RUN mkdir -p "/go/src" "/go/bin" && chmod -R 777 "/go"                                                                                                                                                                                                                                             0.4s 
 => [gocv 3/3] WORKDIR /go                                                                                                                                                                                                                                                                                        0.0s 
 => ERROR [adult-image-detector 1/5] RUN go get -u github.com/pilu/fresh                                                                                                                                                                                                                                         11.9s 
------                                                                                                                                                                                                                                                                                                                 
 > [adult-image-detector 1/5] RUN go get -u github.com/pilu/fresh:                                                                                                                                                                                                                                                     
#0 11.86 # golang.org/x/sys/unix                                                                                                                                                                                                                                                                                       
#0 11.86 src/golang.org/x/sys/unix/syscall.go:83:16: undefined: unsafe.Slice
#0 11.86 src/golang.org/x/sys/unix/syscall_linux.go:2271:9: undefined: unsafe.Slice
#0 11.86 src/golang.org/x/sys/unix/syscall_unix.go:118:7: undefined: unsafe.Slice
#0 11.86 src/golang.org/x/sys/unix/sysvshm_unix.go:33:7: undefined: unsafe.Slice
------
Dockerfile:64
--------------------
  62 |     FROM gocv AS adult-image-detector
  63 |     # ENV GO111MODULE on　
  64 | >>> RUN go get -u github.com/pilu/fresh
  65 |     
  66 |     WORKDIR $GOPATH/src/adult-image-detector
--------------------
ERROR: failed to solve: process "/bin/sh -c go get -u github.com/pilu/fresh" did not complete successfully: exit code: 2

How to upload file via URL?

Is there any way how to upload my file via url?

For example:
curl -i -X POST -F "image=@http://cdn.akc.org/content/article-body-image/samoyed_puppy_dog_pictures.jpg" http://localhost:9191/api/v1/detect

how i train this?

how I train and teach this for detect perfectly?

Consumes more and more memory (and almost never releases it when idle)

Hello!

First, thanks for nice project, I like it, especially docker version, and I'm using it in my Nude Crawler project.

Problem: when using docker image it almost never releases memory, and takes more and more (until will get our of memory).

How to reproduce problem:

# start and see how much memory it uses (tiny 11Mb)
$ sudo docker run -d --rm -p 9191:9191 --name aid --memory=1G opendating/adult-image-detector
$ sudo docker stats --no-stream
CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT   MEM %     NET I/O       BLOCK I/O     PIDS
6b22f14d33a3   aid       0.00%     11.27MiB / 1GiB     1.10%     5.11kB / 0B   11.6MB / 0B   6

# Analyse first file
$ curl -s -i -X POST -F "image=@/tmp/eropicture.jpg" http://localhost:9191/api/v1/detect > /dev/null
$ sudo docker stats --no-stream
CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT   MEM %     NET I/O         BLOCK I/O     PIDS
6b22f14d33a3   aid       0.00%     170MiB / 1GiB       16.60%    315kB / 1.9kB   35.9MB / 0B   10

# and two more
$ curl -s -i -X POST -F "image=@/tmp/eropicture.jpg" http://localhost:9191/api/v1/detect > /dev/null
$ curl -s -i -X POST -F "image=@/tmp/eropicture.jpg" http://localhost:9191/api/v1/detect > /dev/null

# Why now it needs extra 50Mb after two more images analysed?? Ok...  
$ sudo docker stats --no-stream
CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT   MEM %     NET I/O          BLOCK I/O     PIDS
6b22f14d33a3   aid       0.00%     221.7MiB / 1GiB     21.65%    932kB / 5.48kB   35.9MB / 0B   10

# Lets analyse 100 more images
$ for i in `seq 1 100`; do curl -s -i -X POST -F "image=@/tmp/eropicture.jpg" http://localhost:9191/api/v1/detect > /dev/null ; done

# now 500Mb used for something...
$ sudo docker stats --no-stream
CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT   MEM %     NET I/O          BLOCK I/O      PIDS
6b22f14d33a3   aid       0.00%     545.4MiB / 1GiB     53.26%    31.7MB / 174kB   36MB / 614kB   11

# Now it uses about all of available limit, sometimes 1022MiB, sometimes falls back to 976MiB, 
# but about at 700-800 curl request container crashes. 
# If run in foreground, no error messages displayed, last messages are:
2023/03/17 18:34:21 For file 2023-03-17T18:34:21Z_fabede53-3817-47f8-a077-86d712d51602.jpg, openNsfwScore=0.834231
2023/03/17 18:34:21 For file 2023-03-17T18:34:21Z_fabede53-3817-47f8-a077-86d712d51602.jpg, anAlgorithmForNudityDetection=true
2023/03/17 18:34:21 Uploaded file eropicture.jpg, saved as 2023-03-17T18:34:21Z_3c3370b3-ede2-4847-ac04-9ba5957765b0.jpg

Test image size is 299K. All tests I did on docker image I pulled today.

If set RAM limit to 300Mb, it goes to almost full memory much sooner, at about 20-30th request, but continues to work at this memory usage up to ~100th request. (then crashes). When I start it without memory limit, it runs much longer (I have 16Gb RAM), but in the and machine hits OOM anyway.

If stop sending new requests, it keeps almost full memory for long time (I tried to wait for few hours, memory usage is nearly same after few hours of idle time).

In theory, I could restart docker container after every N requests, but it looks ugly. And my script runs as ordinary user account, I would not like to give it root access even to analyze nude woman.

Maybe it's possible to tune up garbage collector or restart process? Or maybe make API call (which I would trigger after every N requests) which will release memory or restart program?