[Bug] immich microservice memory leak kills host about immich HOT 16 OPEN

Thanks for letting me know and I can confirm I am getting the same result.

It is working, and there are no errors in the log; it's just hungry :-)
I will wait for the next release and re-test
Thanks for all your great work.

from immich.

Below are some logs from the microservices; perhaps they might provide some insight:

[Nest] 7  - 05/13/2024, 11:34:38 AM     LOG [EventRepository] WebSocket server initialized.
[Nest] 7  - 05/13/2024, 11:35:06 AM   ERROR [JobService] Failed to execute job handler (thumbnailGeneration/generate-preview): Error: The input file contains an unsupported image format
[Nest] 7  - 05/13/2024, 11:35:06 AM   ERROR [JobService] Error: The input file contains an unsupported image format
    at Sharp.toFile (/usr/src/app/node_modules/sharp/lib/output.js:89:19)
    at MediaRepository.resize (/usr/src/app/dist/repositories/media.repository.js:76:14)
    at MediaService.generateThumbnail (/usr/src/app/dist/services/media.service.js:156:48)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MediaService.handleGeneratePreview (/usr/src/app/dist/services/media.service.js:134:29)
    at async /usr/src/app/dist/services/job.service.js:149:36
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 7  - 05/13/2024, 11:35:06 AM   ERROR [JobService] Affected Job ID:
{
  "id": "5802ddbb-4e01-47fb-b21b-73bc0361f9e8"
}

[Nest] 7  - 05/13/2024, 11:35:07 AM   ERROR [JobService] Failed to execute job handler (thumbnailGeneration/generate-preview): Error: ffprobe exited with code 1
ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
  Built with gcc 12 (Debian 12.2.0-14)
  Configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Invalid sample size -1008
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Error reading header
upload/upload/98a61f7d-3957-465a-b946-79697c36d3b1/59/5d/595d905e-3bde-49ef-a699-17b677622b3c.mp4: Invalid data found when processing input

[Nest] 7  - 05/13/2024, 11:35:07 AM   ERROR [JobService] Error: ffprobe exited with code 1
ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
  Built with gcc 12 (Debian 12.2.0-14)
  Configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Invalid sample size -1008
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Error reading header
upload/upload/98a61f7d-3957-465a-b946-79697c36d3b1/59/5d/595d905e-3bde-49ef-a699-17b677622b3c.mp4: Invalid data found when processing input

    at ChildProcess.<anonymous> (/usr/src/app/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22)
    at ChildProcess.emit (node:events:518:28)
    at ChildProcess._handle.onexit (node:internal/child_process:294:12)
[Nest] 7  - 05/13/2024, 11:35:07 AM   ERROR [JobService] Affected Job ID:
{
  "id": "ee18e5e8-5430-4fcb-aa10-ef4170e2b30e"
}

Limiting the container's resource usage has indeed helped me; it prevents my server from crashing. However, upon examining the microservices' logs, I noticed frequent restarts, indicating that the microservices are indeed exceeding the memory limit. Even after setting a 4GB memory limit, it's still insufficient.

Here's the relevant part of my Docker Compose configuration:

  immich-microservices:
    container_name: immich_microservices
    image: altran1502/immich-server:${IMMICH_VERSION:-release}
    command: ['start.sh', 'microservices']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - "/root/sharedfolder/syncthing/Photo_Album:/mnt/media/Photo_Album:ro"
      - /etc/localtime:/etc/localtime:ro
    devices:
      - /dev/dri:/dev/dri
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 4G

[Nest] 7  - 05/13/2024, 11:35:19 AM     LOG [EventRepository] Initialized websocket server
[Nest] 7  - 05/13/2024, 11:35:33 AM     LOG [EventRepository] Initialized websocket server
[Nest] 7  - 05/13/2024, 11:35:45 AM     LOG [EventRepository] Initialized websocket server
[Nest] 7  - 05/13/2024, 11:35:58 AM     LOG [EventRepository] Initialized websocket server
[Nest] 7  - 05/13/2024, 11:36:11 AM     LOG [EventRepository] Initialized websocket server
[Nest] 7  - 05/13/2024, 11:36:53 AM     LOG [EventRepository] Initialized websocket server
[Nest] 7  - 05/13/2024, 11:37:06 AM     LOG [EventRepository] Initialized websocket server
[Nest] 7  - 05/13/2024, 11:37:19 AM     LOG [EventRepository] Initialized websocket server
[Nest] 7  - 05/13/2024, 11:37:34 AM     LOG [EventRepository] Initialized websocket server
[Nest] 7  - 05/13/2024, 11:37:47 AM     LOG [EventRepository] Initialized websocket server

I'm still hoping to identify and resolve the root cause entirely. If more information is needed, please let me know.

On a side note, how can I persistently save Immich logs? The log path isn't mentioned in the user documentation, so I haven't mapped any volumes, resulting in log loss upon container restarts.

Thank you very much！

from immich.

I can also confirm this issue is affecting the installation on my NAS. Limiting memory causes frequent restarts of the microservices container. I also tried setting the Generate Thumbnails Concurrency to 1, but it didn't help. I found the NAS started overutalising the drives as they were at 100% utilisation. I imagine it was paging out memory to disk; CPU utilization was around 40%. I have increased the RAM in the NAS to 32GB, which has solved the problem for me, and the microservices container now runs constantly, with no restarting at all. The microservices container stabilised at around 28GB, the NAS memory resource monitor shows almost all of that usage is cache memory, the container limit is set to 4GB.
After restarting the container, it takes about 15-20min to stabilise at this level. Disk usage is now around 40% and CPU is around 80%, generating thumbnails at around 30/min. This is with no other jobs running.
I have around 230k assets in the external library, 50%jpg, 50%RAW. I also tried this in docker on a server (TrueNAS scale) and experienced the same issue and outcome.
Overall, I love the way Immich is going. It is going to be a great product. Well done to everyone involved, and thanks for the great work.

from immich.

How many of those assets are RAW, if any? 50% of the images are RAW around 50GB each.
Do you have /tmp configured to be in-memory? No /tmp configured.
Is the increase completely linear, or are there spikes? Mostly linear overlayed with a small saw tooth shape.

I just restarted the container. That smooth section is unusual, and it will go back to the sawtooth.
Any errors in the logs? No Errors, running smoothly.

from immich.

I made a test image for microservices with a possible fix: ghcr.io/immich-app/immich-server:pr-9665. Would you be able to change your image to that and see if it affects RAM usage?

from immich.

Yeah... Something may have gone wrong with the deployment. Let me do some more testing, and maybe get some sleep :-).

from immich.

So sorry! I based that branch off of the latest release, but it turns out that main gets merged into it anyway when the image is built. The error is probably because of that.

You can either wait for the next release to get things back up or restore from a backup. (It's also possible to mess with it more to get it back up, but I think these options are safer.)

from immich.

How much RAM does the server have, and how much is used just after starting thumbnail generation? An increase in memory usage for some period of time during thumbnail generation is normal because of memory fragmentation, but it should plateau after a certain point. If you have any RAW images, they will amplify this effect since they require much more memory.

from immich.

Thank you for your response. The server has a total of 16GB of RAM, and its typical daily memory usage hovers around 6GB. After initiating thumbnail generation, it doesn't immediately consume all memory but gradually fills it up until the program crashes. Your mention of RAW images did remind me that the majority of my gallery consists of RAW files. Is there a way to prevent excessive memory usage in this scenario? I have tried limiting the microservice using Cgroups, but when the limit is exceeded, the microservice restarts. Since I bind the Cgroup settings to the PID, once the microservice restarts, PID changed, the previous Cgroup configurations become ineffective.

from immich.

Given the server's average performance, I have configured all my JOBS to use a single thread, and only one JOB is executed at a time. In theory, processing a single RAW file shouldn't require such an excessive amount of memory.

from immich.

Hmm, that much of an increase is unexpected. It could be related to the issue behind #6542, or possibly #4391. Are there any errors in the microservices logs before the OOM error?

As far as limiting memory usage in the meantime, you can set a limit through Docker, which will force the container to be restarted after reaching a certain usage.

from immich.

I have the same problem (i5 8th gen server, running OMV 6) - am also Looking for a easy way to save immich logs -

from immich.

Thanks for the detailed info!

• How many of those assets are RAW, if any?
• Do you have /tmp configured to be in-memory?
• Is the increase completely linear, or are there spikes?
• Any errors in the logs?

from immich.

Below is an example of the errors I was receiving on the NAS when 4GB was installed and getting constant restarts of the microservices container.

from immich.

Not good news I am afraid. The immich_microservices fails.

immich_microservices
date stream content
2024/05/23 01:09:52 stderr Microservices worker exited with code 1

2024/05/23 01:09:52 stderr }

2024/05/23 01:09:52 stderr routine: 'parserOpenTable'

2024/05/23 01:09:52 stderr line: '1381',
2024/05/23 01:09:52 stderr file: 'parse_relation.c',
2024/05/23 01:09:52 stderr constraint: undefined,
2024/05/23 01:09:52 stderr dataType: undefined,
2024/05/23 01:09:52 stderr column: undefined,
2024/05/23 01:09:52 stderr table: undefined,
2024/05/23 01:09:52 stderr schema: undefined,
2024/05/23 01:09:52 stderr where: undefined,
2024/05/23 01:09:52 stderr internalQuery: undefined,
2024/05/23 01:09:52 stderr internalPosition: undefined,
2024/05/23 01:09:52 stderr position: '128',
2024/05/23 01:09:52 stderr hint: undefined,
2024/05/23 01:09:52 stderr detail: undefined,
2024/05/23 01:09:52 stderr code: '42P01',
2024/05/23 01:09:52 stderr severity: 'ERROR',
2024/05/23 01:09:52 stderr length: 113,
2024/05/23 01:09:52 stderr },
2024/05/23 01:09:52 stderr routine: 'parserOpenTable'

2024/05/23 01:09:52 stderr line: '1381',
2024/05/23 01:09:52 stderr file: 'parse_relation.c',
2024/05/23 01:09:52 stderr constraint: undefined,
2024/05/23 01:09:52 stderr dataType: undefined,
2024/05/23 01:09:52 stderr column: undefined,
2024/05/23 01:09:52 stderr table: undefined,
2024/05/23 01:09:52 stderr schema: undefined,
2024/05/23 01:09:52 stderr where: undefined,
2024/05/23 01:09:52 stderr internalQuery: undefined,
2024/05/23 01:09:52 stderr internalPosition: undefined,
2024/05/23 01:09:52 stderr position: '128',
2024/05/23 01:09:52 stderr hint: undefined,
2024/05/23 01:09:52 stderr detail: undefined,
2024/05/23 01:09:52 stderr code: '42P01',
2024/05/23 01:09:52 stderr severity: 'ERROR',
2024/05/23 01:09:52 stderr length: 113,

from immich.

Hmm, that error is about connecting to Postgres, not related to thumbnail generation.

from immich.