Giter VIP home page Giter VIP logo

webreaper / damselfly Goto Github PK

View Code? Open in Web Editor NEW
1.3K 24.0 72.0 275.5 MB

Damselfly is a server-based Photograph Management app. The goal of Damselfly is to index an extremely large collection of images, and allow easy search and retrieval of those images, using metadata such as the IPTC keyword tags, as well as the folder and file names. Damselfly includes support for object/face detection.

License: GNU General Public License v3.0

C# 63.07% Shell 0.43% HTML 19.96% CSS 5.29% Dockerfile 0.06% JavaScript 11.19%
photography docker image image-processing exif iptc-metadata blazor efcore face-detection

damselfly's Introduction

Damselfly Screenshot

Damselfly - A Photo Management System

Damselfly is a server-based Digital Photograph Management system. Damselfly is designed to manage a large, folder-based collection of photographs, with a particular focus on fast search and keyword-tagging workflow.

Damselfly includes powerful Machine Learning functions which can help you identify photographs and their subjects, including face detection, face recognition and object detection - allowing you to tag people once, and have Damselfly identify other photos in which they appear.

The user-interface and workflow is loosely based on the much-loved Google Picasa app, with a basket to select images for export and other types of processing. Damselfly also provides a desktop/client app which gives closer integraton with your laptop or PC, allowing you to quickly sync a selection of images from the Damselfly basket to a local folder, for editing etc.

Features

  • Support for most image formats including JPG, PNG, HEIC, TIFF, Webp, BMP and DNG/CR2/ORF/NEF (RAW) files.
  • AI / Computer vision image recognition:
    • Facial detection
    • Facial Recognition
    • Object detection and recognition
    • Image Colour Classification
  • Full-text search with multi-phrase partial-word searches
  • Image re-organisation - move/copy images between folders, and delete images (via a trashcan folder)
  • Advanced search - filter by:
    • Find visually similar images
    • Date ranges
    • Objects/Faces/Identified faces
    • Camera/Lens Make/Model
    • File Size
    • Orientation
    • Images with no keyword tags
Advanced Search in Damselfly
  • Focus on extremely fast performance - searching a 500,000-image catalogue returns results in less than a second.
  • Fast keyword tagging workflow with non-destructive EXIF data updates (using ExifTool) - so JPEGs are not re-encoded when keyword-tagged
  • Face detection and recognition, and object recognition in images
  • Multi-user support with user accounts, and role-based entitlements (e.g., users with ReadOnly role are prevented from keyword-tagging images)
  • Download/export processing to watermark images ready for social media, or sending via Email etc.
  • GeoLocation with map display of photo locations where metadata exists
  • Selection basket for saving images from search results:
    • Images within the basket can be downloaded, exported for use within Digikam/PhotoShop/etc., or uploaded to Wordpress
    • Baskets can be user-specific, or shared with other users
  • Server-based deployment, with a web-based front-end UI, so the image library can be accessed via multiple devices without having to copy catalogues or other DBs to local device storage.
  • Exclude images from Damselfly scanning by adding a .nomedia file in any folder.
  • Themes
  • Completely automated background indexing of images, so that the collection is automatically and quickly updated when new images are added or updated
  • Background thumbnail generation
  • Electron.Net Desktop Client for hosted site to allow closer native integration with client OS
    • Desktop Client versions for MacOS (universal), Windows and Linux
    • Synchronise images from server basket select to local filesystem for editing
    • Other integrations coming in future
  • Runs on Windows, Linux and OSX, and in Docker.
  • Built with Microsoft .Net 7, hosted Blazor WebAssembly and EFCore 7.

Update on Damselfly Face Recognition Support

Damselfly used to provide face-recognition via the Microsoft Azure Face API. However, Microsoft have recently locked down the use of this API.

The latest version of Damselfly now supports full face detection and recognition locally and offline.

Want to Support Damselfly?

Damselfly is free, open-source software. But if you find it useful, and fancy buying me a coffee or a slice of pizza, that would be appreciated!

Buy Me A Coffee

Getting in Touch

You can follow Damselfly on Twitter or ask questions/get support on Reddit (r/DamselflyPhotos). If you have problems, feature requests or have found a bug, you can raise an issue. Or if you want to say hello, email me.

Screenshots

Browsing in Damselfly Themes in Damselfly

Viewing images in Damselfly Face and Object Detection in Damselfly
Browsing images grouped by date Damselfly

Planned Features/Enhancements

  • Simple non-destructive editing/manipulation - cropping, colour adjustment etc (in progress)
  • Image de-duplication (in progress)
  • Support for more image formats, and possibly video
  • Support for selection and upload to Alamy Stock Image photo service
  • Synchronisation of local images back to the server
  • If you have ideas for other features - let me know by raising an issue!

How should I use Damselfly? What's the workflow?

The photos live in the library on the server, but whenever you want to work on a picture (e.g., in Photoshop, Digikam or your editing tool of choice) you use the Damselfly Deskop app to add the images to the basket, and choose Download => Save Locally, to sync the pictures across the network to your local folder. Once you've finished editing, copy them back to the server (a future feature enhancement will let Damselfly do this for you) where the server will re-index them to pick up your changes.

Suggested workflow.

  1. Images are copied onto a laptop for initial sorting, quality checks, and IPTC tagging using Picasa or Digikam
  2. Rclone script syncs the new images across the LAN to the network share
  3. Damselfly automatically picks up the new images and indexes them (and generates thumbnails) within 30 minutes
  4. Images are now searchable in Damselfly and can be added to the Damselfly 'basket' with a single click
  5. Images in the basket can be copied back to the desktop/laptop for local editing in Lightroom/On1/Digikam/etc.
    • Use the Damselfly Desktop client to write the files directly to the local filesystem in the same structure as on the server.
    • Export to a zip file to download and extract into the local image folder for additional editing
  6. Re-sync using RClone to push the images back to the collection [Future enhancement: Damselfly Desktop App will do this for you]

More information

Thanks and Credits

  • Microsoft Blazor.Net for allowing me to avoid writing Javascript. ;)
  • PumpkinBeth (my wife) for taking the photo that appears in Damselfly's logo, and for providing the original inspiration to build Damselfly - as well as the 600,000 photos and counting which make excellent test data!
  • SkiaSharp Fast library for Thumbnail generation
  • SixLabors ImageSharp Portable library for Thumbnail generation
  • Drew Noakes' MetadataExtractor, for IPTC and other image meta-data indexing
  • Phil Harvey's ExifTool which Damselfly uses for all of the EXIF write operations
  • ZZZ Project's Entity Framework Extensions for their excellent EFCore extensions.
  • MudBlazor for their excellent Blazor component library
  • FaceONNX for their excellent face detection and recognition library
  • Font Awesome for their Awesome icons
  • Chris Sainty for Blazored Modal and Typeahead, and all his excellent info on Blazor
  • Serilog.Net for logging
  • Wisne for Infinite Scroll inspiration

damselfly's People

Contributors

curiousercreative avatar jongalloway avatar vcsjones avatar webreaper avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

damselfly's Issues

Saved Basket Enhancements

  1. Check for duplicate names and error in the UI
  2. Support deletion of baskets
  3. Improve 'Default' naming convention

Modify Tags on export/download

Would be good to be able to strip EXIF metadata when exporting to Wordpress or downloading images. This could be per-tag, or all tags. Could also be useful to add tags to images when they're exported (e.g., tag all Wordpress uploads with 'wordpress').

Caching of images pulled back from EF

Should we cache the Image objects once they're pulled back from the DB? Or alternatively, can we use the search results collection as an in-memory cache for images?

Exception in ContextMenu code: Cannot read property 'dataset' of null

[11:10:59.718-3323-WRN] Unhandled exception rendering component: Cannot read property 'dataset' of null
TypeError: Cannot read property 'dataset' of null
at Object.f.RegisterTriggerReference (http://192.168.1.120:6363/_content/Blazor.ContextMenu/blazorContextMenu.min.js:1:3579)
at http://192.168.1.120:6363/_framework/blazor.server.js:1:70053
at new Promise ()
at e.beginInvokeJSFromDotNet (http://192.168.1.120:6363/_framework/blazor.server.js:1:70019)
at http://192.168.1.120:6363/_framework/blazor.server.js:1:26303
at Array.forEach ()
at e.invokeClientMethod (http://192.168.1.120:6363/_framework/blazor.server.js:1:26273)
at e.processIncomingData (http://192.168.1.120:6363/_framework/blazor.server.js:1:24211)
at e.connection.onreceive (http://192.168.1.120:6363/_framework/blazor.server.js:1:17296)
at WebSocket.i.onmessage (http://192.168.1.120:6363/_framework/blazor.server.js:1:46513)

Multi-user support

Basket selection, search and other behaviour is global. Need to add the capacity for multiple users to have independent search and basket functionality.

Thumbnail generation on-demand

Rather than batch-processing thumbnails in the background, they should be generated on-demand when browsing. Depends on performance. Perhaps make it an option.

Local FileSystem integration

Investigate whether it's possible to use the Chrome File API to read/write images to a specific file on the local hard disk. For example - to have a button to sync photos into a local folder item (and back again).

Search performance

Investigate better SQL, EF Core queries and indexes for search. This can also include moving more work to Bulk queries.

Proper versioning

Docker container, Desktop App and Blazor app all need to have a correct, aligned version auto-generated as part of the build.

UI design and layout

Need somebody with CSS skillz to be able to tidy up (and possibly redesign) the UI. :)

Allow other exif data to be written

Change KeywordOperations table to be ExifOperations, and include metadata type in the table, so that captions and other info can be edited. For consideration:

  • Caption
  • Description
  • Copyright
  • Rating

Exception when indexing tags

Exception when indexing new image - looks like a race condition loading the tags from the cache and image.

[14:20:11.693-MetaDataThread-ERR] Exception adding ImageTags: System.Collections.Generic.KeyNotFoundException: The given key 'front garden' was not present in the dictionary.

   at System.Collections.Concurrent.ConcurrentDictionary`2.ThrowKeyNotFoundException(Object key)

   at System.Collections.Concurrent.ConcurrentDictionary`2.get_Item(TKey key)

   at Damselfly.Core.Services.IndexingService.<>c__DisplayClass32_1.<AddTags>b__7(String v) in /src/Damselfly.Core/Services/IndexingService.cs:line 356

   at System.Linq.Enumerable.SelectArrayIterator`2.MoveNext()

   at System.Collections.Generic.List`1.InsertRange(Int32 index, IEnumerable`1 collection)

   at System.Linq.Enumerable.SelectManySingleSelectorIterator`2.ToList()

   at Damselfly.Core.Services.IndexingService.AddTags(IDictionary`2 imageKeywords) in /src/Damselfly.Core/Services/IndexingService.cs:line 355

Circuit failure after image download

(index):85 Resource interpreted as Document but transferred with MIME type image/jpeg: "http://poundnas:6363/rawimage/547753".
downloadFile @ (index):85
(anonymous) @ blazor.server.js:1
beginInvokeJSFromDotNet @ blazor.server.js:1
(anonymous) @ blazor.server.js:1
e.invokeClientMethod @ blazor.server.js:1
e.processIncomingData @ blazor.server.js:1
connection.onreceive @ blazor.server.js:1
i.onmessage @ blazor.server.js:1
blazor.server.js:19 [2020-11-10T20:48:45.122Z] Error: Circuit has been shut down due to error.

Optimise DB performance

Check for all calls to SaveChanges and see if it can be replaced with a bulk insert/update.

Investigate Sqlite Stored Procs for performance

May be possible to write a "TagsThatMatch" SP which combines the FTS5 search and a "like '%{term}%" query for tags, and return it as a union list of tags that will be more efficient than the dynamic EFCore SQL I use now.

Deleted folders aren't being picked up/removed during indexing

Child folders of top-level parent folders are not removed when the parent is removed. This leaves dangling orphan folders which never get detected. Should probably make the folder check a complete recursive check of all folders and whether they appear in the DB or not.

Tidy up validation on Config Settings page

Currently the validation for the Wordpress settings means they need to be filled out even if other settings are specified.
Have a checkbox to enable them, and allow the validation to be granular dependent on enabled features.

Tag-related filters

Add filters for

  • The specific tag 'delete' (which should be enabled by default)
  • Images with no tags at all

Rework virtual Scrolling

  • Virtual folder list scrolling should be able to work regardless of the container size. Use the 'All' item to dynamically calculate the item height at runtime, and then use the container size to calculate scroll size and position.
  • Virtual image scroller should be smarter about pre-loading new results before the bottom of the page is hit (i.e., during scrolling)

Google Drive Integration

Ability to send images to a Google Drive folder, optionally watermarking and reducing size/resolution

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.