koher / swift-image Goto Github PK

how to flatten out image to an array of uint 8 pixels?
Is there a helper - I'm not worried about expense of copying.

typealias Byte = UInt8

extension Image where Pixel == UInt8 {
    func byteArray()->[Byte]{
        var pixelIterator = self.makeIterator()
        var pixelArray:[Byte] = []
        while let pixel = pixelIterator.next() {
            pixelArray.append(pixel)
        }
        return pixelArray
    }
}

I'm reviewing a couple of repos that are using tensorflow for swift as a command line app.

I'd like to plug in EasyImagy but unfortuantely - including frameworks with terminal line apps - isn't so supported.

A couple of options came to mind

1. attempt to leverage marathon for using easyimagy
   https://github.com/JohnSundell/Marathon/

2. cherry picking code into a single include file. (may sound bizarre / but consider - no pod - just spit out code and concatenate into a versioned file. )

https://github.com/JulienViguier/CNN_Swift4TensorFlow/blob/master/model.swift
https://github.com/saschaschramm/SwiftReinforce/blob/master/SwiftReinforce/Common/Utilities.swift#L55

//CNN_Swift4TensorFlow

struct Pixel {
    var r: Float
    var g: Float
    var b: Float
    var a: Float
    var row: Int
    var col: Int
    
    init(r: UInt8, g: UInt8, b: UInt8, a: UInt8, row: Int, col: Int) {
        self.r = Float(r)
        self.g = Float(g)
        self.b = Float(b)
        self.a = Float(a)
        self.row = row
        self.col = col
    }
}

func pixelToFloat(_pixelTab: [[Pixel]]) -> [Float] {
    var tab: [Float] = []
    for (_, row) in _pixelTab.enumerated() {
        for (_, cell) in row.enumerated() {
            tab.append(Float(cell.r/255.0))
            tab.append(Float(cell.g/255.0))
            tab.append(Float(cell.b/255.0))
        }
    }
    return tab
}

current solution for swift interfacing to python.

func renderPixels(_ pixels: [UInt8], rows: Int, cols: Int) {
    let sys = Python.import("sys")
    let np = Python.import("numpy")
    let path = "\(NSHomeDirectory())/gym/lib/python2.7/site-packages/"
    sys.path.append(path)
    let image = Python.import("PIL.Image")
    
    let foo = np.array(pixels).reshape([rows,cols])
    let img = image.fromarray(np.uint8(foo))
    img.show()
}

When compiling targeting My Mac (Mac Catalyst), I get the following errors:

swift-image/Sources/SwiftImage/UIKit.swift:56:12: error: invalid redeclaration of 'init(named:)'
    public init?(named name: String) {
           ^
swift-image/Sources/SwiftImage/AppKit.swift:22:12: note: 'init(named:)' previously declared here
    public init?(named name: NSImage.Name) {
           ^
swift-image/Sources/SwiftImage/UIKit.swift:68:12: error: invalid redeclaration of 'init(contentsOfFile:)'
    public init?(contentsOfFile path: String) {
           ^
swift-image/Sources/SwiftImage/AppKit.swift:27:12: note: 'init(contentsOfFile:)' previously declared here
    public init?(contentsOfFile path: String) {
           ^
swift-image/Sources/SwiftImage/UIKit.swift:73:12: error: invalid redeclaration of 'init(data:)'
    public init?(data: Data) {
           ^
swift-image/Sources/SwiftImage/AppKit.swift:32:12: note: 'init(data:)' previously declared here
    public init?(data: Data) {
           ^
swift-image/Sources/SwiftImage/UIKit.swift:83:17: error: invalid redeclaration of 'pngData()'
    public func pngData() -> Data? {
                ^
swift-image/Sources/SwiftImage/AppKit.swift:42:17: note: 'pngData()' previously declared here
    public func pngData() -> Data? {
                ^
swift-image/Sources/SwiftImage/UIKit.swift:91:17: error: invalid redeclaration of 'jpegData(compressionQuality:)'
    public func jpegData(compressionQuality: Double) -> Data? {
                ^
swift-image/Sources/SwiftImage/AppKit.swift:53:17: note: 'jpegData(compressionQuality:)' previously declared here
    public func jpegData(compressionQuality: Double) -> Data? {
                ^
swift-image/Sources/SwiftImage/UIKit.swift:99:17: error: invalid redeclaration of 'data(using:)'
    public func data(using format: ImageFormat) -> Data? {
                ^
swift-image/Sources/SwiftImage/AppKit.swift:64:17: note: 'data(using:)' previously declared here
    public func data(using format: ImageFormat) -> Data? {
                ^
swift-image/Sources/SwiftImage/UIKit.swift:109:17: error: invalid redeclaration of 'write(to:atomically:format:)'
    public func write(to url: URL, atomically: Bool, format: ImageFormat) throws {
                ^
swift-image/Sources/SwiftImage/AppKit.swift:74:17: note: 'write(to:atomically:format:)' previously declared here
    public func write(to url: URL, atomically: Bool, format: ImageFormat) throws {
                ^
swift-image/Sources/SwiftImage/UIKit.swift:117:17: error: invalid redeclaration of 'write(toFile:atomically:format:)'
    public func write<S : StringProtocol>(toFile path: S, atomically: Bool, format: ImageFormat) throws {
                ^
swift-image/Sources/SwiftImage/AppKit.swift:82:17: note: 'write(toFile:atomically:format:)' previously declared here
    public func write<S : StringProtocol>(toFile path: S, atomically: Bool, format: ImageFormat) throws {
                ^

I saw a pretty cool video of wwdc regarding filtering our two images to generate a third one with the difference between the two compared ones... But it's not what I really need for the project I'm working on.

I need to compare two images given a % threshold of difference accepted between them... I would have an uiimage referenced with the expected result and compared with a runtime generated one... I found how to compare pixel by pixel out there, but I'd like to be more soft comparing in % so it does necessarily needs to match 100% the expected and actual results.

Is it possible to compare pixels/images with diff threshold with this lib? Would you have an example case on how to do that with two uiimage objects?

Thanks

Dennis

EasyImagy is a really cool project.

When I open in the current Xcode 8.1 I see 59 warnings.

Is there any way to save images to png format on linux? That would be great :)

I have a program that reads data and plots it using different backends. Each backend implements a protocol and one of the methods in the the protocol is to start clipping. I have used SwiftImage in my png backend, source here. I use image.withCGContext closures to get the context each time I want to do something. What I discovered is that the clipping path I set in one closure was gone in the next.

My initial feeling was that the context should be the same but on the other hand it doesn't seem unreasonable to behave that way. On the off chance that it is a bug I thought I'd let you know.

carthage update returns:
The following build commands failed:
Check dependencies
(1 failure)
A shell task (/usr/bin/xcrun xcodebuild -project /Users//Carthage/Checkouts/EasyImagy/EasyImagy.xcodeproj -scheme EasyImagy-iOS -configuration Release -sdk iphoneos ONLY_ACTIVE_ARCH=NO BITCODE_GENERATION_MODE=bitcode CODE_SIGNING_REQUIRED=NO CODE_SIGN_IDENTITY= CARTHAGE=YES clean build) failed with exit code 65:
** CLEAN FAILED **

The following build commands failed:
Check dependencies
(1 failure)
** BUILD FAILED **

I have noticed since updating to iOS 13 from 12.4.1, the library has been running a lot slower than it was before. I used to consistently clock in around 250ms to flip an image over the x-axis, but now it is constantly at 3.1 seconds for the same exact image. Many other built in functions are slower as well. I tried updating Xcode to 11, Swift to 5, and the library itself to 0.7.0 but nothing has helped, and the code has not changed. Has anyone else had this issue or found a solution?

Hey, just wanna know if this library works with macOS apps? And if not is there a similar library for image (specifically pixel)-manipulation?

EDIT:

I tried importing the macOS Framework to my app but I get a bunch of compiler errors...

/Users/user/Documents/hider/Carthage/Checkouts/EasyImagy/EasyImagy/ImageTyped.swift:171:29: error: same-type requirement makes generic parameter 'Pixel' non-generic
extension Image where Pixel == Int { // Convolution
                            ^
/Users/user/Documents/hider/Carthage/Checkouts/EasyImagy/EasyImagy/ImageTyped.swift:157:29: error: same-type requirement makes generic parameter 'Pixel' non-generic
extension Image where Pixel == UInt8 { // Convolution
                            ^
/Users/user/Documents/hider/Carthage/Checkouts/EasyImagy/EasyImagy/ImageTyped.swift:143:29: error: same-type requirement makes generic parameter 'Pixel' non-generic
extension Image where Pixel == RGBA { // Convolution
                            ^
/Users/user/Documents/hider/Carthage/Checkouts/EasyImagy/EasyImagy/ImageTyped.swift:137:29: error: same-type requirement makes generic parameter 'Pixel' non-generic
extension Image where Pixel == Double { // map with indices

Swift is version 3.0.1

Is there an example to use this with image from URL?

On Xcode 10.0 beta4 I've got two errors Invalid redeclaration of 'countableRange(from:relativeTo:)' for internal func countableRange(from range: Range<Int>, relativeTo collection: CountableRange<Int>) -> CountableRange<Int> and internal func countableRange(from range: ClosedRange<Int>, relativeTo collection: CountableRange<Int>) -> CountableRange<Int> in Util.swift

With the implementation of SE-0143 Conditional conformances in Swift 4.2, CountableRange and CountableClosedRange are not separate types relative to Range, ClosedRange, but their type aliases.

I am trying to compile the package on Linux which fails. I am aware that you are not claiming compatibility with Linux but was hoping that you may know a quick fix that would address the below compilation errors.

First of all I am using it as follows:

func imageDimensions(fromData data: Data) -> (width: Int, height: Int) {
   if let image = Image<RGBA<UInt8>>(data: data) {
        let width = image.width
        let height = image.height
        return (width: width, height: height)
    }
    return (width: -1, height: -1)
}

This method works fine on macOS.

Then when I compile this on Linux I get the following compilation errors:

/project/.build-linux/checkouts/Mercury/Sources/Mercury/ImageDimensions.swift:26:38: 
error: extraneous argument label 'data:' in call
    if let image = Image<RGBA<UInt8>>(data: data) {
                                     ^~~~~~~

/project/.build-linux/checkouts/Mercury/Sources/Mercury/ImageDimensions.swift:26:20: 
error: initializer 'init(_:)' requires the types 'RGBA<UInt8>' 
and 'UInt8' be equivalent
    if let image = Image<RGBA<UInt8>>(data: data) {
                   ^
EasyImagy.ImageProtocol:6:12: note: where 'Self.Element' = 'RGBA<UInt8>', 
'I.Element' = 'UInt8'
    public init<I>(_ image: I) where I : ImageProtocol, Self.Element == I.Element
           ^
/project/.build-linux/checkouts/Mercury/Sources/Mercury/ImageDimensions.swift:26:20: 
error: initializer 'init(_:)' requires that 'Data' conform to 'ImageProtocol'
    if let image = Image<RGBA<UInt8>>(data: data) {
                   ^
EasyImagy.ImageProtocol:6:12: note: where 'I' = 'Data'
    public init<I>(_ image: I) where I : ImageProtocol, Self.Element == I.Element
           ^

Based on the above compilation errors, could you perhaps give a hint on how to fix this?

Hi I am using your library currently to create a steganography tool for macOS using Swift.

Basically I write the message into the last bit of each RGB value (not altering Alpha).
When I try this on an Image that has a transparent background It won't keep the changed RGB values but just sets them all to 0 because the Alpha is set to 0. I can only change the RGB values if the Alpha is all the way up at 255 (which means no transparency).

Either it's a problem with my code or with your library/framework because I created the same program already in Python and in C# where I have had no problem writing values to the pixels if they have transparency.
C# code for reference: https://paste.purpl3.net/16
Python3 code for reference: https://hider.purpl3.net/static/app.py
These two projects encode and decode images and the resulting images / messages are compatible with the two programs. The Swift version can also generate Images that can be read by the C# and Python version but only if they are PNG's without transparency.

Can you maybe help me out? I tried so much already for 3 days straight...

What is weird is that when I write to the debug console the before and after pixels they are exactly what I want but they don't end up in the final image
Maybe I am not saving the images correctly?

The Swfit project is here
The class I am doing all the work in is this

First of all, great library!
There seems so be a memory leak when creating data from images. This causes some trouble in one of my apps. To reproduce, do the following

        for _ in 0..<100 {
            let image = Image<UInt8>(width: 1, height: 1, pixels: [0])
            let _ = image.data(using: .png)!
        }

When setting a breakpoint after the loop and investigate the memory of core graphics one will find all the images, see screenshot.

So what happens in the image.data is basically the following:

            let data: [UInt8] = [0, 0, 0, 0]
            let provider: CGDataProvider = CGDataProvider(data: Data(
                bytes: UnsafeMutableRawPointer(mutating: data),
                count: 4
            ) as CFData)!

            let cgImage = CGImage(
                width: 1,
                height: 1,
                bitsPerComponent: 1 * 8,
                bitsPerPixel: 4 * 8,
                bytesPerRow: 4 * 1,
                space: CGColorSpaceCreateDeviceRGB(),
                bitmapInfo: CGBitmapInfo(),
                provider: provider,
                decode: nil,
                shouldInterpolate: false,
                intent: CGColorRenderingIntent.defaultIntent
            )!

            let imageRep: NSBitmapImageRep = NSBitmapImageRep(cgImage: cgImage)
            imageRep.size = CGSize(width: CGFloat(1), height: CGFloat(1))
            
            // memory leak only with the following line
            let _ = imageRep.representation(using: .png, properties: [:])

This self contained block of code does basically the same as image.data and is reproducing the memory leak (replace the body of the for loop, and you will find the same leak). So the line of code, causing the trouble is the last one. When removing the last line, the memory leak is gone.

Unfortunately, I have no idea what is wrong with this code.

This library looks to be just what I need. Are you planning to release this on CocoaPods?

Currently, warnings are shown when compiling swift-image. It means that swift-image cannot be compiled in Swift 6. Moreover, it seems that Swift 6 will be released in the not-too-distant future.

$ swift build
Building for debugging...
...
/path/to/swift-image/Sources/SwiftImage/AnyImage.swift:63:31: warning: generic parameter 'Pixel' shadows generic parameter from outer scope with the same name; this is an error in Swift 6
    private class AnyImageBox<Pixel> {
                              ^
/path/to/swift-image/Sources/SwiftImage/AnyImage.swift:3:24: note: 'Pixel' previously declared here
public struct AnyImage<Pixel> : ImageProtocol {
                       ^
...

(Xcode 15.2, Swift 5.9.2, macOS 14.3.1 arm64)

Am I using this incorrectly? All pixels appear empty.

let image = UIImage(named: "01", in: .main, with: nil)!
let rgbaImage = SwiftImage.Image<RGBA<UInt8>>(uiImage: image)

(0..<rgbaImage.width).forEach { x in
    (0..<rgbaImage.height).forEach { y in

        guard let pixel = rgbaImage.pixelAt(x: x, y: y) else { return }
        // NOTE: pixel.red, green, blue and alpha are all 0 for every pixel
    }
}

I came across this code base FilterPlay from @AfricanSwift which applies various filters to nsimage.
It maybe nice to some to incorporate some of these into this library.
https://github.com/AfricanSwift/FilterPlay

It maybe something where community can pitch in and add / extend

https://youtu.be/-Ja-vLbHWLc?t=49
https://en.wikipedia.org/wiki/Local_binary_patterns
was thinking

extension Pixel{
func determinePattern()->LBP{
    

     get the surrounding pixels
     do the calculations
     return 01010111



}

digging deeper into this - > there's a suite of these

BG-LBP (BackGround Local Binary Pattern) by Davarpanah et al. (2015)
CS-LBP (First-order Center-Symmetric Local Binary Patterns) by Heikkilä et al. (2006)
CS-LDP (Second-order Center-Symmetric Local Derivative Pattern) by Xue et al. (2011)
CS-SILTP (Center-Symmetric Scale Invariant Local Ternary Patterns) by Wu et al. (2013)
E-LBP (Extended LBP or Circular LBP) by Mdakane and Bergh (2012)
OC-LBP (Opponent Color Local Binary Pattern) by Maenpaa and Pietikainen (2004)
O-LBP (Original LBP) by Ojala et al. (2001)
SCS-LBP (Spatial extended Center-Symmetric Local Binary Pattern) by Xue et al. (2010)
SI-LTP (Scale Invariant Local Ternary Pattern) by Liao et al. (2010)
VAR-LBP (Variance-based LBP) by Ojala et al. (2002)
XCS-LBP (eXtended Center-Symmetric Local Binary Pattern) by Silva et al. (2015)

https://github.com/carolinepacheco/lbplibrary

  void OLBP::OLBP_(const cv::Mat& src, cv::Mat& dst)
  {
    dst = cv::Mat::zeros(src.rows - 2, src.cols - 2, CV_8UC1);
    for (int i = 1; i < src.rows - 1; i++) {
      for (int j = 1; j < src.cols - 1; j++) {
        _Tp center = src.at<_Tp>(i, j);
        unsigned char code = 0;
        code |= (src.at<_Tp>(i - 1, j - 1) > center) << 7;
        code |= (src.at<_Tp>(i - 1, j) > center) << 6;
        code |= (src.at<_Tp>(i - 1, j + 1) > center) << 5;
        code |= (src.at<_Tp>(i, j + 1) > center) << 4;
        code |= (src.at<_Tp>(i + 1, j + 1) > center) << 3;
        code |= (src.at<_Tp>(i + 1, j) > center) << 2;
        code |= (src.at<_Tp>(i + 1, j - 1) > center) << 1;
        code |= (src.at<_Tp>(i, j - 1) > center) << 0;
        dst.at<unsigned char>(i - 1, j - 1) = code;
      }
    }
  }

when I try to install via carthage I get this error:
Parse error: expected submodule commit SHA in output of task (ls-tree -z HEAD EasyImagy) but encountered:

my Cartfile:
github "koher/EasyImagy" ~> 0.5.0

Any ideas how to fix it?

Is it possible to implement this kind of expression in swift?

eg.
I[I == 144] = 0 // would match all elements within the array at value 144.

https://github.com/johndpope/gym/blob/master/lingfei/cartpole/policyGrad.py#L61

def prepro(I):
    """ prepro 210x160x3 uint8 frame into 6400 (80x80) 1D float vector """
    I = I[35:195]  # crop
    I = I[::2, ::2, 0]  # downsample by factor of 2
    I[I == 144] = 0  # erase background (background type 1)
    I[I == 109] = 0  # erase background (background type 2)
    I[I != 0] = 1  # everything else (paddles, ball) just set to 1
    return I.astype(np.float).ravel()

Sample project:
https://github.com/t-ae/EasyImagySaveExperiment

I'm trying to save Image via NSImage/CGImage.
https://github.com/t-ae/EasyImagySaveExperiment/blob/master/EasyImagySaveExperiment/ViewController.swift#L21-L25

Then this NSImage's tiffRepresentation returns nil with error message below,

CGImageDestinationFinalize failed for output type 'public.tiff'

And CGImage also doesn't generate png data.
I suspect this property has a bug.

I wanting to pass all this pixel data to another method but it's asking for an array of pixels.
specifically ->
https://github.com/johndpope/fast-swift/blob/master/FAST.swift

this doesn't quite work.

extension Image{
    func pixels()->[Pixel]{
        return self.enumerated()
    }
}

P.S.

 let image = Image<RGBA<UInt8>>(nsImage: NSImage(named: "fast_1.png")!) -> consider
->
 let image = Image<RGBA<UInt8>>(nsImageNamed: "fast_1.png")  // then coerce this behind scenes.

this
let result = image.map { $0.gray }

should be exposed as just
image.grayScale()

Hi,
I've been trying to load an image (attached) (1.9MB, 2162 × 3843), crop it and then resize the results. It's working, but taking about 30 seconds for the process!
I've seen a closed issue regarding performance in iOS 13, but I've compiled with speed optimizations and I see no change. Here's the code I'm running, along with the time it takes:
Creating image data for image of size: (2162.0, 3843.0):
Code: let imageData = Image<RGBA>(uiImage: image!) ; Time: 6.934875965118408 s
Cropping photo ( 0 < x < 2161, 1316 < y < 2525)
Code: let croppedImageData = Image<RGBA>(data[minX..<maxX, minY..<maxY]) ; Time: 1.0871039628982544
resizing photo to: 2736.0 by 1530.0
Code: let resizedCroppedImageData = croppedImageData.resizedTo(width: 2736, height: 1530) ; Time: 22.0144259929657

My Compiler settings: Optimization level [-O] (Swift compiler - code generation), [-Ofast] (Apple CLang - Code Generation)

The machine I'm running on: iPhone XS, iOS 13.5.1

Is there anything I'm doing wrong? Any help would be appreciated!