tucan9389 / poseestimation-coreml Goto Github PK

The example project of inferencing Pose Estimation using Core ML

Home Page: https://github.com/motlabs/awesome-ml-demos-with-ios

License: MIT License

Swift 99.04% Python 0.96%

poseestimation-coreml's Introduction

This project is Pose Estimation on iOS with Core ML.
If you are interested in iOS + Machine Learning, visit here you can see various DEMOs.

한국어 README

Jointed Keypoints	Hatmaps	Still Image

Video source:

https://www.youtube.com/watch?v=EM16LBKBEgI

Pose Capturing	Pose Matching

Features

Estimate body pose on a image
Inference with camera's pixel buffer on real-time
Inference with a photo library's image
Visualize as heatmaps
Visualize as lines and points
Pose capturing and pose matching

How it works

Requirements

Xcode 9.2+
iOS 11.0+
Swift 4

Model

Get PoseEstimationForMobile's model

Download this temporary models from following link.

☞ Download Core ML model model_cpm.mlmodel or hourglass.mlmodel.

input_name_shape_dict = {"image:0":[1,192,192,3]} image_input_names=["image:0"]
output_feature_names = ['Convolutional_Pose_Machine/stage_5_out:0']

－in https://github.com/tucan9389/pose-estimation-for-mobile

Model Size, Minimum iOS Version, Input/Output Shape

Model	Size (MB)	Minimum iOS Version	Input Shape	Output Shape
cpm	2.6	iOS11	`[1, 192, 192, 3]`	`[1, 96, 96, 14]`
hourhglass	2	iOS11	`[1, 192, 192, 3]`	`[1, 48, 48, 14]`

Infernece Time (ms)

Model vs. Device	11 Pro	XS Max	XR	X	8	8+	7	7+	6S+	6+
cpm	5	27	27	32	31	31	39	37	44	115
hourhglass	3	6	7	29	31	32	37	42	48	94

Total Time (ms)

Model vs. Device	11 Pro	XS Max	XR	X	8	8+	7	7+	6S+	6+
cpm	23	39	40	46	47	45	55	58	56	139
hourhglass	23	15	15	38	40	40	48	55	58	106

FPS

Model vs. Device	11 Pro	XS Max	XR	X	8	8+	7	7+	6S+	6+
cpm	15	23	23	20	20	21	17	16	16	6
hourhglass	15	23	23	24	23	23	19	16	15	8

Get your own model

Or you can use your own PoseEstimation model

Build & Run

1. Prerequisites

1.1 Import pose estimation model

Once you import the model, compiler generates model helper class on build path automatically. You can access the model through model helper class by creating an instance, not through build path.

1.2 Add permission in info.plist for device's camera access

2. Dependencies

No external library yet.

3. Code

3.1 Import Vision framework

import Vision

3.2 Define properties for Core ML

// properties on ViewController
typealias EstimationModel = model_cpm // model name(model_cpm) must be equal with mlmodel file name
var request: VNCoreMLRequest!
var visionModel: VNCoreMLModel!

3.3 Configure and prepare the model

override func viewDidLoad() {
    super.viewDidLoad()

    visionModel = try? VNCoreMLModel(for: EstimationModel().model)
	request = VNCoreMLRequest(model: visionModel, completionHandler: visionRequestDidComplete)
	request.imageCropAndScaleOption = .scaleFill
}

func visionRequestDidComplete(request: VNRequest, error: Error?) {
    /* ------------------------------------------------------ */
    /* something postprocessing what you want after inference */
    /* ------------------------------------------------------ */
}

3.4 Inference 🏃‍♂️

// on the inference point
let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer)
try? handler.perform([request])

Performance Test

1. Import the model

You can download cpm or hourglass model for Core ML from tucan9389/pose-estimation-for-mobile repo.

2. Fix the model name on `PoseEstimation_CoreMLTests.swift`

3. Run the test

Hit the ⌘ + U or click the Build for Testing icon.

poseestimation-coreml's People

Contributors

Stargazers

Watchers

Forkers

zysyyz ashwinrajendraprasad lewisling jaswers liamwalsh jiqiaochun kycho broderickhigby agtlucas jacobwjs nkjzm abubakhr glavin001 younatics tarsbase bangslosan scotthavird tlhchutang noseeevil sergeimeza dkpoulsen amanmeet gaoyiyeah soccergame iamharshtrivedi daydreamer2023 wuyuebupt alianglan hammingcube dahburj xraycoder gonzalj3 haowang1013 macro-dadt nomardata albutko noticeable morsamatias swetharajagopalan17 hackcoin1 ashyanagisawa linhaosunny edvardhua zhongxingpeng trendingtechnology toddrimes ommmmooooo ponhan scaiado harshbshah yongjinleee daluethi kenstack tailongfei 8secz-johndpope weacw yak0xff johndpope jauharibill leegof samsgates abhijitdalavi diwakarthapa hongj77 irekzayniev thebridge hgoycoolea delort schliffen japita-se verebes1 sabuj-cse11 h-jia vivek-mittal hsmwfgie jacklongking sky9743 mohan158 madenmud codezakh longddsf lobster-king todosthing vicioux daijuting kimcaprio osh3516 johnren1111 pushkarravi vikramank herryliq fallending sblincov mumer92 rhaon-virtual-reality-sports-3d linkonbsmrstu pxlshpr ductringuyen justbcuz naturezhm

poseestimation-coreml's Issues

Pose Estimation for Core ML logo

Back camera not working in poseMatchingViewController

When I set up camera with back position it does not work in poseMatchingViewController

How to add normalization to the image in preprocess?

Hello

I need to add normalization to the image in the preprocess

This is how it's being done in python (From here):

    parser.add_argument('--std', type=list, default=[0.229, 0.224, 0.225],
                        help='the std used to normalize your images')
    parser.add_argument('--mean', type=list, default=[0.485, 0.456, 0.406],
                        help='the mean used to normalize your images')

    normalize = transforms.Normalize(std=args.std,
                                     mean=args.mean)

    transformation = transforms.Compose([transforms.ToTensor()])

    man_normalize = transformation(man_resize)

This is how it's being done in Android:

@Override
  | protected void addPixelValue(int pixelValue) {
  | imgData.putFloat((((pixelValue >> 16) & 0xFF)/255f - IMAGE_MEAN[0]) / IMAGE_STD[0]);
  | imgData.putFloat((((pixelValue >> 8) & 0xFF)/255f - IMAGE_MEAN[1]) / IMAGE_STD[1]);
  | imgData.putFloat(((pixelValue & 0xFF)/255f - IMAGE_MEAN[2]) / IMAGE_STD[2]);
  | }

How can I do similar thing in your Swift application?

Thanks

Understanding performance test

Hello. Thanks for the amazing repository!

I built the project and it works great.

I am trying to understand the performance test.
I would be glad if you can elaborate about what was measured.

Looking at Measure.swift:

func 🎬🤚() {
       🏷(for: index, with: "end")
       
       let beforeMeasurement = getBeforeMeasurment(for: index)
       let currentMeasurement = measurements[index]
       if let startTime = currentMeasurement["start"],
           let endInferenceTime = currentMeasurement["endInference"],
           let endTime = currentMeasurement["end"],
           let beforeStartTime = beforeMeasurement["start"] {
           delegate?.updateMeasure(inferenceTime: endInferenceTime - startTime,
                                   executionTime: endTime - startTime,
                                   fps: Int(1/(startTime - beforeStartTime)))
       }
       
   }

startTime is the moment that the image is received. It is a pixelBuffer, (640x480) image.
endInferenceTime is the moment when the inference was stopped, receiving a (14, 96, 96) heatmap from the CPM model.
beforeStartTime is the start time of the previous frame. Why do we measure it?

If so I understand what is:

inferenceTime: endInferenceTime - startTime - from pixelBuffer, (640x480) image to (14, 96, 96) heatmap .
executionTime: endTime - startTime - from pixelBuffer, (640x480), including post-process (convertToPredictedPoints, and moving average filter. BTW - what is this filter?)

But I don't understand what is fps: Int(1/(startTime - beforeStartTime))) - why is the beforeStartTime relevant?

To summarize, my questions are:

Did I understand correctly the inferenceTime and executionTime?
Why is the fps calculated with beforeStartTime?
Why do you have moving average filter in the post-process code? Is it instead the gaussian_filter in the original CPM code?

Thank you very much.

Minimal example for image inference

Hello

I am trying to use other model and it gets complicated.

I was wondering whether someone haev minimal example on how to run inference on existing image from memory? This way it will be simpler to understand how to adapt different model.

Thanks for the repo!

What is the coordinate space used for the model output MultiArray?

Per the above image, in the core ml model, the output is specified as a multi array and I am wondering which coordinate space is being used for this output since iOS has different spaces like ULO and LLO.

Performance Test

Model Size (MB), Minimum iOS Version

Model	Size	Minimum iOS Version
cpm	2.6	iOS11
hourhglass	2	iOS11

Infernece Time (ms)

Model vs. Device	XS	XS Max	XR	X	8	8+	7	7+	6S+	6+
cpm	-	27	27	32	31	31	39	37	44	115
hourhglass	-	6	7	29	31	32	37	42	48	94

Total Time (ms)

Model vs. Device	XS	XS Max	XR	X	8	8+	7	7+	6S+	6+
cpm	-	39	40	46	47	45	55	58	56	139
hourhglass	-	15	15	38	40	40	48	55	58	106

FPS

Model vs. Device	XS	XS Max	XR	X	8	8+	7	7+	6S+	6+
cpm	-	23	23	20	20	21	17	16	16	6
hourhglass	-	23	23	24	23	23	19	16	15	8

Support apple's posenet.mlmodel

https://developer.apple.com/machine-learning/models/

Separate the human pose estimation domain dependency from the project

The repo's code is coupled with human pose estimation output format(assume 14 keypoints output and draw the static fixed joint point). We need to change to be able to configure the keypoints number and joint info.

remove fixed 14 keypoints output format
remove fixed joint relationships

Why did you add 0.5 when transpose predicted point to (1.0, 1.0)? What is 0.5?

How to add another CoreML model? Detection output feed to another CoreML model (GAN)?

Hi, my group is trying to plug in another CoreML model into this app (a GAN model to generate new targets with same poses, paper: everybody dance now). However, we are all newbies with Swift, and we struggled a lot and still could not do it. Basically we just want to feed the skeletons into the new model and display the generated image/video from GAN.

I am wondering do you have any advice as for which part should we modify/add/remove and etc? We would much appreciate your help, thank you so much!

ps: We are working on Swift version of shufflenet-like OpenPose 17 keypoint detection and would love to merge it after everything is finished!

Can't build project

Keep running into

Unknown class _TtC21PoseEstimation_CoreML19JointViewController in Interface Builder file.

Error

Tried running on Iphone 6 and Iphone 11 simulators

Support objective-C

Need some suggestion in library

Hello @tucan9389 @taehyen ,

Thanks for creating wonderful library for pose detection. But I need to know that can we draw dots only selected part of body instead full body while full body visible in front of camera.

Thanks and Regards

use of undeclared type 'model_cpm'

Support PCKh and compare to each model(PoseEstimationForMobile, PoseNet, Fritz's pose estimation, ARKit3's body tracking etc.)

plz give me any good idea 😎

Dataset

COCO 2017 dataset? (19GB.. OMG)

Metric

[email protected]
Inference time for each device

Devices

iPhone XS, iPhone XS Max
iPhone X, iPhone 8, iPhone 8+
iPad...

Fever problem

My iphone's back is very hot when i run this demo after 10 seconds,
Could you tell me how to solve this problem?
(sorry, my english is not very well, Please forgive my my words)

GIF image upload issue

joint demo	heatmap demo

This gif will be uploaded on README

How show in front camera?

I am tried this project. I am using with front camera, but when i move left hand, the right hand of model moves. It is like mirror. How to fix it?

Need Help with pose capturing and pose matching technique

hello @tucan9389 i'm flutter developer i don't understand swift language and i just want to know about how you captured the poses with only skeleton drawn and compared it with live feed i'm using different posenet moel here please guide me with this.

Since this is not an issue so you can close this and please reply here email : [email protected]

Thanks :)

Quantization test

Need measure accuracy(PCKh) in mobile.

퀀타이제이션을 사용했을 때 얼마나 성능이 떨어지는지 확인하기 위해.
Need to check the accuracy when use Quantization of Core ML 2.

Implement moving average filter

Separate Measure.swift module from project

Hourglass model

Try to use hourglass model in here.

input_image_shape: [1, 192, 192, 3]
output_shape: [1, 48, 48, 14]
input_node_name: image
output_node_name: hourglass_out_3

I think it will speed up 3x than old model and the precision is acceptable.

Tomorrow I will update my repo.

Look forward to your replay ( ? FPS on ipx )

Support the model having points array output

The current project assumes that the output of the model is the heatmaps format(Array<Array<Array<Double>>>). I want to support the points array output format(Array<CGPoint>).

Pretrained CPM model output resolution is only 24x24

Please see my issue at: edvardHua/PoseEstimationForMobile#61

For the pretrained CPM model downloaded here, the resolution of keypoint prediction appears to be only 1/4th of the expected 96x96.

I think this is related to the neural network, which is why I posted it to the PoseEstimationForMobile repo. However, I could be wrong. Is there something I'm missing?

Inferencing with .pb file ?

The model folder contains protobuf, coreml and tflite files for both hourglass and cpm.
Should I expect the model performance to be similar in all three files?
I yes, is there an inference code to test the .pb file (or any other) in Colab and visualise the results ?

Is the model latest？ How do I update it?

Hi, I try the project,it is very good.
But, I found that it spend 51ms on my iPhoneX, not 27ms.
So,what is the problem?
Should you update the model to latest, and how?

Implement heatmap tap

Use Metal texture when drawing heatmaps

Include toe keypoint - Body-Foot Estimation

Hello!
Is there any way to include the foot (toe keypoints) in your model or are they excluded for a reason? I noticed in the OpenPose sample here:
https://github.com/CMU-Perceptual-Computing-Lab/openpose

There is a toe keypoint but I am unsure how to convert that model to CoreML.
If it can be included in your current model that would be amazing!

@tucan9389 any insight on this would be greatly appreciated !

lightweight-human-pose-estimation-3d-demo.pytorch

repo: https://github.com/Daniil-Osokin/lightweight-human-pose-estimation-3d-demo.pytorch

Models

pytorch model: human-pose-estimation-3d.pth.zip
onnx model: human-pose-estimation-3d.onnx.zip
mlmodel: human-pose-estimation-3d.mlmodel.zip

Metadata

input: a 256x448 color image
outputs:
- features:
- heatmaps:
- pafs:

Parsing Reference

Support cocoapods

Wrapper library or helper library?

Interface

// for use
typealias InferencePoint = (CGFloat, CGPoint) // maximum confidence, position of maximum confidence
func convert(result: MLMultiArray) -> [InferencePoint]

// for debug
class HeatmapView: UIView { }
class JointView: UIView { }

Use GPU on post-processing like argmax

I'm having trouble trying to test your code

hi i was searching for a pose estimation code for ios and i found that your demos were great

and i downloaded your code and doesn't work

here's what i've done

as you instructed i downloaded models and put them added cpm_model/model.mlmode as model_cpm.mlmodel and hourglass_model/model.mlmodel as model_hourglass.model

and

even though i have errors i could run the simulator and i got

2019-07-09 15:36:26.736876+0900 PoseEstimation-CoreML[4719:560970] libMobileGestalt MobileGestalt.c:890: MGIsDeviceOneOfType is not supported on this platform.
2019-07-09 15:36:27.116772+0900 PoseEstimation-CoreML[4719:560970] [espresso] [Espresso::handle_ex_] exception=Unsupported engine type
Error: no video devices available
2019-07-09 15:36:27.205463+0900 PoseEstimation-CoreML[4719:560970] *** Terminating app due to uncaught exception 'NSGenericException', reason: '*** -[AVCaptureSession startRunning] startRunning may not be called between calls to beginConfiguration and commitConfiguration'
*** First throw call stack:
(
	0   CoreFoundation                      0x0000000104cdb6fb __exceptionPreprocess + 331
	1   libobjc.A.dylib                     0x0000000103e44ac5 objc_exception_throw + 48
	2   AVFoundation                        0x00000001047b4bed -[AVCaptureSession startRunning] + 206
	3   PoseEstimation-CoreML               0x00000001034e886c $s21PoseEstimation_CoreML12VideoCaptureC5startyyF + 140
	4   PoseEstimation-CoreML               0x00000001034dbec7 $s21PoseEstimation_CoreML19JointViewControllerC14viewWillAppearyySbF + 391
	5   PoseEstimation-CoreML               0x00000001034dbf13 $s21PoseEstimation_CoreML19JointViewControllerC14viewWillAppearyySbFTo + 51
	6   UIKitCore                           0x000000010fde2437 -[UIViewController _setViewAppearState:isAnimating:] + 687
	7   UIKitCore                           0x000000010fde2ba2 -[UIViewController __viewWillAppear:] + 131
	8   UIKitCore                           0x000000010fd092fe -[UITabBarController viewWillAppear:] + 293
	9   UIKitCore                           0x000000010fde2437 -[UIViewController _setViewAppearState:isAnimating:] + 687
	10  UIKitCore                           0x000000010fde2ba2 -[UIViewController __viewWillAppear:] + 131
	11  UIKitCore                           0x000000010fde438d -[UIViewController viewWillMoveToWindow:] + 696
	12  UIKitCore                           0x0000000110895195 -[UIView(Hierarchy) _willMoveToWindow:withAncestorView:] + 656
	13  UIKitCore                           0x00000001108a6069 -[UIView(Internal) _addSubview:positioned:relativeTo:] + 456
	14  UIKitCore                           0x0000000110414fbc -[UIWindow addRootViewControllerViewIfPossible] + 1027
	15  UIKitCore                           0x0000000110415327 -[UIWindow _setHidden:forced:] + 289
	16  UIKitCore                           0x0000000110427f86 -[UIWindow makeKeyAndVisible] + 42
	17  UIKitCore                           0x00000001103d7f1c -[UIApplication _callInitializationDelegatesForMainScene:transitionContext:] + 4555
	18  UIKitCore                           0x00000001103dd0c6 -[UIApplication _runWithMainScene:transitionContext:completion:] + 1617
	19  UIKitCore                           0x000000010fc226d6 __111-[__UICanvasLifecycleMonitor_Compatability _scheduleFirstCommitForScene:transition:firstActivation:completion:]_block_invoke + 904
	20  UIKitCore                           0x000000010fc2afce +[_UICanvas _enqueuePostSettingUpdateTransactionBlock:] + 153
	21  UIKitCore                           0x000000010fc222ec -[__UICanvasLifecycleMonitor_Compatability _scheduleFirstCommitForScene:transition:firstActivation:completion:] + 236
	22  UIKitCore                           0x000000010fc22c48 -[__UICanvasLifecycleMonitor_Compatability activateEventsOnly:withContext:completion:] + 1091
	23  UIKitCore                           0x000000010fc20fba __82-[_UIApplicationCanvas _transitionLifecycleStateWithTransitionContext:completion:]_block_invoke + 782
	24  UIKitCore                           0x000000010fc20c71 -[_UIApplicationCanvas _transitionLifecycleStateWithTransitionContext:completion:] + 433
	25  UIKitCore                           0x000000010fc259b6 __125-[_UICanvasLifecycleSettingsDiffAction performActionsForCanvas:withUpdatedScene:settingsDiff:fromSettings:transitionContext:]_block_invoke + 576
	26  UIKitCore                           0x000000010fc26610 _performActionsWithDelayForTransitionContext + 100
	27  UIKitCore                           0x000000010fc2571d -[_UICanvasLifecycleSettingsDiffAction performActionsForCanvas:withUpdatedScene:settingsDiff:fromSettings:transitionContext:] + 223
	28  UIKitCore                           0x000000010fc2a6d0 -[_UICanvas scene:didUpdateWithDiff:transitionContext:completion:] + 392
	29  UIKitCore                           0x00000001103db9a8 -[UIApplication workspace:didCreateScene:withTransitionContext:completion:] + 514
	30  UIKitCore                           0x000000010ff92dfa -[UIApplicationSceneClientAgent scene:didInitializeWithEvent:completion:] + 361
	31  FrontBoardServices                  0x000000010d43b125 -[FBSSceneImpl _didCreateWithTransitionContext:completion:] + 448
	32  FrontBoardServices                  0x000000010d444ed6 __56-[FBSWorkspace client:handleCreateScene:withCompletion:]_block_invoke_2 + 283
	33  FrontBoardServices                  0x000000010d444700 __40-[FBSWorkspace _performDelegateCallOut:]_block_invoke + 53
	34  libdispatch.dylib                   0x0000000108910db5 _dispatch_client_callout + 8
	35  libdispatch.dylib                   0x00000001089142ba _dispatch_block_invoke_direct + 300
	36  FrontBoardServices                  0x000000010d476146 __FBSSERIALQUEUE_IS_CALLING_OUT_TO_A_BLOCK__ + 30
	37  FrontBoardServices                  0x000000010d475dfe -[FBSSerialQueue _performNext] + 451
	38  FrontBoardServices                  0x000000010d476393 -[FBSSerialQueue _performNextFromRunLoopSource] + 42
	39  CoreFoundation                      0x0000000104c42be1 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
	40  CoreFoundation                      0x0000000104c42463 __CFRunLoopDoSources0 + 243
	41  CoreFoundation                      0x0000000104c3cb1f __CFRunLoopRun + 1231
	42  CoreFoundation                      0x0000000104c3c302 CFRunLoopRunSpecific + 626
	43  GraphicsServices                    0x000000010ac262fe GSEventRunModal + 65
	44  UIKitCore                           0x00000001103deba2 UIApplicationMain + 140
	45  PoseEstimation-CoreML               0x00000001034fdf4b main + 75
	46  libdyld.dylib                       0x0000000108985541 start + 1
	47  ???                                 0x0000000000000001 0x0 + 1
)
libc++abi.dylib: terminating with uncaught exception of type NSException
(lldb)

can you help me how to launch your project ... thank you

AVLayerVideoGravity.resizeAspectFill

if I set AVLayerVideoGravity.resizeAspectFill，then pose is wrong; what can I do ?please help me!

tucan9389 / poseestimation-coreml Goto Github PK

poseestimation-coreml's Introduction

Features

How it works

Requirements

Model

Get PoseEstimationForMobile's model

Model Size, Minimum iOS Version, Input/Output Shape

Infernece Time (ms)

Total Time (ms)

FPS

Get your own model

Build & Run

1. Prerequisites

1.1 Import pose estimation model

1.2 Add permission in info.plist for device's camera access

2. Dependencies

3. Code

3.1 Import Vision framework

3.2 Define properties for Core ML

3.3 Configure and prepare the model

3.4 Inference 🏃‍♂️

Performance Test

1. Import the model

2. Fix the model name on PoseEstimation_CoreMLTests.swift

3. Run the test

See also

poseestimation-coreml's People

Contributors

Stargazers

Watchers

Forkers

poseestimation-coreml's Issues

Model Size (MB), Minimum iOS Version

Infernece Time (ms)

Total Time (ms)

FPS

Dataset

Metric

Devices

lightweight-human-pose-estimation-3d-demo.pytorch

Models

Metadata

Parsing Reference

Interface

Recommend Projects

Recommend Topics

Recommend Org

2. Fix the model name on `PoseEstimation_CoreMLTests.swift`