tucan9389 / poseestimation-coreml Goto Github PK
View Code? Open in Web Editor NEWThe example project of inferencing Pose Estimation using Core ML
Home Page: https://github.com/motlabs/awesome-ml-demos-with-ios
License: MIT License
The example project of inferencing Pose Estimation using Core ML
Home Page: https://github.com/motlabs/awesome-ml-demos-with-ios
License: MIT License
hello @tucan9389 i'm flutter developer i don't understand swift language and i just want to know about how you captured the poses with only skeleton drawn and compared it with live feed i'm using different posenet moel here please guide me with this.
Since this is not an issue so you can close this and please reply here email : [email protected]
Thanks :)
hi i was searching for a pose estimation code for ios and i found that your demos were great
and i downloaded your code and doesn't work
here's what i've done
as you instructed i downloaded models and put them added cpm_model/model.mlmode as model_cpm.mlmodel and hourglass_model/model.mlmodel as model_hourglass.model
even though i have errors i could run the simulator and i got
2019-07-09 15:36:26.736876+0900 PoseEstimation-CoreML[4719:560970] libMobileGestalt MobileGestalt.c:890: MGIsDeviceOneOfType is not supported on this platform.
2019-07-09 15:36:27.116772+0900 PoseEstimation-CoreML[4719:560970] [espresso] [Espresso::handle_ex_] exception=Unsupported engine type
Error: no video devices available
2019-07-09 15:36:27.205463+0900 PoseEstimation-CoreML[4719:560970] *** Terminating app due to uncaught exception 'NSGenericException', reason: '*** -[AVCaptureSession startRunning] startRunning may not be called between calls to beginConfiguration and commitConfiguration'
*** First throw call stack:
(
0 CoreFoundation 0x0000000104cdb6fb __exceptionPreprocess + 331
1 libobjc.A.dylib 0x0000000103e44ac5 objc_exception_throw + 48
2 AVFoundation 0x00000001047b4bed -[AVCaptureSession startRunning] + 206
3 PoseEstimation-CoreML 0x00000001034e886c $s21PoseEstimation_CoreML12VideoCaptureC5startyyF + 140
4 PoseEstimation-CoreML 0x00000001034dbec7 $s21PoseEstimation_CoreML19JointViewControllerC14viewWillAppearyySbF + 391
5 PoseEstimation-CoreML 0x00000001034dbf13 $s21PoseEstimation_CoreML19JointViewControllerC14viewWillAppearyySbFTo + 51
6 UIKitCore 0x000000010fde2437 -[UIViewController _setViewAppearState:isAnimating:] + 687
7 UIKitCore 0x000000010fde2ba2 -[UIViewController __viewWillAppear:] + 131
8 UIKitCore 0x000000010fd092fe -[UITabBarController viewWillAppear:] + 293
9 UIKitCore 0x000000010fde2437 -[UIViewController _setViewAppearState:isAnimating:] + 687
10 UIKitCore 0x000000010fde2ba2 -[UIViewController __viewWillAppear:] + 131
11 UIKitCore 0x000000010fde438d -[UIViewController viewWillMoveToWindow:] + 696
12 UIKitCore 0x0000000110895195 -[UIView(Hierarchy) _willMoveToWindow:withAncestorView:] + 656
13 UIKitCore 0x00000001108a6069 -[UIView(Internal) _addSubview:positioned:relativeTo:] + 456
14 UIKitCore 0x0000000110414fbc -[UIWindow addRootViewControllerViewIfPossible] + 1027
15 UIKitCore 0x0000000110415327 -[UIWindow _setHidden:forced:] + 289
16 UIKitCore 0x0000000110427f86 -[UIWindow makeKeyAndVisible] + 42
17 UIKitCore 0x00000001103d7f1c -[UIApplication _callInitializationDelegatesForMainScene:transitionContext:] + 4555
18 UIKitCore 0x00000001103dd0c6 -[UIApplication _runWithMainScene:transitionContext:completion:] + 1617
19 UIKitCore 0x000000010fc226d6 __111-[__UICanvasLifecycleMonitor_Compatability _scheduleFirstCommitForScene:transition:firstActivation:completion:]_block_invoke + 904
20 UIKitCore 0x000000010fc2afce +[_UICanvas _enqueuePostSettingUpdateTransactionBlock:] + 153
21 UIKitCore 0x000000010fc222ec -[__UICanvasLifecycleMonitor_Compatability _scheduleFirstCommitForScene:transition:firstActivation:completion:] + 236
22 UIKitCore 0x000000010fc22c48 -[__UICanvasLifecycleMonitor_Compatability activateEventsOnly:withContext:completion:] + 1091
23 UIKitCore 0x000000010fc20fba __82-[_UIApplicationCanvas _transitionLifecycleStateWithTransitionContext:completion:]_block_invoke + 782
24 UIKitCore 0x000000010fc20c71 -[_UIApplicationCanvas _transitionLifecycleStateWithTransitionContext:completion:] + 433
25 UIKitCore 0x000000010fc259b6 __125-[_UICanvasLifecycleSettingsDiffAction performActionsForCanvas:withUpdatedScene:settingsDiff:fromSettings:transitionContext:]_block_invoke + 576
26 UIKitCore 0x000000010fc26610 _performActionsWithDelayForTransitionContext + 100
27 UIKitCore 0x000000010fc2571d -[_UICanvasLifecycleSettingsDiffAction performActionsForCanvas:withUpdatedScene:settingsDiff:fromSettings:transitionContext:] + 223
28 UIKitCore 0x000000010fc2a6d0 -[_UICanvas scene:didUpdateWithDiff:transitionContext:completion:] + 392
29 UIKitCore 0x00000001103db9a8 -[UIApplication workspace:didCreateScene:withTransitionContext:completion:] + 514
30 UIKitCore 0x000000010ff92dfa -[UIApplicationSceneClientAgent scene:didInitializeWithEvent:completion:] + 361
31 FrontBoardServices 0x000000010d43b125 -[FBSSceneImpl _didCreateWithTransitionContext:completion:] + 448
32 FrontBoardServices 0x000000010d444ed6 __56-[FBSWorkspace client:handleCreateScene:withCompletion:]_block_invoke_2 + 283
33 FrontBoardServices 0x000000010d444700 __40-[FBSWorkspace _performDelegateCallOut:]_block_invoke + 53
34 libdispatch.dylib 0x0000000108910db5 _dispatch_client_callout + 8
35 libdispatch.dylib 0x00000001089142ba _dispatch_block_invoke_direct + 300
36 FrontBoardServices 0x000000010d476146 __FBSSERIALQUEUE_IS_CALLING_OUT_TO_A_BLOCK__ + 30
37 FrontBoardServices 0x000000010d475dfe -[FBSSerialQueue _performNext] + 451
38 FrontBoardServices 0x000000010d476393 -[FBSSerialQueue _performNextFromRunLoopSource] + 42
39 CoreFoundation 0x0000000104c42be1 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
40 CoreFoundation 0x0000000104c42463 __CFRunLoopDoSources0 + 243
41 CoreFoundation 0x0000000104c3cb1f __CFRunLoopRun + 1231
42 CoreFoundation 0x0000000104c3c302 CFRunLoopRunSpecific + 626
43 GraphicsServices 0x000000010ac262fe GSEventRunModal + 65
44 UIKitCore 0x00000001103deba2 UIApplicationMain + 140
45 PoseEstimation-CoreML 0x00000001034fdf4b main + 75
46 libdyld.dylib 0x0000000108985541 start + 1
47 ??? 0x0000000000000001 0x0 + 1
)
libc++abi.dylib: terminating with uncaught exception of type NSException
(lldb)
can you help me how to launch your project ... thank you
Hello
I need to add normalization to the image in the preprocess
This is how it's being done in python (From here):
parser.add_argument('--std', type=list, default=[0.229, 0.224, 0.225],
help='the std used to normalize your images')
parser.add_argument('--mean', type=list, default=[0.485, 0.456, 0.406],
help='the mean used to normalize your images')
normalize = transforms.Normalize(std=args.std,
mean=args.mean)
transformation = transforms.Compose([transforms.ToTensor()])
man_normalize = transformation(man_resize)
This is how it's being done in Android:
@Override
| protected void addPixelValue(int pixelValue) {
| imgData.putFloat((((pixelValue >> 16) & 0xFF)/255f - IMAGE_MEAN[0]) / IMAGE_STD[0]);
| imgData.putFloat((((pixelValue >> 8) & 0xFF)/255f - IMAGE_MEAN[1]) / IMAGE_STD[1]);
| imgData.putFloat(((pixelValue & 0xFF)/255f - IMAGE_MEAN[2]) / IMAGE_STD[2]);
| }
How can I do similar thing in your Swift application?
Thanks
Hello
I am trying to use other model and it gets complicated.
I was wondering whether someone haev minimal example on how to run inference on existing image from memory? This way it will be simpler to understand how to adapt different model.
Thanks for the repo!
The repo's code is coupled with human pose estimation output format(assume 14 keypoints output and draw the static fixed joint point). We need to change to be able to configure the keypoints number and joint info.
My iphone's back is very hot when i run this demo after 10 seconds,
Could you tell me how to solve this problem?
(sorry, my english is not very well, Please forgive my my words)
퀀타이제이션을 사용했을 때 얼마나 성능이 떨어지는지 확인하기 위해.
Need to check the accuracy when use Quantization of Core ML 2.
Wrapper library or helper library?
// for use
typealias InferencePoint = (CGFloat, CGPoint) // maximum confidence, position of maximum confidence
func convert(result: MLMultiArray) -> [InferencePoint]
// for debug
class HeatmapView: UIView { }
class JointView: UIView { }
Hello. Thanks for the amazing repository!
I built the project and it works great.
I am trying to understand the performance test.
I would be glad if you can elaborate about what was measured.
Looking at Measure.swift
:
func 🎬🤚() {
🏷(for: index, with: "end")
let beforeMeasurement = getBeforeMeasurment(for: index)
let currentMeasurement = measurements[index]
if let startTime = currentMeasurement["start"],
let endInferenceTime = currentMeasurement["endInference"],
let endTime = currentMeasurement["end"],
let beforeStartTime = beforeMeasurement["start"] {
delegate?.updateMeasure(inferenceTime: endInferenceTime - startTime,
executionTime: endTime - startTime,
fps: Int(1/(startTime - beforeStartTime)))
}
}
startTime
is the moment that the image is received. It is a pixelBuffer
, (640x480) image.
endInferenceTime
is the moment when the inference was stopped, receiving a (14, 96, 96)
heatmap from the CPM model.
beforeStartTime
is the start time of the previous frame. Why do we measure it?
If so I understand what is:
inferenceTime: endInferenceTime - startTime
- from pixelBuffer
, (640x480) image to (14, 96, 96)
heatmap .executionTime: endTime - startTime
- from pixelBuffer
, (640x480), including post-process (convertToPredictedPoints
, and moving average filter. BTW - what is this filter?)But I don't understand what is fps: Int(1/(startTime - beforeStartTime)))
- why is the beforeStartTime
relevant?
To summarize, my questions are:
inferenceTime
and executionTime
?fps
calculated with beforeStartTime
?gaussian_filter
in the original CPM code?Thank you very much.
Try to use hourglass model in here.
I think it will speed up 3x than old model and the precision is acceptable.
Tomorrow I will update my repo.
Look forward to your replay ( ? FPS on ipx )
Hello @tucan9389 @taehyen ,
Thanks for creating wonderful library for pose detection. But I need to know that can we draw dots only selected part of body instead full body while full body visible in front of camera.
Thanks and Regards
Keep running into
Unknown class _TtC21PoseEstimation_CoreML19JointViewController in Interface Builder file.
Error
Tried running on Iphone 6 and Iphone 11 simulators
I am tried this project. I am using with front camera, but when i move left hand, the right hand of model moves. It is like mirror. How to fix it?
The model folder contains protobuf, coreml and tflite files for both hourglass and cpm.
Should I expect the model performance to be similar in all three files?
I yes, is there an inference code to test the .pb file (or any other) in Colab and visualise the results ?
Hello!
Is there any way to include the foot (toe keypoints) in your model or are they excluded for a reason? I noticed in the OpenPose sample here:
https://github.com/CMU-Perceptual-Computing-Lab/openpose
There is a toe keypoint but I am unsure how to convert that model to CoreML.
If it can be included in your current model that would be amazing!
@tucan9389 any insight on this would be greatly appreciated !
Please add model_cpm.mlmodel or hourglass.mlmodel. I can`t find it anywhere.
Please see my issue at: edvardHua/PoseEstimationForMobile#61
For the pretrained CPM model downloaded here, the resolution of keypoint prediction appears to be only 1/4th of the expected 96x96.
I think this is related to the neural network, which is why I posted it to the PoseEstimationForMobile repo. However, I could be wrong. Is there something I'm missing?
Hi, my group is trying to plug in another CoreML model into this app (a GAN model to generate new targets with same poses, paper: everybody dance now). However, we are all newbies with Swift, and we struggled a lot and still could not do it. Basically we just want to feed the skeletons into the new model and display the generated image/video from GAN.
I am wondering do you have any advice as for which part should we modify/add/remove and etc? We would much appreciate your help, thank you so much!
ps: We are working on Swift version of shufflenet-like OpenPose 17 keypoint detection and would love to merge it after everything is finished!
repo: https://github.com/Daniil-Osokin/lightweight-human-pose-estimation-3d-demo.pytorch
Model | Size | Minimum iOS Version |
---|---|---|
cpm | 2.6 | iOS11 |
hourhglass | 2 | iOS11 |
Model vs. Device | XS | XS Max |
XR | X | 8 | 8+ | 7 | 7+ | 6S+ | 6+ |
---|---|---|---|---|---|---|---|---|---|---|
cpm | - | 27 | 27 | 32 | 31 | 31 | 39 | 37 | 44 | 115 |
hourhglass | - | 6 | 7 | 29 | 31 | 32 | 37 | 42 | 48 | 94 |
Model vs. Device | XS | XS Max |
XR | X | 8 | 8+ | 7 | 7+ | 6S+ | 6+ |
---|---|---|---|---|---|---|---|---|---|---|
cpm | - | 39 | 40 | 46 | 47 | 45 | 55 | 58 | 56 | 139 |
hourhglass | - | 15 | 15 | 38 | 40 | 40 | 48 | 55 | 58 | 106 |
Model vs. Device | XS | XS Max |
XR | X | 8 | 8+ | 7 | 7+ | 6S+ | 6+ |
---|---|---|---|---|---|---|---|---|---|---|
cpm | - | 23 | 23 | 20 | 20 | 21 | 17 | 16 | 16 | 6 |
hourhglass | - | 23 | 23 | 24 | 23 | 23 | 19 | 16 | 15 | 8 |
When I set up camera with back position it does not work in poseMatchingViewController
use of undeclared type 'model_cpm'
Run Joint View sample in Xcode 10.3, the FPS is 15 ~ 20, But run same project in Xcode 11.3, the FPS dropdown to 6~7, my device is iPhone 7, Is anything setting I need to change ?
Hi, I try the project,it is very good.
But, I found that it spend 51ms on my iPhoneX, not 27ms.
So,what is the problem?
Should you update the model to latest, and how?
plz give me any good idea 😎
The current project assumes that the output of the model is the heatmaps format(Array<Array<Array<Double>>>
). I want to support the points array output format(Array<CGPoint>
).
if I set AVLayerVideoGravity.resizeAspectFill,then pose is wrong; what can I do ?please help me!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.