Giter VIP home page Giter VIP logo

stagesepx's Introduction

stage sep(aration) x

detect stages in video automatically


Type Status
package version PyPI version
python version PyPI - Python Version
auto test CI Status
code maintainability Maintainability
code coverage codecov
code style Code style: black
stat Downloads Downloads Downloads

2022/08/13:在 0.18.0 之后,ffmpeg 以 imageio_ffmpeg 方式内置(请自行评估其LICENSE影响)。此版本为解决opencv版本及M1芯片的限制,并不再需要额外安装ffmpeg。详见 #178

2022/03/30:官方不维护任何诸如微信群、QQ群的多人群组,所有信息请通过issue公开交流。不需要任何捐赠支持,如遇到涉及金钱相关的信息请不要相信。

2021/12/15:在 0.16.0 之后,stagesepx 将开始提供部分功能测试的支持,详见 #158

2020/05/21:目前,该项目已经达到一个较为稳定的状态,并顺利在多家头部公司、团队落地,作为正式工具存在。Bug跟踪与建议请优先通过 issue 联系我,感谢所有支持过这个项目的人。欢迎有心优化的同学、落地成功的团队共同建设:)


English README here

这段视频展示了一个应用的完整启动过程:

video_readme.gif

将视频传递给 stagesepx,它将自动分析拆解,得到视频中所有的阶段。包括变化的过程及其耗时,以及在稳定的阶段停留的时长:

taobao_startup.png

你可以据此得到每个阶段对应的精确耗时。

跨端运作

当然,它是天然跨端的,例如web端。甚至,任何端:

sugar.gif

sugar

高准确度

与视频一致的高准确度。以秒表为例:

accuracy.png

可以看到,与秒表的表现几乎没有差异。请注意,这里的准确度指的是 stagesepx 能够精确还原视频本身的数据与表现。而对于现象(例如某某时间点出现什么状态)而言,准确度很大程度上取决于视频本身,如fps/分辨率等。

彻底解耦 & 可编程

如果比起报告,更希望亲自处理原始数据,进而进行二次开发,你可以直接将 report 部分去除。如此做,你将得到一个 python 对象供你随意使用。它提供了大量的API,例如转换成字典:

{
	"data": [{
		"data": null,
		"frame_id": 1,
		"stage": "0",
		"timestamp": 0.0,
		"video_path": "../demo.mp4"
	}, {
		"data": null,
		"frame_id": 2,
		"stage": "0",
		"timestamp": 0.04,
		"video_path": "../demo.mp4"
	}, {
		"data": null,
		"frame_id": 3,
		"stage": "0",
		"timestamp": 0.08,
		"video_path": "../demo.mp4"
	}, {
	
  ...

从这个字典中我们可以知道,每一帧分别对应的:

  • 被分类到哪一个类别
  • 时间戳
  • 帧编号
  • ...

用户可以随意处理这些数据,无论是保存或是交给下一段代码。

完整自动化支持 & 规模化

  • 既然它是可编程的,那么它必然是朝着彻底替代人力的方向演进的。这也是它最强大的特性;
  • 它允许用户利用自己的训练集进行模型训练,利用神经网络进行规模化、全自动化的特定阶段耗时计算;
  • 此方案能够被广泛应用到各类业务迭代中,与持续集成配合,有效降低人力消耗;
  • 一些方向参考:
    • 为你的应用建立高频次的性能回归测试,形成benchmark
    • 对模型进行补足,为一系列同类应用(如小程序、小游戏,etc.)构建巡检能力
    • ...

具体可参见 将 stagesepx 应用到实际业务中


  • 标准模式下无需前置训练与学习
  • 更少的代码需要
  • 高度可配置化,适应不同场景
  • 支持与其他框架结合,融入你的业务
  • 所有你需要的,只是一个视频

开始

正式使用

在正式落地时,推荐使用 完整的python脚本 而不是命令行,以保证更高的可编程性。完整的落地例子另外单独开了一个 repo 存放,传送门。 请一定配合 这篇文章 使用,基本能解决90%的问题。

命令行

你也可以直接通过命令行使用,而无需编写脚本:

stagesepx analyse your_video.mp4 report.html

基于此,你可以非常方便地利用 shell 建立工作流。以 android 为例:

adb shell screenrecord --time-limit 10 /sdcard/demo.mp4
adb pull /sdcard/demo.mp4 .
stagesepx analyse demo.mp4 report.html

关于结果不准确的问题请参考 #46

配置化运行(0.15.0)

当然,通常因为场景差异,我们需要对参数进行修改使其达到更好的效果。这使得用户需要投入一些精力在脚本编写上。在 0.15.0 之后,配置化运行的加入使用户能够在不需要编写脚本的情况下直接使用所有能力,大大降低了接入门槛。

{
  "output": ".",
  "video": {
    "path": "./PATH_TO_YOUR/VIDEO.mp4",
    "fps": 30
  }
}

命令行运行:

stagesepx run YOUR_CONFIG.json

即可达到与脚本相同的效果。其他的配置项可以参考:work_with_stagesepx

安装

标准版(pypi)

pip install stagesepx

预览版(github):

pip install --upgrade git+https://github.com/williamfzc/stagesepx.git

常见问题

最终我还是决定通过 issue 面板维护所有的 Q&A ,毕竟问题的提出与回复是一个强交互过程。如果在查看下列链接之后你的问题依旧没有得到解答:

  • 新建issue
  • 或在相关的 issue 下进行追问与补充
  • 你的提问将不止帮助到你一个人 :)

问题列表:

不仅是问题,如果有任何建议与交流想法,同样可以通过 issue 面板找到我。我们每天都会查看 issue 面板,无需担心跟进不足。

相关文章

架构

structure

参与项目

规划

在 1.0版本 之前,我们接下来的工作主要分为下面几个部分:

标准化

随着越来越多的业务落地,我们开始思考它是否能够作为行业级别的方案。

  • 基于实验室数据的准确度对比(未公开)
  • 规范且适合落地的例子
  • 边界情况下的确认
  • 代码覆盖率 95%+
  • API参数相关文档

新需求的收集与开发

该部分由 issue 面板管理。

贡献代码

欢迎感兴趣的同学为这个项目添砖加瓦,三个必备步骤:

  • 请在开始编码前留个 issue 告知你想完成的功能,因为可能这个功能已经在开发中或者已有;
  • commit规范我们严格遵守 约定式提交
  • 该repo有较为完善的单测与CI以保障整个项目的质量,在过去的迭代中发挥了巨大的作用。所以请为你新增的代码同步新增单元测试(具体写法请参考 tests 中的已有用例)。

联系我们

Changelog / History

see CHANGELOG.md

Thanks

Thank you JetBrains for supporting the project with free product licenses.

License

MIT

stagesepx's People

Contributors

dependabot-preview[bot] avatar mutexz avatar williamfzc avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

stagesepx's Issues

hook option: change the origin frame or not

目前的 hook 不会对原始的 frame 造成影响,也不会影响 cut 与 classify 的过程。如果该功能作为一个选项出现,hook将可以做到更多的事情(例如自定义的图片放缩、模糊与锐化等等)。

当多个阶段有重复时,分类会有问题

例如:

  • 从桌面进入应用
  • 从应用返回桌面

此时,阶段1与阶段3的表现是一致的(都是桌面),如果用这样的训练结果去分析视频可能会出现不准确的情况。

compare_videos.py 对比2个视频相似度时,脚本报错了

Traceback (most recent call last): File "D:/pythonworkpace/stagesepx/example/compare_videos.py", line 13, in <module> stable, _ = res.get_range(limit=3) File "D:\pythonworkpace\stagesepx\stagesepx\cutter.py", line 197, in get_range unstable_range_list = self.get_unstable_range(limit, **kwargs) File "D:\pythonworkpace\stagesepx\stagesepx\cutter.py", line 168, in get_unstable_range if change_range_list[-1].start > merged_change_range_list[-1].end: IndexError: list index out of range
拉去的是最新代码

classifier 增加 load_range 方法

目前 classifier 只能通过 切割好的图片 进行分类,而不能直接利用 cutter 的返回结果。
在不需要中间结果的情况下,这会造成一定的IO浪费。

对range进行采样时尽量不选用边界帧

2019-08-22 15:16:27.751 | DEBUG    | stagesepx.cutter.cut_range:pick:102 - pick 5 frames from 50(0.8833333333333334) to 112(1.8833333333333333) on video ../demo.mp4
2019-08-22 15:16:28.407 | INFO     | stagesepx.cutter.cut_result:pick_and_save:284 - pick [50, 62, 75, 87, 100] in range <VideoCutRange [50-112] ssim=[1.0]>

首帧始终会被采样到。而对于稳定阶段而言,选择首帧是有风险的(可能它并没有彻底变化完毕,只是低于阈值)。尤其当采样张数较少时,它的权重会更大,从而导致效果不准确。

使用自己录制的视频报错 RuntimeError: frame -1 not found in video

出错日志如下:
2019-08-08 21:03:44.516 | DEBUG | stagesepx.cutter.cutter:convert_video_into_ssim_list:73 - ssim between 1180 & 1181: 0.8632647465179197
2019-08-08 21:03:44.678 | DEBUG | stagesepx.cutter.cutter:convert_video_into_ssim_list:72 - part 0: 0.9526492946983928
2019-08-08 21:03:44.709 | DEBUG | stagesepx.cutter.cutter:convert_video_into_ssim_list:72 - part 1: 0.9496711932140692
2019-08-08 21:03:44.737 | DEBUG | stagesepx.cutter.cutter:convert_video_into_ssim_list:72 - part 2: 0.8699858194411924
2019-08-08 21:03:44.766 | DEBUG | stagesepx.cutter.cutter:convert_video_into_ssim_list:72 - part 3: 0.956238770687535
2019-08-08 21:03:44.766 | DEBUG | stagesepx.cutter.cutter:convert_video_into_ssim_list:73 - ssim between 1181 & 1182: 0.8699858194411924
2019-08-08 21:03:44.911 | INFO | stagesepx.cutter.cutter:cut:112 - cut finished: ../boot.mp4
2019-08-08 21:03:44.930 | DEBUG | stagesepx.cutter.cut_result:get_unstable_range:70 - unstable range of [../boot.mp4]: [<VideoCutRange [1-35] ssim=[0.9181243658534559, 0.9252134306342404, 0.9050137559964935, 0.8908822591219767, 0.9148245557737175, 0.9487921674036248, 0.9490449310631784, 0.9144310146601791, 0.8778321803540652, 0.8509602102781896, 0.8338616019149875, 0.8551517699414735, 0.8825341430750818, 0.879468603985351, 0.8817909240488079, 0.9007143940379229, 0.8816951452667183, 0.8331507396734783, 0.8365107038245857, 0.8317524424284333, 0.8283431672148087, 0.8240149695572643, 0.8242853123615972, 0.8206763519559634, 0.8363518252907702, 0.8591345425489371, 0.8913256705312311, 0.8854700721409169, 0.8852091751961552, 0.8941897296160934, 0.893845546176238, 0.9111032072067128, 0.92115168086549, 0.9248221517196584]>, <VideoCutRange [36-54] ssim=[0.9420438109386463, 0.8879572430444717, 0.8904208184925245, 0.8691167000981724, 0.8995547271371701, 0.9165024337047402, 0.9078909745225309, 0.9130993683687919, 0.9153993393865681, 0.8964063241039606, 0.908572486695875, 0.8984784204612064, 0.8957711815660545, 0.8938650128215719, 0.890478744243276, 0.9082329769289119, 0.9258385267639464, 0.9224706212269499]>, <VideoCutRange [55-56] ssim=[0.947602905774728]>, <VideoCutRange [59-61] ssim=[0.938503410397641, 0.9314480675180605]>, <VideoCutRange [65-70] ssim=[0.9132356760564576, 0.8989709053962114, 0.9374417754199166, 0.9412690049439755, 0.9483129284864441]>, <VideoCutRange [71-75] ssim=[0.9125561507310234, 0.8849652489230537, 0.9132766974174759, 0.9495159705261856]>, <VideoCutRange [76-79] ssim=[0.9365373979076361, 0.9335255469226493, 0.9365607023700473]>, <VideoCutRange [82-85] ssim=[0.934846137850981, 0.9070579488291513, 0.9458062913233082]>, <VideoCutRange [87-91] ssim=[0.9115731050440241, 0.9305024839067251, 0.9314827625843571, 0.9407777027900649]>, <VideoCutRange [94-119] ssim=[0.916047299877974, 0.916395861031223, 0.8986614307258266, 0.8470687905415466, 0.8749162410711698, 0.9267084013668724, 0.9323863782676106, 0.902467677886002, 0.8801576308895069, 0.8828267262603481, 0.9363772038287094, 0.9278188632682642, 0.9004508642873242, 0.8909708647236211, 0.9075580369403887, 0.8797254519382434, 0.8596581314536917, 0.8581459222287048, 0.8939484695859322, 0.9152129598627761, 0.9275797052557968, 0.9223943000326825, 0.8929038089009321, 0.8794212849755111, 0.9232257790692727]>, <VideoCutRange [120-121] ssim=[0.9394397216950161]>, <VideoCutRange [123-136] ssim=[0.9134202718379061, 0.8929735350059819, 0.8521767988869599, 0.8626348209820505, 0.8880416572747505, 0.887639578085367, 0.8519529166718843, 0.8305632776333839, 0.8245712179159321, 0.9207230486846629, 0.896735709806936, 0.8358142525195348, 0.9071819110696226]>, <VideoCutRange [137-138] ssim=[0.9244215987082557]>, <VideoCutRange [139-153] ssim=[0.9294310180177454, 0.848478680111593, 0.8178458616011756, 0.7945456836417215, 0.8254825227478304, 0.8811278650062359, 0.8528254371737461, 0.846040360978211, 0.8413486790204538, 0.8645000680369821, 0.8995605721274038, 0.8852028256856235, 0.9096100226453765, 0.89290776040117]>, <VideoCutRange [154-168] ssim=[0.9140033554049425, 0.8699266132008844, 0.9032812608295469, 0.8703124669701822, 0.8548256615189435, 0.9033900624026308, 0.8977769428609743, 0.9353557894077562, 0.939298584336759, 0.9489880096893573, 0.8503240698366538, 0.8135336642969553, 0.8118722801034252, 0.8671877242040686]>, <VideoCutRange [169-178] ssim=[0.8900970729165435, 0.8619357871884182, 0.8901984386458397, 0.8987742001662775, 0.939710742325725, 0.9075027610172571, 0.8804409569871495, 0.8814913899332463, 0.923318861869869]>, <VideoCutRange [180-182] ssim=[0.9161308033789792, 0.9497778665341052]>, <VideoCutRange [187-192] ssim=[0.8958913072269997, 0.9038886484198713, 0.8728455336629659, 0.8879995289551622, 0.93177824899818]>, <VideoCutRange [193-202] ssim=[0.9092900811092816, 0.8627192672031541, 0.8117456523498704, 0.8457502469913444, 0.9004879707263446, 0.8778171082856986, 0.9133006619790786, 0.9162964093703343, 0.857413497404563]>, <VideoCutRange [203-214] ssim=[0.9277546582293905, 0.8717887241002067, 0.9011987398099255, 0.8761211360056683, 0.9360223153894827, 0.8864624221910704, 0.8765853943424892, 0.8751838048510959, 0.8815478700215329, 0.9164133664512916, 0.899366777946475]>, <VideoCutRange [215-220] ssim=[0.9122911059583063, 0.8752830155420894, 0.8665543861838086, 0.8700208206158795, 0.9173240381673746]>, <VideoCutRange [222-223] ssim=[0.9461697063992119]>, <VideoCutRange [227-248] ssim=[0.9374144974140738, 0.8448081508350217, 0.8545683534813803, 0.8807639045363673, 0.9124155928448431, 0.8859629618001678, 0.8660371964772295, 0.9233695315291022, 0.8849869851041774, 0.9341475331418605, 0.8750125470623177, 0.9303723170404243, 0.8902102626983182, 0.8556637098387213, 0.8905888149419433, 0.9311006854117257, 0.88500072957843, 0.8954163496667421, 0.9472951939302661, 0.9211122932306743, 0.8764018908439832]>, <VideoCutRange [249-250] ssim=[0.9476055157106805]>, <VideoCutRange [251-252] ssim=[0.9364378952921397]>, <VideoCutRange [253-256] ssim=[0.90842912924813, 0.8279834832799743, 0.8986826868956954]>, <VideoCutRange [258-264] ssim=[0.9495718710713129, 0.8801361415955551, 0.8678612398196311, 0.8629864949939425, 0.9018083371795317, 0.9403474201180007]>, <VideoCutRange [269-271] ssim=[0.9082887334587378, 0.9041532895740388]>, <VideoCutRange [273-276] ssim=[0.9217264114221823, 0.9266358229483561, 0.9298028445926105]>, <VideoCutRange [280-285] ssim=[0.9451423337751736, 0.9165878149104183, 0.9258442291920371, 0.8970289099863208, 0.9148999081314619]>, <VideoCutRange [288-300] ssim=[0.9354793839762884, 0.9288102590342234, 0.9209744284016556, 0.9220281614665765, 0.8796493839678632, 0.8980896933359289, 0.9224058019906783, 0.9128885507655224, 0.8680174714482094, 0.8747325836413995, 0.9364658774296861, 0.9376115159136487]>, <VideoCutRange [301-304] ssim=[0.9059120804090965, 0.8752413991955051, 0.9236440219018974]>, <VideoCutRange [305-327] ssim=[0.9289514427784339, 0.9213488085829686, 0.8881844522550958, 0.8651669651742009, 0.8515745315711588, 0.8508347023468081, 0.867915445398801, 0.8895093877244898, 0.887530225036962, 0.8674842297222931, 0.9456311992861018, 0.9464474613835402, 0.8967599165153687, 0.8417346723514519, 0.8187480782702747, 0.8620739192443237, 0.9284814430336455, 0.880847728261245, 0.8313400820235934, 0.8440838660478267, 0.9402011595775511, 0.8694321055474096]>, <VideoCutRange [329-337] ssim=[0.9205605870558403, 0.9299757127971737, 0.8934303859012572, 0.8917437555646135, 0.9044526973565408, 0.8559038233160416, 0.8945879077057481, 0.8916294812323253]>, <VideoCutRange [340-352] ssim=[0.9483860702480006, 0.8720659707822579, 0.9101828992228944, 0.9397991825746153, 0.9334971409750048, 0.9319786290596163, 0.8505198055454132, 0.844982967416731, 0.8778232828948758, 0.8390984705042408, 0.8274468744733717, 0.8089276614166195]>, <VideoCutRange [353-355] ssim=[0.9034374621864268, 0.894475776396278]>, <VideoCutRange [356-358] ssim=[0.9261745783809148, 0.9464140436045768]>, <VideoCutRange [360-374] ssim=[0.9045004800092085, 0.8694420610592535, 0.9202714985270322, 0.8488055974896023, 0.8331903419027629, 0.840128682250825, 0.8705451322682305, 0.8515139155987062, 0.8299057491389306, 0.824338731795251, 0.8241758925268123, 0.8544703389254413, 0.849459069179199, 0.8820111236867648]>, <VideoCutRange [378-388] ssim=[0.908964078677052, 0.9217514016490111, 0.8826313870472627, 0.8524661360946173, 0.803565351581368, 0.8401237687286791, 0.8599633063680036, 0.8491758534341126, 0.8977840465613822, 0.9358163719388369]>, <VideoCutRange [389-395] ssim=[0.9020710369612064, 0.9162835924389539, 0.9079883168024493, 0.8655188896693191, 0.8435844933921318, 0.9015380407476711]>, <VideoCutRange [397-398] ssim=[0.9304397570324507]>, <VideoCutRange [399-405] ssim=[0.889034675095657, 0.8500540017288875, 0.8521856983186764, 0.9014981335795242, 0.948138238280019, 0.9247130891072151]>, <VideoCutRange [406-408] ssim=[0.9210469016922199, 0.9202201200311455]>, <VideoCutRange [413-422] ssim=[0.9368692276930569, 0.9068012136512253, 0.8664582567701314, 0.8922037941894168, 0.8502738466424976, 0.8222292437037741, 0.8059631012974496, 0.8268044012256176, 0.9004254264832453]>, <VideoCutRange [424-429] ssim=[0.8916842000342899, 0.930982463062723, 0.9196897254772819, 0.8512136153836147, 0.905319452112295]>, <VideoCutRange [431-433] ssim=[0.9462789086431618, 0.9226733392556463]>, <VideoCutRange [435-438] ssim=[0.9439421073386318, 0.9152986449721161, 0.9276132607883923]>, <VideoCutRange [443-444] ssim=[0.9480504253275646]>, <VideoCutRange [445-452] ssim=[0.933770500136377, 0.9115190294728539, 0.9296329097863201, 0.905641948148159, 0.8871331244814475, 0.8899166801175115, 0.9376797236808186]>, <VideoCutRange [457-465] ssim=[0.9147052307353014, 0.9022742782718115, 0.928234922502605, 0.9045065348800911, 0.9373208184950615, 0.94150656014396, 0.91663293910306, 0.911092394047512]>, <VideoCutRange [466-485] ssim=[0.8873115355863911, 0.8255390582399224, 0.8389152194824806, 0.8822692693816129, 0.8665634645479956, 0.8926855574193683, 0.8629047816456885, 0.8337331958826111, 0.837740542125943, 0.8335796730621203, 0.8610546850989047, 0.8773044515582772, 0.8702353090363294, 0.8413645562564512, 0.8850898464015068, 0.8274672193148183, 0.8138275356464764, 0.8271126248683188, 0.9053474394388885]>, <VideoCutRange [487-490] ssim=[0.8835589145227666, 0.8599559856877769, 0.9240427689194268]>, <VideoCutRange [491-493] ssim=[0.9465202989042918, 0.9423305743126497]>, <VideoCutRange [495-498] ssim=[0.9356239394671415, 0.892995054649378, 0.9431273020785965]>, <VideoCutRange [501-503] ssim=[0.9239691392932715, 0.9381924625631833]>, <VideoCutRange [509-514] ssim=[0.9059703585694155, 0.873700822096657, 0.8780853378608363, 0.8879716665030188, 0.9290314024553921]>, <VideoCutRange [524-528] ssim=[0.9478127836109926, 0.8811505374397318, 0.861288037741847, 0.9373828197040298]>, <VideoCutRange [529-532] ssim=[0.9168527418555864, 0.8516199259275475, 0.8842325939494318]>, <VideoCutRange [533-544] ssim=[0.8968692079606908, 0.8946762494151175, 0.9126258141074342, 0.9349963783453182, 0.9415311444014706, 0.8934680066797938, 0.8643896432488484, 0.9054412730675832, 0.9412882249303269, 0.902969080764062, 0.9466808166331391]>, <VideoCutRange [545-547] ssim=[0.9434774629897873, 0.9338067153258096]>, <VideoCutRange [548-559] ssim=[0.9064459761915927, 0.8647310690109892, 0.935039352087305, 0.897594750078401, 0.8675410729912896, 0.8553597505740221, 0.8744997270188736, 0.8593409103206029, 0.8669258764231983, 0.8942654139613747, 0.9310592461748264]>, <VideoCutRange [566-582] ssim=[0.8826476241774445, 0.8649876556368308, 0.9170958259451594, 0.9310466546450579, 0.9423997918290264, 0.9237156483721088, 0.873460820626001, 0.8759197294128991, 0.8667082179588854, 0.8954109583191852, 0.9198430000990403, 0.9440166350425673, 0.9099168456599794, 0.8991619003243175, 0.9015269707436879, 0.9251243387963914]>, <VideoCutRange [585-586] ssim=[0.9332982613672635]>, <VideoCutRange [591-592] ssim=[0.9382452986892347]>, <VideoCutRange [593-596] ssim=[0.917221496440849, 0.8315118693223038, 0.9458815901635077]>, <VideoCutRange [600-616] ssim=[0.898805274578831, 0.8569713554279064, 0.8740773481397693, 0.9043016747091999, 0.8724067119927983, 0.8864893713622435, 0.8822620741748328, 0.8754489433259423, 0.8715629996611622, 0.9058536320964645, 0.9146580056815639, 0.8604695495555754, 0.8470467318860516, 0.8371487863891194, 0.8443209010657505, 0.9390228768361568]>, <VideoCutRange [617-618] ssim=[0.947162838699479]>, <VideoCutRange [623-625] ssim=[0.9296499581355954, 0.9019321189473875]>, <VideoCutRange [627-631] ssim=[0.9050210559962425, 0.9045797273276996, 0.9249710046157598, 0.9437905940532048]>, <VideoCutRange [635-637] ssim=[0.9187872900408469, 0.8796891141416748]>, <VideoCutRange [638-653] ssim=[0.9457327657962354, 0.8613933361585248, 0.8159989273062435, 0.8092695720421543, 0.8423951303032474, 0.8730820334519326, 0.8970818500079881, 0.91867965174877, 0.8902002888801663, 0.8952545755331156, 0.9244885596547244, 0.9203105002511481, 0.9441592538819347, 0.9277796147240458, 0.9407151769541762]>, <VideoCutRange [655-659] ssim=[0.8940794240019716, 0.8313467553727953, 0.8804681240411167, 0.9458189274471525]>, <VideoCutRange [663-666] ssim=[0.93146016671682, 0.9311122240360732, 0.9132109770501475]>, <VideoCutRange [668-678] ssim=[0.9230459464586764, 0.9191755684047082, 0.9326682612297419, 0.9370627199181669, 0.9276403821508946, 0.8618696781562908, 0.8585904826013697, 0.9354059212239474, 0.8542270768143121, 0.8642611664512977]>, <VideoCutRange [679-709] ssim=[0.8938096020607573, 0.8944075435126131, 0.8981538961118527, 0.9038952608395202, 0.869942766241637, 0.8406549703104464, 0.8307413415574989, 0.8523094883146869, 0.8870667186830079, 0.9297061050224894, 0.8920609298702413, 0.8933373180961214, 0.8614407672995864, 0.8389937047702736, 0.8324986177440564, 0.8183249998407467, 0.8266913827544792, 0.8427286969411171, 0.8697795969139793, 0.8635830870791261, 0.8887886776809801, 0.8833716024597617, 0.8375373927583256, 0.8708899612140645, 0.8687414895706435, 0.8460352895087973, 0.8386837006563117, 0.8666523616720507, 0.9275685748534643, 0.9367056704314307]>, <VideoCutRange [711-713] ssim=[0.919656348006361, 0.9438952440842463]>, <VideoCutRange [714-716] ssim=[0.9481980687107721, 0.9439539454685558]>, <VideoCutRange [719-721] ssim=[0.8596792174381301, 0.8888547691491194]>, <VideoCutRange [722-753] ssim=[0.8152956032833527, 0.8111894907369074, 0.8575681804217653, 0.8711006520089064, 0.9205106254133278, 0.8688436249053145, 0.8513564514502996, 0.8634599756820427, 0.7983889545447123, 0.7941248850060756, 0.801677012249432, 0.8020658313669452, 0.8560512061257864, 0.8783842052108646, 0.9039564832135994, 0.8892567239241517, 0.8625949529002374, 0.8487072488735241, 0.8560844709552423, 0.8740382201820481, 0.8706823340881767, 0.8630398234504868, 0.8777380022134909, 0.9115125059602605, 0.929091942710012, 0.909307366144371, 0.8549409398333172, 0.8939882994093337, 0.9326680735864133, 0.9442543775760343, 0.9479639326539914]>, <VideoCutRange [755-756] ssim=[0.9470182082136234]>, <VideoCutRange [757-758] ssim=[0.9497700687330342]>, <VideoCutRange [761-764] ssim=[0.889924211851352, 0.8570629114614624, 0.9098925359797402]>, <VideoCutRange [768-775] ssim=[0.9435810231680293, 0.9344331410371985, 0.9456859001396146, 0.9408486002996506, 0.9192998475226893, 0.9004775642184731, 0.9163017773467073]>, <VideoCutRange [776-777] ssim=[0.9357605179930808]>, <VideoCutRange [779-786] ssim=[0.9002820346720098, 0.8839124702814881, 0.8786273041357743, 0.8779116129249075, 0.9005122843835663, 0.9312293897742348, 0.9434085175639941]>, <VideoCutRange [793-794] ssim=[0.9424570551883233]>, <VideoCutRange [795-798] ssim=[0.9268508964682853, 0.9426531623858935, 0.9113684600008766]>, <VideoCutRange [799-801] ssim=[0.9466992625091026, 0.9221289049322856]>, <VideoCutRange [803-805] ssim=[0.9028150082532631, 0.9180150092092728]>, <VideoCutRange [810-819] ssim=[0.905404476323141, 0.8758223597745403, 0.9343212101718261, 0.9440689360869445, 0.8723129616838718, 0.8917536911146663, 0.8996007477769458, 0.8994171511072621, 0.9438073627806045]>, <VideoCutRange [821-823] ssim=[0.8907846565825827, 0.9338449216790483]>, <VideoCutRange [826-830] ssim=[0.8938494675055583, 0.9290850668636755, 0.8817798988544061, 0.841809183952398]>, <VideoCutRange [832-865] ssim=[0.9497781731793566, 0.9036555053479131, 0.9035729621474275, 0.913499124448536, 0.887541301266079, 0.8537740505644017, 0.8651931894701571, 0.825163376883814, 0.8062620677203294, 0.8135435592419289, 0.8503608969762342, 0.863058285733028, 0.8576904417692252, 0.8521911272561156, 0.8449405305268988, 0.865721360011569, 0.8279433892440146, 0.8069893725114491, 0.8193354906810751, 0.8728096740377417, 0.9354087768859974, 0.9419119548136581, 0.8874981072404012, 0.8775081731123501, 0.8909930076317765, 0.8850916451799296, 0.903771008773303, 0.9127169642377612, 0.882403268278562, 0.8828589816126015, 0.9197354950041999, 0.914380468302607, 0.902853149610288]>, <VideoCutRange [866-877] ssim=[0.9482916997236315, 0.8910387708003015, 0.8556988021831362, 0.8967740409854877, 0.8839524359694992, 0.8572265939448594, 0.8635260225695623, 0.8595869944882519, 0.8729166083613505, 0.91183324537747, 0.9498192418204197]>, <VideoCutRange [878-880] ssim=[0.9399682152787949, 0.9262122758282618]>, <VideoCutRange [886-890] ssim=[0.8776922347373791, 0.8272963378172297, 0.8265276891342865, 0.894294171508783]>, <VideoCutRange [891-896] ssim=[0.8913545727900511, 0.8542777986462451, 0.9024753469895676, 0.9142560740939463, 0.9308814366195813]>, <VideoCutRange [900-936] ssim=[0.9198011842814673, 0.9467728095962745, 0.8909882733630997, 0.8343960528914529, 0.8621180642655171, 0.8796668655103007, 0.8881685143690452, 0.9018050595620946, 0.8720226052600611, 0.8847048115756426, 0.9142054348782664, 0.927105738457402, 0.8755791579506641, 0.8585805424405332, 0.9034337623533307, 0.8875120572856032, 0.9183927642525744, 0.912777160842122, 0.9034441740520057, 0.9291452402256416, 0.878572360514613, 0.851799943176435, 0.8939972332724873, 0.9328538365278485, 0.8818296024737894, 0.861620421390159, 0.8461602136376278, 0.8268982009305494, 0.8209140039883873, 0.8253579660154233, 0.8581346615241656, 0.8438294838761795, 0.8461864391922872, 0.8947501797129476, 0.8975903692221429, 0.9195705082782996]>, <VideoCutRange [939-943] ssim=[0.866547752241125, 0.8546324013512915, 0.9108410735169309, 0.8966257580756022]>, <VideoCutRange [944-946] ssim=[0.9288775358545607, 0.9305635703148251]>, <VideoCutRange [949-951] ssim=[0.9204582539686503, 0.9334277301460655]>, <VideoCutRange [952-961] ssim=[0.8829721247907536, 0.8657769388216152, 0.832569030677543, 0.825498097082563, 0.8348699830250975, 0.8437044204226672, 0.913318518660412, 0.9325347348682279, 0.9386016794394554]>, <VideoCutRange [962-970] ssim=[0.932109731618047, 0.8861805335668472, 0.8315184592848889, 0.9020978723725283, 0.9327100756029562, 0.9398078184384614, 0.9221103754714078, 0.9234833870133776]>, <VideoCutRange [972-975] ssim=[0.9461337253351592, 0.9246335560913169, 0.9303903487516243]>, <VideoCutRange [976-979] ssim=[0.8796815355172819, 0.8574075927593574, 0.9196643635997797]>, <VideoCutRange [980-991] ssim=[0.9201430085983604, 0.8948770086937307, 0.8548221151146561, 0.8231704913512738, 0.8339846795309062, 0.8303058359112415, 0.8828150884754162, 0.8873142497287798, 0.8851241189407923, 0.8792639943003658, 0.8816194383032732]>, <VideoCutRange [993-996] ssim=[0.8481433779606206, 0.82819544381876, 0.8780530999077077]>, <VideoCutRange [999-1036] ssim=[0.9021088338878132, 0.8439345830034864, 0.8122104864552475, 0.8012777522808433, 0.8213870691174333, 0.8902908471973925, 0.9054097596470879, 0.9069159205766527, 0.928481723890282, 0.8428272901720049, 0.8269271389835722, 0.8323064105906846, 0.8137088150262268, 0.8163369032726366, 0.8373540463033557, 0.832022370834878, 0.834964047640812, 0.8327453786205329, 0.8393628687921669, 0.8446965507649976, 0.8462924705475026, 0.8441331323213942, 0.8414834091438749, 0.8648394338099822, 0.8627311532604859, 0.8506416379734875, 0.856733774690456, 0.8809164686775155, 0.9026636017718463, 0.8978973841189188, 0.843222211344476, 0.8970973230431188, 0.9135049173878326, 0.8794307463060904, 0.8869139123158498, 0.8999378610983279, 0.9267418786378968]>, <VideoCutRange [1039-1046] ssim=[0.8965104705467255, 0.866210499275293, 0.8507370179523148, 0.9029850175370409, 0.9412207039525108, 0.9306024223983564, 0.9205103437449235]>, <VideoCutRange [1048-1063] ssim=[0.8850476301947425, 0.867286830453785, 0.8533341823169079, 0.8649365461688545, 0.8589542734450348, 0.8507962092694065, 0.8355429705938386, 0.832151686801738, 0.8351945837543192, 0.8463750044205267, 0.8924960956702307, 0.8458968107517036, 0.853862234393965, 0.8926972587359329, 0.9404369824822219]>, <VideoCutRange [1064-1066] ssim=[0.9337430051848495, 0.9485445472169387]>, <VideoCutRange [1067-1074] ssim=[0.904777770611344, 0.9198752081395798, 0.8999945744862357, 0.911333647499567, 0.8748087539134144, 0.9268799769370024, 0.9316383808740144]>, <VideoCutRange [1079-1091] ssim=[0.9269957477590604, 0.8669993787936, 0.9117628469729593, 0.8841693250296185, 0.8522786907308836, 0.8621037553914498, 0.8554140811381713, 0.8435815702471774, 0.8294220037944929, 0.8595023289080738, 0.9104647893708415, 0.939341564446912]>, <VideoCutRange [1092-1093] ssim=[0.9335624336444992]>, <VideoCutRange [1094-1098] ssim=[0.9193405879283291, 0.8736799989076443, 0.9378308554568376, 0.9101265227882159]>, <VideoCutRange [1100-1115] ssim=[0.9349324111271454, 0.9038841192948003, 0.8938413308169831, 0.903395017083237, 0.9229255144472905, 0.8977293734899444, 0.8740679635826715, 0.852301953993077, 0.8542147727522732, 0.8971661255620301, 0.8991362225832089, 0.8868781799192135, 0.9015302541505905, 0.9359133414929117, 0.9470193994920513]>, <VideoCutRange [1119-1128] ssim=[0.9095540454031426, 0.8212552332385732, 0.5631660269333111, 0.8090959223618649, 0.8565985758265745, 0.885134091174482, 0.9267166874126134, 0.9381303904897996, 0.9479077893356675]>, <VideoCutRange [1130-1131] ssim=[0.9447442855287157]>, <VideoCutRange [1132-1148] ssim=[0.9133557367706386, 0.8821354743016915, 0.8768338598781673, 0.8806029364224163, 0.921979558899471, 0.8801706529016198, 0.8442989685368619, 0.833165679111592, 0.8770430624423303, 0.8750905641659973, 0.8210279866799485, 0.8176366527254532, 0.8260942183970337, 0.9039326642629166, 0.9419427270830297, 0.9380884809419646]>, <VideoCutRange [1149-1158] ssim=[0.7995059454891394, 0.7618970442740022, 0.7983324575595944, 0.826169305834383, 0.8260546703898662, 0.8046866316071338, 0.7990330706885145, 0.8820629471962438, 0.9052113868454351]>, <VideoCutRange [1161-1162] ssim=[0.824272589676426]>, <VideoCutRange [1163-1182] ssim=[0.8581314334881646, 0.8437381874267059, 0.8438084961304544, 0.9226115522276603, 0.8363695590725925, 0.9061433549206566, 0.9054492066154005, 0.8928011202283437, 0.9204247232735968, 0.866334985826736, 0.7938396223253756, 0.8091568714394337, 0.8198893745575723, 0.8478278434858042, 0.856499193832517, 0.9120901761760489, 0.8827618675332891, 0.8632647465179197, 0.8699858194411924]>]
Traceback (most recent call last):
File "D:/PycharmProjects/stagesepx/example/cut_and_classify.py", line 9, in
stable, unstable = res.get_range()
File "C:\Python37\lib\site-packages\stagesepx\cutter\cut_result.py", line 114, in get_range
self.get_target_range_by_id(first_stable_range_end_id - 1).start_time,
File "C:\Python37\lib\site-packages\stagesepx\cutter\cut_result.py", line 23, in get_target_range_by_id
raise RuntimeError(f'frame {frame_id} not found in video')
RuntimeError: frame -1 not found in video

VideoCutRange改造

目前 VideoCutRange中的属性有:

self.video_path = video_path
self.start = start
self.end = end
self.ssim = ssim

时间戳

事实上,视频与帧位置都已经确定了,对应的时间戳其实也已经确定,完全可以在其中加入起始与结束的时间戳。

SSIM列表

VideoCutRange里应该是SSIM列表,而不是一个已经评估完的SSIM具体值

thumbnail 内置

for each in unstable:
    r.add_thumbnail(
        f'{each.start}({each.start_time}) - {each.end}({each.end_time})',
        res.thumbnail(each))

r.draw(
    classify_result,
    cut_result=res,
)

目前 thumbnail 与 ssim+psnr趋势图 都是分别可选的,而他们在debug中举足轻重。
保留一处开关即可。

阶段极短时有误差

2019-07-25 15:51:40.061 | INFO     | stagesepx.cutter:pick_and_save:208 - pick [1, 10, 19, 28, 37] in range <VideoCutRange [1-45] ssim=0>
2019-07-25 15:51:40.061 | INFO     | stagesepx.cutter:pick_and_save:208 - pick [50, 50, 50, 50, 50] in range <VideoCutRange [50-49] ssim=0>
2019-07-25 15:51:40.061 | INFO     | stagesepx.cutter:pick_and_save:208 - pick [52, 57, 63, 69, 75] in range <VideoCutRange [52-80] ssim=0>

50-49

blur变换难以检测

SSIM原生对高斯模糊不敏感,而目前非常多的应用带有blur类型(类似毛玻璃)的动画效果。

分析前对帧预处理

如果视频是拍摄所得,难免会有噪声影响。为了降低噪声对分析结果的影响,应当在分析前进行合理的预处理(例如降噪)

hook in classifier & cutter

在分类时可以同时对每一帧进行自定义操作,例如检测视频中是否包含黑白屏。需要兼容cutter与classifier。

可能的形式:

class YourHook(CustomHook):
    # do something ...
    # and result collection ...

hook = YourHook()
cutter.add_hook(hook)
cutter.cut()
result = hook.get_result()

测试用视频收集

为了提高该库的稳定性与适用性,我们每次发布前都会在内部视频测试集上运行以确保效果正常。但测试集的数量有限,我们很难保证所有场景的使用效果。

欢迎所有的开发者提交你觉得目前 stagesepx 支持得不够好的测试视频(当然,尽量不要涉密,可以是你的使用场景也可以是你认为分析不准确的场景),把这个工具做得更好。

视频请不要超过半分钟,短视频、MP4格式为佳。

请以邮件形式发送到:[email protected]
或以附件形式贴到该issue中。

report 内容改进

  • 除了每个阶段的占用时间,从一个稳态切换到另一个稳态的耗时通常对于开发者也相当重要
  • 当前的stage图是类似方波形式的,而这很容易误导人,后续改为常规折线图
  • report中没有展示每个稳定阶段对应的状态图

thumbnail中加入分割线

image

快照是直接由帧紧凑拼接而成,中间无间隔。而当背景一致时,人眼阅读比较困难。

step非1时列表越界

主要因为部分计算直接用了 ssim_list 的索引

Traceback (most recent call last):
  File "F:/stagesepx/example/cut_and_classify.py", line 9, in <module>
    stable, unstable = res.get_range()
  File "F:\stagesepx\stagesepx\cutter.py", line 228, in get_range
    self.ssim_list[end_stable_range_start_id - 1].end_time,
IndexError: list index out of range

Process finished with exit code 1

assert ret AssertionError

Traceback (most recent call last):
File "test01.py", line 10, in
data_home = res.pick_and_save(stable, 3)
File "/python/lib/python3.6/site-packages/stagesepx/cutter.py", line 186, in pick_and_save
each_frame = toolbox.get_frame(cap, each_frame_id - 1)
File "/python/lib/python3.6/site-packages/stagesepx/toolbox.py", line 48, in get_frame
assert ret
AssertionError

关于分类结果修正

cutter会将你的视频自动切割分类出一系列的阶段,在 pick_and_save 之后通常是这样的形式:

image

如图,stagesepx将你的视频分成了两个阶段0与1。此时,如果你发现1中有部分图片你认为分错了(例如你认为应该属于阶段0或者应该被分成一个新阶段),你可以手动将他们移动到你认为的分类中去(例如你认为应该分类为0,你可以将这些图片移动到文件夹0下;如果你认为应该分类为新分类2,你可以新建一个文件夹2然后将这些图片移动到下面)。

在此之后利用 classifier 去分类,就可以达到误差修正的效果。你可以将该模型保存起来以便下次分析。

limit参数调整

对于稳态与非稳态,limit参数的价值完全是不同的。而目前的做法是一个limit参数会同时影响到他们两者的过滤,需要将他们区分开。

帧预处理移交到hook中

目前compress方法带了一个简单的降噪流程,这应该以hook形式实现以达到便于拼接的目的。

偶现range区间重复

unstable range of [../file3.mp4]: [<VideoCutRange [49-63] ssim=0.9289527845042277>, <VideoCutRange [102-104] ssim=0.8636853400069384>,<VideoCutRange [103-104] ssim=0.8636853400069384>]

关于判定时间与实际不符的问题

非常值得注意的是,分析结果的准确性与如何录制视频有较大关联性。以android为例,通常使用adb shell screenrecord 录制出来的视频,fps是不稳定的。

The frame rate is variable, not fixed. Every time the screen is updated, one frame is recorded. If the screen is not updated, no frame is recorded. Therefore there is no setting for the frame rate, because it's determined by how quickly the system updates the screen.

而这种不稳定很可能带来非常巨大的误差,绝对不应该被忽视。

image

image

如上述例子,同一个场景下,不同的录制方法导致的误差达到了3s之多。

目前推荐两种视频录制方式:

对于已有视频,请参考:

ffmpeg -i input.mp4 -r 60 output.mp4

timestamp error

2019-08-15 22:11:48.006 | INFO     | stagesepx.cutter.cut_result:pick_and_save:274 - pick [0] in range <VideoCutRange [0-18] ssim=[1.0]>
2019-08-15 22:11:48.006 | DEBUG    | stagesepx.cutter.cut_range:pick:102 - pick 1 frames from 20(0.8) to 21(0.8) on video ../demo.mp4
2019-08-15 22:11:48.051 | INFO     | stagesepx.cutter.cut_result:pick_and_save:274 - pick [20] in range <VideoCutRange [20-21] ssim=[1.0]>
2019-08-15 22:11:48.051 | DEBUG    | stagesepx.cutter.cut_range:pick:102 - pick 1 frames from 30(1.2) to 30(1.2) on video ../demo.mp4

frame 20 and 21 have the same timestamps.

working with imageai

for further usage, i choose imageai as AI backend, which was built on the top of keras and tensorflow.

stagesepx have no plan of offering an built-in deep learning toolbox (too much dependenices it needs). stagesepx is a lightweight tool for analysing videos only and supports running on the most of platforms.

but we will offer a workflow (example) to show how to make it work with popular AI frameworks.

report可读性调整

image

extras中的参考值有点不明所以

image

这个链接的存在是否真的有必要

image

图表没有文字介绍

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.