juliadocs / democards.jl Goto Github PK

I expected the following try-catch block works, but it doesn't. When line 152 errors, it throws a warning, and then immediately throws an error

https://github.com/johnnychen94/DemoCards.jl/blob/12c66103c43912afd22bba989149c95189ef97e8/src/types/julia.jl#L146-L160

For example, in JuliaImages/juliaimages.github.io#94

$ julia --project=. --color=yes docs/make.jl
WARNING: Method definition _bcs1(Any, Any) in module Broadcast at broadcast.jl:439 overwritten in module ImageFiltering at /home/travis/.julia/packages/ImageFiltering/y5YkM/src/ImageFiltering.jl:28.
WARNING: Method definition _bcs1(Any, Any) in module Broadcast at broadcast.jl:439 overwritten in module ImageFiltering at /home/travis/.julia/packages/ImageFiltering/y5YkM/src/ImageFiltering.jl:28.
WARNING: Method definition _bcs1(Any, Any) in module Broadcast at broadcast.jl:439 overwritten in module ImageFiltering at /home/travis/.julia/packages/ImageFiltering/y5YkM/src/ImageFiltering.jl:28.
WARNING: Method definition _bcs1(Any, Any) in module Broadcast at broadcast.jl:439 overwritten in module ImageFiltering at /home/travis/.julia/packages/ImageFiltering/y5YkM/src/ImageFiltering.jl:28.
[ Info: SetupDemoCardsDirectory: setting up examples directory.
┌ Warning: Executing demo docs/examples/color_channels/color_separations_svd.jl fails.
└ @ DemoCards ~/.julia/packages/DemoCards/67iDl/src/types/julia.jl:159
ERROR: LoadError: LoadError: UndefVarError: eachslice not defined
in expression starting at string:1
when executing the following code block

svdfactors = svd.(eachslice(channels; dims=1))
imgs = map((10, 50, 100)) do k
    colorview(RGB,
              rank_approx(svdfactors[1], k),
              rank_approx(svdfactors[2], k),
              rank_approx(svdfactors[3], k))
end
vcat([img imgs[1]], [imgs[2] imgs[3]])

in expression starting at /home/travis/build/JuliaImages/juliaimages.github.io/docs/make.jl:7

The complete log can be found in https://travis-ci.org/JuliaImages/juliaimages.github.io/jobs/619407603

It would be a nice feature to integrate with Literate.jl and directly generate well-written Julia source codes into demos.

This feature enables the following pipeline:

• write a runnable demo in julia format
• test the julia file in CI
• if test passes, convert it to markdown files
• provide a downloadable link to the julia file and jupyter notebook file.

This is a valid YAML syntax

title: mnist_elastic
id: mnist_elastic
description: >
             In this example we are going to use Augmentor on the famous **MNIST database of handwritten digits**
             [^MNIST1998] to reproduce the elastic distortions discussed in [^SIMARD2003].

but it fails to parse when I add this to the julia script as a frontmatter:

# ---
# title: mnist_elastic
# id: mnist_elastic
# description: >
#              In this example we are going to use Augmentor on the famous **MNIST database of handwritten digits**
#              [^MNIST1998] to reproduce the elastic distortions discussed in [^SIMARD2003].
# ---

similar to #48, I should not remove the leading whitespace in
https://github.com/johnnychen94/DemoCards.jl/blob/a2ee88b77147f1f17c1e3c1a973f98d0b5dc7230/src/utils.jl#L256

At begining,in repl:
julia>include("docs/make.jl")
everything is ok.
Then I want to change the demo1,for example adding some cover,something is wrong.

[ Info: SetupDemoCardsDirectory: setting up democards directory.
[ Info: SetupBuildDirectory: setting up build directory.
ERROR: LoadError: IOError: stat: permission denied (EACCES) for file "build"
Stacktrace:
[1] stat(::String) at .\stat.jl:69
[2] isdir at .\stat.jl:311 [inlined]
[3] runner(::Type{Documenter.Builder.SetupBuildDirectory}, ::Documenter.Documents.Document) at C:\Users\26947
.julia\packages\Documenter\pjwqp\src\Builder.jl:101
[4] dispatch(::Type{Documenter.Builder.DocumentPipeline}, ::Documenter.Documents.Document) at C:\Users\26947.
julia\packages\Documenter\pjwqp\src\Utilities\Selectors.jl:167
[5] #2 at C:\Users\26947.julia\packages\Documenter\pjwqp\src\Documenter.jl:241 [inlined]
[6] cd(::Documenter.var"#2#3"{Documenter.Documents.Document}, ::String) at .\file.jl:93
[7] #makedocs#1 at C:\Users\26947.julia\packages\Documenter\pjwqp\src\Documenter.jl:240 [inlined]
[8] top-level scope at D:\ATOM\atom-x64-windows\Location\Code\MyTestPkg\docs\make.jl:18
[9] include(::String) at .\client.jl:457
Unless I remove the file "build" and "src/democards",rebuild "docs/make.jl".

Wish author will repair this bug.

Currently, the same script is executed three times, which is not efficient especially when it comes with a lot of demos:

• Documenter rendering
• assets(e.g., cover) generation
• jupyter notebook generation

This issue plans to add a new keyword "notebook" to optionally disable jupyter notebook generation. I have plans to disable the assets generation as well, but I need to play with it first as it is deeply involved with the DemoCards design.

The jupyter notebook feature is perhaps not as attractive as I originally thought when I wrote this package.

This configuration needs to

• be able to configure/override at all levels (page/section/card)
• propagate to all children items
• parent configuration has lower priority than its children configuration

As an example:

examples/
├── color_channels
│   ├── config.json
│   ├── color_separations_svd.jl
│   ├── indexed_image.jl
│   ├── rgb_grayscale.jl
│   └── rgb_hsv_thresholding.jl
├── config.json
├── index.md
└── visualization
    └── block_visualization.jl

• If we disable the execution in "examples/config.json", then all the Julia demos are disabled by default.
• If we again enable the execution for examples/color_channels/config.json, then all demos in this section are enabled. It has a higher priority than its parent page settings.
• Similarly, if we configure this in the julia script(e.g., rgb_grayscale.jl), then it has the highest priority.

keywords for these will be case insensitive full match:

• enable: "enable", "yes", "true"
• disabled: "disable", "no", "false"

Example

For page/section:

{
    "notebook": "no",
}

For card:

---
notebook: no
---

JuliaImages/juliaimages.github.io#98

The current pipeline doesn't support assets generated after the building, for example, assets/demo.jpg is generated after Documenter.jl

---
title: Otsu
---

'''@example
using Images, ImageBinarization, TestImages
img = testimage("cameraman")
alg = Otsu()
img₀₁ = binarize(img, alg)
save("assets/demo.jpg", hcat(img, img₀₁)) # hide
'''

![result](assets/demo.jpg)

There will be cases that we only want to show some images without linked urls (e.g., https://zygmuntszpak.github.io/ImageBinarization.jl/v0.2/)

Affected by Documenter v0.25.1, when credit=true, duplicate credits are rendered now:

It would nice to host an entrance gallery to different well-maintained Julia packages as an alternative to https://pkg.julialang.org/docs/.

For cards that don't need a cover, the current implementation adds a blank cover, which is ugly. A better solution for this issue is to add another theme that doesn't render the card cover at all, e.g., lists, tables.

Currently, it is only possible to pass a description item to config.json for each section, but it is not very useful if the description is very long. A better solution is to support a section level template, just like how we provide the index.md file to
the page.

This package is developed to replace the current demo page of https://juliaimages.org/latest/demos/

The "new" demo page is expected to be:

• easy to deploy; seamlessly integrate with Documenter.jl and Literate.jl
• easy to write; demo authors (not necessary package maintainers) only need to provide a single file (with assets) with no need to understand the entire documentation mechanism.
• CI supported;

Roadmap:

• functionalities
  • decouple {demo_image}, {demo_title} and {demo_file}; currently they're of the same name.
  • integrate with Literate.jl to support CI
  • provide a default style sheet
• beautify the demo page
  • custom sections and cards order; currently it's sorted alphabetically.
  • make a grid of demo cards; currently it's listed vertically.
• nice to have:
  • multiple demo pages
  • simplify the card template

assets folder is not needed in this case.

shell> tree $(dirname(preview_demos("test1.jl", require_html=false)))
[ Info: SetupDemoCardsDirectory: setting up preview_page directory.
/var/folders/c0/p23z1x6x3jg_qtqsy_r421y40000gn/T/jl_V7tiuv/src/democards/preview_page
├── covers
│   └── test1.png
├── index.md
└── preview_section
    ├── assets
    ├── test1.ipynb
    ├── test1.jl
    └── test1.md

shell> tree $(dirname(preview_demos("test2.md", require_html=false)))
[ Info: SetupDemoCardsDirectory: setting up preview_page directory.
/var/folders/c0/p23z1x6x3jg_qtqsy_r421y40000gn/T/jl_V7tiuv/src/democards/preview_page
├── covers
│   └── test2.png
├── index.md
└── preview_section
    └── test2.md

By doing this we could pass more structural information to Documenters and thus get a better-looking sidebar list.

• really decouple cardtheme and makedemos
• add a theme for list type
• remove cover from card struct #33

Edit:

Seems like it would be better to return a dict of name-path pair

e.g. https://johnnychen94.github.io/DemoCards.jl/stable/democards/quickstart/index.html

A naive solution to support documentation with multiple languages, is to provide folder structure as follows:

docs/
├── Project.toml
├── build
├── make.jl
└── src
    ├── en
    │   └── index.md
    ├── index.md
    └── zh-cn
        └── index.md

In this way, https://johnnychen94.github.io/DemoCards.jl/dev/en and https://johnnychen94.github.io/DemoCards.jl/dev/zh-cn would point to contents written in different languages.

For terminology, I'll call en the main page, and zh-cn as the mirror page of en.

This solution works, but it is not working very nice. Here are some key challenges:

• there isn't a way to keep track of the contents to make sure they are updated. For example, it is very likely that mirror pages are out-dated because of various reasons. This is even harder if we consider the case that different pages might have different localization contents.
• mirror pages are usually developed distributedly in different repos. Usually, we can not assume that the main page is aware of the existence of mirror pages.

First, it is worth noting that, whether the multiple versions of the documentation are spread in different repos, or in one single big repo, or the mixture of them, it can be solved quite perfectly by #72.

To make sure whether contents are updated. We only need one more metadata in mirror pages. For example, in docs/src/zh-cn/config.json

{
    "upstream": "../en"
}

By doing this, we can link two page together, and generate a page mapping from en to zh-cn. Several tasks can be done with this map:

1. If a page is only available in en, it will be directly copied to zh-cn without modification. This often happens when the translation is incomplete.
2. given that each page contains a date timestamp, this page mapping can be used to dynamically generate an outdated warning/info at the beginning of each page of zh-cn.

Note that it only affects downstream mirrors (e.g., zh-cn), it won't affect upstream repo en. We could not assume that upstream is aware of the existence of downstream mirrors.

Of course, upstream can be remote page https://github.com/JuliaImages/juliaimages.github.io.git/[docs/src/pkgs] as is described in #72.

Note that this "upstream" configuration is per page. It is not a per-site upstream. To mirror the whole site, one has to organize the original site into multiple pages, and then configure multiple upstreams for them. For example, there can be three mirror pages for "tutorials", "pkgs", and "demos" in JuliaImages. Besides that, no synchronization is required and developers can be free to write content for better localization purposes (e.g., index.md). Again, given that #72 clones all repo in docs/.democards folder, the same repo won't be cloned multiple times.

#4 is fixed by #7 with the introduction of a post-processing callback function.

There's an EditURL meta recognized by Documenter. So an alternative way of #7 is to insert such information to available md files.

https://juliadocs.github.io/Documenter.jl/dev/man/syntax/#@meta-block-1

A workaround is to manually write them as assets.

Currently, it uses case-insensitive alphabetic order for demos, it might be helpful to allow users to specify a more "meaningful" order preset. For example:

{
    "order": "alphabetic"
}

This doesn't break the current syntax, as the previous order is defined as a list (not a string):

{
  "order": [
    "basics",
    "julia_demos"
    ]
}

Here are presets that I guess would be useful:

• "Alphabetic": case-sensitive alphabetic order.
• "xxx-rev": reversibly order by preset xxx.
• "modified_date": order by git commit dates

Suggestions for different order preset are welcomed.

#md ## References
#md
#md # ```@docs
#md # FlipX
#md # FlipY
#md # ```

outputs

``@example flip
# References
#md
``

``@docs
FlipX
FlipY
``

For example, the following contents are not parsed correctly at present.

#md ```@docs
#md AdditiveWhiteGaussianNoise
#md ```

Currently, demo build failure is suppressed and shown as a warning. For CI/CD purposes, it would be good to explicitly throw an error with perhaps throw_error=true keyword.

The demo script will be executed twice:

The first execution is for the cover generation (I think we can optionally skip this)

https://github.com/johnnychen94/DemoCards.jl/blob/07aa3b70d72e3e3f2332e9d9733c078af69b3a8b/src/types/julia.jl#L145-L146

The second execution is for the Jupyter notebook generation:

https://github.com/johnnychen94/DemoCards.jl/blob/07aa3b70d72e3e3f2332e9d9733c078af69b3a8b/src/types/julia.jl#L170-L175

cc: @findmyway

Currently, all demo files are generated together with the documentation.

In any non-trivial project, there might be tens of demos together with pages, so when someone wants to add a new demo, a lot of time would be spent on generating other demos, especially he/she only makes some typo updates.

I imagine to support the following pipeline:

julia> makedemos("mydemo.jl") # generates a demo page with only one card in a temp dir
a single demo page is generated in <URL>

Users can then view the demo by simply click the URL.

docs/examples
├── color_channels
│   ├── color_separations_svd.jl
│   └── rgb_hsv_thresholding.jl
└── index.md

If there isn't an asset folder in color_channels, the following errors are raised, and an empty cover is generated:

[ Info: SetupDemoCardsDirectory: setting up examples directory.
Errors encountered while saving "assets/color_separations_svd.png".
All errors:
===========================================
WriteBlob Failed `assets/color_separations_svd.png' @ error/png.c/MagickPNGErrorHandler/1641
===========================================
SystemError: opening file assets/color_separations_svd.png: No such file or directory 
===========================================

Fatal error:
┌ Warning: Executing demo docs\examples\color_channels\color_separations_svd.jl fails.
└ @ DemoCards D:\Julia\DemoCards\src\types\julia.jl:170
Errors encountered while saving "assets/rgb_hsv_thresholding.png".
All errors:
===========================================
WriteBlob Failed `assets/rgb_hsv_thresholding.png' @ error/png.c/MagickPNGErrorHandler/1641
===========================================
SystemError: opening file assets/rgb_hsv_thresholding.png: No such file or directory
===========================================

Fatal error:
┌ Warning: Executing demo docs\examples\color_channels\rgb_hsv_thresholding.jl fails.
└ @ DemoCards D:\Julia\DemoCards\src\types\julia.jl:170

Weave.jl as a featured alternative to Literate.jl

https://fredrikekre.github.io/Literate.jl/stable/generated/example/

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

For example: https://johnnychen94.github.io/DemoCards.jl/dev/democards/gallery_of_packages/

As an improvement to the current date: 2020-12-5 usage.

It's very likely that a demo has more meta info than the predefined ones, for example, category, lang, etc.

DemoCards could add them to the source code, but there should be a public protocol that enables users to define their own.

I now believe it will be better if we hold all the meta info into a Dict (instead of a tuple-like thing)

• build/modification time in demo page
• author in demo page
• julia compatibility badge in demo page
• expected reading time in both cover and demo page

and fix possible bugs

Let's take JuliaImages as an example.

Images.jl in JuliaImages consists of different packages, some of which (e.g., ImageFiltering) already have their own documentation. In the meantime, there is a centralized documentation in https://juliaimages.org. It is unclear whether a centralized version or a distributed version is good.

This proposal aims to collect the distributed documentation and rebuild it in a centralized way.

This solution includes two steps:

First, configure things in config.json. For example,

src
├── index.md
└── pkg
    ├── axes
    ├── config.json
    └── segmentation

with docs/src/pkg/config.json filled with following items:

{
    "remote":{
        "filter": [
            "git",
            "https://github.com/JuliaImages/ImageFiltering.jl.git",
            "docs/src"
        ],
        "binarization": "https://github.com/zygmuntszpak/ImageBinarization.jl.git/[docs/src]"
    }
}

Two valid syntaxes can be supported here:

• the first item is a little bit verbose, it says: use git protocol to clone https://github.com/JuliaImages/ImageFiltering.jl.git and collect the whole docs/src folder as one section filter in src/pkg.
• the second item is a more compact version for git, which follows the subdir syntax in Pkg.jl. i.e., pkg> add /tmp/testproject/dev/FooBar.git/[otherfolder/subfolder/Bar]

The type hierarchy for this is:

abstract AbstractRemotePath end

struct GitRemote{T} <: AbstractRemotePath
    path::String
    item::T
end

This type models how remote (git) repo is cloned into local folders during deployment.

Remote folders are not cloned directly into docs/src/pkg folder because that's not friendly to folder arrangement. Instead, all remote repos are cloned into .democards folder. This also enables repo sharing.

Because generally, DemoCards does not allow indirect paths. Not cloneing remote folders into docs/src/pkg requires a new type LocalRemote to break such requirement and models runtime folder link/move/copy.

The workflow is:

1. when DemoPage("src/pkg") notices the existence of remote pages/sections, it will immediately clone those repo into docs/.democards folder, and then generate a LocalRemote placeholder for it.
2. generate(::LocalRemote) would copy the contents at runtime, doing normal generation, and deleting the copied contents to restore the original folder structure.

using DemoCards
template,theme = cardtheme("grid")
MyDemoCards,postprocess_cb = makedemos("democards", template)

makedocs(;
modules = [MyTestPkg],
authors = "DeathGodBXX and contributors",
repo = "https://github.com/DeathGodBXX/MyTestPkg.jl/blob/{commit}{path}#L{line}",
sitename = "MyTestPkg.jl",
format = Documenter.HTML(;
prettyurls = get(ENV, "CI", "false") == "true",
canonical = "https://DeathGodBXX.github.io/MyTestPkg.jl",
edit_link = "master",
assets = String[theme],
),
pages = [
"Home" => "index.md",
"MyDemoCards" => MyDemoCards,
],
)

postprocess_cb()

demo2.jl has generated .md ,.ipynb,*.html ,but don't has nbviewer,what should I do in make.jl ?

The current demo cards examples are so trivial that it doesn't show the real power of it

• Theme: None currently does not respect hidden property, which is a bug.
• If all demos in the folder are hidden, then it should not appear in the sidebar

In demo4.jl,when I want to generate the cover autoly
# ---
# id: demo4
# cover: assets/demo4.png
# description: plot cameraman
# ---

the content of demo4.jl:
using Images,TestImages
t = testimage("cameraman")
#-
save("assets/demo4.png",t)

Somthing is wrong.Before inserting these code,everything is ok.

julia> include("docs/make.jl")
ERROR: LoadError: expected '' but found YAML.BlockMappingStartToken at nothing
Stacktrace:
[1] parse_document_start(::YAML.EventStream) at C:\Users\26947.julia\packages\YAML\e31Qv\src\parser.jl:167
[2] peek(::YAML.EventStream) at C:\Users\26947.julia\packages\YAML\e31Qv\src\parser.jl:54
[3] compose(::YAML.EventStream) at C:\Users\26947.julia\packages\YAML\e31Qv\src\composer.jl:39
[4] load(::YAML.TokenStream, ::Nothing) at C:\Users\26947.julia\packages\YAML\e31Qv\src\YAML.jl:23
[5] load at C:\Users\26947.julia\packages\YAML\e31Qv\src\YAML.jl:28 [inlined]
[6] load at C:\Users\26947.julia\packages\YAML\e31Qv\src\YAML.jl:71 [inlined] (repeats 2 times)
[7] parse(::Val{:Julia}, ::DemoCards.JuliaDemoCard) at C:\Users\26947.julia\packages\DemoCards\XlSmE\src\util
s.jl:136
[8] parse at C:\Users\26947.julia\packages\DemoCards\XlSmE\src\utils.jl:146 [inlined]
[9] load_config(::DemoCards.JuliaDemoCard, ::String) at C:\Users\26947.julia\packages\DemoCards\XlSmE\src\typ
es\card.jl:41

[10] DemoCards.JuliaDemoCard(::String) at C:\Users\26947.julia\packages\DemoCards\XlSmE\src\types\julia.jl:67

[11] democard(::String) at C:\Users\26947.julia\packages\DemoCards\XlSmE\src\types\card.jl:19
[12] iterate at .\generator.jl:47 [inlined]
[13] _collect(::Array{String,1}, ::Base.Generator{Array{String,1},typeof(DemoCards.democard)}, ::Base.EltypeUn
[23] DemoCards.DemoPage(::String) at C:\Users\26947.julia\packages\DemoCards\XlSmE\src\types\page.jl:100
[24] makedemos(::String, ::Dict{String,Mustache.MustacheTokens}; root::String, destination::String, src::Strin
g, build::String, branch::String, edit_branch::String, credit::Bool) at C:\Users\26947.julia\packages\DemoCard
s\XlSmE\src\generate.jl:86
[25] makedemos(::String, ::Dict{String,Mustache.MustacheTokens}) at C:\Users\26947.julia\packages\DemoCards\X
lSmE\src\generate.jl:78
[26] top-level scope at D:\ATOM\atom-x64-windows\Location\Code\MyTestPkg\docs\make.jl:11
[27] include(::String) at .\client.jl:457
[28] top-level scope at REPL[2]:1
in expression starting at D:\ATOM\atom-x64-windows\Location\Code\MyTestPkg\docs\make.jl:11

Unlike JuliaImages, other repo might not need a layout of covers for their demos. Instead, they might just want the entire demo structure layer inserted into the sidebar generated by Documenter.

The generated index.md might not be wanted, too.

Need to implement #27 first.

Currently, it is limited to local image file, but it would be nice to also support remote image URL as the cover image.

I see two ways to do so:

• Doesn't trigger the asset generation process if there aren't missing assets.
• generate assets in one common process; this avoids reloading large packages such as Images. 9cd009e

For example, I'd like to change the following line:

https://github.com/johnnychen94/DemoCards.jl/blob/e56c25843ea5584e62c9272a0a65146e47dc31f0/src/Themes/grid/style.css#L36

to display:inline.

Currently, DemoCards does not support folders with files other than .jl and .md, so if there're .py files in the folder, it errors.

There're two possible ways to enhance this:

1. provide a filename filter API so that users can decide what filename rules should DemoCards respect.
2. provide a handler API so that users can decide how DemoCards process the file( if the handler is a NoOp then it means this file will be discarded by DemoCards.)

The second way seems more flexible and powerful speaking of extensibility.

The following content is a valid markdown content, but the indentation information is lost here because whitespaces are trimmed

# !!! note
#    Although `Gray.(img)` could infer the storage type as well, `ConvertEltype(Gray)` on the other
#    hand, does not do so. To get better performance, it is recommended to specify the storage type
#    when you use this operation, e.g., `ConvertEltype(Gray{Float32})`.

The only reason I used config.json when writing this package is that I'm not familiar with the YAML syntax :)

In JuliaImages/juliaimages.github.io#159, even though the "order" is specified:

{
    "order":[
        "introduction",
        "axes",
        "metadata",
        "features",
        "segmentation",
        "transformations"
    ]
}

it uses some unknown order instead of the customized order.