Describe the bug

context
When I install other pip packages I get conflicts because of the restriction from rst-to-myst.

expectation
I expected it to not have this restriction.

$ pip install otherpackage
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
rst-to-myst 0.3.2 requires click~=7.1, but you have click 8.1.3 which is incompatible.

problem
This is a problem for people wanting to have other packages installed.

Reproduce the bug

Install rst-to-myst and try to install for example molecule[docker].

List your environment

No response

Describe the problem

The following input fails to convert:

.. sidebar:: _`Some label`

   ABC

The error is:

test.rst -> test.md
FAILED:
error: block-level target has children

Link to your repository or website

No response

Steps to reproduce

rst2myst convert test.rst with the document above

The version of Python you're using

3.8

Your operating system

GNU/Linux (Mint)

Versions of your packages

$ rst2myst --version
rst2myst, version 0.3.2
$ python3 --version
Python 3.8.10
$ sphinx-build --version
sphinx-build 3.5.4
$ pip show myst_parser | grep Version
Version: 0.15.1

Additional context

This is the corresponding pseudoXML:

<document source="test.rst">
    <sidebar>
        <title>
            <target ids="some-label" names="some\ label">
                Some label
        <paragraph>
            ABC

There is not yet a direct equivalent in MyST Markdown for line blocks (see https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#line-blocks)

This could/should be added though, via executablebooks/mdit-py-plugins#10, then adding a new extension in myst-parser

Describe the bug

When using on Windows10 with the command rst2myst convert *.rst
it produces:

Usage: rst2myst convert [OPTIONS] [PATHS]...
Try 'rst2myst convert -h' for help.

Error: Invalid value for '[PATHS]...': Path '*.rst' does not exist.

I triple-checked the if I am in the right directory, where my .rst files are stored.
Also tried quoting the path.

Does anybody experienced that issue? On Linux it worked as expected.

Reproduce the bug

1. in a conda environment install with pip install rst-to-myst[sphinx]
2. in terminal navigate to the folder where *.rst files are stored
3. execute the command rst2myst convert *.rst

List your environment

No response

Context

Dependabot is flagging (GHSA-vrjv-mxr7-vjf8) and (GHSA-jrwr-5x3p-hvc3) as vulnerabilities, but is unable to provide a resolution, as it requires markdown-it-py to be at version 2.2.0 or greater, while this project has "markdown-it-py~=1.0" in pyproject.toml

Proposal

Update markdown-it-py version to 2.2.0, assuming there are no breaking API changes.

Tasks and updates

No response

Describe the problem

Sphinx uses docinfo blocks to include information like author name or date. When these are converted from reST to MyST they become front-matter, but these isn't interpreted the same way by the MyST compiler.

For example, this document:

=======
 Title
=======

:author: Me
:date: Today

Becomes this when converted:

---
author: Me
date: Today
---

# Title

But the first one renders as this (with Sphinx):

while the second one renders as that:

This could be a bug in the MyST compiler, too.

Link to your repository or website

No response

Steps to reproduce

sphinx-quickstart, then add the contents above to a new file.

The version of Python you're using

3.8

Your operating system

GNU/Linux (Mint)

Versions of your packages

$ rst2myst --version
rst2myst, version 0.3.2
$ python3 --version
Python 3.8.10
$ sphinx-build --version
sphinx-build 3.5.4
$ pip show myst_parser | grep Version
Version: 0.15.1

Additional context

No response

These are notes from converting the documentation of the https://github.com/jupyterhub/zero-to-jupyterhub-k8s project. I just wrote that came to mind along as I used it for the first time with no prior knowledge other than its a cli to help me transform rst to myst.

• roles list command output line separated roles instead of a big string for better human readability.
• roles show help string sais the same as list.
• directives list/show same as above
• --help could perhaps say something more relevant than CLI for rst-to-myst, such as the focus of its purpose.

  Usage: rst2myst [OPTIONS] COMMAND [ARGS]...
  
  CLI for rst-to-myst
  

• parse --help or the help of the parse --conversion flag could perhaps provide some reference or be a bit more verbal about it maybe.
• parse --extensions help text doesn't describe how to specify the extensions, comma separated list to load?
• `parse --sphinx/--no-sphinx doesn't make it clear what is the default
• Trailing blank lines.
  • The conversion of a .rst file with only hello world followed by a blank line leads to a .md output of hello world with 3 blank lines
  • The conversion of the following

    .. image:: ../_static/images/architecture.png
       :align: center
    

    Led to a blank new line in the image directive

    ```{image} ../_static/images/architecture.png
    :align: center
    
    ```
    

• I noted that my generated myst directives was using colon-syntax for directives like note, and important, but not for code-block. I ended up with a mix of colon-syntax and backtick syntax for my myst directives. I had not enabled colon-syntax manually in any way or by configuration.
• It would be nice to have a --validate flag or similar for a parsing, giving hints about the transform rather than just outputting the output without warnings etc.
• I found myself using rst2myst parse -f myfile.rst > myfile.md a lot, it could be nice to have some flag to modify the file in place or similar perhaps. -i, --in-place for example.
• I learned that it wasn't possible to do rst2myst parse -f test.md > test.md for some reason I don't get. I would guess this would be reasonable to have working in general though, for example cat test.md | sed replacesomething > test.md. Perhaps its because we started printing content before we had finished reading it?
• Having used some double ticks in rst translated to doubleticks in md where I expected to have only one
• I'm surprised to see this conversion of links...

  # rst
  `Google <https://cloud.google.com/>`_)
  
  # myst - why is there a <> remaining? It seems fine though.
  [Google](<https://cloud.google.com/>)
  

• I failed to transform the glossary directive to myst, but the automated generated MyST content containing a eval-rst block worked.
• Converting a .. toctree:: directive with a :titlesonly: argument worked in a way, but the generated MyST content was invalid. It became :titlesonly: None which made a difference to the toctree where nothing was shown instead of only the titles.

Is your feature request related to a problem? Please describe.

The dollarmath feature of MyST is huge for readability of math-intensive writing. The current rst2myst parse produces a direct translation that has similar verbosity to ReST.

Describe the solution you'd like

I'd like to have an option that enables inline $\pi$ and displayed

$$
e = mc^2
$$ (eqn:best)

This is the best equation {eq}`eqn:best`

We may be able to help with the implementation, but have a couple questions:

1. Since dollarmath is an optional feature, I'm curious how you would like this gated in the interface. I could see it being managed by new keys in directives.yml, though a new --myst-extensions dollarmath would be more convenient.
2. Any notes on preferred implementation strategy.

Cc: @DiffeoInvariant

Using https://github.com/phha/click_config_file would be particularly helpful with extension lists

When converting to myst syntax that requires extensions, it would be good in case the user was made aware of these required extensions somehow.

The description below was my original post, which thought the conversion was buggy because I wasn't aware of the extension colon_fence.

Describe the bug

When converting e.g. a figure directive

.. _fig_intro_workchain_graph:
.. figure:: include/images/basics_workchain_graph.png
    :scale: 30
    :align: center

    Provenance Graph of a basic AiiDA WorkChain.

This gets converted into

(fig-intro-workchain-graph)=
:::{figure} include/images/basics_workchain_graph.png
:scale: 30
:align: center

Provenance Graph of a basic AiiDA WorkChain.

:::

Expected behavior

Doesn't myst-nb need backticks for the directives? I.e. if I convert this into

(fig-intro-workchain-graph)=

```{figure} include/images/basics_workchain_graph.png
:scale: 30
:align: center

Provenance Graph of a basic AiiDA WorkChain.
```#

This works fine for me.

Similar issues arise for note and important directives (and possibly others that are not in the file I'm converting).

Environment

• Python Version [e.g. 3.7.1]: 3.8.6
• Package versions or output of jupyter-book --version:

  $ pip freeze | grep myst
  myst-nb==0.13.0a1
  myst-parser==0.14.0a2
  rst-to-myst==0.1.2

• Operating System: macOS Big Sur 11.3.1

Is your feature request related to a problem? Please describe.

When trying to convert a rubric directive, e.g.:

.. rubric:: Footnotes

.. [#f1] We purposefully do not provide advanced commands for crystal structure manipulation in AiiDA, because python packages that accomplish such tasks already exist (such as ASE or pymatgen).

rst2myst parse -f produces:

:::{rubric} Footnotes

:::

[^f1]: We purposefully do not provide advanced commands for crystal structure manipulation in AiiDA, because python packages that accomplish such tasks already exist (such as ASE or pymatgen).

This causes the build to fail with the following warning:

WARNING: toctree contains reference to document 'sections/fundamentals/basics' that doesn't have a title: no link will be generated

Proposed solution

I tried using the extended markdown syntax for footnotes, but maybe mystnb doesn't support this? If no support can be added, maybe a warning would be useful during parsing, since the warning returned during the build does not point towards the footnotes issue.

I'm not sure exactly when this happens, but I noticed that sometimes the proper indentation isn't applied to the output content if the original was indented. For example:

2. It must explicitly specify a tag in the image you source.

   When sourcing a pre-existing Docker image with ``FROM``,
   **a tag is required**. The tag *cannot* be ``latest``. Note that tag
   naming conventions differ between images, so we recommend using
   the SHA tag of the image.

converts to

2. It must explicitly specify a tag in the image you source.

   When sourcing a pre-existing Docker image with `FROM`,
**a tag is required**. The tag *cannot* be `latest`. Note that tag
   naming conventions differ between images, so we recommend using
   the SHA tag of the image.

Perhaps this is related to the ** characters?

Describe the problem

Table conversions that rely on eval-rst are using wrong directive "eval_rst" which doesn't actually appear to do anything in sphinx. Notice the difference in the dash and underscore.

Link to your repository or website

No response

Steps to reproduce

generate document with a table in rst that is not convertible to markdown, table with column span for example.

Simple table:

=====  =====  ======
   Inputs     Output
------------  ------
  A      B    A or B
=====  =====  ======
False  False  False
True   False  True
False  True   True
True   True   True
=====  =====  ======

the converted output will have incorrect directive (notice the underscore)

```{eval_rst}
Simple table:

=====  =====  ======
   Inputs     Output
------------  ------
  A      B    A or B
=====  =====  ======
False  False  False
True   False  True
False  True   True
True   True   True
=====  =====  ======
```

hand editing the underscore to be a dash allows sphinx to correctly build the converted markdown.

The version of Python you're using

No response

Your operating system

No response

Versions of your packages

No response

Additional context

No response

One challenge of us introducing MyST to the Sphinx world is that almost all of the documentation is written in rST, not MyST Markdown, and many folks will likely want to keep rST in their documentation rather than switch to MyST entirely.

For that group, it would be great if there were a way to display MyST syntax without requiring them to re-write their example usage.

So I wonder if we could use this tool to provide a directive like:

.. rst-myst-demo::
   .. admonition::
      My admonition!

That would effectively mimic the following behavior:

.. tabbed:: reStructuredText
   .. code-block:: rst
      .. admonition::
         My admonition `with a link <https://google.com>`_!

.. tabbed:: MyST Markdown
   .. code-block:: md
      ```{admonition}
      My admonition [with a link](https://google.com).
      ```

.. admonition::
   My admonition!

This way, people could take their pre-existing rST examples, and simply wrap them in .. rst-myst-demo:: and they'd get MyST generated for their examples as well.

An example of what this could look like in practice is @pradyunsg's excellent reference material for Furo here:

https://pradyunsg.me/furo/reference/admonitions/

where he shows "split" views of rST and MyST markdown for all of the directives.

(ideally, it would be possible to do this for MyST markdown -> rst as well, but I don't believe that functionality exists currently)

Describe the problem

The way rst2myst translates rst substitutions to Markdown does not seem to work properly.
So far 2 inconsistencies have been identified:

1. rst2myst relies on MyST ability to use Jinja2 for substitutions. However Jinja2 requires a valid identifier (i.e., valid Python variable names), which is a much more restrictive condition, and therefore not compatible with rst (e.g. rst accepts strings with spaces)

2. rst2myst fails on translating links with substitutions (e.g. |ref|_), which are a very common use case when writing rst (e.g. for creating links with bold, italic or monospaced text elements)

This issue was first identified in pyscaffold/pyscaffoldext-markdown#25. A simplified description of the problem and steps to reproduce it can be found in this gist.

Link to your repository or website

https://gist.github.com/abravalheri/a9cdfa09f2809182fccca659b413da4a

Steps to reproduce

The following is a simplified example to reproduce the problem:

1. Assuming I have the following (example) in ReStructuredText:

   #. To avoid any problems with your installed Python packages, consider using |virtual environments|_
   
   #. If everything works fine, push your local branch to |the repository service| with::
   
       git push -u origin my-feature
   
   #. Go to the web page of your fork and click |contribute button| to send your changes for review.
   
   .. |virtual environments| replace:: ``virtualenv``
   .. |the repository service| replace:: GitHub
   .. |contribute button| replace:: "Create pull request"
   .. _virtual environments: https://virtualenv.pypa.io/en/stable/

2. When I try to convert it using the CLI (rst2myst stream original.rst > converted.md), I obtain the following MyST-flavoured markdown:

   ---
   substitutions:
     contribute button: '"Create pull request"'
     the repository service: GitHub
     virtual environments: '`virtualenv`'
   ---
   
   1. To avoid any problems with your installed Python packages, consider using [virtual environments]
   
   2. If everything works fine, push your local branch to {{ the repository service }} with:
   
      ```
      git push -u origin my-feature
      ```
   
   3. Go to the web page of your fork and click {{ contribute button }} to send your changes for review.
   
   [virtual environments]: https://virtualenv.pypa.io/en/stable/

As we can see, converted markdown contains invalid Jinja2 syntax, e.g. {{ the repository service }}, and the original link with substitution was converted to a simple link, without a substitution.

The version of Python you're using

3.8.0

Your operating system

Ubuntu 18.04.5 LTS

Versions of your packages

Direct installation:

myst-parser==0.15.2
rst-to-myst[sphinx]==0.3.2
Sphinx==4.1.2

The remaining packages where installed as dependencies of those 3 in an empty virtual environment created with:

$ virtualenv -p py38 .venv
$ source .venv/bin/activate
$ pip install sphinx myst-parser 'rst-to-myst[sphinx]'

Expected output

Two different behaviours could be expected from rst2myst:

1. The "rst substitution key" could be pre-processed to produce a valid Python identifier and in turn, that identifier could be used in Jinja2.
   This include replacing or removing invalid characters, e.g. |the repository service| could be translated to
   {{ the_repository_service }}.
   This approach in shown in expected_option1 bellow.
2. The "rst substitution key" could be nested inside a dict value with an arbitrary name (e.g. __),
   e.g. |the repository service| could be translated to {{ __["the repository service"] }}.
   This approach in shown in expected_option2 bellow

Regarding links with substitutions, CommonMark's full reference links in combination with Jinja2 seem to be a good alternative, e.g. |ref|_ could be translated to [{{ ref }}][ref]. This approach is shown in both expected_option1 and expected_option2.

expected_option1

---
substitutions:
  contribute_button: '"Create pull request"'
  the_repository_service: GitHub
  virtual_environments: '`virtualenv`'
---

1. To avoid any problems with your installed Python packages, consider using [{{ virtual_environments }}][virtual environments]

2. If everything works fine, push your local branch to {{ the_repository_service }} with:

   `git push -u origin my-feature`

3. Go to the web page of your fork and click {{ contribute_button }} to send your changes for review.

[virtual environments]: https://virtualenv.pypa.io/en/stable/

expected_option2

---
substitutions:
  "__":
    contribute button: '"Create pull request"'
    the repository service: GitHub
    virtual environments: '`virtualenv`'
---

1. To avoid any problems with your installed Python packages, consider using [{{ __["virtual environments"] }}][virtual environments]

2. If everything works fine, push your local branch to {{ __["the repository service"] }} with:

   `git push -u origin my-feature`

3. Go to the web page of your fork and click {{ __["contribute button"] }} to send your changes for review.

[virtual environments]: https://virtualenv.pypa.io/en/stable/

In expected_option2, instead of using a nested dict __ inside of substitutions, the substititions dict itself could also be directly assigned to __ in the Jinja2 template context, this would allow users using any substitution key directly if they are valid identifiers and invalid identifiers via the __ helper.

(code blocks around git push ... in expected_option1|2 omitted due to GitHub's inability of dealing with nested code blocks)

Describe the bug

context
I have a project where I want to export the output to both html and commonmark, but the usage of sphinx version <5 makes is impossible to use with other sphinx extensions that require newer versions (furo is an example).

expectation
I should be able to run rst2myst alongside furo.

bug
But instead I see an error when trying to install them together that states that rst-to-myst and furohave incompatible sphinx version requirements.

problem
This is a problem for people installing rst-to-myst and furo or other extensions that depend on sphinx>=5.

Reproduce the bug

$ poetry add sphinx rst-to-myst furo
Using version ^6.2.1 for sphinx
Using version ^0.3.4 for rst-to-myst
Using version ^2023.3.27 for furo

Updating dependencies
Resolving dependencies... (1.0s)

Because no versions of rst-to-myst match >0.3.4,<0.4.0
 and rst-to-myst (0.3.4) depends on docutils (>=0.15,<0.18), rst-to-myst (>=0.3.4,<0.4.0) requires docutils (>=0.15,<0.18).
And because sphinx (6.2.1) depends on docutils (>=0.18.1,<0.20)
 and no versions of sphinx match >6.2.1,<7.0.0, rst-to-myst (>=0.3.4,<0.4.0) is incompatible with sphinx (>=6.2.1,<7.0.0).
So, because myproject depends on both sphinx (^6.2.1) and rst-to-myst (^0.3.4), version solving failed.

List your environment

Python 3.11.3
Poetry 1.4.1

@choldgraf @mmcky @AakashGfude

As mentioned on Slack, this is in part a replacement for https://github.com/executablebooks/sphinx-tomyst.
The key difference/improvement is that it directly uses and adapts the docutils/rst parsing code to produce a lossless AST, notably not actually running directives or roles; so you don't have to reverse engineer what they were.
You also don't need to run the whole sphinx process (or need a conf.py); just point it at a file/piece of RST and it will convert it.

Additionally, it has some other tricks up its sleeve, such as being able to list all available roles/directives and showing the options for a particular one.

The documentation is all in the README, so yeh give it a go and report back:
if you have any good exemplar RST files to try against it (and maybe add to the tests), or if any other feature requirements come to mind?

Describe the bug

Lists that have been indented in .rst are converted in such a way that they aren't rendered properly.

To Reproduce

When trying to convert the following .rst snippet:

This is a numbered list!

#. Step 1
#. Step 2
#. Step 3
#. Step 4

This is a numbered list with indentation!

    #. Step 1
    #. Step 2
    #. Step 3
    #. Step 4

This is a regular list with indentation!

    * Step 1
    * Step 2
    * Step 3
    * Step 4

You obtain

This is a numbered list!

1. Step 1
2. Step 2
3. Step 3
4. Step 4

This is a numbered list with indentation!

> > 1. Step 1
> 2. Step 2
> 3. Step 3
> 4. Step 4

This is a regular list with indentation!

> > * Step 1
> * Step 2
> * Step 3
> * Step 4

Expected behavior

The > should probably be removed, even though this means there is no difference between the indented and non-indented case (which is different for .rst).

Environment

• Python Version [e.g. 3.7.1]: 3.8.6
• Package versions or output of jupyter-book --version:

Sphinx 2.4.4
rst-to-myst                   0.1.2
mypy-extensions               0.4.3
myst-nb                       0.13.0a1
myst-parser                   0.14.0a2

• Operating System: macOS Big Sur 11.3.1

Is your feature request related to a problem? Please describe.

When converting all files, a new file is created as a sibling to the original. I must copy all these files to separate directory to build the converted files and compare to the original.

Describe the solution you'd like

It would be nice to be able to designate an output directory separate from the source directory. This would make it easier to build the converted files.

Describe alternatives you've considered

There might be an existing option to redirect output of which I am not aware, whether that is with this app or a shell command. I might create a shell script that uses rsync to move files into a separate directory to build the converted docs.

Describe the bug

Here

Reproduce the bug

.

List your environment

No response

Add a command to recurse through a whole folder and convert all .rst files to .md ones

Describe the bug

Underscores _ inside a target header seem to get converted into hyphens -, but not for the references. E.g. this .rst snippet:

.. _test_snippet:

.. code-block:: Python

    # This is a test snippet.

Dude, check out that :ref:`test snippet <test_snippet>`!

Gets converted into:

(test-snippet)=

```{code-block} Python

# This is a test snippet.

```#

Dude, check out that {ref}`test snippet <test_snippet>`!

Expected behavior

Either the underscores should be preserved, or both the target header and references should be converted into hyphens, in case this is preferred.

Environment

• Python Version [e.g. 3.7.1]: 3.8.6
• Package versions or output of jupyter-book --version:

  $ pip freeze | grep myst
  myst-nb==0.13.0a1
  myst-parser==0.14.0a2
  rst-to-myst==0.1.2

• Operating System: macOS Big Sur 11.3.1