pappasam / toml-sort Goto Github PK

I am using toml-sort==0.23.1 with the below pyproject.toml config:

[tool.tomlsort]
all = true
in_place = true
trailing_comma_inline_array = true

I integrate toml-sort into my pre-commit like so:

  - repo: https://github.com/pappasam/toml-sort
    rev: v0.23.1
    hooks:
      - id: toml-sort-fix

Running identify to see the associations on poetry.lock:

> identify-cli poetry.lock
["file", "non-executable", "text", "toml"]

We can see it's considered TOML, so toml-sort will pick it up. However, poetry.lock is made programmatically, so it should not be touched by toml-sort.

How can one skip a particular file? I don't see anything in toml-sort --help that will exclude or skip files.

Is the only route right now to use an exclude within pre-commit?

Hi, in the context of pretty-format-toml I was recently surprised by the disappearance of a multi-paragraph comment. After investigation I discovered that it was interpreted by toml-sort as an orphan comment.

For example,

# paragraph 1

# paragraph 2
[a-section]

and then # paragraph 1 vanishes.

I ended up solving this with

# paragraph 1
#
# paragraph 2
[a-section]

Would it make sense to add an option which changes the behavior so that orphan comments are added to the next section rather than deleted?

Hey,
first thanks a lot for creating this awesome tool.

I have a feature request and I'm happy to to provide a PR for it. If the tool is used in CI but also locally, you might want to have common options in addition to context sensitive options. For example, in the CI you use --check while locally you use -i. But in both cases the toml files are the same but you need to specify them twice.

I would suggest to offer configuration in the toml file

[tool.toml_sort]
files = ["foo.toml"]
no_header = true

Basically, the tool will first check if there is a pyproject.toml with a toml_sort configuration and load the configuration if applicable. Second, the CLI is used and will override any value if specified there

TOML file:

# foo.toml

[tool.foo]
bar = []

Exception:

$ python -V
Python 3.11.0

$ toml-sort --version
0.22.0

$ toml-sort foo.toml
Traceback (most recent call last):
  File "/private/tmp/foo/.venv/bin/toml-sort", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/private/tmp/foo/.venv/lib/python3.11/site-packages/toml_sort/cli.py", line 356, in cli
    ).sorted()
      ^^^^^^^^
  File "/private/tmp/foo/.venv/lib/python3.11/site-packages/toml_sort/tomlsort.py", line 634, in sorted
    sorted_toml = self.toml_doc_sorted(toml_doc)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/foo/.venv/lib/python3.11/site-packages/toml_sort/tomlsort.py", line 620, in toml_doc_sorted
    item.key, self.toml_elements_sorted(item, sorted_document)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/foo/.venv/lib/python3.11/site-packages/toml_sort/tomlsort.py", line 439, in toml_elements_sorted
    item.key, self.toml_elements_sorted(item, previous_item)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/foo/.venv/lib/python3.11/site-packages/toml_sort/tomlsort.py", line 435, in toml_elements_sorted
    for item in self.sorted_children_table(original.children):
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/foo/.venv/lib/python3.11/site-packages/toml_sort/tomlsort.py", line 387, in sorted_children_table
    non_tables = self.sort_items(
                 ^^^^^^^^^^^^^^^^
  File "/private/tmp/foo/.venv/lib/python3.11/site-packages/toml_sort/tomlsort.py", line 359, in sort_items
    item.value = self.sort_item(item.value)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/foo/.venv/lib/python3.11/site-packages/toml_sort/tomlsort.py", line 306, in sort_item
    return self.sort_array(item, indent_depth=indent_depth)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/foo/.venv/lib/python3.11/site-packages/toml_sort/tomlsort.py", line 284, in sort_array
    new_array_value[-1].comma = Whitespace("")
    ~~~~~~~~~~~~~~~^^^^
IndexError: list index out of range

Hi! thanks for creating great tool :)
I think I have encountered an issue whiling upgrading version 0.20.1 -> 0.22.2

it seems it is not compatible with tomlkit 0.8.0 and poetry doesn't upgrade it

I would like the option of using a different config file (not the CLI) that isn't a pyproject.toml. Mainly because gives the impression it's a python code base which in my (current) use case it isn't.

Ruff made some config changes that required adding .lint but to some options. Once you do this, when toml-sort runs, it introduces some breaking changes by removing the prefixes:

I was not able to find any config option that might prevent this from happening so I was forced to disable the hook for the moment.

Taking an example from the PEP documentation.

[project]
urls.homepage = "https://example.com"
urls.documentation = "https://readthedocs.org"}

Running toml-sort results in

Traceback (most recent call last):
  File "/Users/paddy/mambaforge/bin/toml-sort", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/Users/paddy/mambaforge/lib/python3.11/site-packages/toml_sort/cli.py", line 297, in cli
    args = get_parser().parse_args(args=arguments)  # strip command itself
           ^^^^^^^^^^^^
  File "/Users/paddy/mambaforge/lib/python3.11/site-packages/toml_sort/cli.py", line 289, in get_parser
    parser.set_defaults(**load_config_file())
                          ^^^^^^^^^^^^^^^^^^
  File "/Users/paddy/mambaforge/lib/python3.11/site-packages/toml_sort/cli.py", line 89, in load_config_file
    document = tomlkit.parse(content)
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/paddy/mambaforge/lib/python3.11/site-packages/tomlkit/api.py", line 83, in parse
    return Parser(string).parse()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/paddy/mambaforge/lib/python3.11/site-packages/tomlkit/parser.py", line 158, in parse
    key, value = self._parse_table()
                 ^^^^^^^^^^^^^^^^^^^
  File "/Users/paddy/mambaforge/lib/python3.11/site-packages/tomlkit/parser.py", line 999, in _parse_table
    item = self._parse_item()
           ^^^^^^^^^^^^^^^^^^
  File "/Users/paddy/mambaforge/lib/python3.11/site-packages/tomlkit/parser.py", line 242, in _parse_item
    return self._parse_key_value(True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/paddy/mambaforge/lib/python3.11/site-packages/tomlkit/parser.py", line 338, in _parse_key_value
    cws, comment, trail = self._parse_comment_trail()
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/paddy/mambaforge/lib/python3.11/site-packages/tomlkit/parser.py", line 284, in _parse_comment_trail
    raise self.parse_error(UnexpectedCharError, c)
tomlkit.exceptions.UnexpectedCharError: Unexpected character: '}' at line 3 col 46

This can be modified to

[project]
urls = {homepage = "https://example.com", documentation = "https://readthedocs.org"}

and now toml-sort will work as expected. However, from the error message, it is not at all obvious that this is what is required.

Because toml-sort does not explicitly pass a newline parameter to open() in cli.read_file() and cli.write_file(), Python defaults to universal newlines mode.

This means that formatted files always get LF line endings on most operating systems and CRLF line endings on MS Windows. Would it be possible to either keep the original newlines or add an option to force a specific line ending?

This bug was reported here macisamuele/language-formatters-pre-commit-hooks#198, but it is actually a bug in toml-sort.

The minimal reproducible toml file for this error is

[x]
v.w = ""

[x.r]

which gets formatted to:

[x]

[x.r]
v.w = ""

As is, --check doesn't change the file that is supposed to be sorted, despite --help explicitly stating:

"Check if an original file is changed by the formatter."

which suggests changes are being made to the original file (they aren't).

Proposed solution: either rename option to indicate it only checks if the file is considered sorted or not - --check-only or --is-sorted, with the latter suggesting no changes and just true/false return value explicitly.

Alternatively, I'd suggest changing the --help message to be more clear:

"Check if an original file would be changed by the formatter."

If you give toml-sort a file that only contains comments, then it will print the contents of that file followed by a newline followed by the contents of the file again:

$ cat just-a-comment.toml 
# This TOML file only contains a comment.
$ toml-sort just-a-comment.toml 
# This TOML file only contains a comment.

# This TOML file only contains a comment.
$

The comment

# init_command = [
#     'python',
#     'examples/pytorch/s3_init.py',
#     '--config-json=examples/pytorch/s3_init_config.json',
#     '--linter=clang-format',
#     '--dry-run={{DRYRUN}}',
#     '--output-dir=.lintbin',
#     '--output-name=clang-format',
# ]

is formatted as

# init_command = [
# 'python',
# 'examples/pytorch/s3_init.py',
# '--config-json=examples/pytorch/s3_init_config.json',
# '--linter=clang-format',
# '--dry-run={{DRYRUN}}',
# '--output-dir=.lintbin',
# '--output-name=clang-format',
# ]

It would be great if the leading spaces are preserved.

pylint provides an examples module that has several example configs (pylintrc, pyproject.toml). I use these to speed up adding new config options (and it promotes awareness of the available config options).

A huge benefit of this is also revealing the default behavior of all config options.

I think it would be cool if toml-sort had something similar. Fwiw, https://github.com/pappasam/toml-sort/tree/b9b6210da457c38122995e434b314f4c4a4a923e#configuration-file has an example config, but it doesn't document if it's the "default" config.

Proposal

The request here is to add example config(s) somewhere to docs/

I tried out your library and it is really helpful. Thank you for your contribution.

Question: Do you think it is worthwhile to add an option to preserve (multi-line) comments that precede a key-value pair?

Example (from a pyproject.toml in one of my projects):

[tool.poetry.dependencies]
click = "^7.1.2"
coverage = {extras = ["toml"], optional = true, version = "^5.1"}
pytest = {optional = true, version = "^5.4.3"}
pytest-cov = {optional = true, version = "^2.10.0"}
pytest-xdist = {optional = true, version = "^1.32.0"}
python = "^3.8"
tox = {optional = true, version = "^3.16.1"}

[tool.poetry.extras]
# line1
# line2
# line3
develop = "..."
testing = "..."

[tool.poetry.scripts]
version = "..."

I'm working on a cookiecutter template where we have something like this

[project]
classifiers = [
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3 :: Only",
    {%- for python_version in range(
       cookiecutter.min_python_version | replace('3.', '') | int,
       cookiecutter.max_python_version | replace('3.', '') | int + 1
    ) %}
    "Programming Language :: Python :: 3.{{ python_version }}",
    {%- endfor %}
]

[tool.tomlsort]
all = true
spaces_indent_inline_array = 4
trailing_comma_inline_array = true
overrides."project.classifiers".inline_arrays = false

This works fine but toml-sort panics (understandably so). For now, I'm excluding the templated pyproject.toml from the pre-commit hooks at the top level and creating a dummy project inside that and linting in there, i.e. https://github.com/UCL-ARC/python-template/blob/paddys-improvement/.github/workflows/linting.yml. Is there a way for this to work nicely?

Hey,

Great tool, one slight improvement I would suggest if it isn't too hard is around quoted keys.

Firstly, I think it is great that if you have something like:

'key1' = 'value1'
"key2" = "value2"

It will get changed to:

'key1' = 'value1'
"key2" = "value2"

That is great, given that there is no need to put those keys in quotes.
However, I have a case where I do require a . within a key.
So for that I define it as a literal quote (single quotes), i.e.,

'namespace.package' = 'my_value'

And currently toml-sort changes that to:

"namespace.package" = 'my_value'

So it changes the quoted key from a literal string to a basic string. This isn't the end of the world, but some others I work with have a strong preference for using literal strings in toml rather than basic strings so it looks a bit odd having a file full of single quotes and one string with double quotes.

Also, toml-sort changes the following (one dumb example, and one string from the quoted keys section of the toml spec:

'C:\Users\nodejs\templates' = true
'quoted "value"' = "value"

to:

"C:\Users\nodejs\templates" = true
"quoted \"value\"" = "value"

One would argue that changing 'quoted "value"' to "quoted "value"" has decreased the readability of that string which is not ideal.
The other string (I accept, no one should ever have a path as a key... but technically it is still valid toml) is more problematic. After toml-sort has changed that from single to double quotes then tomlkit can no longer parse the file.
Interestingly, even if I set that key to "C:\\Users\\nodejs\\templates" then toml-sort changes it back to "C:\Users\nodejs\templates" which then can't be parsed.

Again, no one should have such a stupid key, and I don't have anything like that (I just have one key with a . in it).
But toml-sort also shouldn't take a parsable input and end up with a non-parsable output.
And it would also be great if toml-sort preserved the string type of the key if a quoted key is required.

See toml-lang/toml#904

When running github action pre-commit hook I get

Pretty format TOML............................................................Failed
- hook id: pretty-format-toml
- exit code: 1
Input File pyproject.toml is not a valid TOML file: TomlSort.__init__() got an unexpected keyword argument 'only_sort_tables'

This is the GH action

      - uses: pre-commit/[email protected]
        env:
          # yamllint disable-line rule:line-length
          SKIP: detect-aws-credentials,protect-first-parent,terraform_docs,terraform_providers_lock,terraform_tflint,terraform_tfsec,terraform_checkov,terrascan

If a key's value is not a table, it would be nice if we didn't need to sort it. For example:

[b]
name = "B great day"
info = "Some great info about b"

[a]
name = "A great day"
info = "Some great info about a"

currently sorts to

[a]
info = "Some great info about a"
name = "A great day"

[b]
info = "Some great info about b"
name = "B great day"

This is fine, but it would be nice to have the option to sort it to:

[a]
name = "A great day"
info = "Some great info about a"

[b]
name = "B great day"
info = "Some great info about b"

Currently, the toml file is always written independently if anything was changed during sorting/formatting. It would be nice if this is only done if something changed.

I was hoping I could use toml-sort to maintain & preserve formatting of something like this:

[a.config]
list1 = [
    "abc",
    "def",
    "ghi",
]

value1 = 123 # abc


[b.config]
value2 = 456

list2 = [
    7, # def
    8,
    # ghi
    9,
]

context: I can use toml-sort as a TOML formatting tool with the --no-sort-tables option, haven't found anything better for Python so far. Just have to find a way to work this issue, and avoid orphaned comments like I described in: #59 (comment)

Possible workaround solution though ugly with some more comments added (with --spaces-indent-inline-array & --trailing-comma-inline-array options, of course):

# configure some things

[a.config]
list1 = [
    "abc",
    "def",
    "ghi",
]
# ---
value1 = 123 # abc

# ---
# configure specific things for b
[b.config]
value2 = 456
# ---
list2 = [
    7, # def
    8,
    # ghi
    9,
]

I may try to develop & propose a better solution, time permitting.

If I try to sort a TOML file with boolean values, the sorter fails.

Traceback (most recent call last):
  File "/pythonpath/bin/toml-sort", line 8, in <module>
    sys.exit(cli())
  File "/pythonpath/lib/python3.8/site-packages/toml_sort/cli.py", line 226, in cli
    sorted_toml = TomlSort(
  File "/pythonpath/lib/python3.8/site-packages/toml_sort/tomlsort.py", line 161, in sorted
    sorted_toml = self.toml_doc_sorted(toml_doc)
  File "/pythonpath/lib/python3.8/site-packages/toml_sort/tomlsort.py", line 153, in toml_doc_sorted
    for key, value in self.sorted_children_table(original):
  File "/pythonpath/lib/python3.8/site-packages/toml_sort/tomlsort.py", line 100, in sorted_children_table
    table_items = [
  File "/pythonpath/lib/python3.8/site-packages/toml_sort/tomlsort.py", line 101, in <listcomp>
    (key, convert_tomlkit_buggy_types(parent[key], parent, key))
  File "/pythonpath/lib/python3.8/site-packages/toml_sort/tomlsort.py", line 49, in convert_tomlkit_buggy_types
    return parent.value.item(key)
AttributeError: 'dict' object has no attribute 'item'

Here are the contents of the test file:

bool1 = true

Currently Uppercase are grouped and sorted differently from lowercases.

e.g.

[tool.poetry.dependencies]
Django = "^2.2"
Jinja2 = "2.11"
Markdown = "^3.2"
arrow = "^0.15"
django-filter = "^2.3"

expected

[tool.poetry.dependencies]
arrow = "^0.15"
Django = "^2.2"
django-filter = "^2.3"
Jinja2 = "2.11"
Markdown = "^3.2"

Currently with this pre-commit config below toml-sort will fail because it doesn't support checking multiple files at the time, see:

Sample of .pre-commit-config.yaml

repos:
  - repo: local
    hooks:
      - id: toml-sort
        name: toml-sort
        entry: toml-sort --check
        language: system
        types: [toml]

Actual result

$ touch foo.toml

$ pre-commit run --all-files toml-sort
toml-sort................................................................Failed
- hook id: toml-sort
- exit code: 2

Usage: toml-sort [OPTIONS] [FILENAME]
Try 'toml-sort --help' for help.
Error: Got unexpected extra arguments (pyproject.toml foo.toml)
Usage: toml-sort [OPTIONS] [FILENAME]
Try 'toml-sort --help' for help.

Would you be interested in this feature? If it a yes I could send a PR shortly. Thanks for the awesome linter!

I would like to use the --all option, but it reorders "python", which is first by convention for poetry packages.

> poetry new tmp
> cd tmp
> poetry add antigravity
> cat pyproject.toml
[tool.poetry]
name = "tmp"
version = "0.1.0"
description = ""
authors = ["Kyle King <[email protected]>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.11"
antigravity = "^0.1"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
> toml-sort pyproject.toml -i -a
> cat pyproject.toml
[build-system]
build-backend = "poetry.core.masonry.api"
requires = ["poetry-core"]

[tool.poetry]
authors = ["Kyle King <[email protected]>"]
description = ""
name = "tmp"
readme = "README.md"
version = "0.1.0"

[tool.poetry.dependencies]
antigravity = "^0.1"
python = "^3.11"

Where, if sorted manually or by poetry-plugin-sort, I would expect the dependencies to be sorted as:

[tool.poetry.dependencies]
python = "^3.11"
antigravity = "^0.1"

Would you be interested in a feature that would recognize tool.poetry.dependencies in pyproject.toml and keep the python dependency first? If so, I would be happy to try to contribute and if so, would probably add some sort of generic hook to override the sort order for edge cases like these

I'm going through moving any config from .pre-commit-config.yaml into other config files wherever possible to make sure that running tools from terminal has the same result as running in the pre-commit hook.

So I changed from having:

  - repo: https://github.com/pappasam/toml-sort
    rev: v0.22.1
    hooks:
      - id: toml-sort
        args:
          - --all
          - --in-place

to:

  - repo: https://github.com/pappasam/toml-sort
    rev: v0.22.1
    hooks:
      - id: toml-sort

And adding the following to pyproject.toml

[tool.tomlsort]
all = true
in_place = true

This then works fine when running something like toml-sort pyproject.toml and that fixes the file in-place.
However, when I run the pre-commit hook it doesn't run the files in-place, it just uses the --check argument from https://github.com/pappasam/toml-sort/blob/main/.pre-commit-hooks.yaml instead as command line arguments take precedence over pyproject.toml arguments. This means that both check and in_place are True and as args.check is checked first (line 357 of cli.py) so my setting in_place = true basically has no effect.

I'm not sure of the best way to fix this.
Having a standard args of --check for the pre-commit does make sense as either --check or --in-place needs to be defined when running multiple files. And defaulting to an option that doesn't result in overwritten files is best. But it would also be good if it was possible for the in_place setting in pyproject.toml to work when running toml-sort from pre-commit.
Maybe the easiest way is to change the tool behaviour so that if both in-place and check are true then it runs in-place, rather than check.
That would just mean swapping

        if args.check:
            if original_toml != sorted_toml:
                check_failures.append(filename)
        elif args.in_place:
            write_file(filename, sorted_toml)

to:

        if args.in_place:
            write_file(filename, sorted_toml)
        elif args.check:
            if original_toml != sorted_toml:
                check_failures.append(filename)

Obviously that is a fairly large change in behaviour, though I imagine very few people would have both --check and --in-place set other than in my scenario where in_place is set in pyproject.toml and --check is only being set as the default pre-commit args.
Obviously for now I can workaround this easily enough by changing the pre-commit args to define --in-place and not --check, but I would prefer not having to set the same setting twice.

If a TOML file contains a blank comment line, then toml-lint will add a trailing whitespace to that line:

$ sed 's/ /␠/g' space-on-blank-comment.toml 
#␠This␠next␠comment␠line␠has␠no␠whitespace␠(other␠than␠the␠line
#␠terminator):
#

foo␠=␠"bar"

$ toml-sort space-on-blank-comment.toml | sed 's/ /␠/g'
#␠This␠next␠comment␠line␠has␠no␠whitespace␠(other␠than␠the␠line
#␠terminator):
#␠

foo␠=␠"bar"

In order to make the difference easier to see, I used sed to replace the spaces with ␠ characters.

Hello! I have a tool that "un-inlines" comments in a TOML list like so (coming from astral-sh/rye#1255 (comment)):

foo = [
---    "a",  # Comment
+++    "a",
+++    # Comment
    "b",
]

I am wondering, would toml-sort be willing to add a flag that turns on re-inlining of comments? So it can essentially do this:

foo = [
---    "a",
---    # Comment
+++    "a",  # Comment
    "b",
]

It would be great to have the ability to selectively disable sorting for an array (via a comment or a flag in the config). I think the sorting feature is great, but I can only solve this by disabling it for all files, like in #37.

[tool.tomlsort]
all = false
sort_inline_arrays = false

For example, with Coverage for the following to work src must appear before the .tox line, but sorting will change it.

[tool.coverage]
paths.source = [
    "src",
    ".tox*/*/lib/python*/site-packages",
]