In use cases where python is part of an application install such as blender it is useful to run shed as a python module. Currently the result is as below.

 .\python.exe -m shed
C:\Program Files\Blender Foundation\Blender 2.92\2.92\python\bin\python.exe: No module named shed.__main__; 'shed' is a package and cannot be directly executed

The problem is:

• pre-commit caches installed hook, the cache key is repo tag
• black published 22.1.0 release 3 days ago
• https://pre-commit.ci uses cached shed==0.8.0 which has black>=21.12b0, (black==21.12b0 was cached by pre-commit.ci internally)
• when I run pre-commit hooks locally it uses black==22.1.0 if installed from fresh (or after pre-commit clean command)

As the result, https://pre-commit.ci and my local pre-commit install have different black versions and reformat files differently.
The black version bump in setup.py solves the problem.
To prevent it in future I can suggest pinning all requirements to strict versions in pre-commit-hooks.yaml. additional_dependencies: [".[pre-commit]"] record can enforce pre-commit extra dependency that pins all requirements from open-ranged deps to strict ones, e.g. autoflake >= 1.4 from install_requires should be converted into autoflake == 1.4 in extra_requires section.
Keeping extra deps versions can be tedious a little, I believe Dependabot can automate everything except publishing a new tag maybe (it is up to you, I personally prefer to forbid bots initiating a new release process).

If you agree with the proposed strategy and have no time for implementing it yourself, I'm happy to master a PR. Should be an easy task, sure.

It would be nice to have shed also available in conda-forge for anaconda/miniconda users

By default, autoflake will remove unused imports in __init__.py files. However, importing modules into these files and not using them is quite a common pattern, e.g. isort __init__.py.

The safest thing would be to disable this behaviour with this autoflake flag: --ignore-init-module-imports. Perhaps shed should use this flag?

psf/black#664 proposes breaking very long with-statements using backslashes (< 3.9) or parentheses (>= 3.9), and shows the ugly current formatting for such statements.

Once that's done, re-enabling FixTrivialNestedWiths will be an obvious win. If Black initially only supports parenthesised context managers we can re-enable this refactoring for py39+ only.

A live online demo of shed like https://black.vercel.app would be awesome, and with Pyodide it's pretty easy to do without a server!

We can probably crib most of the code we need from https://github.com/Zac-HD/Zac-HD.github.io/tree/master/ghostwriter, and likely embed it in a markdown file for display on GitHub.

Since shed has the ability to specify which files to format, I would like to request this feature. autoflake, isort, and black all have this feature, so it would not be a bad idea for shed to have this feature.

If implemented like autoflake, it would be like:

shed -r /path/to/folder

If implemented like isort or black:

shed /path/to/folder

I am sorry in advance, but I will not have time to implement this feature. If you don't like it, please close it.

When running shed i receive the following error:
Internal error formatting 'vworker/helpers/amp.py': AttributeError: module 'pyupgrade._main' has no attribute '_fix_py36_plus' Please report this to https://github.com/Zac-HD/shed/issues

Tested this on multiple files and using all possible flags to run shed.

Shed looks amazing, and i want to adopt it into some existing projects instead of the cludge of making isort/black/etc play nice each time, but i have hit a problem with python 3.9 code that uses type annotations at runtime.

the problem comes from pyupgrade and specifically this rule:

https://github.com/asottile/pyupgrade#pep-604-typing-rewrites

in short it converts:

from __future__ import annotations
import typing

def foo(bar: typing.Optional[str]) -> None:
    ...

into

from __future__ import annotations

def foo(bar: str | None) -> None:
    ...

this breaks any runtime code interpreting the types (e.g. pydantic, possibly hypothesis.infer too?) as the | format is not supported at runtime until python 3.10.

pyupgrade offers --keep-runtime-typing to prevent it doing this update, but there seems to be no way to plumb that in via shed considering pyupgrade lacks a config file?

maybe this and any other potentially dangerous pyupgrade changes could be gated behind the --refactor cli param?

i suspect there is probably a similar problem for python 3.8 and this rule:

https://github.com/asottile/pyupgrade#pep-585-typing-rewrites

I'm failing to run pre-commit on this repo, it saying .pre-commit-config.yaml is not a file (also failing to run it by installing and such). Is it supposed to work with just a hooks file?

Every project with aspirations to greatness needs a logo, and shed is no exception. Are you the generous designer who can help?

• The project is named shed as a three-way reference to "shedding" the old formatting as a snake sheds old skin, an end to "bikeshedding" style or configuration choices, and Australian sheds as places where things get made.
• I like Prettier's logo much more than Black's, though shed is built on the latter.
• Once shed is out of beta and has a logo I like, I'll be printing it on stickers - and will send you some wherever you are if you would like some.
• The logo need not include the project name, but it would be nice if the sticker design could include the name or shed.zhd.dev (which will host the homepage once we're out of alpha).

Ideas or sketches are welcome, not just finished proposals 😁

https://github.com/snok/pep585-upgrade would be nice to have for Python 3.9+ code 😁

Python code cells in Jupyter notebooks can contain code that is not valid Python, like

%load_ext my extension

or

!command

shed currently fails on Jupyter notebooks in Jupytext that contain such blocks with an error like

Could not parse ```python markdown block in file 'myfile.md'

Example notebook

---
jupyter:
  jupytext:
    text_representation:
      extension: .md
      format_name: markdown
      format_version: '1.3'
      jupytext_version: 1.13.4
  kernelspec:
    display_name: Python 3 (ipykernel)
    language: python
    name: python3
---

```python
%load_ext autoreload
```

```python
!ls /dev
```

```python
valid = 'python'
```

It's not entirely clear to me who's at fault here - jupytext or shed.
The code blocks declare themselves as python (which they are not). I guess in the medium term one might also consider adding a new, more specific MarkDown code-block identifier for Jupyter notebook cells.

If passed an .ipynb file, or a repo containing them, apply shed to each code cell using https://github.com/jupyter/nbformat This should be an optional extra, with the caveat that shed will raise an error telling you to install shed[notebook] if run on a .ipynb file otherwise (or print a warning if they're just in a repo).

Shipping a Jupyter plugin to automatically format code cells immediately before execution would be very cool, but perhaps out of scope for now.

I have a line of code in my project that looks like:

logging.info(f"{OMEGA.message_cache=}, {len(OMEGA.user_cache)=}, "
                 f"{OMEGA.message_cache / max(len(OMEGA.user_cache), 1)=}")

shed works fine but when using shed --refactor it stalls here:

Internal error formatting 'main.py': ParserSyntaxError: Syntax Error @ 794:31.
Incomplete input. Encountered '=', but expected '!', ':', or '}'.

        f"{OMEGA.message_cache=}, {len(OMEGA.user_cache)=}, "
                              ^
    Please report this to https://github.com/Zac-HD/shed/issues

seems to have been removed in the recent pyupgrade v3 release: asottile/pyupgrade@57eb8ab

which leads to:

Internal error formatting '...': AttributeError: module 'pyupgrade._main' has no attribute 'IMPORT_REMOVALS'
    Please report this to https://github.com/Zac-HD/shed/issues

I know this is closed but I don't think it's working as intended:

$ shed ./box_resize.py 
fatal: not a git repository (or any parent up to mount point /home)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

When passed in a positional file argument on the command line it is resorting to the git approach. This seems like a bug, as the README says it can be used without a git repo, thought I should let you know

Hello, I just installed shed and received this error:

shed src/app.py
Internal error formatting 'src/app.py': _fix_py36_plus() missing 1 required keyword-only argument: 'min_version'

I guess __init__.py#36 should be: source_code = pyupgrade._main._fix_py36_plus(source_code, min_version=pyupgrade_min)

"""
# We want to get to this, but it takes some manual editing...

def f(
    value: Literal["Option1", "Option2", "Option3", "Option4", "Option5"] | None = None,
):
    pass
"""

from typing import Literal, Optional


def f(
    # The `value` argument is poorly upgraded to the Python 3.10 syntax
    value: Optional[
        Literal["Option1", "Option2", "Option3", "Option4", "Option5"]
    ] = None,
):
    pass


## need to nicely reformat arguments like 
helpful_file: None | (
        str
    ) = "some very long string aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
# because they show up all the time when upgrading from Optional[str] =

I think this output comes from pyupgrade and fixing it there is probably infeasible, but we could add custom LibCST logic to fix them after the fact.

• Run tox -e deps
• Add Python 3.11 back to the list in .github/workflows/ci.yml
• Open a PR and fix any resulting failures that seem like bugs (if in doubt, ask)

Bonus: check whether we can also run against PyPy in CI.

Somehow tox failed to install com2ann for 3.8 and dataclasses for black for python 3.7. Have you seen the same problem? The problematic builds are here: https://travis-ci.com/github/terminusdb/terminusdb-client-python/builds/182622894

This is a bit annoying. It doesn't seem to contain very much: just .hypothesis/unicode_data/13.0.0/charmap.json.gz.

In asottile/pyupgrade#627, _fix_py36_plus was removed from pyupgrade/_main.py, and pyupgrade was released as 2.32.0 (asottile/pyupgrade@v2.31.1...v2.32.0). This causes shed to fail for us with module 'pyupgrade._main' has no attribute '_fix_py36_plus' (terminusdb/terminusdb-client-python#298).

Thanks for the great tool!

As far as I can see, shed always exits with 0 code (success). It's not always desired behavior. For example, in CI it would be useful to know if anything changed or not. Many formatters have such an option:

• black --check;
• isort --check-only.

Consider adding a similar option or just changing the default behavior of the tool to return different status codes. I don't mind if shed actually formats files instead of just checking. The only thing I care about is the exit code. Otherwise, it's difficult to use shed in CI.

with (a() as b, c() as d):
    pass

@(lambda f: f)
def g():
    pass

is valid Python 3.9 syntax which triggers psf/black#1948 and, in --refactor mode, Instagram/LibCST#486 and Instagram/LibCST#490. Once these upstream bugs are fixed, we should:

• update the minimum version of Black
• update the minimum version of LibCST

The readme.md / help text only gives the following information about the file argument:

positional arguments:
  file         File(s) to format, instead of autodetection

It's not clear what this file format is. Are globs allowed, can we do ** subdirectory matching? Could somebody expand the documentation to explain what kinds of values are acceptable?

I've installed shed and when trying to run it via command line I get the following error on each of my files:

Internal error formatting '<FILE>.py': module 'pyupgrade' has no attribute 'IMPORT_REMOVALS'

It looks like the new version of pyupgrade 2.8.0 has broken compatibility with IMPORT_REMOVALS? Downgrading to 2.7.4 works as expected.

Here are the relevant package versions:

autoflake-1.4 isort-5.7.0 libcst-0.3.16 pybetter-0.3.6.1 pyemojify-0.2.0 pyupgrade-2.8.0 pyyaml-5.4.1 shed-0.3.1 tokenize-rt-4.1.0 toml-0.10.2 typing-extensions-3.7.4.3 typing-inspect-0.6.0

• Split up assert a and b into two separate assertions, for easier debugging. Preserve comments.
• Break heavily-annotated function arguments over multiple lines, like #22 (comment) - this should be in a separate PR so I can evaluate the heuristic on some real large code corpora
• NoRedundantLambdaRule (impl): "de-proxy" patterns like lambda x: whatever(x) to just whatever.
  • special case for literals: pie807: prefer list builtin on [], plus {}, ().
• pie801: prefer simple return
• In sorted(list(...)), remove the inner call to list() (etc; see flake8-comprehensions for more)
  • likewise in "...".join(list(...)), remove the call to list() (or tuple())
  • ... and similarly in all(), dict(), sum(), min(), and max()
• pie787: no len condition
• pie788: no bool condition
• raise NotImplemented should be raise NotImplementedError
• Remove redundant parens directly around a function call, or around the argument to a function
• Replace assert False with raise AssertionError, handling the optional arg
• Reorder Union and Literal contents so that None is last
• Flatten syntatically nested Union or Literal instances
• Reorder new-style unions using | so that None is last
• Expand convert_none_cmp() to handle True and False too, ala CompareSingletonPrimitivesByIsRule
• Replace builtin calls with literals: we do this for comprehensions, but should also do e.g. tuple()->(), list("")->[], etc.
• Remove list() (or tuple) call in comprehensions such as [x**2 for x in list(...)] - we're about to iterate anyway. Only applies to list, set, or dict comprehensions; generator expressions might not consume the whole iterable.
• Replace map(lambda ..., x) with a comprehension, like adamchainz/flake8-comprehensions#409
  • And likewise rewrite filter(lambda ..., x)
• Nontrivial cases of CollapseIsinstanceChecksRule (impl): isinstance(x, y) or isinstance(x, z) -> isinstance(x, (y, z))
• If the last except ...: clause just does raise, remove it and repeat for the preceeding clause. If it's the only except clause and there's no finally:, remove the try: block entirely.
• pie797: no unnecessary ternary
• pie800: no unnecessary spread
• pie804: no unnecessary dict kwargs
• Replace sorted(...)[0] with min(...) (or [-1] with max(...)). Invert min/max if passed reverse=True (or truthy constant); ignore reverse=False (or falsey); skip if the reverse= argument is present with any non-constant value.

Variously inspired by linters (flake8, flake8-comprehensions, fixit, etc.) and my own observations of some typed code.

blacken-docs recently added support for .. code-block:: pycon entries in .rst files (or docstrings), and it would be awesome if shed supported that too.

This should probably include turning off the deletion of "unused" imports and variables, since they might in fact be used in a later line. (maybe we should investigate keeping those referenced in later blocks, too, for python code blocks as well?)

Thanks for a very helpful project!

I have found that pyupgrade does not work as well with converting old strings to f strings as another project called flynt that is dedicated to that task only.

I'd like to suggest running flynt as part of the process in addition to pyupgrade which does a lot of other stuff as well.

https://github.com/Instagram/Fixit looks like a nice complement to pybetter for --refactor mode.

# isort: skip_file
import json

1+2# should have spaces added

Isort skips the file by raising a special FileSkipComment exception, which shed needs to catch - currently this simple example triggers an internal error and doesn't format the file.

This is due to a bug in how Black analyses decorator syntax - see psf/black#2181. Unfortunately the only workaround for now is to use an older version of Black 😢

We're interested in running shed as a Github action to check if contributors ran shed, but we are hoping to have check support (running each of the linters in --check mode). Have you considered support for check? I think it would mean passing through the flag to each of the underlying tools, but I'm not certain that all of them support check (I know isort/black do).

Thanks!

It's bitten me several times, so I'd like to fix this before the next time I work on this.

Example:
tests/recorded/foo.txt

bool(len([1, 2]))
foo((5))

================================================================================

bool([1, 2])
foo(5)

but if we introduce a syntax error, the reformatter isn't run and all tests would pass with a file like this:

bool(len([1, 2]))
foo((5)

================================================================================

bool(len([1, 2]))
foo((5)

Why should you remove isort?

At the moment, isort depends on the virtual environment it runs on. It means that running isort on different machines causes different sorting for the imports. This is especially important in CI/CD processes.

This is a bug that will be fixed only on version 5.0.0 of isort, so until then I suggest removing the usage of isort here.

A little side-note.

I started to develop my own tool that does exactly what you do here!
It is called Eddington-static and it runs Black, flake8, mypy, pylint and pydocstyle.
It also configurable using commands.toml file, so you can add/remove/edit commands as much as you want.
Maybe we can consider joining forces and make a single tool ;)

there is no way to set known_first_party or custom sections when using shed.

# pyproject.toml
[tool.isort]
sections = ['FUTURE', 'STDLIB', 'THIRDPARTY', 'TEST', 'FIRSTPARTY', 'LOCALFOLDER']
known_test = ["pytest"]

isort will work, but shed will then reorder the items back into third party.

same when isort needs help with first party modules.

echo 1/ > t.py && shed t.py reports an internal error, because the file contents are not syntactically valid Python. We should just emit a warning in this case. CC @rsokl 😉

With shed 0.9.5 and Python 3.10.4, running shed --refactor over pytest, I get:

Internal error formatting 'doc/en/example/pythoncollection.rst': SyntaxError: multiple exception types must be parenthesized (<unknown>, line 5)
    Please report this to https://github.com/Zac-HD/shed/issues
Internal error formatting 'doc/en/how-to/writing_plugins.rst': SyntaxError: positional argument follows keyword argument (<unknown>, line 1)
    Please report this to https://github.com/Zac-HD/shed/issues

Using refactor() instead of rewrite_source would allow teyit to output in a deterministic manner (since in rare cases some cases might require double folding [e.g assertIs(a > b, True) => assertTrue(a > b) => assertGreater(a, b)]). This is why we have refactor_until_deterministic. But since you don't care about the statistics (only for debug purposes), I highly advocate usage of refactor() API.

https://github.com/isidentical/teyit/blob/c9391311a26eb9fc66514be78fefeeebe5b28acd/teyit.py#L286-L288

P.S: I'll also try to bring support for 3.6+!

One thing I've noticed about using shed inside of pre-commit is that it doesn't list the files that it reformatted/fixed the way that Black (or https://github.com/pre-commit/pre-commit-hooks) do.

This can make it a little bit hard to observe what needed to be reformatted or what the formatter did.

Black:

black....................................................................Failed
- hook id: black
- files were modified by this hook

reformatted cherry_picker.py

Shed:

shed.....................................................................Failed
- hook id: shed
- files were modified by this hook

I realized this is probably more difficult because it involves combining the actions of different formatters, but think it would be nice to have a flag for this. (Or, imo, just make it the default behavior -- I think this would be a further advantage of shed over a variety of separate formatters, because you would get one complete list of Python files changed).

I recently found https://github.com/life4/awesome-python-code-formatters, and was inspired: it would be great if shed could integrate docformatter, blacken-docs, teyit, and parts of pybetter.

Ruff supports a considerable number of auto-fixes. It might be worth using for Shed. It's ‘extremely fast’, and may be able to replace some of the other things Shed runs, as well as adding extra rewrites.

autoflake.is_python_file doesn't care about .pyi as a file ending, but I don't see why that's the case. Running Black or isort manually they both reformat .pyi, but shed uses autoflake's check and totally ignores them. I'm not sure if autoflake should fix it upstream, and/or if shed should reimplement is_python_file itself (or take it from isort/black/etc if they have one).

When I run shed in a repo that uses pattern matching, I get the following error:

Could not parse file './some/file/with/pattern/matching'
    The syntax is valid Python, so please report this as a bug.

Let me know if there's anything I can do to assist.

ls = [1, 2, 3]
maybe_num = None

num = maybe_num or len(ls)

In this case dropping the len() is incorrect; there are presumably other similar cases still latent. CC @jakkdl

See these commits; the problem is that it removes the whitespace (which includes comments) too. Should just be one new condition in remove_pointless_parens_around_call(); "if node contains comment node: return node without changes" which can be copied from remove_trailing_comma().

Current I'm using black to auto-format my code on save when using VSCode, there is a setting in .vscode/settings.json here you specify the path to the black binary if it isn't in your path. I tried the following, but it doesn't work as I had hoped it to:

"python.formatting.blackPath": "./.venv/bin/shed",
"python.formatting.provider": "black",

Is there another way to hook up shed to VSCode or is there possibly a way for my above example to work?

If you have a moderately complex repository setup (perhaps a monorepo with multiple component subpackages, you're using a VCS other than Git, or you want to combine shed running on Python source code with a linting tool running over non-Python code or other artefacts like XML/JSON/YAML), and seek to manage the complexity of this using a Makefile task like:

format:
	poetry run shed --refactor --py310-plus my_package/**/*.py tests/**/*.py

.PHONY: format

This will fail because Make uses sh which won't expand the globs.

The immediate fix for this is either fixing the Makefile to use find to construct the list of files for processing, or ensuring Make uses a shell other than sh (bash, say).

It might be useful to output a warning if shed has filenames passed in that contain * or ** in them, so the user knows the globs have been passed through without being expanded. Alternatively, perhaps if a list of files is passed through and none of the filenames actually resolve to a Python file, some kind of warning might be useful.

We recently added Shed to our project via pre-commit and are pretty happy with it.

However, our Windows users discovered that shed won't run on any Python file that contain Unicode:

skipping 'code/dictation.py' due to 'charmap' codec can't decode byte 0x9d in position 3481: character maps to <undefined>"

The root issue has more detail, but basically it seems like Windows' default behavior for open() is to use the platform-default encoding (cp1252) instead of UTF-8, so it fails at this open() call.

It looks like Black uses the tokenize library to open Python files, which automatically detects encoding, so perhaps all we need to do is replace it with tokenize.open if the file is Python?

Alternatively, if the goal is for Shed to be "maximally opinionated" then maybe we should just add encoding="utf8" to that open call? :)

Reproduction steps:

1. Clone https://github.com/knausj85/knausj_talon on Windows
2. Run shed, it should fail with the above error because of the unicode characters in https://github.com/knausj85/knausj_talon/blob/master/code/dictation.py