Giter VIP home page Giter VIP logo

pyupgrade's Introduction

build status pre-commit.ci status

pyupgrade

A tool (and pre-commit hook) to automatically upgrade syntax for newer versions of the language.

Installation

pip install pyupgrade

As a pre-commit hook

See pre-commit for instructions

Sample .pre-commit-config.yaml:

-   repo: https://github.com/asottile/pyupgrade
    rev: v3.16.0
    hooks:
    -   id: pyupgrade

Implemented features

Set literals

-set(())
+set()
-set([])
+set()
-set((1,))
+{1}
-set((1, 2))
+{1, 2}
-set([1, 2])
+{1, 2}
-set(x for x in y)
+{x for x in y}
-set([x for x in y])
+{x for x in y}

Dictionary comprehensions

-dict((a, b) for a, b in y)
+{a: b for a, b in y}
-dict([(a, b) for a, b in y])
+{a: b for a, b in y}

Replace unnecessary lambdas in collections.defaultdict calls

-defaultdict(lambda: [])
+defaultdict(list)
-defaultdict(lambda: list())
+defaultdict(list)
-defaultdict(lambda: {})
+defaultdict(dict)
-defaultdict(lambda: dict())
+defaultdict(dict)
-defaultdict(lambda: ())
+defaultdict(tuple)
-defaultdict(lambda: tuple())
+defaultdict(tuple)
-defaultdict(lambda: set())
+defaultdict(set)
-defaultdict(lambda: 0)
+defaultdict(int)
-defaultdict(lambda: 0.0)
+defaultdict(float)
-defaultdict(lambda: 0j)
+defaultdict(complex)
-defaultdict(lambda: '')
+defaultdict(str)

Format Specifiers

-'{0} {1}'.format(1, 2)
+'{} {}'.format(1, 2)
-'{0}' '{1}'.format(1, 2)
+'{}' '{}'.format(1, 2)

printf-style string formatting

Availability:

  • Unless --keep-percent-format is passed.
-'%s %s' % (a, b)
+'{} {}'.format(a, b)
-'%r %2f' % (a, b)
+'{!r} {:2f}'.format(a, b)
-'%(a)s %(b)s' % {'a': 1, 'b': 2}
+'{a} {b}'.format(a=1, b=2)

Unicode literals

-u'foo'
+'foo'
-u"foo"
+'foo'
-u'''foo'''
+'''foo'''

Invalid escape sequences

 # strings with only invalid sequences become raw strings
-'\d'
+r'\d'
 # strings with mixed valid / invalid sequences get escaped
-'\n\d'
+'\n\\d'
-u'\d'
+r'\d'
 # this fixes a syntax error in python3.3+
-'\N'
+r'\N'

is / is not comparison to constant literals

In python3.8+, comparison to literals becomes a SyntaxWarning as the success of those comparisons is implementation specific (due to common object caching).

-x is 5
+x == 5
-x is not 5
+x != 5
-x is 'foo'
+x == 'foo'

.encode() to bytes literals

-'foo'.encode()
+b'foo'
-'foo'.encode('ascii')
+b'foo'
-'foo'.encode('utf-8')
+b'foo'
-u'foo'.encode()
+b'foo'
-'\xa0'.encode('latin1')
+b'\xa0'

extraneous parens in print(...)

A fix for python-modernize/python-modernize#178

 # ok: printing an empty tuple
 print(())
 # ok: printing a tuple
 print((1,))
 # ok: parenthesized generator argument
 sum((i for i in range(3)), [])
 # fixed:
-print(("foo"))
+print("foo")

constant fold isinstance / issubclass / except

-isinstance(x, (int, int))
+isinstance(x, int)

-issubclass(y, (str, str))
+issubclass(y, str)

 try:
     raises()
-except (Error1, Error1, Error2):
+except (Error1, Error2):
     pass

unittest deprecated aliases

Rewrites deprecated unittest method aliases to their non-deprecated forms.

 from unittest import TestCase


 class MyTests(TestCase):
     def test_something(self):
-        self.failUnlessEqual(1, 1)
+        self.assertEqual(1, 1)
-        self.assertEquals(1, 1)
+        self.assertEqual(1, 1)

super() calls

 class C(Base):
     def f(self):
-        super(C, self).f()
+        super().f()

"new style" classes

rewrites class declaration

-class C(object): pass
+class C: pass
-class C(B, object): pass
+class C(B): pass

removes __metaclass__ = type declaration

 class C:
-    __metaclass__ = type

forced str("native") literals

-str()
+''
-str("foo")
+"foo"

.encode("utf-8")

-"foo".encode("utf-8")
+"foo".encode()

# coding: ... comment

as of PEP 3120, the default encoding for python source is UTF-8

-# coding: utf-8
 x = 1

__future__ import removal

Availability:

  • by default removes nested_scopes, generators, with_statement, absolute_import, division, print_function, unicode_literals
  • --py37-plus will also remove generator_stop
-from __future__ import with_statement

Remove unnecessary py3-compat imports

-from io import open
-from six.moves import map
-from builtins import object  # python-future

import replacements

Availability:

  • --py36-plus (and others) will replace imports

see also reorder-python-imports

some examples:

-from collections import deque, Mapping
+from collections import deque
+from collections.abc import Mapping
-from typing import Sequence
+from collections.abc import Sequence
-from typing_extensions import Concatenate
+from typing import Concatenate

rewrite mock imports

Availability:

-from mock import patch
+from unittest.mock import patch

yield => yield from

 def f():
-    for x in y:
-        yield x
+    yield from y
-    for a, b in c:
-        yield (a, b)
+    yield from c

Python2 and old Python3.x blocks

 import sys
-if sys.version_info < (3,):  # also understands `six.PY2` (and `not`), `six.PY3` (and `not`)
-    print('py2')
-else:
-    print('py3')
+print('py3')

Availability:

  • --py36-plus will remove Python <= 3.5 only blocks
  • --py37-plus will remove Python <= 3.6 only blocks
  • so on and so forth
 # using --py36-plus for this example

 import sys
-if sys.version_info < (3, 6):
-    print('py3.5')
-else:
-    print('py3.6+')
+print('py3.6+')

-if sys.version_info <= (3, 5):
-    print('py3.5')
-else:
-    print('py3.6+')
+print('py3.6+')

-if sys.version_info >= (3, 6):
-    print('py3.6+')
-else:
-    print('py3.5')
+print('py3.6+')

Note that if blocks without an else will not be rewritten as it could introduce a syntax error.

remove six compatibility code

-six.text_type
+str
-six.binary_type
+bytes
-six.class_types
+(type,)
-six.string_types
+(str,)
-six.integer_types
+(int,)
-six.unichr
+chr
-six.iterbytes
+iter
-six.print_(...)
+print(...)
-six.exec_(c, g, l)
+exec(c, g, l)
-six.advance_iterator(it)
+next(it)
-six.next(it)
+next(it)
-six.callable(x)
+callable(x)
-six.moves.range(x)
+range(x)
-six.moves.xrange(x)
+range(x)


-from six import text_type
-text_type
+str

-@six.python_2_unicode_compatible
 class C:
     def __str__(self):
         return u'C()'

-class C(six.Iterator): pass
+class C: pass

-class C(six.with_metaclass(M, B)): pass
+class C(B, metaclass=M): pass

-@six.add_metaclass(M)
-class C(B): pass
+class C(B, metaclass=M): pass

-isinstance(..., six.class_types)
+isinstance(..., type)
-issubclass(..., six.integer_types)
+issubclass(..., int)
-isinstance(..., six.string_types)
+isinstance(..., str)

-six.b('...')
+b'...'
-six.u('...')
+'...'
-six.byte2int(bs)
+bs[0]
-six.indexbytes(bs, i)
+bs[i]
-six.int2byte(i)
+bytes((i,))
-six.iteritems(dct)
+dct.items()
-six.iterkeys(dct)
+dct.keys()
-six.itervalues(dct)
+dct.values()
-next(six.iteritems(dct))
+next(iter(dct.items()))
-next(six.iterkeys(dct))
+next(iter(dct.keys()))
-next(six.itervalues(dct))
+next(iter(dct.values()))
-six.viewitems(dct)
+dct.items()
-six.viewkeys(dct)
+dct.keys()
-six.viewvalues(dct)
+dct.values()
-six.create_unbound_method(fn, cls)
+fn
-six.get_unbound_function(meth)
+meth
-six.get_method_function(meth)
+meth.__func__
-six.get_method_self(meth)
+meth.__self__
-six.get_function_closure(fn)
+fn.__closure__
-six.get_function_code(fn)
+fn.__code__
-six.get_function_defaults(fn)
+fn.__defaults__
-six.get_function_globals(fn)
+fn.__globals__
-six.raise_from(exc, exc_from)
+raise exc from exc_from
-six.reraise(tp, exc, tb)
+raise exc.with_traceback(tb)
-six.reraise(*sys.exc_info())
+raise
-six.assertCountEqual(self, a1, a2)
+self.assertCountEqual(a1, a2)
-six.assertRaisesRegex(self, e, r, fn)
+self.assertRaisesRegex(e, r, fn)
-six.assertRegex(self, s, r)
+self.assertRegex(s, r)

 # note: only for *literals*
-six.ensure_binary('...')
+b'...'
-six.ensure_str('...')
+'...'
-six.ensure_text('...')
+'...'

open alias

-with io.open('f.txt') as f:
+with open('f.txt') as f:
     ...

redundant open modes

-open("foo", "U")
+open("foo")
-open("foo", "Ur")
+open("foo")
-open("foo", "Ub")
+open("foo", "rb")
-open("foo", "rUb")
+open("foo", "rb")
-open("foo", "r")
+open("foo")
-open("foo", "rt")
+open("foo")
-open("f", "r", encoding="UTF-8")
+open("f", encoding="UTF-8")
-open("f", "wt")
+open("f", "w")

OSError aliases

 # also understands:
 # - IOError
 # - WindowsError
 # - mmap.error and uses of `from mmap import error`
 # - select.error and uses of `from select import error`
 # - socket.error and uses of `from socket import error`

 def throw():
-    raise EnvironmentError('boom')
+    raise OSError('boom')

 def catch():
     try:
         throw()
-    except EnvironmentError:
+    except OSError:
         handle_error()

TimeoutError aliases

Availability:

  • --py310-plus for socket.timeout
  • --py311-plus for asyncio.TimeoutError
 def throw(a):
     if a:
-        raise asyncio.TimeoutError('boom')
+        raise TimeoutError('boom')
     else:
-        raise socket.timeout('boom')
+        raise TimeoutError('boom')

 def catch(a):
     try:
         throw(a)
-    except (asyncio.TimeoutError, socket.timeout):
+    except TimeoutError:
         handle_error()

typing.Text str alias

-def f(x: Text) -> None:
+def f(x: str) -> None:
     ...

Unpacking list comprehensions

-foo, bar, baz = [fn(x) for x in items]
+foo, bar, baz = (fn(x) for x in items)

Rewrite xml.etree.cElementTree to xml.etree.ElementTree

-import xml.etree.cElementTree as ET
+import xml.etree.ElementTree as ET
-from xml.etree.cElementTree import XML
+from xml.etree.ElementTree import XML

Rewrite type of primitive

-type('')
+str
-type(b'')
+bytes
-type(0)
+int
-type(0.)
+float

typing.NamedTuple / typing.TypedDict py36+ syntax

Availability:

  • --py36-plus is passed on the commandline.
-NT = typing.NamedTuple('NT', [('a', int), ('b', Tuple[str, ...])])
+class NT(typing.NamedTuple):
+    a: int
+    b: Tuple[str, ...]

-D1 = typing.TypedDict('D1', a=int, b=str)
+class D1(typing.TypedDict):
+    a: int
+    b: str

-D2 = typing.TypedDict('D2', {'a': int, 'b': str})
+class D2(typing.TypedDict):
+    a: int
+    b: str

f-strings

Availability:

  • --py36-plus is passed on the commandline.
-'{foo} {bar}'.format(foo=foo, bar=bar)
+f'{foo} {bar}'
-'{} {}'.format(foo, bar)
+f'{foo} {bar}'
-'{} {}'.format(foo.bar, baz.womp)
+f'{foo.bar} {baz.womp}'
-'{} {}'.format(f(), g())
+f'{f()} {g()}'
-'{x}'.format(**locals())
+f'{x}'

note: pyupgrade is intentionally timid and will not create an f-string if it would make the expression longer or if the substitution parameters are sufficiently complicated (as this can decrease readability).

subprocess.run: replace universal_newlines with text

Availability:

  • --py37-plus is passed on the commandline.
-output = subprocess.run(['foo'], universal_newlines=True)
+output = subprocess.run(['foo'], text=True)

subprocess.run: replace stdout=subprocess.PIPE, stderr=subprocess.PIPE with capture_output=True

Availability:

  • --py37-plus is passed on the commandline.
-output = subprocess.run(['foo'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+output = subprocess.run(['foo'], capture_output=True)

remove parentheses from @functools.lru_cache()

Availability:

  • --py38-plus is passed on the commandline.
 import functools

-@functools.lru_cache()
+@functools.lru_cache
 def expensive():
     ...

shlex.join

Availability:

  • --py38-plus is passed on the commandline.
-' '.join(shlex.quote(arg) for arg in cmd)
+shlex.join(cmd)

replace @functools.lru_cache(maxsize=None) with shorthand

Availability:

  • --py39-plus is passed on the commandline.
 import functools

-@functools.lru_cache(maxsize=None)
+@functools.cache
 def expensive():
     ...

pep 585 typing rewrites

Availability:

  • File imports from __future__ import annotations
    • Unless --keep-runtime-typing is passed on the commandline.
  • --py39-plus is passed on the commandline.
-def f(x: List[str]) -> None:
+def f(x: list[str]) -> None:
     ...

pep 604 typing rewrites

Availability:

  • File imports from __future__ import annotations
    • Unless --keep-runtime-typing is passed on the commandline.
  • --py310-plus is passed on the commandline.
-def f() -> Optional[str]:
+def f() -> str | None:
     ...
-def f() -> Union[int, str]:
+def f() -> int | str:
     ...

remove quoted annotations

Availability:

  • File imports from __future__ import annotations
-def f(x: 'queue.Queue[int]') -> C:
+def f(x: queue.Queue[int]) -> C:

use datetime.UTC alias

Availability:

  • --py311-plus is passed on the commandline.
 import datetime

-datetime.timezone.utc
+datetime.UTC

pyupgrade's People

Contributors

adamchainz avatar asottile avatar atugushev avatar cj81499 avatar danialkeimasi avatar danielchabrowski avatar flashcode avatar graingert avatar hamdanal avatar hannseman avatar hugovk avatar jdufresne avatar jellezijlstra avatar jkittner avatar marcogorelli avatar mhils avatar michael-k avatar mxr avatar neutrinoceros avatar peterjclaw avatar piazzesiniccolo avatar pre-commit-ci[bot] avatar sadikkuzu avatar sbrugman avatar scop avatar senpos avatar tusharsadhwani avatar unknownplatypus avatar verhovsky avatar vmarkovtsev avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pyupgrade's Issues

IndexError with pytest's fixtures.py

% pyupgrade --py3-only --keep-percent-format src/_pytest/fixtures.py
Traceback (most recent call last):
  File "…/Vcs/pytest/.venv/bin/pyupgrade", line 11, in <module>
    sys.exit(main())
  File "…/Vcs/pytest/.venv/lib/python3.7/site-packages/pyupgrade.py", line 1428, in main
    ret |= fix_file(filename, args)
  File "…/Vcs/pytest/.venv/lib/python3.7/site-packages/pyupgrade.py", line 1402, in fix_file
    contents_text = _fix_py3_plus(contents_text)
  File "…/Vcs/pytest/.venv/lib/python3.7/site-packages/pyupgrade.py", line 1244, in _fix_py3_plus
    _replace_call(tokens, i, end, func_args, template)
  File "…/Vcs/pytest/.venv/lib/python3.7/site-packages/pyupgrade.py", line 1171, in _replace_call
    src = tmpl.format(args=arg_strs, rest=rest)
IndexError: list index out of range

print(repr(tmpl), repr(arg_strs), repr(rest)) shows:

'raise {args[1]}.with_traceback({args[2]})' ['*err'] ''

pyupgrade is introducing a raw string where a line continuation is being used.

Hi.

The pyupgrade tool is introducing a raw string where a line continuation is being used.

Check this example below:

import textwrap


def test_importerror(tmpdir):
    print(tmpdir)
    p1 = tmpdir.join("test1.txt")
    p2 = tmpdir.join("test2.txt")
    p1.write(
        textwrap.dedent(
            """\
        import doesnotexist

        x = 1
        """
        )
    )
    p2.write(
        textwrap.dedent(
            """
        import doesnotexist

        x = 2
        """
        )
    )

after running pyupgrade on this file, a r""" is introduced as it's demonstrated above.

def test_importerror(tmpdir):
    print(tmpdir)
    p1 = tmpdir.join("test1.txt")
    p2 = tmpdir.join("test2.txt")
    p1.write(
        textwrap.dedent(
            r"""\
        import doesnotexist

        x = 1
        """
        )
    )
    p2.write(
        textwrap.dedent(
            """
        import doesnotexist

        x = 2
        """
        )
    )

I'm running pyupgrade (1.10.0) with python Python 3.6.2 on Windows 10, version 1709

f-string roadmap?

You mention that f-string support was planned in the README, and I'm just curious if you had a rough idea of when that might be.

Big fan of the project by the way.

consider fixing invalid escape sequences

$ python2 -W once -c 'print("\d test")'
\d test
$ python3 -W once -c 'print("\d test")'
<string>:1: DeprecationWarning: invalid escape sequence \d
\d test
$ python3 -W once -c 'print(r"\d test")'
\d test

multiline strings don't get fstringified

on 1.5.1

do("foo {}".format(bar))
do("foo {}"
   .format(bar))

the first line gets fstringified, not the second one.
(imagine very long strings, such that it made sense to break a line before the method call)

Fix `is (int, float, bytes, str)` automatically

Similar to #47, let's automatically fix this as it becomes a SyntaxWarning in python3.8

$ python3.8 -Werror
Python 3.8.0a3 (default, Mar 27 2019, 03:46:44) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> x = 5
>>> x is 5
  File "<stdin>", line 1
SyntaxError: "is" with a literal. Did you mean "=="?
>>> 

Removes necessary extra parens from coroutine yield expression

Currently this is happening, but it causes a SyntaxError -- the extra set of parens is necessary here:

 def run_tests():
     url = options.url + '/getCaseCount'
     control_ws = yield websocket_connect(url, None)
-    num_tests = int((yield control_ws.read_message()))
+    num_tests = int(yield control_ws.read_message())

py3.3+: IOError (and others) -> OSError

In Python 3.3+ the following exceptions are now simple aliases of OSError: IOError, EnvironmentError, WindowsError, mmap.error, socket.error and select.error.

The aliases have been removed from the documented hierarchy: https://docs.python.org/3/library/exceptions.html#exception-hierarchy

I think pyupgrade should rewrite these aliases as the canonical OSError.

References:

https://docs.python.org/3/whatsnew/3.3.html#pep-3151-reworking-the-os-and-io-exception-hierarchy

https://docs.python.org/3/library/exceptions.html#OSError

--py36-plus ditches format variable

Python 3.7.0
pyupgrade==1.4.0

Before

bad = 1

print ("Test FAIL: ".format(bad))

Run

$ pyupgrade 1.py --py36-plus
Rewriting 1.py

After

bad = 1

print (f"Test FAIL: ")

Add switch to avoid %s -> {} when the format strings contains braces

When the format string contains braces, converting %s to {} can be quite unwieldy:

$ echo '"{%s, %s}" % (x, y)' >! /tmp/test.py
$ pyupgrade --py36-plus test.py
$ cat test.py
f"{{{x}, {y}}}"

A real case (from the Matplotlib examples), involving the generation of TeX strings:

-        label = r'$%s_{_{\mathrm{%s}}}$' % (orig_label[0], orig_label[1:])
+        label = r'${}_{{_{{\mathrm{{{}}}}}}}$'.format(orig_label[0], orig_label[1:])

I would thus suggest adding a switch --avoid-brace-brace (name up to bikeshedding).

123L -> 123

Somewhere between python2.0 and python2.4 the L prefix became unnecessary for long literals

SyntaxError rewriting 'and' to and

Python 3.7.2
pyupgrade 1.11.0

Input:

given_keyword = "a"
and_keyword = "b"

input = '%(given)s a string with Given in it   \n%(and)s another string' % {
    'given': given_keyword,
    'and': and_keyword,
}

Output:

given_keyword = "a"
and_keyword = "b"

input = '{given} a string with Given in it   \n{and} another string'.format(
    given=given_keyword,
    and=and_keyword,
)

Running the output causes:

    and=and_keyword,
      ^
SyntaxError: invalid syntax

Rewrite `if PY2:` blocks in `--py3-plus`

Ideally handle these different cases:

# input
from six import PY2
if PY2:
   x = 5
else:
   x = 6
# output
x = 6

But also for

sys.version_info[0] == 2
sys.version_info < (3,)

etc.

  • six.PY2
  • six.PY3
  • not six.PY2
  • not six.PY3
  • sys.version_info >= (3,)
  • sys.version_info < (3,)
  • sys.version_info[0] == 2
  • sys.version_info[0] == 3

Support use on directories

In all my uses of pyupgrade, I want to apply it to every Python file in the repo - in general, some directory. pyupgrade $(find . -name '*.py') works OK on *nix systems, but I'd be happier if I could just write pyupgrade . and have it work on Windows too!

Personally I'd prefer to follow Black and flake8 (etc) by having recursive-by-default for directories, but if not a -r/--recursive flag is also fairly common.

py3-plus: .encode('utf-8') → .encode() & .decode('utf-8') → .decode()

In Python 3, the first positional argument to .encode() and .decode() defaults to 'utf-8'. This was not the case with Python 2:

https://docs.python.org/3/library/stdtypes.html#str.encode
https://docs.python.org/3/library/stdtypes.html#bytes.decode

As this is the most common value passed, there could be a transform to simplify the calls to just .encode() or .decode() (without the first positional argument). This could help clean up and simplify modern Python code by removing noise.

Behaviour change after upgrading '%9s' to '{:9}'

This pyupgrade change (https://github.com/hugovk/yamllint/commit/25e8e9be0838bff930b3b204360fc24385b6edbf) fails indentation-related unit tests which previously passed:

-            output += '%9s %s\n' % (token_type,
+            output += '{:9} {}\n'.format(token_type,
                                     self.format_stack(context['stack']))

https://travis-ci.org/hugovk/yamllint/jobs/459076859#L860

Small example:

Python 3.7.1 (default, Nov  6 2018, 18:45:35)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> '%9s %s\n' % ("abc", "def")
'      abc def\n'
>>> '{:9} {}\n'.format("abc", "def")
'abc       def\n'

It would have worked if they were integers:

>>> '%9s %s\n' % (123, 456)
'      123 456\n'
>>> '{:9} {}\n'.format(123, 456)
'      123 456\n'

Another issue with this example, it'd be nice if the indentation of the second line was adjusted, as this often causes flake8 errors. But I understand if this is out of scope of pyupgrade.

For example, instead of this:

-            output += '%9s %s\n' % (token_type,
+            output += '{:9} {}\n'.format(token_type,
                                     self.format_stack(context['stack']))

Do this:

-            output += '%9s %s\n' % (token_type,
-                                    self.format_stack(context['stack']))
+            output += '{:9} {}\n'.format(token_type,
+                                         self.format_stack(context['stack']))

Doesn't handle indexed format strings

The following code which I tried to reduce (which I had no idea was possible):

thing = {'a': 1, 'b': 2}
print('{0[a]} and {0[b]}'.format(thing))

leads to

Traceback (most recent call last):
  File "/home/ethan/venv/bin/pyupgrade", line 10, in <module>
    sys.exit(main())
  File "/home/ethan/venv/lib/python3.7/site-packages/pyupgrade.py", line 1396, in main
    ret |= fix_file(filename, args)
  File "/home/ethan/venv/lib/python3.7/site-packages/pyupgrade.py", line 1372, in fix_file
    contents_text = _fix_fstrings(contents_text)
  File "/home/ethan/venv/lib/python3.7/site-packages/pyupgrade.py", line 1348, in _fix_fstrings
    tokens[i] = token._replace(src=_to_fstring(token.src, node))
  File "/home/ethan/venv/lib/python3.7/site-packages/pyupgrade.py", line 1310, in _to_fstring
    name = ''.join((params[k or str(i)], dot, rest))
KeyError: '0[a]'

This is a very esoteric format string usage, but 🤷‍♂️

Remove all future imports

I have projects that have this line at the start of every file, that I now want to upgrade to Python 3 only:

from __future__ import absolute_import, print_function, division

That's pretty common, everyone dropping Python 2 support from a codebase will want to do that cleanup, no?

It would be great if there was an option to just delete those lines (reviewing / adjusting the rest of the files would be up to me & automated tests to check).

#42 is related, but I thought I'd ask separately for this feature to remove any __future__ import lines.

2to3 can do it, but it leaves an empty line, doesn't delete the future line.
If you don't want to add something here, then maybe the 2to3 option could be mentioned in the README?

Run on a file or directory

Thanks for this handy tool!

It would be nice to be able to run pyupgrade on a directory, and have it process all .py files in that directory and subdirectories.

For example:

pyupgrade .

This would be similar to other tools like Flake8 and Black.

In the meantime, this works:

pyupgrade `find . -name "*.py"`

Behaviour change after upgrading '%.4g' to '{:.4g}'

This change hugovk/cartopy@f1c5b7a fails unit tests, which passed before:

-        return u'%.4g, %.4g (%f\u00b0%s, %f\u00b0%s)' % (x, y, abs(lat),
-                                                         ns, abs(lon), ew)
+        return u'{:.4g}, {:.4g} ({:f}\u00b0{}, {:f}\u00b0{})'.format(x, y, abs(lat),
+                                                                     ns, abs(lon), ew)

https://travis-ci.org/hugovk/cartopy/jobs/463956921#L1111

Small example:

Python 3.7.1 (default, Nov  6 2018, 18:45:35)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> x, y = np.array([-969100.0]), np.array([-4457000.0])
>>>
>>> # Before
...
>>> u"%.4g, %.4g" % (x, y)
'-9.691e+05, -4.457e+06'
>>>
>>> # After
...
>>> u"{:.4g}, {:.4g}".format(x, y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported format string passed to numpy.ndarray.__format__
>>>

py3.6+: Rewrite OrderedDict as dict literals

Python 3.6 has an insertion order preserving dict. Python 3.7 has this defined as a language feature.

This obviates collections.OrderedDict usage, so this tool could rewrite OrderedDict as dict literals when --py36-plus is passed.

for x in y: yield x => yield from y

No need for async for support according to this

Here's an ast parser to detect this, just need the tokenization to rewrite it:

class Visitor(ast.NodeVisitor):
    def __init__(self, filename):
        self.filename = filename

    def visit_For(self, node: ast.For):
        if (
            isinstance(node.target, ast.Name) and
            len(node.body) == 1 and
            isinstance(node.body[0], ast.Expr) and
            isinstance(node.body[0].value, ast.Yield) and
            isinstance(node.body[0].value.value, ast.Name) and
            node.target.id == node.body[0].value.value.id
        ):
            print(f'{self.filename}:{node.lineno}: could be yield from')

        self.generic_visit(node)

python3.8: many tests failing

I suspect something has changed in offset information in ast. See if it's a fix or a bug and report before 3.8 releases

$ pytest tests -v | grep FAILED
tests/pyupgrade_test.py::test_sets[set((1, 2))-{1, 2}] FAILED            [  7%]
tests/pyupgrade_test.py::test_sets[set([x for x in y])-{x for x in y}] FAILED [  8%]
tests/pyupgrade_test.py::test_sets[set((x for x in y))-{x for x in y}] FAILED [  8%]
tests/pyupgrade_test.py::test_sets[set(((1, 2)))-{1, 2}] FAILED          [  9%]
tests/pyupgrade_test.py::test_sets[set((a, b) for a, b in y)-{(a, b) for a, b in y}] FAILED [  9%]
tests/pyupgrade_test.py::test_sets[set(((1, 2), (3, 4)))-{(1, 2), (3, 4)}] FAILED [  9%]
tests/pyupgrade_test.py::test_sets[set((((1, 2),),))-{((1, 2),)}] FAILED [ 11%]
tests/pyupgrade_test.py::test_sets[set(\n(1, 2))-{\n1, 2}] FAILED        [ 11%]
tests/pyupgrade_test.py::test_sets[set((\n1,\n2,\n))\n-{\n1,\n2,\n}\n] FAILED [ 12%]
tests/pyupgrade_test.py::test_sets[set((frozenset(set((1, 2))), frozenset(set((3, 4)))))-{frozenset({1, 2}), frozenset({3, 4})}] FAILED [ 12%]
tests/pyupgrade_test.py::test_sets[set((1,))-{1}] FAILED                 [ 12%]
tests/pyupgrade_test.py::test_sets[set((1, ))-{1}] FAILED                [ 13%]
tests/pyupgrade_test.py::test_sets[set((x for x in y),)-{x for x in y}] FAILED [ 13%]
tests/pyupgrade_test.py::test_sets[set(\n    (x for x in y),\n)-{\n    x for x in y\n}] FAILED [ 14%]
tests/pyupgrade_test.py::test_dictcomps[dict((a, b) for a, b in y)-{a: b for a, b in y}] FAILED [ 16%]
tests/pyupgrade_test.py::test_dictcomps[dict((a, b,) for a, b in y)-{a: b for a, b in y}] FAILED [ 17%]
tests/pyupgrade_test.py::test_dictcomps[dict((a, b, ) for a, b in y)-{a: b for a, b in y}] FAILED [ 17%]
tests/pyupgrade_test.py::test_dictcomps[dict([a, b] for a, b in y)-{a: b for a, b in y}] FAILED [ 17%]
tests/pyupgrade_test.py::test_dictcomps[dict(((a, b)) for a, b in y)-{a: b for a, b in y}] FAILED [ 18%]
tests/pyupgrade_test.py::test_dictcomps[dict([(a, b) for a, b in y])-{a: b for a, b in y}] FAILED [ 18%]
tests/pyupgrade_test.py::test_dictcomps[dict([(a, b), c] for a, b, c in y)-{(a, b): c for a, b, c in y}] FAILED [ 18%]
tests/pyupgrade_test.py::test_dictcomps[dict(((a), b) for a, b in y)-{(a): b for a, b in y}] FAILED [ 19%]
tests/pyupgrade_test.py::test_dictcomps[dict((k, dict((k2, v2) for k2, v2 in y2)) for k, y2 in y)-{k: {k2: v2 for k2, v2 in y2} for k, y2 in y}] FAILED [ 19%]
tests/pyupgrade_test.py::test_dictcomps[dict((a, b)for a, b in y)-{a: b for a, b in y}] FAILED [ 19%]
tests/pyupgrade_test.py::test_dictcomps[dict(\n    (\n        a,\n        b,\n    )\n    for a, b in y\n)-{\n        a:\n        b\n    for a, b in y\n}] FAILED [ 20%]
tests/pyupgrade_test.py::test_dictcomps[x(\n    dict(\n        (a, b) for a, b in y\n    )\n)-x(\n    {\n        a: b for a, b in y\n    }\n)] FAILED [ 21%]
tests/pyupgrade_test.py::test_percent_format["%s" % ("%s" % ("nested",),)-"{}".format("{}".format("nested"))] FAILED [ 63%]
tests/pyupgrade_test.py::test_main_changes_a_file FAILED                 [ 97%]
tests/pyupgrade_test.py::test_main_keeps_line_endings FAILED             [ 97%]

Seems to be related to tuple offset calculation -- which I'm pretty sure was buggy before

Consider more complex format-string replacement.

I have the following piece of code:

            if preview_version_string:
                actions.append(
                    'ttfix --name-version="{v}" -o "{target}" "{target}"'.format(
                        target=target_path, v=preview_version_string
                    )
                )

            if kerntable_file:
                actions.append(
                    (
                        "kerntable --ufo {ufo} --glyphset {kerntable} "
                        '"{target}" "{target}"'
                    ).format(ufo=ufo_path, target=target_path, kerntable=kerntable_file)
                )
                file_deps.append(kerntable_file)

pyupgrade --py36-plus did not upgrade those to f-strings.

Recursive mode?

Currently I run pyupgrade with something like find ... | xargs pyupgrade. A recursive flag to drill down directories would be nice (and in line with a lot of other formatters like black, yapf, and isort).

KeyError running with --py36-plus on matplotlib's ticker.py

Running pyupgrade 1.5 on matplotlib's ticker.py as of, say, mpl3.0rc2 (https://github.com/matplotlib/matplotlib/blob/v3.0.0rc2/lib/matplotlib/ticker.py) gives me

Traceback (most recent call last):
  File "/usr/bin/pyupgrade", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python3.7/site-packages/pyupgrade.py", line 907, in main
    ret |= fix_file(filename, args)
  File "/usr/lib/python3.7/site-packages/pyupgrade.py", line 884, in fix_file
    contents_text = _fix_fstrings(contents_text)
  File "/usr/lib/python3.7/site-packages/pyupgrade.py", line 858, in _fix_fstrings
    tokens[i] = token._replace(src=_to_fstring(token.src, node))
  File "/usr/lib/python3.7/site-packages/pyupgrade.py", line 824, in _to_fstring
    name = ''.join((params[k or str(i)], dot, rest))
KeyError: '3'

Percent printf not rewritten

A file containing

a = 2
b = "%s" % a

is not rewritten to use .format() unless I wrap the a into a one-element tuple like b = "%s" % (a,).

TokenError

Thank you very much for writing this tool!!

I want to run pyupgrade over a project that probably still contains code that raises a syntax error when imported in a python3 interpreter.

pyupgrade --py36-plus outputs is the following in a python3.7 conda env:

filename1.py
filename2.py
Traceback (most recent call last):
  File "/home/thomas/miniconda/envs/py37/bin/pyupgrade", line 10, in <module>
    sys.exit(main())
  File "/home/thomas/miniconda/envs/py37/lib/python3.7/site-packages/pyupgrade.py", line 1428, in main
    ret |= fix_file(filename, args)
  File "/home/thomas/miniconda/envs/py37/lib/python3.7/site-packages/pyupgrade.py", line 1397, in fix_file
    contents_text = _fix_format_literals(contents_text)
  File "/home/thomas/miniconda/envs/py37/lib/python3.7/site-packages/pyupgrade.py", line 107, in _fix_format_literals
    tokens = src_to_tokens(contents_text)
  File "/home/thomas/miniconda/envs/py37/lib/python3.7/site-packages/tokenize_rt.py", line 44, in src_to_tokens
    ) in tokenize.generate_tokens(tokenize_target.readline):
  File "/home/thomas/miniconda/envs/py37/lib/python3.7/tokenize.py", line 579, in _tokenize
    raise TokenError("EOF in multi-line statement", (lnum, 0))
tokenize.TokenError: ('EOF in multi-line statement', (350, 0))

Unfortunately this exception is not very helpful, since it doesn't really tell me which file lead to this exception. Note that filename2.py is not the culprit, because it doesn't have a line with number 350.

IMO pyupgrade should avoid showning tracebacks by defaults, but instead output filename+linum of the faulty files.

Tested it with

(py37) > $ conda list pyupgrade                                                                                 [±master ●●]
# packages in environment at /home/thomas/miniconda/envs/py37:
#
# Name                    Version                   Build  Channel
pyupgrade                 1.17.0                     py_0    conda-forge

BTW pyupgrade doesn't have a --version cli flag.

six: burn the bridges

Let's try and automatically un-six python source. Here's a quick audit of what six does and how to undo it in python 3:

From the docs:

trivial replacements

  • six.text_type => str
  • six.binary_type => bytes
  • six.class_types => (type,)
  • six.string_types => (str,)
  • six.integer_types => (int,)
  • six.unichr => chr
  • six.iterbytes => iter
  • six.print_(...) => print(...)
  • six.exec_(c, g, l) => exec(c, g, l)
  • six.advance_iterator(it) => next(it)
  • six.next(it) => next(it)
  • six.callable(x) => callable(x)

trivial removals:

  • python_2_unicode_compatible (remove decorator)

    @six.python_2_unicode_compatible
    class C: pass

requires parsing a function call

  • six.u('foo') => 'foo'
  • six.byte2int(bs) => bs[0]
  • six.indexbytes(bs, i) => bs[i]
  • six.iteritems(dct) => dct.items() # also {iter,view}{items,keys,values}
  • six.create_unbound_method(fn, cls) => fn
  • six.get_unbound_method(meth) => meth
  • six.get_method_function(meth) => meth.__func__
  • six.get_method_self(meth) => meth.__self__
  • six.get_function_closure(fn) => fn.__closure__
  • six.get_function_code(fn) => fn.__code__
  • six.get_function_defaults(fn) => fn.__defaults__
  • six.get_function_globals(fn) => fn.__globals__
  • six.assertCountEqual(self, arg1, arg2) => self.assertCountEqual(arg1, arg2)
  • six.assertRaisesRegex(self, fn, *a, **k) => self.assertRaisesRegex(fn, *a, **k)
  • six.assertRegex(self, *a) => self.assertRegex(*a)

maybe replace with literal?

  • six.b => replace with bytes literal?

need to know that you're not in a place where a raise statement is invalid

  • six.raise_from(exc, exc_from) => raise exc from exc_from
  • six.reraise(tp, exc, tb) => raise exc.with_traceback(tb)

probably can reuse the "new style class" logic

  • class C(six.Iterator): => class C:

requires multiple function call parses, probably hard

  • class C(six.with_metaclass(Meta, Base)): => class C(Base, metaclass=Meta):

  • (rewriting add_metaclass)

    @six.add_metaclass(mcs)
    class C(...): pass

    to

    class C(..., metaclass=mcs): pass

probably can't do (requires introducing an import)

Note that for six.moves I'm planning on handling that in reorder-python-imports

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.