blakenaccarato / copier-python Goto Github PK

Dependabot runs over in blakeNaccarato/tooling. When a pull request is merged over there, I want the updated dependency to show up as a commit on this repo. That can be done by writing an update job into main.yml that needs all the other jobs in main.yml to succeed. Then, update will propagate changes back to blakeNaccarato/copier-python/template. This automates that section of the "loop" between the two repos.

The other end of the "loop", updating blakeNaccarato/tooling whenever the template changes, is less automatable. The questions/answers might change so the automation would be fragile. There is currently no way to not run a job if a certain file changes.

The update job shouldn't leak into my other projects, so it should be protected behind Jinja conditionals. That conditional could be tied to the gitignore_dependencies question which should probably be generalized to are_we_in_tooling or something.

Once this automation is established, it should become easier to maintain tooling, as bumped Dependencies will come as new commits.

This is the only file where it still matters that it doesn't get copied into runners due to a bug with Copier. It will be necessary to unignore this file, but also not to keep track of it in Dependabot runs for repos outside of this template repo.

e.g. pipx install copier=={{ _min_copier_version }}

Add an action to the flake8 workflow that generates in-line code annotations right in the GitHub interface.

Only attempt this after successfully implementing #12.

Instead of an outright override, Dynaconf could be used to update the user's tooling configs.

This script would ask the user if they have pipx installed, if they have Copier installed, and install them if not already there. Then it will run setup.ps1.

This will probably be the best way to override files like launch.json or pyrightconfig.json for specific projects. One could agree to answer a set of override questions, and answer yes/no to each one. This would programmatically exclude that file from being synced to that repo. Merges/updates could be applied by changing a yes to a no, bumping the template, and then changing the no back to a yes.

These scripts are mostly the same. We should rewrite it as a script that takes a boolean argument for something like "first time setup".

The project will use Copier tasks to ask the same questions as the copier_python template, populate initial file structures, then bootstrap the copier_python template (after changing _src_path in copier_answers.yml). There could be a GitHub action to keep the copier.yml file up-to-date so that it informs the follow-on template.

For those files not migrated in #2, make sure they're shipped with the template

It turns out that we can create a .github "magic repo" that will inform the default workflows, issue templates, pull request templates, etc. for all repos. This may be a better source for managing .github specifics, reducing the overhead of tooling.

However, there may be value in keeping such things in the template, as others can get those things in their repos for free. But maybe we keep Funding out of the template, as it's not guaranteed to be set up for all users.

There may be fewer growing pains if we bite the bullet and use poetry. Although I am concerned about user-friendliness to newcomers. Perhaps instruction about installing poetry with pipx would help somewhat. After all, copier must already be installed with pipx.

• Migrate to poetry
• Include standard PyPI metadata like Python version, docs page, issues page, etc.

This is a quality-of-life update. Tab-completion for tooling/update.ps1 locally is intercepted by setup_cicd.sh because of the underscore. Let's rename the CICD shell script.

Pretty sure this is unnecessary.

Having it in blakeNaccarato/.github puts it everywhere, even forks. We need a way to explicitly put it on repos.

The canonical way to override defaults in copier is to ask a question about it. A question would need to be asked for every file that might need to be overridden, and ugly Jinja "ifs" will have to be placed on folders/filenames and in files.

The user experience could be marginally improved by hiding these questions behind a conditional, "Do you want to answer advanced questions?" but the last time I tried this, the subkeys protected by such an optional question don't even get set when the user declines. Perhaps that's different now, or I was mistaken.

An alternative would be to allow users to override by a custom list of exclusions that is processed by a Copier action script, but these actions take place after the copying process, so that might be difficult to orchestrate.

Finally, certain files can be overridden simply by un-gitignoring them (mainly tooling), and then rejecting changes either at the copying stage or before committing. But this would also require custom handling, especially in CI/CD.

But this should be looked into, because linter configs and requirements files often differ by project.

The last resort would be to put them in the _skip_if_exists flag, which makes them subject to #76.

Currently I recommend a lot of extensions. I will need to segment these into absolutely essential extensions and recommended ones. An additional question in copier.yml could be posed as well.

These files technically have a broken python-version field since they're meant to be suffixed by .jinja.

Dependabot currently only manages Python dependencies in requirements.txt files and the pyproject.toml file, but only for Poetry. So maybe we should make Poetry the default tooling for now. If we go this route, we should keep a known good branch of flit to return to if/when PEP 508 is supported.

The Code of Conduct that I want to use is repo-dependent. This means that I cannot use the default blakeNaccarato/.github repo for it. I will have to template it with Copier.

This workflow should trigger upon new releases. It should use appropriately setup secrets.

The "Project-specific settings" heading is not quite right, since we ship these settings to all projects. Let's revisit this file and figure out how to streamline it.

This will automate dependency bumping in the Dependabot-unsupported pyproject.toml format source from the Dependabot-supported requirements.txt format.

I'm unsure of exactly what causes this, but it happens on the lab computer when a .venv folder is already present. Need to revisit this.

For example, the line requires-python = ">={{ python_version }}" will fall out of sync with other uses of {{ python_version }} due to pyproject.toml.jinja only being used to re-populate pyproject.toml if it doesn't already exist. We don't want our template overwriting a package's pyproject.toml, because that is highly customized per project. On the other hand, we want Python version in pyproject.toml getting bumped when it gets bumped elsewhere.

Perhaps this is a use-case for a so-called "Task" in Copier, which would allow for us to touch just the relevant lines of pyproject.toml and other "one-shot" files with templated values. Instead of a Copier Task, it could also be handled in CICD workflows similar to the bumping of pyproject.toml when requirements.txt changes via template/tooling/bump_pyproject.py (currently a workaround for unsupported flit-flavored pyproject.toml).

My PowerShell profile activates virtual environments detected in the current environment. I could instruct first-time contributors to modify their profile in CONTRIBUTING.md, but it would be nice to ship the appropriate profile options.

Maybe we can include a profile mix-in that can be dot-sourced by users.

When bumping the template in a destination repository, changes must be staged before whitespace-only changes are recognized by git and GUI-based tools informed by git (e.g. VSCode's Source Control UI). I suspect this is due to platform-specific whitespace leaking into some of my template files. I should go through and make sure all files have UNIX-style line endings. If they do, then the problem happens during templating with Jinja, and I should find/report an associated issue.

Early in the development of a package (especially at __version__ = 0.0.0), it's possible that a package's dependencies are bumped in pyproject.toml without the package itself having its version incremented. When using flit install -s in this fashion, an error that looks something like the following may appear:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
package 0.0.0 requires dependency~=#.#.#, but you have dependency #.#.# which is incompatible.

Perhaps the advice on "developmental releases" from PEP 440 should be followed. Whenever a dependency is bumped, also increment the number "X' in #.#.#.devX. This could be implemented in the bump.yml workflow supplied by this template.

Tools to add

• isort
• black
• sourcery
• #331
• Add template question for the platform to run the workflows on
• Minimize setup steps for workflows.
  • For example, publish workflow doesn't need to install the package

Okay this seems to be a larger issue. On a GitHub Actions runner, copier seems to run successfully, however, top-level files copied over in the template (such as .flake8) don't actually get copied over. This is the source of flake8 behaving differently in Actions than locally.

Manual touch and mkdir calls work fine, it is something to do with copier failing to properly create top-level files, despite thinking that it did so. Files nested in at least one folder copy over just fine, however. A workaround could be to keep master versions in .tooling, and run CI/CD off of that, but it still doesn't solve the issue of Copier improperly copying top-level files.

It may be related to copier-org/copier#273 somehow. That Copier is careful when dealing with folders that it did not create, and this somehow leads to a file not being properly created at root? Will have to look further.

The tooling/.pythonrc.py file will run whenever python or ipython is run. This allows us to customize REPL colors, import niceties from willmcgugan/rich, and so on.

For example when an issue receives the "good first issue" label, a bot should leave a comment giving contribution tips to newcomers. A document should also be maintained that lists all the good first issues.

These files are generated when updating the template and a file is ignored by _skip_if_exists. Until conflict resolution is changed in copier-org/copier#613, let's scrub these files away after updating the template.

This would permit its use as a project grows from idea to package. The src folder and pyproject.toml would be omitted until the user is ready to commit to a package.

Before then, all the benefits of the tooling, and even tests would be available. This would also facilitate the template functioning as a general source of truth for tooling.

This will allow for customizing the Jupyter experience. We will need to point to our templated directory in the IPYTHONDIR environment variable. We will need to duplicate pythonrc.py here. Also, we can't have the dot-prefix because for some reason Jupyter doesn't load the file then!

We will use the startup folder of the IPython profile folder to configure Jupyter.

Would be useful for black and isort, at the very least.

For example:

jobs:
  job:
    runs-on: ubuntu-20.04
    steps:
      - uses: actions/[email protected]
      - uses: actions/[email protected]
        with:
          python-version: 3.9                            < here

So the "tmp" directory of the template is leaking weirdness into the diff, which results in a task seeming to run before completing the migration? And that task tries to act on something that doesn't exist, perhaps because of an absolute path to the wrong place is occurring?

See referenced comment for the logic flow that may be causing this issue?

Maybe due to the paths being generated from the terminal instance in the current directory rather than the expected tmp directory?

copier-org/copier#239 (comment)

It's easy to mess up a noqa statement, so let's introduce this meta linter that makes sure it's done properly.

This will streamline GitHub Actions to not run the entire Copier routine, now that most necessary configs are present. It will just have to install the Python dependencies and run the appropriate tool.

This could be the first pre-commit hook to be installed. Other pre-commit use cases are simple configuration validation checks. We probably shouldn't force linters to pass in pre-commit, so that workflows can catch them instead. Aggressive pre-commit hooks might discourage novice contributions.

Do so by migrating as many files as possible into a tooling folder. Linter configs and the like should stay in root to cut down on the CLI clutter that would be necessary when invoking non-default config paths.

See: https://stackoverflow.com/questions/16827858/npm-warn-package-json-no-repository-field/16827990#16827990