Giter VIP home page Giter VIP logo

python-fsutil's Introduction

python-fsutil

high-level file-system operations for lazy devs.

Installation

pip install python-fsutil

Usage

Just import the main module and call its methods.

import fsutil

Methods

assert_dir

# Raise an OSError if the given path doesn't exist or it is not a directory.
fsutil.assert_dir(path)

assert_exists

# Raise an OSError if the given path doesn't exist.
fsutil.assert_exists(path)

assert_file

# Raise an OSError if the given path doesn't exist or it is not a file.
fsutil.assert_file(path)

assert_not_dir

# Raise an OSError if the given path is an existing directory.
fsutil.assert_not_dir(path)

assert_not_exists

# Raise an OSError if the given path already exists.
fsutil.assert_not_exists(path)

assert_not_file

# Raise an OSError if the given path is an existing file.
fsutil.assert_not_file(path)

clean_dir

# Clean a directory by removing empty sub-directories and/or empty files.
fsutil.clean_dir(path, dirs=True, files=True)

convert_size_bytes_to_string

# Convert the given size bytes to string using the right unit suffix.
size_str = fsutil.convert_size_bytes_to_string(size)

convert_size_string_to_bytes

# Convert the given size string to bytes.
size_bytes = fsutil.convert_size_string_to_bytes(size)

copy_dir

# Copy the directory at the given path and all its content to dest path.
# If overwrite is not allowed and dest path exists, an OSError is raised.
# More informations about kwargs supported options here:
# https://docs.python.org/3/library/shutil.html#shutil.copytree
fsutil.copy_dir(path, dest, overwrite=False, **kwargs)

copy_dir_content

# Copy the content of the directory at the given path to dest path.
# More informations about kwargs supported options here:
# https://docs.python.org/3/library/shutil.html#shutil.copytree
fsutil.copy_dir_content(path, dest, **kwargs)

copy_file

# Copy the file at the given path and its metadata to dest path.
# If overwrite is not allowed and dest path exists, an OSError is raised.
# More informations about kwargs supported options here:
# https://docs.python.org/3/library/shutil.html#shutil.copy2
fsutil.copy_file(path, dest, overwrite=False, **kwargs)

create_dir

# Create directory at the given path.
# If overwrite is not allowed and path exists, an OSError is raised.
fsutil.create_dir(path, overwrite=False)

create_file

# Create file with the specified content at the given path.
# If overwrite is not allowed and path exists, an OSError is raised.
fsutil.create_file(path, content="", overwrite=False)

create_tar_file

# Create tar file at path compressing directories/files listed in content_paths.
# If overwrite is allowed and dest tar already exists, it will be overwritten.
fsutil.create_tar_file(path, content_paths, overwrite=True, compression="gzip")

create_zip_file

# Create zip file at path compressing directories/files listed in content_paths.
# If overwrite is allowed and dest zip already exists, it will be overwritten.
fsutil.create_zip_file(path, content_paths, overwrite=True, compression=zipfile.ZIP_DEFLATED)

delete_dir

# Alias for remove_dir.
fsutil.delete_dir(path)

delete_dir_content

# Alias for remove_dir_content.
fsutil.delete_dir_content(path)

delete_dirs

# Alias for remove_dirs.
fsutil.delete_dirs(*paths)

delete_file

# Alias for remove_file.
fsutil.delete_file(path)

delete_files

# Alias for remove_files.
fsutil.delete_files(*paths)

download_file

# Download a file from url to the given dirpath and return the filepath.
# If dirpath is not provided, the file will be downloaded to a temp directory.
# If filename is provided, the file will be named using filename.
# It is possible to pass extra request options (eg. for authentication) using **kwargs.
filepath = fsutil.download_file(url, dirpath=None, filename="archive.zip", chunk_size=8192, **kwargs)

exists

# Check if a directory of a file exists at the given path.
value = fsutil.exists(path)

extract_tar_file

# Extract tar file at path to dest path.
# If autodelete, the archive will be deleted after extraction.
# If content_paths list is defined, only listed items will be extracted, otherwise all.
fsutil.extract_tar_file(path, dest, content_paths=None, autodelete=False)

extract_zip_file

# Extract zip file at path to dest path.
# If autodelete, the archive will be deleted after extraction.
# If content_paths list is defined, only listed items will be extracted, otherwise all.
fsutil.extract_zip_file(path, dest, content_paths=None, autodelete=False)

get_dir_creation_date

# Get the directory creation date.
date = fsutil.get_dir_creation_date(path)

get_dir_creation_date_formatted

# Get the directory creation date formatted using the given format.
date_str = fsutil.get_dir_creation_date_formatted(path, format='%Y-%m-%d %H:%M:%S')

get_dir_hash

# Get the hash of the directory at the given path using
# the specified algorithm function (md5 by default).
hash = fsutil.get_dir_hash(path, func="md5")

get_dir_last_modified_date

# Get the directory last modification date.
date = fsutil.get_dir_last_modified_date(path)

get_dir_last_modified_date_formatted

# Get the directory last modification date formatted using the given format.
date_str = fsutil.get_dir_last_modified_date_formatted(path, format="%Y-%m-%d %H:%M:%S")

get_dir_size

# Get the directory size in bytes.
size = fsutil.get_dir_size(path)

get_dir_size_formatted

# Get the directory size formatted using the right unit suffix.
size_str = fsutil.get_dir_size_formatted(path)

get_file_basename

# Get the file basename from the given path/url.
basename = fsutil.get_file_basename(path)

get_file_creation_date

# Get the file creation date.
date = fsutil.get_file_creation_date(path)

get_file_creation_date_formatted

# Get the file creation date formatted using the given format.
date_str = fsutil.get_file_creation_date_formatted(path, format="%Y-%m-%d %H:%M:%S")

get_file_extension

# Get the file extension from the given path/url.
extension = fsutil.get_file_extension(path)

get_file_hash

# Get the hash of the file at the given path using
# the specified algorithm function (md5 by default).
filehash = fsutil.get_file_hash(path, func="md5")

get_file_last_modified_date

# Get the file last modification date.
date = fsutil.get_file_last_modified_date(path)

get_file_last_modified_date_formatted

# Get the file last modification date formatted using the given format.
date_str = fsutil.get_file_last_modified_date_formatted(path, format="%Y-%m-%d %H:%M:%S")

get_file_size

# Get the file size in bytes.
size = fsutil.get_file_size(path)

get_file_size_formatted

# Get the file size formatted using the right unit suffix.
size_str = fsutil.get_file_size_formatted(path)

get_filename

# Get the filename from the given path/url.
filename = fsutil.get_filename(path)

get_parent_dir

# Get the parent directory for the given path going up N levels.
parent_dir = fsutil.get_parent_dir(path, levels=1)

get_permissions

# Get the file/directory permissions.
permissions = fsutil.get_permissions(path)

get_unique_name

# Get a unique name for a directory/file ath the given directory path.
unique_name = fsutil.get_unique_name(path, prefix="", suffix="", extension="", separator="-")

is_dir

# Determine whether the specified path represents an existing directory.
value = fsutil.is_dir(path)

is_empty

# Determine whether the specified path represents an empty directory or an empty file.
value = fsutil.is_empty(path)

is_empty_dir

# Determine whether the specified path represents an empty directory.
value = fsutil.is_empty_dir(path)

is_empty_file

# Determine whether the specified path represents an empty file.
value = fsutil.is_empty_file(path)

is_file

# Determine whether the specified path represents an existing file.
value = fsutil.is_file(path)

join_filename

# Create a filename joining the file basename and the extension.
filename = fsutil.join_filename(basename, extension)

join_filepath

# Create a filepath joining the directory path and the filename.
filepath = fsutil.join_filepath(dirpath, filename)

join_path

# Create a path joining path and paths.
# If path is __file__ (or a .py file), the resulting path will be relative
# to the directory path of the module in which it's used.
path = fsutil.join_path(path, *paths)

list_dirs

# List all directories contained at the given directory path.
dirs = fsutil.list_dirs(path)

list_files

# List all files contained at the given directory path.
files = fsutil.list_files(path)

make_dirs

# Create the directories needed to ensure that the given path exists.
# If a file already exists at the given path an OSError is raised.
fsutil.make_dirs(path)

make_dirs_for_file

# Create the directories needed to ensure that the given path exists.
# If a directory already exists at the given path an OSError is raised.
fsutil.make_dirs_for_file(path)

move_dir

# Move an existing dir from path to dest directory.
# If overwrite is not allowed and dest path exists, an OSError is raised.
# More informations about kwargs supported options here:
# https://docs.python.org/3/library/shutil.html#shutil.move
fsutil.move_dir(path, dest, overwrite=False, **kwargs)

move_file

# Move an existing file from path to dest directory.
# If overwrite is not allowed and dest path exists, an OSError is raised.
# More informations about kwargs supported options here:
# https://docs.python.org/3/library/shutil.html#shutil.move
fsutil.move_file(path, dest, overwrite=False, **kwargs)

read_file

# Read the content of the file at the given path using the specified encoding.
content = fsutil.read_file(path, encoding="utf-8")

read_file_from_url

# Read the content of the file at the given url.
content = fsutil.read_file_from_url(url, **kwargs)

read_file_json

# Read and decode a json encoded file at the given path.
data = fsutil.read_file_json(path, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None)

read_file_lines

# Read file content lines.
# It is possible to specify the line indexes (negative indexes too),
# very useful especially when reading large files.
content = fsutil.read_file_lines(path, line_start=0, line_end=-1, strip_white=True, skip_empty=True, encoding="utf-8")

read_file_lines_count

# Read file lines count.
lines_count = fsutil.read_file_lines_count(path)

remove_dir

# Remove a directory at the given path and all its content.
# If the directory is removed with success returns True, otherwise False.
# More informations about kwargs supported options here:
# https://docs.python.org/3/library/shutil.html#shutil.rmtree
fsutil.remove_dir(path, **kwargs)

remove_dir_content

# Removes all directory content (both sub-directories and files).
fsutil.remove_dir_content(path)

remove_dirs

# Remove multiple directories at the given paths and all their content.
fsutil.remove_dirs(*paths)

remove_file

# Remove a file at the given path.
# If the file is removed with success returns True, otherwise False.
fsutil.remove_file(path)

remove_files

# Remove multiple files at the given paths.
fsutil.remove_files(*paths)

rename_dir

# Rename a directory with the given name.
# If a directory or a file with the given name already exists, an OSError is raised.
fsutil.rename_dir(path, name)

rename_file

# Rename a file with the given name.
# If a directory or a file with the given name already exists, an OSError is raised.
fsutil.rename_file(path, name)

rename_file_basename

# Rename a file basename with the given basename.
fsutil.rename_file_basename(path, basename)

rename_file_extension

# Rename a file extension with the given extension.
fsutil.rename_file_extension(path, extension)

replace_dir

# Replace directory at the specified path with the directory located at src.
# If autodelete, the src directory will be removed at the end of the operation.
# Optimized for large directories.
fsutil.replace_dir(path, src, autodelete=False)

replace_file

# Replace file at the specified path with the file located at src.
# If autodelete, the src file will be removed at the end of the operation.
# Optimized for large files.
fsutil.replace_file(path, src, autodelete=False)

search_dirs

# Search for directories at path matching the given pattern.
dirs = fsutil.search_dirs(path, pattern="**/*")

search_files

# Search for files at path matching the given pattern.
files = fsutil.search_files(path, pattern="**/*.*")

set_permissions

# Set the file/directory permissions.
fsutil.set_permissions(path, 700)

split_filename

# Split a filename and returns its basename and extension.
basename, extension = fsutil.split_filename(path)

split_filepath

# Split a filename and returns its directory-path and filename.
dirpath, filename = fsutil.split_filepath(path)

split_path

# Split a path and returns its path-names.
path_names = fsutil.split_path(path)

transform_filepath

# Trasform a filepath by applying the provided optional changes.
filepath = fsutil.transform_filepath(path, dirpath=None, basename=lambda b: slugify(b), extension="webp")

write_file

# Write file with the specified content at the given path.
fsutil.write_file(path, content, append=False, encoding="utf-8", atomic=False)

write_file_json

# Write a json file at the given path with the specified data encoded in json format.
fsutil.write_file_json(path, data, encoding="utf-8", atomic=False, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, cls=None, indent=None, separators=None, default=None, sort_keys=False)

Testing

# clone repository
git clone https://github.com/fabiocaccamo/python-fsutil.git && cd python-fsutil

# create virtualenv and activate it
python -m venv venv && . venv/bin/activate

# upgrade pip
python -m pip install --upgrade pip

# install requirements
python -m pip install -r requirements.txt -r requirements-test.txt

# install pre-commit to run formatters and linters
pre-commit install --install-hooks

# run tests using tox
tox

# or run tests using unittest
python -m unittest

License

Released under MIT License.


Supporting

See also

  • python-benedict - dict subclass with keylist/keypath support, I/O shortcuts (base64, csv, json, pickle, plist, query-string, toml, xml, yaml) and many utilities. ๐Ÿ“˜

  • python-fontbro - friendly font operations on top of fontTools. ๐Ÿงข

python-fsutil's People

Contributors

dependabot[bot] avatar fabiocaccamo avatar github-actions[bot] avatar pre-commit-ci[bot] avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

python-fsutil's Issues

Add `replace_file` method.

  • move new file to the same dir using a non-existing filename
  • copy old file filename
  • remove old file
  • rename new file using old file filename

fsutil write_file atomic=True destroys created file on exit

Python version
3.10

Package version
latest

Current behavior (bug description)
write_file atomic=True destroys the created file on exit.
Obviously fsutil is not designed for running on Windows. Cannot handle directory names etc.

Expected behavior
created file should stay

Test Results:
pytest_result.txt

Upvote & Fund

  • We're using Polar.sh so you can upvote and help fund this issue.
  • We receive the funding once the issue is completed & confirmed by you.
  • Thank you in advance for helping prioritize & fund our backlog.
Fund with Polar

Add possibility to write file atomically.

Upvote & Fund

  • We're using Polar.sh so you can upvote and help fund this issue.
  • We receive the funding once the issue is completed & confirmed by you.
  • Thank you in advance for helping prioritize & fund our backlog.
Fund with Polar

Add `pathlib.Path` path arguments support.

Upvote & Fund

  • We're using Polar.sh so you can upvote and help fund this issue.
  • We receive the funding once the issue is completed & confirmed by you.
  • Thank you in advance for helping prioritize & fund our backlog.
Fund with Polar

Tests folder included in fsutil release

Python version
Python 3.9

Package version
Since release 0.9

Current behavior (bug description)
In my project, we are using python-codicefiscale which has for dependency python-fsutil. When launching our Django app, we load our settings located in tests.[...].settings.
Python is taking the fsutil tests folder, located in my virtual environment, as the folder under which my settings are located.
The bug was introduced in this commit. The testsfolder is not excluded as the wheel is built.

Expected behavior
Update the setup.cfg file to exclude the tests folder.

File permissions get lost when using write_file with atomic=True

Python version
3.9

Package version
0.12.0

Current behavior (bug description)
When writing files using write_file with atomic=True, the file permissions of the original file get lost and are always set to 600.

Expected behavior
The file permissions of the file that is written to should be kept as is, no matter if you use atomic write or not.

Upvote & Fund

  • We're using Polar.sh so you can upvote and help fund this issue.
  • We receive the funding once the issue is completed & confirmed by you.
  • Thank you in advance for helping prioritize & fund our backlog.
Fund with Polar

Bug report related to _write_file_atomic() in Windows environment

Dear python-fsutil developers

I encountered a file write error on Windows environment while using the django-maintenance-mode package.
First, I reported it on the django-maintenance-mode issue tracker(#180), and since it's actually an fsutil issue, fabiocaccamo asked me to report it here, so I'll report it here.

Environment

  • OS: Windows 11 Home 23H2
  • pyenv-win 3.1.1
  • Python 3.10.11
  • Django 3.2
  • Django maintenance mode 0.21.1
  • Python-fsutil 0.13.1

Current behavior (bug description)
I confirmed 2 defects.

(1) _write_file_atomic() fails to create files:
When fsutil.write_file() function was called, _write_file_atomic() was called to create maintenance_mode_state.txt, but maintenance_mode_state.txt was not created. No exception is raised in _write_file_atomic(), but afterward assert_file() is called in fsutil.read_file() afterwards and raises an OSError exception. Actually, FileNotFoundError: [WinError 2] occurs in the isfile() function in the builtin genericpath.py, which is called by assert_file().

(2) _write_file_atomic() deletes existing files:
Due to the above issue, I manually created the maintenance_mode_state.txt file in advance and then ran the command below.
$ python manage.py maintenance_mode on
This command uses _write_file_atomic() to write characters to the maintenance_mode_state.txt file, but I confirmed that the maintenance_mode_state.txt file disappeared during this function.

No Problem In Linux Environment
I have confirmed that there is no problem described above in Linux environment.

That's all I've come across. Thank you for your time and assistance.

Upvote & Fund

  • We're using Polar.sh so you can upvote and help fund this issue.
  • We receive the funding once the issue is completed & confirmed by you.
  • Thank you in advance for helping prioritize & fund our backlog.
Fund with Polar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.