Giter VIP home page Giter VIP logo

python-btrfs's Introduction

python-btrfs

What is python-btrfs?

python-btrfs is a Python 3 library that provides ways to interact programmatically with an online btrfs file system.

It provides a pure python shadow implementation of data structures used in btrfs together with convenient wrappers around the collection of kernel functions that are available in the btrfs kernel API.

Using them, we can examine the secret inner world of a btrfs file system for educational purposes.

Where do I get it?

Your favourite GNU/Linux distro probably has it packaged as either python-btrfs or python3-btrfs package.

This git repository with source code can also directly be used with python 3. No dependencies other than the python standard library are needed.

Should I be using this?

The target audience for using the library is system administrators and developers who want to discover more about the internals of a btrfs file system, or want to create adjusted monitoring or administration tools that are optimized for their specific use cases.

Of course, it's python, so, this is for who prefers programming python over programming C for quickly building fun stuff.

I have a broken file system, can I repair it using python-btrfs?

python-btrfs does not directly access disk storage, it only uses functions available in the kernel interface, using system calls. This also means that python-btrfs can not be used to repair a broken filesystem whenever the running Linux kernel cannot properly mount it.

What can I do with python-btrfs?

Using it allows one to take a peek behind the curtains of the regular functionality provided by the btrfs-progs programs and the libbtrfsutil C and Python library.

You can basically do anything that btrfs-progs or libbtrfsutil can do with an online file system. However, at the same time we're operating on a bit lower abstraction level. However again, that allows us to also be creative and make optimized utilities for our own special use cases.

An example is the btrfs-balance-least-used program that you can find in the bin directory. It's a modified algorithm for using btrfs balance to compact allocated space (i.e. defragment free space) as fast and efficient as possible by taking the usage ratio of the individual allocations of raw disk space into account.

Show me some example code!

Let's for example have a look at the equivalent of the btrfs fi df / command:

>>> import btrfs
>>> with btrfs.FileSystem('/') as fs:
...     for space in fs.space_info():
...         print(space)
... 
Data, single: total=839.01GiB, used=838.47GiB
System, DUP: total=8.00MiB, used=112.00KiB
Metadata, DUP: total=4.00GiB, used=2.38GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

Well, that was easy! But, say, instead of this text, you want to create a pie chart out of it. Now, instead of writing a horrible program that parses back the text output of the btrfs fi df command, we can access the values directly.

>>> spaces = fs.space_info()
>>> len(spaces)
4

The space_info() function calls the SPACE_INFO kernel function, which returns a list of SpaceInfo objects. By feeding one of those to the pretty printer in the utils module, we can see all contents. The attributes are directly accessible in our code:

>>> btrfs.utils.pretty_print(spaces[0])
<btrfs.ioctl.SpaceInfo>
flags: Data, single
total_bytes: 839.01GiB
used_bytes: 838.50GiB

>>> spaces[0].flags
1
>>> btrfs.utils.block_group_flags_str(spaces[0].flags)
'DATA'
>>> spaces[0].total_bytes
900877778944

So, using these values, we could create a nice picture using an imaging library.

More examples!

The bin and examples directory in the source code contain an example collection of programs that are built using the library.

Documentation

Reference documentation of the stable API of the library is written in Sphinx autodoc format. An online version of the HTML documentation is also available.

In general, the btrfs.FileSystem object, shown above, is the best starting point for exploring available functionality.

Tutorial style documentation will be added in the future.

License

The python-btrfs library itself is licensed under the LGPL-3.0.

Example scripts in the bin directory are licensed under the MIT License (Expat). Feel free to use all the ideas and code from them to build new stuff using python-btrfs!

python-btrfs's People

Contributors

cebtenzzre avatar endolith avatar garamotte avatar jorti avatar knorrie avatar lorddoskias avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

python-btrfs's Issues

! [new branch] debian/master -> origin/debian/master (unable to update local ref)

PSA: I'm converting the debian branches of this repo to DEP14 format.

That means that we'll have debian/master and debian/stretch-backports etc.

For whoever has cloned this repo before, this will result in the error shown in the subject, when trying to fetch updates:

error: cannot lock ref 'refs/remotes/origin/debian/master': 'refs/remotes/origin/debian' exists; cannot create 'refs/remotes/origin/debian/master'
 ! [new branch]      debian/master            -> origin/debian/master  (unable to update local ref)
error: cannot lock ref 'refs/remotes/origin/debian/stretch-backports': 'refs/remotes/origin/debian' exists; cannot create 'refs/remotes/origin/debian/stretch-backports'
 ! [new branch]      debian/stretch-backports -> origin/debian/stretch-backports  (unable to update local ref)

The git client will suggest:

error: some local refs could not be updated; try running
 'git remote prune origin' to remove any old, conflicting branches

As long as you don't have a local branch based on the old upstream debian, just follow the instruction once, and you'll be fine.

"Promote" useful examples to proper programs in /usr/bin

I'm thinking about these ones:

  • show_usage.py -> btrfs-usage-report
  • balance_least_used.py-> btrfs-balance-least-used
  • show_orphan_cleaner_progress.py -> btrfs-orphan-cleaner-progress
  • space_calculator.py -> btrfs-space-calculator

After I do some more work on the tutorial documentation, 99% of the other content of current examples/, or maybe all of it can be deleted anyway (or, should be converted to a less poor way of doing regression testing).

All of them need a bit better commandline argument handling, not explode with a stack trace if you point it to a non-btrfs filesystem, and a basic manual page.

Subvolumes (!= '/' or '/@') break munin plugin

Hi there!
I just found python-btrfs (v8) and enabled the munin plugin on some test nodes.

While working fine on a few nodes, I got problems on my workstation:

linux # /etc/munin/plugins/btrfs_usage 
Traceback (most recent call last):
  File "/etc/munin/plugins/btrfs_usage", line 119, in <module>
    main()
  File "/etc/munin/plugins/btrfs_usage", line 106, in main
    for fs in btrfs.utils.mounted_filesystems():
  File "/usr/lib/python3/dist-packages/btrfs/utils.py", line 34, in mounted_filesystems
    fs = btrfs.ctree.FileSystem(path)
  File "/usr/lib/python3/dist-packages/btrfs/ctree.py", line 461, in __init__
    _fs_info = self.fs_info()
  File "/usr/lib/python3/dist-packages/btrfs/ctree.py", line 467, in fs_info
    return btrfs.ioctl.fs_info(self.fd)
  File "/usr/lib/python3/dist-packages/btrfs/ioctl.py", line 113, in fs_info
    fcntl.ioctl(fd, IOC_FS_INFO, buf)
OSError: [Errno 25] Inappropriate ioctl for device

So I took a look at the code (that's reading /proc/self/mounts to get btrfs mounted filesystems).

Output in case of the broken machine is this

linux # cat /proc/self/mounts | grep " btrfs "
/dev/mapper/linux9--vg-root / btrfs rw,relatime,ssd,space_cache,subvolid=257,subvol=/@ 0 0
/dev/mapper/linux9--vg-root /home btrfs rw,relatime,ssd,space_cache,subvolid=258,subvol=/@home 0 0
/dev/sdd1 /proj.stand/speed btrfs rw,relatime,nodatasum,nodatacow,ssd,space_cache,subvolid=5,subvol=/ 0 0
/dev/sda1 /proj.stand/scratch btrfs rw,relatime,space_cache,subvolid=5,subvol=/ 0 0
/dev/mapper/linux9--vg-root /var/lib/docker/btrfs btrfs rw,relatime,ssd,space_cache,subvolid=257,subvol=/@/var/lib/docker/btrfs 0 0
/dev/mapper/crypto /proj.stand/scratch/backup-mnt btrfs rw,relatime,space_cache,subvolid=5,subvol=/ 0 0

Only thing different on my workstation is the usage of (non-standard) subvols.
(Machine is Ubuntu 16.04 with /home (and docker) on separate subvol.)

So I extended the applied filtering to only allow subvols / and /@ and the plugin started working.

Diff is here:

linux # diff /usr/lib/python3/dist-packages/btrfs/utils.py  /usr/lib/python3/dist-packages/btrfs/utils.py.ok
33c33,34
<     for path in [mount[1] for mount in mounts if mount[2] == 'btrfs']:
---
>     for path in [mount[1] for mount in mounts if (mount[2] == 'btrfs' and mount[3].endswith('subvol=/') or mount[3].endswith('subvol=/@') )]:

Maybe the other subvolumes just need to be handled differently (or do not provide any additional information anyway). Hope this helps to fix this behaviour.

Request: subvolume qgroup stats

I have a script which lists all subvolumes for a filesystem and saves their usage referenced and exclusive values to a database for further analysis. Currently this calls btrfs subvolume show from btrfs-progs for each subvolume. It would be nice if I could use python-btrfs for this instead.

Ideally there would be an easy way to retrieve all the information provided by btrfs subvolume show.

Find non-mounted volumes

Hi,
I am very new to btrfs, so maybe I am using wrong the terminology or am thinking in the wrong direction altogether.
I would like discover all block devices that are formatted with btrfs, even if the volume isn't mounted, yet, as I'd like to get a list of all devices of type btrfs and the information to which volume they belong.

The command line tool allows me to do this:

$ btrfs filesystem show
Label: none  uuid: 4f4bb4b2-69e2-4395-ac4e-08fefa6715ff
    Total devices 1 FS bytes used 128.00KiB
    devid    1 size 512.00MiB used 20.00MiB path /dev/loop0

For this it is using internally the btrfs_scan_devices() function.
Am I correct that python-btrfs is not implementing access to this functionality?

show_inode.py crash

v8

linux-gtrk:/host/home/src/python-btrfs/examples # ./show_inode.py 262 256 /
inode generation 15 transid 15874 size 24 nbytes 0 block_group 0 mode 40755 nlink 1 uid 0 gid 0 rdev 0 flags 0x0(none)
inode ref list size 1
    inode ref index 0 name utf-8 ..
Traceback (most recent call last):
  File "./show_inode.py", line 37, in <module>
    raise Exception("Whoa, key {}".format(btrfs.ctree.key_type_str(header.type)))
Exception: Whoa, key DIR_ITEM

Documentation for python-btrfs

It would be nice if there was some documentation included this project instead of the current terse README text.

So... my question is... (thinking out loud)

  • Are you using python-btrfs for anything?
  • If so, for what?
  • What kind of extra information to help you start to script things in python for your btrfs filesystem management would you like to see?

For anyone interested in technical things about btrfs, the commit messages for the full history of the project contain a wealth of information already, and the examples cover all implemented functionality.

But, maybe it would be nice to have some tutorial-style documentation pages to show you around the world of btrfs with code examples and explanation about why metadata is organized the way it is. (Like, what is a chunk? What's the difference between a chunk and a block group? Why are all those names so confusing? How can I see what chunks I have? etc...)

See for example my linux networking tutorials for an idea about how the writing style would look like. One of my all time favourite tutorials is the Linux Advanced Routing & Traffic Control documentation, which helped me learning the basics of networking a long time ago. The page about Exploring your current configuration has been the first inspiration point for me to learn how to write fun documentation, showing a user around to discover what's happening.

So, the existence of this issue is an opportunity for anyone out there to comment on these ideas and provide feedback.

btrfs-space-calculator crashes with raid5

bor@bor-Latitude-E5450:~/src/python-btrfs$ git describe HEAD
v11
bor@bor-Latitude-E5450:~/src/python-btrfs$ ./bin/btrfs-space-calculator -m raid0 -d raid5 3T 1T 2T
Traceback (most recent call last):
  File "./bin/btrfs-space-calculator", line 189, in <module>
    main()
  File "./bin/btrfs-space-calculator", line 158, in main
    usage = btrfs.fs_usage.FsUsage(fs)
  File "/home/bor/src/python-btrfs/bin/btrfs/fs_usage.py", line 449, in __init__
    dev_extent_length = btrfs.volumes.chunk_to_dev_extent_length(chunk)
  File "/home/bor/src/python-btrfs/bin/btrfs/volumes.py", line 212, in chunk_to_dev_extent_length
    return chunk_length_to_dev_extent_length(chunk.type, chunk.num_stripes, chunk.length)
  File "/home/bor/src/python-btrfs/bin/btrfs/volumes.py", line 192, in chunk_length_to_dev_extent_length
    dev_extent_length = raw_data_bytes // num_data_stripes
ZeroDivisionError: integer division or modulo by zero
bor@bor-Latitude-E5450:~/src/python-btrfs$ ./bin/btrfs-space-calculator -m raid0 -d raid6 3T 1T 2T
Target metadata profile: RAID0
Target data profile: RAID6
Mixed block groups: False
Total raw filesystem size: 5.46TiB
Device sizes:
  Device 1: 2.73TiB
  Device 2: 931.32GiB
  Device 3: 1.82TiB
Metadata to data ratio: 1:200
Estimated virtual space to use for metadata: 5.00GiB
Estimated virtual space to use for data: 929.66GiB
Total unallocatable raw amount: 2.73TiB
Unallocatable raw bytes per device:
  Device 1: 1.82TiB
  Device 2: 0.00B
  Device 3: 931.32GiB
bor@bor-Latitude-E5450:~/src/python-btrfs$ 

It does not matter whether I use raid5 for data or metadata. Other profiles seem to work after quick testing.

btrfs-space-calculator device size is not optional

Help output makes impression device sizes can be omitted. They cannot (or is there usage where they can?).

bor@bor-Latitude-E5450:~/src/python-btrfs$ git describe HEAD
v11
bor@bor-Latitude-E5450:~/src/python-btrfs$ ./bin/btrfs-space-calculator -h
usage: btrfs-space-calculator [-h] -m METADATA -d DATA [-M] [-r RATIO]
                              [sizes [sizes ...]]
...
bor@bor-Latitude-E5450:~/src/python-btrfs$ ./bin/btrfs-space-calculator -m single -d single
Traceback (most recent call last):
  File "./bin/btrfs-space-calculator", line 189, in <module>
    main()
  File "./bin/btrfs-space-calculator", line 158, in main
    usage = btrfs.fs_usage.FsUsage(fs)
  File "/home/bor/src/python-btrfs/bin/btrfs/fs_usage.py", line 520, in __init__
    self._simulate_chunk_allocations(device_sizes)
  File "/home/bor/src/python-btrfs/bin/btrfs/fs_usage.py", line 675, in _simulate_chunk_allocations
    ratio = self._data_metadata_ratio()
  File "/home/bor/src/python-btrfs/bin/btrfs/fs_usage.py", line 603, in _data_metadata_ratio
    used_fraction = self.virtual_used / self.total
ZeroDivisionError: division by zero
bor@bor-Latitude-E5450:~/src/python-btrfs$ 

ctree.py:543: SyntaxWarning: invalid escape sequence '\_'

I just noticed the following warning while bumping the Gentoo package to v14 using python-3.12:

/usr/lib/python3.12/site-packages/btrfs/ctree.py:543: SyntaxWarning: invalid escape sequence '\_'

The actual line is here but I'm not sure what to do about this in a portable manner (across Python versions) or exactly how the pydoc comments are supposed to work.

show_usage crash on newly created filesystem

Just download version 6.

10:~/src/python-btrfs-6/examples # mkfs -t btrfs /dev/sdb1
btrfs-progs v4.10.2+20170406
See http://btrfs.wiki.kernel.org for more information.

Performing full device TRIM /dev/sdb1 (5.00GiB) ...
Label:              (null)
UUID:               a50f32b1-7c10-4bd1-b5d5-7f43e9f68832
Node size:          16384
Sector size:        4096
Filesystem size:    5.00GiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP             255.94MiB
  System:           DUP               8.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     5.00GiB  /dev/sdb1

10:~/src/python-btrfs-6/examples # mount /dev/sdb1 /mnt
10:~/src/python-btrfs-6/examples # mkdir /mnt/test
10:~/src/python-btrfs-6/examples # chattr +C /mnt/test
10:~/src/python-btrfs-6/examples # ./show_dev_extents.py /mnt
devid 1 type DATA pstart 12582912 length 8388608 pend 20971520 vaddr 12582912
devid 1 type SYSTEM|DUP pstart 20971520 length 8388608 pend 29360128 vaddr 20971520
devid 1 type SYSTEM|DUP pstart 29360128 length 8388608 pend 37748736 vaddr 20971520
devid 1 type METADATA|DUP pstart 37748736 length 268369920 pend 306118656 vaddr 29360128
devid 1 type METADATA|DUP pstart 306118656 length 268369920 pend 574488576 vaddr 29360128
10:~/src/python-btrfs-6/examples # ./show_usage.py  /mnt
dev item devid 1 total bytes 5367660544 bytes used 561905664
block group vaddr 12582912 transid 7 length 8388608 flags DATA used 65536 used_pct 1
Traceback (most recent call last):
  File "./show_usage.py", line 17, in <module>
    print("    " + str(chunk))
  File "/root/src/python-btrfs-6/examples/btrfs/ctree.py", line 623, in __str__
    "num_stripes {self.num_stripes}".format(self=self)
  File "/root/src/python-btrfs-6/examples/btrfs/ctree.py", line 619, in flags_str
    return btrfs.utils.flags_str(self.flags, _block_group_flags_str_map)
AttributeError: 'Chunk' object has no attribute 'flags'

Scrub

Any chance you would consider adding the scrub ioctls to the library?

(When the kernel 5 scrub cancel problem is fixed...) I would love to be able to rewrite my btrfs-scrub-slowly script (part of btrfs-balance-slowly) in python.

Rewriting balance-slowly in python using your library has improved it significantly. Although scrub-slowly is simpler, the timers are hell and it would be great to be able to get access in the script to things like the progress information as I am thinking about adjusting delays if things are running particularly slowly (implying a lot else is going on).

Just hoping you might consider adding it to your ever-growing list :-)

Error building sphinx documentation with python-3.10

The builds of the python-btrfs v13 package have begun to fail in Fedora rawhide with Python 3.10 with this error:

+ /usr/bin/make -O -j8 V=1 VERBOSE=1 html
sphinx-build -b html -d build/doctrees   source build/html
Running Sphinx v4.1.2
making output directory... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 3 source files that are out of date
updating environment: [new config] 3 added, 0 changed, 0 removed
reading sources... [ 33%] btrfs
WARNING: html_static_path entry '_static' does not exist

Exception occurred:
  File "/usr/lib/python3.10/site-packages/sphinx/util/docfields.py", line 369, in transform
    new_list += fieldtype.make_field(fieldtypes, self.directive.domain, items,
TypeError: patched_make_field() got an unexpected keyword argument 'inliner'
The full traceback has been saved in /tmp/sphinx-err-dfd4p4s4.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
make: *** [Makefile:53: html] Error 2

This is the full sphinx backtrace in the file /tmp/sphinx-err-dfd4p4s4.log:

# Sphinx version: 4.1.2
# Python version: 3.10.0rc1 (CPython)
# Docutils version: 0.16 release
# Jinja2 version: 3.0.1
# Last messages:
#   Running Sphinx v4.1.2
#   making output directory...
#   done
#   building [mo]: targets for 0 po files that are out of date
#   building [html]: targets for 3 source files that are out of date
#   updating environment:
#   [new config]
#   3 added, 0 changed, 0 removed
#   reading sources... [ 33%] btrfs
# Loaded extensions:
#   sphinx.ext.mathjax (4.1.2) from /usr/lib/python3.10/site-packages/sphinx/ext/mathjax.py
#   sphinxcontrib.applehelp (1.0.2) from /usr/lib/python3.10/site-packages/sphinxcontrib/applehelp/__init__.py
#   sphinxcontrib.devhelp (1.0.2) from /usr/lib/python3.10/site-packages/sphinxcontrib/devhelp/__init__.py
#   sphinxcontrib.htmlhelp (2.0.0) from /usr/lib/python3.10/site-packages/sphinxcontrib/htmlhelp/__init__.py
#   sphinxcontrib.serializinghtml (1.1.5) from /usr/lib/python3.10/site-packages/sphinxcontrib/serializinghtml/__init__.py
#   sphinxcontrib.qthelp (1.0.3) from /usr/lib/python3.10/site-packages/sphinxcontrib/qthelp/__init__.py
#   alabaster (0.7.12) from /usr/lib/python3.10/site-packages/alabaster/__init__.py
#   sphinx.ext.autodoc.preserve_defaults (1.0) from /usr/lib/python3.10/site-packages/sphinx/ext/autodoc/preserve_defaults.py
#   sphinx.ext.autodoc.type_comment (4.1.2) from /usr/lib/python3.10/site-packages/sphinx/ext/autodoc/type_comment.py
#   sphinx.ext.autodoc (4.1.2) from /usr/lib/python3.10/site-packages/sphinx/ext/autodoc/__init__.py
#   sphinx.ext.viewcode (4.1.2) from /usr/lib/python3.10/site-packages/sphinx/ext/viewcode.py
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/sphinx/cmd/build.py", line 280, in build_main
    app.build(args.force_all, filenames)
  File "/usr/lib/python3.10/site-packages/sphinx/application.py", line 343, in build
    self.builder.build_update()
  File "/usr/lib/python3.10/site-packages/sphinx/builders/__init__.py", line 293, in build_update
    self.build(to_build,
  File "/usr/lib/python3.10/site-packages/sphinx/builders/__init__.py", line 307, in build
    updated_docnames = set(self.read())
  File "/usr/lib/python3.10/site-packages/sphinx/builders/__init__.py", line 414, in read
    self._read_serial(docnames)
  File "/usr/lib/python3.10/site-packages/sphinx/builders/__init__.py", line 435, in _read_serial
    self.read_doc(docname)
  File "/usr/lib/python3.10/site-packages/sphinx/builders/__init__.py", line 475, in read_doc
    doctree = read_doc(self.app, self.env, self.env.doc2path(docname))
  File "/usr/lib/python3.10/site-packages/sphinx/io.py", line 188, in read_doc
    pub.publish()
  File "/usr/lib/python3.10/site-packages/docutils/core.py", line 217, in publish
    self.document = self.reader.read(self.source, self.parser,
  File "/usr/lib/python3.10/site-packages/sphinx/io.py", line 108, in read
    self.parse()
  File "/usr/lib/python3.10/site-packages/docutils/readers/__init__.py", line 77, in parse
    self.parser.parse(self.input, document)
  File "/usr/lib/python3.10/site-packages/sphinx/parsers.py", line 100, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 170, in run
    results = StateMachineWS.run(self, input_lines, input_offset,
  File "/usr/lib/python3.10/site-packages/docutils/statemachine.py", line 241, in run
    context, next_state, result = self.check_line(
  File "/usr/lib/python3.10/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 2769, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 327, in section
    self.new_subsection(title, lineno, messages)
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 393, in new_subsection
    newabsoffset = self.nested_parse(
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 281, in nested_parse
    state_machine.run(block, input_offset, memo=self.memo,
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/usr/lib/python3.10/site-packages/docutils/statemachine.py", line 241, in run
    context, next_state, result = self.check_line(
  File "/usr/lib/python3.10/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 2769, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 327, in section
    self.new_subsection(title, lineno, messages)
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 393, in new_subsection
    newabsoffset = self.nested_parse(
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 281, in nested_parse
    state_machine.run(block, input_offset, memo=self.memo,
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/usr/lib/python3.10/site-packages/docutils/statemachine.py", line 241, in run
    context, next_state, result = self.check_line(
  File "/usr/lib/python3.10/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 2342, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 2354, in explicit_construct
    return method(self, expmatch)
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 2096, in directive
    return self.run_directive(
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 2146, in run_directive
    result = directive_instance.run()
  File "/usr/lib/python3.10/site-packages/sphinx/ext/autodoc/directive.py", line 173, in run
    result = parse_generated_content(self.state, params.result, documenter)
  File "/usr/lib/python3.10/site-packages/sphinx/ext/autodoc/directive.py", line 116, in parse_generated_content
    nested_parse_with_titles(state, content, node)
  File "/usr/lib/python3.10/site-packages/sphinx/util/nodes.py", line 335, in nested_parse_with_titles
    return state.nested_parse(content, 0, node, match_titles=1)
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 281, in nested_parse
    state_machine.run(block, input_offset, memo=self.memo,
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/usr/lib/python3.10/site-packages/docutils/statemachine.py", line 241, in run
    context, next_state, result = self.check_line(
  File "/usr/lib/python3.10/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 2344, in explicit_markup
    self.explicit_list(blank_finish)
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 2369, in explicit_list
    newline_offset, blank_finish = self.nested_list_parse(
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 318, in nested_list_parse
    state_machine.run(block, input_offset, memo=self.memo,
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/usr/lib/python3.10/site-packages/docutils/statemachine.py", line 241, in run
    context, next_state, result = self.check_line(
  File "/usr/lib/python3.10/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 2647, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 2354, in explicit_construct
    return method(self, expmatch)
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 2096, in directive
    return self.run_directive(
  File "/usr/lib/python3.10/site-packages/docutils/parsers/rst/states.py", line 2146, in run_directive
    result = directive_instance.run()
  File "/usr/lib/python3.10/site-packages/sphinx/domains/__init__.py", line 286, in run
    return super().run()
  File "/usr/lib/python3.10/site-packages/sphinx/directives/__init__.py", line 212, in run
    DocFieldTransformer(self).transform_all(contentnode)
  File "/usr/lib/python3.10/site-packages/sphinx/util/docfields.py", line 251, in transform_all
    self.transform(child)
  File "/usr/lib/python3.10/site-packages/sphinx/util/docfields.py", line 369, in transform
    new_list += fieldtype.make_field(fieldtypes, self.directive.domain, items,
TypeError: patched_make_field() got an unexpected keyword argument 'inliner'

Unmounted (or unmountable) filesystems

Hi, I'm trying to somehow fix my currently broken raid10 array. Unfortunately, filesystem is unmountable, but with btrfs tools I was able to locate blocks I think are broken (invalid checksums, wrong parent key), but I don't have enough knowledge to at least try to fix them. Your library looks very promising, because I'm much more familiar with python than C, and it already helped me to better understand some things.

Is it possible to use this library to dig into unmounted/unmountable/broken filesystem? When trying to use with unmounted device (/dev/mapper/data0 is first of four) I get following error (IPython output):

Python 3.7.2 (default, Jan 10 2019, 23:51:51) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.2.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import btrfs                                                                                                  

In [2]: with btrfs.FileSystem('/dev/mapper/data0') as fs: 
   ...:     pass 
   ...:                                                                                                               
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-2-5ee87a216dac> in <module>
----> 1 with btrfs.FileSystem('/dev/mapper/data0') as fs:
      2     pass
      3 

~/src/array/.venv/lib/python3.7/site-packages/btrfs/ctree.py in __init__(self, path)
    632         self.path = path
    633         self.fd = os.open(path, os.O_RDONLY)
--> 634         _fs_info = self.fs_info()
    635         self.fsid = _fs_info.fsid
    636         self.nodesize = _fs_info.nodesize

~/src/array/.venv/lib/python3.7/site-packages/btrfs/ctree.py in fs_info(self)
    645         :rtype: :class:`btrfs.ioctl.FsInfo`
    646         """
--> 647         return btrfs.ioctl.fs_info(self.fd)
    648 
    649     def dev_info(self, devid):

~/src/array/.venv/lib/python3.7/site-packages/btrfs/ioctl.py in fs_info(fd)
    149     """
    150     buf = bytearray(ioctl_fs_info_args.size)
--> 151     fcntl.ioctl(fd, IOC_FS_INFO, buf)
    152     return FsInfo(buf)
    153 

OSError: [Errno 25] Inappropriate ioctl for device

Kernel: 4.20.7 (Arch Linux)
Btrfs-progs: v4.20.1

Support for block-group-tree

The new block-group-tree support in 6.1 is ๐ŸŒˆ awesomesauce ๐ŸŒˆ.
Currently it still needs to be explicitly enabled when building progs (via --enable-experimental), but it really works as advertised and cuts mount times to practically zero.

An unfortunate side effect is that this feature is not recognized by python-btrfs:

Traceback (most recent call last):
  File "/usr/local/bin/btrfs-debugfs", line 12, in <module>
    bg = fs.block_group(chunk.vaddr, chunk.length)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/btrfs/ctree.py", line 874, in block_group
    raise ItemNotFoundError("No block group at vaddr {}".format(vaddr))
btrfs.ctree.ItemNotFoundError: No block group at vaddr 1928125743104

Recognizing this feature would be super lovely.

change parent_uuid and parent_transid before btrfs receive

wondering if there's an easy way to change these values on the send stream encoded data of a snapshot? went down the BTRFS_IOC_SET_RECEIVED_SUBVOL route before realizing the fd has to be a mounted subvolume :/

trying to implement something where i rebase the snapshot on a new and either identical or almost identical subvolume on another machine before calling btrfs receive.

seems crazy to me this isn't a supported behavior.

Breakage with Python 3.7: 'str' object has no attribute 'decode'

Hooray, it's Python time ๐Ÿ’ฉ

Tinkering around with the examples and everything works fine with python 3.6:

PYTHONPATH=/tmp/python-btrfs/build/lib/ sudo python3.6 fs_info.py /mnt/backup
max_id 1 num_devices 1 fsid d163af2f-6e03-4972-bfd6-30c68b6ed312 nodesize 16384 sectorsize 4096 clone_alignment 4096
devid 1 uuid 01d2fa5e-0994-4d29-add7-8596122817ca bytes_used 1348686839808 total_bytes 4000785960960 path /dev/sdc1
devid 1 write_errs 0 read_errs 0 flush_errs 0 generation_errs 0 corruption_errs 0

Let's try the new hotness:

PYTHONPATH=/tmp/python-btrfs/build/lib/ sudo python3.7 fs_info.py /mnt/backup
Traceback (most recent call last):
  File "fs_info.py", line 3, in <module>
    import btrfs
  File "/tmp/python-btrfs/examples/btrfs/__init__.py", line 23, in <module>
    from btrfs.ctree import FileSystem  # noqa
  File "/tmp/python-btrfs/examples/btrfs/ctree.py", line 355, in <module>
    import btrfs.ioctl  # noqa
  File "/tmp/python-btrfs/examples/btrfs/ioctl.py", line 215, in <module>
    ioctl_search_key.format.decode(), 4096 - ioctl_search_key.size))
AttributeError: 'str' object has no attribute 'decode'

I dug through the Python 3.7 Changelog and think the relevant change is here:
"The struct.Struct.format type is now str instead of bytes."

Reading the bug discussion it seems this was actually a bug in <3.7, and the new behaviour is "correct". Not sure how to handle this so that it works with any version, maybe just check the type.

corruption_errors are reported as generation_errors in check_btrfs

Hello,

today one of our disks started to report corruption error:

$ btrfs dev stats /
[/dev/sdf2].write_io_errs   0
[/dev/sdf2].read_io_errs    0
[/dev/sdf2].flush_io_errs   0
[/dev/sdf2].corruption_errs 1
[/dev/sdf2].generation_errs 0

But Nagios plugin check_btrfs is reporting that wrongly as generation errors:

$ ./check_btrfs -m /
BTRFS CRITICAL: Device 1: generation_errs: 1
Physical size: 74.43GiB, Allocatable: 74.43GiB, Allocated: 30.05GiB (40%), Unallocatable: 0.00B (Reclaimable: 0.00B), Virtual used: 22.09GiB (30%), 1 device(s)

Misleading show_file.py error for non-regular file

v8

linux-gtrk:/host/home/src/python-btrfs/examples # ./show_file.py /etc
/etc is not a filename!
linux-gtrk:/host/home/src/python-btrfs/examples # 

Well, it is obviously filename (in the broad sense of Unix), it is just filename of a file for which show_file cannot return information. Probably "not a regular file" or similar would be more clear; even better would be to actually return information that is possible in this case. Thanks.

bblu --kthxbye

So, there's btrfs-balance-least-used, or bblu as we might call it. The reason this example program was created was to try defragment free space as efficient and quick as possible. I needed this to fight or to recover from situations in which the old -o ssd allocator was being used.

So what's the tool still good for now? Well, users still regularly ask for something that they can run periodically to prevent getting into unexpected ENOSPC situations because of whatever other reason. bblu could of course be used for this, by telling it to compact stuff until at least all block groups are at some % of usage. But, that would likely mean that it's often doing a lot of unnecessary work.

It would be interesting to make it a bit smarter so that it executes the minimal amount of work necessary with a goal of making sure there's actually usable unallocated raw disk space present. How hard can it be? Well, for example, if we're having 100G of unallocated disk space, but it's on 1 disk of 2 and the target profile is RAID1... fail.

What I'm thinking about is some fire-and-forget mode to run it in, in the title jokingly called --kthxbye, but maybe something like --auto. It should use a clear set of rules that we think need to be met.

Now, the python-btrfs library already has the fsusage module which provides a large amount of interesting information that can be used: https://python-btrfs.readthedocs.io/en/stable/btrfs.html#btrfs.fs_usage.FsUsage The btrfs-usage-report tool simply displays almost everything it can tell you.

  • There's estimated_allocatable_virtual_data, estimated_allocatable_virtual_metadata, which tell you (even taking current data/metadata usage ration in account) how much more of both you can store on the fs inside new chunk allocations that can still be done. This seems like an easy one to look at and try to get it on or above some limit.
  • If we're under that limit, we can just start doing basic bblu that goes compact stuff, and stop when ready, but it still might do way too much work on the wrong disk to e.g. overcome the above example with one full disk and RAID1.
  • In fsuage, there's also unallocatable_reclaimable which tells us how much unallocatable space can be recovered for use because of unbalanced allocations. If needed, we can go that path to get more unallocated space available. It would of course be more interesting to see what needs to be done to in a smart way for that. (feeding stuff to balance from the disk that has lowest unallocatable number?)

Next: but how do we figure out which block groups exactly need to be fed to balance to fix the unbalanced situation?

Adding a program to easily do small targeted metadata searches

After moving a bunch of interesting example programs into the bin/ location I'm looking at the rest of the random pile of code in examples/

What about a tool that allows searching any metadata you want? And what if it already has a bunch of presets that select ranges from trees automatically?

btrfs-search-metadata [-l|--long] <type> [<moar> ..] <path>

So, e.g.:

  • btrfs-search-metadata chunks /mountpoint -> show list of chunks
  • btrfs-search-metadata blockgroups /mountpoint -> show list of block groups
  • btrfs-search-metadata inode /path/to/file -> show everything related to directory or file inode
  • btrfs-search-metadata devices /mountpoint -> show list of devices with their info
  • btrfs-search-metadata orphans /mountpoint -> show list of orphaned stuff, like subvolume roots that the cleaner is working on
  • btrfs-search-metadata roots /mountpoint -> show list of tree roots present with some info
  • btrfs-search-metadata tree X [--min '(A B C)'] [--max '(D E F)'] /mountpoint -> dump any range of metadata objects from any tree on the screen, with key '(A B C)' and '(D E F)' as min/max search key. So e.g. '(31337 0 0)' and '(80085 -1 -1)' would show all metadata objects in that range. When no min or max given, just dump entire tree X.

All these things can be done in just a few lines of code using lib functionality. \o/

Doing things like the whole if/else tree inside examples/show_directory_contents.py is made easier by just using the new 'recursive object printer using str()' that's in develop branch now. The -l option for more elaborate output would just switch to the object pretty printer instead, which recursively prints all attributes of all nested objects that it gets fed.

Feedback / more thoughts anyone? This will remove another 80% of crap from the examples/ and when it's in distro packages it will be available to answer a lot of questions that people keep asking on the mailing list and on IRC. "How can I see which extents this file has and if they're compressed or not?" "What does my list of blockgroups look like?" etc.

Question: Managing Subvolumes

I'm looking to write a script to manage subvolumes locally and remotely. The idea is a create a "Time Machine" like setup with the additional benefit of copying good blocks from a backup should a checksum fail. I've perused your documentation, but I didn't find any mention of create or deleting subvolumes. I may have missed it, if so could you point me towards it? Is this something your library supports?

Add some percentages to btrfs-usage-report output

When having a quick look, it would be pretty convenient if there were some percentages added, for example here:

Total physical space usage:
|
| Total filesystem size: 15.00TiB
| Allocated bytes: 14.93TiB
| Allocatable bytes remaining: 67.97GiB (x.y%)

...and probably in a few other places as well.

Note that in the above example, the percentage should be calculated, not based on total filesystem size (sum of size of all attached block devices), but based on actual allocatable raw disk space, which can be less in case there's 'soft' or 'hard' unallocatable disk space that cannot be used. So, that's similar to how e.g. the nagios monitoring plugin calculates stuff.

check btrfs: some alternative check modes

Hey.

Right now, check_btrfs allows to check for allocated/unallocated space.
btw: The option names and their descriptions may be a bit ambiguous with respect to allocated vs. unallocated.

First:
I'm not an expert, but is it (still) necessarily a problem if btrfs runs out of unallocated space?
I mean it has the global reserve... and if there is plenty of free space left in data respectively meta block groups, not having any unallocated space may not mean much, unless perhaps that balance will no longer be possible (which may however not be an issue).
And if there's still enough free space in the already allocated data respectively meta data block groups, ... most other things should continue to run just fine?

At the university here I run a large data centre for the LHC at CERN, and recently stumbled into the following situation:
https://lore.kernel.org/linux-btrfs/CAHzMYBR9dFVTw5kJ9_DfkcuvdrO4x+koicfiWgVNndh8qU2aEw@mail.gmail.com/T/#t
which I would like to be detectable by check_btrfs.

What happened there is basically that unallocated space was completely used up, quite some space (~800GB) was still free in the data block groups, but nothing (usable) was free in the meta-data block groups.
So while there would have been space for the file data itself, nothing was left for new file metadata, so one got ENOSPC.

Of course, just checking for low unallocated space would also detect it, but would also ring the bell too often (namely when there's still enough left in the meta-data block groups.

So not sure how one could detect this better... maybe a check when unallocated space is below some threshold (or even 0) and free metadata space is also below some other threshold, while free data space is still above some threshold (otherwise, it would also report a fs, that's simply full).

Any ideas?

Cheers,
Chris.

./show_metadata_tree_sizes.py crash

10:/home/bor/python-btrfs/examples # ./show_metadata_tree_sizes.py /
Traceback (most recent call last):
File "./show_metadata_tree_sizes.py", line 61, in
for extent in fs.extents(min_vaddr, max_vaddr, load_metadata_refs=True):
File "/home/bor/python-btrfs/examples/btrfs/ctree.py", line 771, in extents
extent = MetaDataItem(header, data, load_refs=load_metadata_refs)
File "/home/bor/python-btrfs/examples/btrfs/ctree.py", line 1431, in init
self._load_refs(data[ExtentItem._extent_item.size:])
File "/home/bor/python-btrfs/examples/btrfs/ctree.py", line 1441, in _load_refs
self.tree_block_refs._append(InlineTreeBlockRef(inline_ref_offset))
AttributeError: 'list' object has no attribute '_append'
10:/home/bor/python-btrfs/examples # rpm -q python3
python3-3.7.2-2.2.x86_64

openSUSE Tumbleweed

On commit b9da652

check btrfs: support an automatic detection of all available btrfs filesystems

Hey.

It would be nice, if there was a switch that includes all available btrfs filesystems in the check, so that one doesn't have to configure each check manually if different hosts have many (different) btrfs filesystems at different mount points.

Maybe with some --exclude-fs switch that allows to exclude certain fs. That switch could then allow to specify via mountpoint or via UUID.

Not sure what the best way would be to detect all btrfs,... I mean one could of course parse /proc/self/mountinfo (but beware that its format is tricky: it has varying number of optional columns in the middle), but one would want to pick the same btrfs more than once, (which could easily happen with subvolumes and multiple mounts of these).

Cheers,
Chris.

struct.error: unpack_from requires a buffer of at least 12 bytes

Uhoh...

./show_block_group_contents.py 155713536000 /
[...]
extent vaddr 156097769472 length 16384 refs 3 gen 35254 flags DATA
    extent data backref root 257 objectid 150469 offset 0 count 1
    shared data backref parent 183848517632 count 1
    shared data backref parent 33333886976 count 1
extent vaddr 156109320192 length 4096 refs 9 gen 34379 flags DATA
    extent data backref root 319 objectid 574623 offset 6778880 count 1
    shared data backref parent 184193253376 count 1
    shared data backref parent 183844847616 count 1
    shared data backref parent 183840768000 count 1
    shared data backref parent 34028601344 count 1
    shared data backref parent 33983873024 count 1
    shared data backref parent 33752367104 count 1
    shared data backref parent 33468841984 count 1
    shared data backref parent 33417953280 count 1
extent vaddr 156109971456 length 4096 refs 1 gen 34379 flags DATA
    extent data backref root 319 objectid 574623 offset 4620288 count 1
extent vaddr 156111749120 length 4096 refs 1 gen 34383 flags DATA
    shared data backref parent 638205952 count 1
Traceback (most recent call last):
  File "../python-btrfs/examples/show_block_group_contents.py", line 11, in <module>
    for extent in fs.extents(vaddr, vaddr + block_group.length - 1):
  File "/home/knorrie/src/git/python-btrfs/examples/btrfs/ctree.py", line 381, in extents
    extent.append_shared_data_ref(header, data)
  File "/home/knorrie/src/git/python-btrfs/examples/btrfs/ctree.py", line 535, in append_shared_data_ref
    self.shared_data_refs.append(SharedDataRef(data, 0))
  File "/home/knorrie/src/git/python-btrfs/examples/btrfs/ctree.py", line 559, in __init__
    self.parent, self.count = SharedDataRef.shared_data_ref.unpack_from(data, pos)
struct.error: unpack_from requires a buffer of at least 12 bytes

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.