Impossibly large file at ~/.orbstack/data/data.img about orbstack HOT 59 CLOSED

Yes, data.img is a sparse file. This is expected behavior, not a bug. I've added it to the docs, which you can read for more info: https://docs.orbstack.dev/faq#data-img

The 8 TB data.img file in OrbStack's data is a virtual disk image that stores Docker and Linux data. Don't worry about the size; it's a special type of file known as a sparse file, meaning that it only takes as much space as you use, and automatically shrinks when you delete data. It does not take up 8 TB of disk space.

Sparse files are an APFS feature that provides a fast, efficient, and flexible way to store large files with varying usage. Time Machine supports them, so your backups will not be affected. The large 8 TB size allows OrbStack to manage the disk dynamically so you don't have to worry about having too much or too little disk space.

To find the true size of the file, run du -h ~/.orbstack/data/data.img, or look at "size on disk" in Finder's info panel. Sparse files have two sizes: an apparent size (8 TB in this case), and size on disk (actual usage). Many tools show the apparent size, but the size on disk is what matters for disk space.

If you have third-party apps complaining about the file, please see the GitHub issue. Since macOS uses sparse files extensively, we'd like to move the ecosystem forward and fix compatibility issues with them.

I understand that it may be surprising, but several factors went into my decision to ship an 8 TiB image:

• Sparse raw images are the only disk format supported by by Apple's framework
• Using large images is part of what makes OrbStack's seamless and dynamic disk management possible
• Time Machine handles them well

That being said, I was afraid that some third-party apps might have trouble with it. Some ideas for reducing the size:

• Reduce the image size to match the size of your Mac's SSD. I plan to do this in the long term anyway, but it hasn't been a priority so far. (example: 2 TB Mac with 1 TB free = 2 TB image size)
• Reduce it to the amount of free space, plus some %. (example: 2 TB Mac with 1 TB free = 1.2 TB image size)

I'm not sure it's worth the complexity though. No matter what, the image would always be bigger than the free space on your Mac, so apps that choke on it for size reasons would have the same problem either way. Reducing size further would adversely impact the dynamic disk feature.

In the long term, I think the best solution is to advance the macOS ecosystem by weeding out sparse file compatibility issues in apps. We could start by reporting the issue to Backblaze. In the meantime, you can safely exclude the file to work around the issue. It looks like they already exclude .vhd and .sparsebundle by default, so .img should likely just be added to the exclusion list as a quick fix.

If you don't have apps complaining about the file, don't worry — it won't cause any problems or take up any more space than what you're using on the Docker/Linux side. APFS (the current macOS filesystem) is designed to support sparse files, and Apple uses them extensively. In Finder, look at "size on disk" for the true space taken on macOS.

from orbstack.

In the future the data image size will also likely be reduced to match the size of your disk, rather than always being 8 TB.

from orbstack.

For those who want to workaround the issue with Backblaze — you can add .orbstack to the exclusion list.

@kdrag0n I'm curious if changing the file extension from data.img to data.sparseimage could change the behavior of Finder.

from orbstack.

In the next version:

• For new users: data image is excluded from Time Machine backups by default (and other backup apps that check the xattr)
• For old users:
  • If Time Machine is enabled: data image is NOT excluded by default, so existing working setups with APFS will continue to work
  • If not: since almost no third-party backup apps handled this correctly on APFS, data image IS excluded by default

Inclusion/exclusion can be adjusted at any time in Settings.

Imperfect solution, but this should solve most of the issues mentioned here. Sorry that it took so long and resulted in instances of data loss — there are a lot of potential solutions and non-solutions with different tradeoffs, but I think this is the best compromise for now.

from orbstack.

but I think this is the best compromise for now.

It was the best compromise in July 2023 when the problem and that solution became completely obvious.

from orbstack.

@kenn .sparseimage files aren't sparse at the filesystem level (it's a higher level sparse image format), so they don't really have a special meaning here. But it might make sense to use the extension anyway in order to be auto-excluded from Backblaze and other backup apps.

from orbstack.

I’m just going to chime in an additional postscript to my above comments.

@kdrag0n Recently I installed and started trying out orbstack on my mac. So far, I think it is an excellent alternative to Docker Desktop. Very useful features, much better performance. Great work.

An extremely unfortunate side effect of simply installing and running orbstack in my case was the deletion of EVERYTHING ON MY TIME MACHINE BACKUP DISK. That was historical backups for at least a year - probably a lot longer, I’m not actually sure. That historical data was valuable to me.

If something I built had a side-effect like that I’d be pretty horrified and I would 1. feel indebted to whoever was affected and 2. not sleep until I had shipped code that no longer had that side effect and did everything humanly possible to alert anyone who might have prior versions that their backup data could be at risk.

Last July, @bombich correctly (I strongly opine) noted the following:

The real solution is for backup apps to detect and skip holes, thus preserving sparsity like tar and Time Machine.

No. Speaking from the POV of the developer of backup software, I disagree that this is the "real" solution.

And gave you an extremely simple solution to this issue:

you can attach an extended attribute named "com.apple.metadata:com_apple_backup_excludeItem" to the file

I understand that if your Time Machine backup disk is APFS formatted then the large sparse image file is not a problem and I also understand that APFS formatting is a best practice for Time Machine backup disks. But HFS is still 1. working, 2. supported, and 3. common enough for those who have been using a mac for a bit.

Therefore, the right thing to do for you last July would have been to have orbstack generate that data.img file with the com.apple.metadata:com_apple_backup_excludeItem attribute so it would not BY DEFAULT do irreparable damage to backup data for people who happen to have HFS formatted backup disks and then to provide some documentation about the situation so that people could make whatever choices and carry out whatever remediations make the most sense in their case.

First do no harm. Then documentation.

Documentation about how to not lose data is not at all helpful after that data is gone.

from orbstack.

@kdrag0n Thanks for the hard work on this; while I wasn't affected by it, it doesn't go unnoticed to this (enterprise paying) customer that you're continually improving the product.

from orbstack.

FWIW, this 8.7 TB sparse file (~/.orbstack/data/data.img) that’s not really taking up that much space on disk caused Time Machine on my mac to delete a year’s worth of historical backups (gone forever) to try to make space on my 4 TB backup disk to back up the mac. Then after trying to back up to the now empty 4 TB backup disk, Time Machine puked and told me there was not enough room to back up my disk which has about 750 GB on it.

I think it would really be a good idea for orbstack to automatically exclude this sparse image from Time Machine backups by default.

from orbstack.

This is a real problem for anybody that uses HFS+ for their Time Machine drives. This wiped out six years of backup snapshots and completely obliterated Time Machine's ability to back my system up. I will have to reformat my Time Machine drive to get this working again. Surely excluding this by default is the right thing to do here.

from orbstack.

Thanks for the explanation. I noticed the file when I scanned my disk with Disk Map, which (mis)reported it as 8TB. I already reached out to the developer to properly handle sparse files.

Thinking about the issue a bit more, I am almost inclined to say that 8TB is better than the proposed solutions of matching either the disk size or the free space, because when you see 8TB file on a 2TB machine it's obvious that the file can't be taking 8TBs...

from orbstack.

If there's no big performance issue with it, I'd for sure like the option to limit the size. I don't want my Docker taking up that much space, so if it goes above my limit, I should clean stuff up

from orbstack.

If there's no big performance issue with it, I'd for sure like the option to limit the size. I don't want my Docker taking up that much space, so if it goes above my limit, I should clean stuff up

@probablykasper If you just want to limit the max disk size, there are better ways. Feel free to open a new feature request for that.

from orbstack.

That's good to know, @kdrag0n ! I reported this thread to the Backblaze team — hopefully they take note and address the problem in the near future.

from orbstack.

The real solution is for backup apps to detect and skip holes, thus preserving sparsity like tar and Time Machine.

No. Speaking from the POV of the developer of backup software, I disagree that this is the "real" solution. FWIW, my software can handle sparse files just fine; this only came to my attention because some people back up to NAS devices, and those don't support sparse files. This inevitably leads to either "why wasn't this file copied, can you not handle large files?" or "Why is my NAS reporting a full disk?", so eventually complaints about orbstack files show up on my Help Desk.

The sparse file size should be limited to the capacity of the disk that the file is stored on. It doesn't make any sense to make it larger than that, because the file will never be able to consume that much space. Arguably you might copy the file to a larger disk (e.g. Migration Assistant to a new Mac), but if you did, it's easy to just grow the file (e.g. truncate(fd, newLargerSize)). Making it ridiculously large is lazy and only begs for logistical issues. The performance of seeking and reading through such an enormous file also leads to slower backups. What is the gain in orbstack to use an 8TB file vs. a {disk capacity} file?

from orbstack.

Separately, I wondered about this previous comment:

For those who want to workaround the issue with Backblaze — you can add .orbstack to the exclusion list.

Should these sparse orbstack files be excluded from backups by default? IOW, do you as the developer recommend that people not back up these files? If that is the case, you can attach an extended attribute named "com.apple.metadata:com_apple_backup_excludeItem" to the file to indicate that the file doesn't need to be included in a backup.

from orbstack.

In the case of backing up to a NAS, I'm not sure if a mostly empty $DISK_SIZE TB file would be that much better than 8 TB, especially for people with 4/8 TB Macs. If it would, then I'd be happy to work on this sooner.

More often than not people have larger backup disks than source disks (because they want to retain backup history, and because rotational storage is cheaper). So in many cases we'll see a 1TB SSD source and a 2 or 4TB rotational backup disk. We always recommend using APFS for the destination, but it's not required and not always applicable. I have seen the occasional 8TB sources out there, but those are a lot less common.

from orbstack.

Adding this comment so that others might find this thread via search — this issue causes the Druva inSync client to lose its mind, as well (at least the GovCloud v7.1.0 client, which is what we run). As soon as I installed OrbStack, inSync started failing to complete a single backup, and reporting a total-amount-needing-to-be-backed-up in excess of 8 Tb. Of course, I didn't know it was correlated to OrbStack until the same thing started happening to a colleague of mine, and we figured out the correlation.

Interestingly, we could prevent our computers from sleeping and get inSync to think it completed the main part of the backup — but then it would sit at a "backing up live data" step (showing the OrbStack data.img file as what it was working on) basically forever, preventing final completion.

Excluding ~/.orbstack/data/ returned inSync to functional.

from orbstack.

other than that, there's com.apple.metadata:com_apple_backup_excludeItem extended attribute that TimeMachine, and few backup software respect too

https://eclecticlight.co/2017/12/13/xattr-com-apple-metadatacom_apple_backup_excludeitem-exclude-from-backups/

from orbstack.

@delfuego Hi, yes, thanks I read through the comments above and I understand. What my comment was meant to indicate was that it was an extremely unwelcome surprise to me to have lost a year of historical backups merely by having installed and started up orbstack.

That seems like a consequential and highly negative side effect which could probably be avoided by programmatically excluding the sparse image file from Time Machine backups on installation and then popping up an explanation to the user that includes the highly relevant information that if your Time Machine backups happen to be stored on an HFS volume that you should go ahead and leave that sparse image file excluded, but if your backup is to an APFS volume then you can safely un-exclude it if you want to back it up.

from orbstack.

From the FAQ:

Time Machine supports them, so your backups will not be affected.

If Time Machine only handles them under certain circumstances, perhaps this line needs to be updated.

from orbstack.

I also have an 8TB file in case that matters:

% ls -hla ~/.orbstack/data/data.img
-rw-r--r--  1 josediazgonzalez  staff   8.0T Mar 17 19:51 /Users/josediazgonzalez/.orbstack/data/data.img

from orbstack.

Thanks for the reply @kdrag0n. I admit I reported it as more of a documentation issue than a bug, but I figured it would make sense to report it as a bug. It seems like it might be a bug in Backblaze then in that case! I've added the folder to my Backblaze ignore list so it shouldn't report it again.

Thanks for adding it as documentation, I did check the docs beforehand but couldn't find anything, so I figured it was worth a report! 😄

Thanks again, Orbstack is so SO much better than docker desktop!

from orbstack.

Using large images is part of what makes OrbStack's seamless and dynamic disk management possible

If there was an option to set a custom sparse file size, how much of an impact would it have? Let's say I set it to 10GB and the size-on-disk is 3GB.

from orbstack.

Using large images is part of what makes OrbStack's seamless and dynamic disk management possible

If there was an option to set a custom sparse file size, how much of an impact would it have? Let's say I set it to 10GB and the size-on-disk is 3GB.

@probablykasper The total Docker and machine storage would be limited to the max file size. It's possible to allow setting a limit like this, but we're interested in avoiding it if at all possible.

from orbstack.

Describe the bug After installing orbstack on my macOS monterey, I created two guest virtual machines for testing, they were all very small. Later, when I make a Timemachine backup for my MBP, a few day later, I can't restore from the backup, I investigated the issue then I found that the Users/Myuser/.orbstack folder was a whopping 8TB in size(1.45GB on disk), which means I can't restore the user data from Time Machine because no MacBook Pro comes with an 8TB disk.

To Reproduce Steps to reproduce the behavior:

1. Insert my Time machine drive,
2. Open the folder of backup,then go to Users/Myname/
3. Right click .orgstack directory, then click 'Get info'
4. See the size '8.1TB'

Expected behavior Applications on macOS should handle hard links properly so that they do not abnormally inflate disk usage.

Screenshots

@wdde I see what you mean:

This is just an issue with Finder's free disk space check when copying files across volumes. You should be able to restore it by using tar (which is aware of sparse files) to copy the file:

tar cf - path/to/backup/data.img | tar -C dest/ xf -

More details in #29 (comment).

from orbstack.

Just hit this as it blew out my backups etc...

I have other sparse images that are fine - so I suspect it's being created with borked metadata.

.orbstack/data/data.img: DOS/MBR boot sector; partition 1 : ID=0xee, start-CHS (0x0,0,2), end-CHS (0x3ff,255,63), startsector 1, 4294967295 sectors, extended partition table (last)

from orbstack.

It's not related to metadata. OrbStack's data image is a sparse file at the file system level, which can improve performance significantly and is the only possible form of storage for the underlying VM (until macOS Sonoma). DMGs, sparsebundles, and sparseimages are not actually sparse at the file system level so they don't have this issue; they handle sparsity at a higher level instead. It also helps that backup apps often exclude those file extensions by default.

The real solution is for backup apps to detect and skip holes, thus preserving sparsity like tar and Time Machine.

from orbstack.

@bombich Thanks for chiming in and sorry if this caused trouble for you. You're right, there's no benefit to 8 TB compared to matching the size of the host filesystem it's on. This issue is still open because the plan is to implement resizing at some point. It's a bit more complicated than it appears, but definitely doable.

The main reason it hasn't been done yet is because I don't see how it would help that much in cases like this. The total apparent size of all files on the host filesystem will always exceed its capacity if there's a sparse file equal in size to the capacity, so it doesn't help much with size calculations. In the case of backing up to a NAS, I'm not sure if a mostly empty $DISK_SIZE TB file would be that much better than 8 TB, especially for people with 4/8 TB Macs. If it would, then I'd be happy to work on this sooner.

(And making it smaller than the host filesystem would have noticeable effects on UX, so I'd rather not go that route.)

Should these sparse orbstack files be excluded from backups by default? IOW, do you as the developer recommend that people not back up these files? If that is the case, you can attach an extended attribute named "com.apple.metadata:com_apple_backup_excludeItem" to the file to indicate that the file doesn't need to be included in a backup.

It depends. Some users wouldn't bat an eye if the data is gone, while others would definitely mind, and there's no way to determine that.

My suggestion up there was just a workaround for those affected by this issue, but it's also a perfectly good solution for users who don't care about the data.

from orbstack.

FWIW, I brought this up with Backblaze, and got the following:

Indeed, the "low on disk space" message would require that you have a temporary disk drive with at least 8.8TB of free space in order to complete the upload of your sparse image.

Backblaze CAN back up the sparse disk image when it's not in use, but it'll be the whole 8TB? What would happen if I tried to restore it, since I don't have an 8+ TB drive?

Yes, and you would need to restore the entire sparse image together. You can order it sent to you on a USB hard drive, though that actually has a maximum size of 7.5TB. Your only option for restore a bundle of that size would be to save it to a B2 snapshot and download a 8TB zip file, which would require over 16TB of free space on a single volume to unzip.

So while theoretically we can backup sparse image bundles, realistically there is no practical means of restoring sparse images over a certain size. Unfortunately Backblaze computer backup would not be a solution for your use case.

So my takeaway is that they can't backup just the actual data of a sparse image like Orbstack's, and won't be able to any time soon.

from orbstack.

For those who want to workaround the issue with Backblaze — you can add .orbstack to the exclusion list.

Came here due to Blackblaze prompt. Glad to see there's a solution. All my docker images are ephemeral; so, it's good to know there's a way to exclude its data in my backups! I've added the full .orbstack directory to my excludes list.

Should these sparse orbstack files be excluded from backups by default?

@bombich IMO, yes. In this instance, I think Orbstack is too new for Backblaze to see and exclude it by default.

from orbstack.

All my docker images are ephemeral

I don't think it's just the Docker images, IIUC, but the volumes as well. If you have any non-trivial data setup (like I do), you may want to back it up. Assuming OrbStack volumes should be excluded from backups by default is mistaken, I think. Definitely not a universal opinion, anyway.

from orbstack.

If you have any non-trivial data setup (like I do), you may want to back it up.

What's the reason behind not using a persistent volume? Even Docker volumes are meant to be ephemeral.

from orbstack.

What's the reason behind not using a persistent volume? Even Docker volumes are meant to be ephemeral.

Because it wasn't the default in the code I inherited, it was convenient not to set it up, and while suboptimal, that would have worked just fine for my purposes. Or rather, it did until I learned that it wouldn't be backed up.

So, now I have a problem: do I exclude it from backups, or take the time to learn the ins and outs of Docker volumes, Mariadb backups, and whatever else it will take to set this up properly? (The data is a hassle to replicate, but not mission-critical, either, so it's not a clearcut decision.)

Mind you, I have no idea how long it will take to wrangle docker/orbstack, since it's new to me; I could easily imagine spending an afternoon or two learning and doing all this, which I'd rather spend on more important issues.

For now, I'm going to bet that my estimate of time lost replicating the data (an hour or two × p(corrupted volume)) is less than fiddling with Docker, so I've added .orbstack to the ignore list.

from orbstack.

macOS uses these "magic"names to exclude from various operations (https://michaelbach.de/2019/03/19/MacOS-nosync-noindex-nobackup.html) if you name the file/directory with .nobackup it is supposed to be picked up automatically by many backup software too

from orbstack.

@krzyzanowskim I'm not sure that there's any consistency to other software picking up that hint; I can only really find one reference online to any other backup software that respects it. Given that Time Machine already is doing the right thing with this sparse file (e.g., it doesn't get confused by its theoretical maximum size), I'm not sure that .nobackup would help deal with the other apps' inability to handle it.

from orbstack.

This is not something that needs to be fixed per-app but at the source. A lot of backup systems are unhappy with this file.

from orbstack.

Not sure what "source" you're referring to, @mfouesneau — if it's OrbStack, they're using sparse disk image files exactly to spec and as they're supposed to be used. If an app (like Druva InSync, in my case) is having an issue with them, that's a bug in those apps, not in what OrbStack is doing.

We certainly filed a bug report with Druva InSync (we fortunately have a very large enterprise contract with them) to fix their handling of the file; if you are seeing issues with other apps that are mishandling large sparse disk image files, then it's worth you filing a bug report with those apps' vendors. In the end, that actually solves the problem "at the source", since when a given backup app fixes its buggy handling of sparse disk image files, it resolves issues not just with OrbStack's file, but with sparse disk image files created or managed by any and all other apps as well.

from orbstack.

@delfuego, exactly my issue with Druva. I realize my message was not clear, sorry.

from orbstack.

hi,
i ran into a similar issue, which might be connected.
just after i installed orbstack and started using it, my time maschine backup
kept running longer than usual. i noticed that it deleted all of the previous backups
to gain free disk space.
when there was no older backup folder left to delete (but enough space for a whole backup of the
computer disk), the backup failed at about 87%, saying it does not have enough space.

How can i check if this is related to the new swap data img file?

In the time maschine options i cannot ignore the orbstack data.img file.

thanks for helping.

from orbstack.

I just ran into the exact same issue as @hhentschel. All my old Time Machine backups got wiped and then the new backup failed at 70%. Ignored the .orbstack directory in Time Machine preferences, and then the backup completed quickly and successfully. It seems something is off with this sparse disk image file setup, at least with Time Machine on MacOS Ventura.

from orbstack.

Yes, i am also on MacOS Ventura (13.2.1).
I disconnected the external drive, erased all the options in the time maschine settings, plugged in the external drive again. after that the back-up works like before.

from orbstack.

@jalendport @hhentschel Maybe it's because you have/had old backup drives formatted as HFS+ instead of APFS?

from orbstack.

In my case, yes. But so far everything is running ok.

from orbstack.

@jalendport @hhentschel Maybe it's because you have/had old backup drives formatted as HFS+ instead of APFS?

@kdrag0n ah, yes my drive isn't APFS formatted. Wiping the whole thing now (since I've already lost all my backups 🤦‍♂️ ) and trying again with APFS as the new format. Thanks!

from orbstack.

Just as a follow up info.
i did not change the formatting, as it started working fine at one point.
but now only for about a week, now it starts deleting older versions again,
probably the free disk space has gone now under 800gb.

from orbstack.

Unfortunately the feature request was closed two weeks ago: #242.

I'm running in the issue, that this file is physically blocking half of my hard disk for no reason. I don't have much images on my system, but it still blocking 123G of my disk space.

Usually I wouldn't care, but the Mac has 8G free space left and this results in stuttering mouse pointer and some apps are crazy slow. (It's a combination of virtual monitor with display link manager and insufficient disk space).

I would appreciate to reopen the feature request and implement an option, which also limit the file size of that sparse file.

from orbstack.

@sebb-m0 What you're seeing is not related to this issue. I've replied in #242.

from orbstack.

Is ~/Orbstack a recent addition? I had already excluded ~/.orbstack from Backblaze, but today I got a macOS notice that I was running out of disk space.

After checking, I finally found that Backblaze had created 143 GB of bzfilelist log files in the last couple days. I scanned them, and it's all errors and warnings about stuff under /Users/matthew/OrbStack/docker/containers/

I don't really care who's at fault, I just want a workable backup system. Does anyone know of a whole disk backup service that won't choke on Orbstack?

from orbstack.

Does anyone know of a whole disk backup service that won't choke on Orbstack?

I don't normally plug my own software in forums, but since you asked – CCC can handle these files just fine. Be sure to back up to an APFS-formatted destination, otherwise the sparsefiles get inflated to their logical size.

from orbstack.

Well, hold a sec — this is now two separate issues.

~/.orbstack/ is the Orbstack config and metadata directory, and the sparse disk image file that's the primary subject of this thread lives here.

~/Orbstack/ is the directory that was added in v1.2.0 back in December 2023, and surfaces the native file system(s) of any containers that are running; it's described in the docs here and was announced here.

There's no magic here; if you don't want your backup system to include these files, then you just have to set it to exclude that directory.

from orbstack.

@delfuego

Knowing how to make Backblaze exclude these dirs is not the underlying problem. It's trivial to exclude once you know about it.

But twice now I've been surprised by a Backblaze/Orbstack interaction and had various things blow up. This last time nearly crashed my computer for lack of disk space, and only fast clean-up saved it.

Why is this issue still open? Is it an issue that Orbstack can fix, or is it Backblaze's fault, and thus should be closed (and maybe pinned for others) as WONTFIX?

(I like using Orbstack more than I like Backblaze, but I haven't switched off BB yet because I thought maybe this would get addressed.)

from orbstack.

@pjv See the comments above in the thread — Time Machine natively supports this file just fine, but depends on your backup medium being formatted with APFS to fully do the right thing.

from orbstack.

@kdrag0n Docker Desktop has a simple, easy-to-use slider to set the max size of the VM disk. The only complication is that it has to erase the disk to make it smaller. Otherwise, it is a very straightforward GUI interface, easy to use, easy to understand. And it completely solves the backup problem, first by adding another GUI item, a checkbox to exclude the VM from backups (or not) via the com.apple.metadata:com_apple_backup_excludeItem attribute, and second by limiting it to a practical size so that it can be backed up to non-APFS file systems.

If you want Orbstack to be a Docker Desktop replacement, these are 2 very easy-to-implement features you want to include for feature parity.

from orbstack.

FYI rsync --sparse does not handle this file correctly. This is more of an informational message for other folks who are curious about how rsync will handle this file (and other sparse files), rather than a complaint that orbstack's data image is somehow breaking rsync.

This seems unintuitive me, and I haven't dug into why rsync --sparse wouldn't handle this file as expected, since --sparse seems to be exactly the flag you'd expect to use to handle this kind of file. 🤷‍♂️

EDIT: I did some more digging I think I was wrong that rsync --sparse was not working as expected. I was using rsync --inline --sparse without --whole-file which can have different effects depending on the environment. See man 1 rsync --sparse section for more details.

from orbstack.

In case anyone's curious, the iDrive backup service can't handle this either. It attempted to backup all 8TB. Carbonite said they don't support the use case, but didn't say whether they could technically handle it or not. As mentioned above, Backblaze not only fails, it fills up the entire disk with warning logs, potentially crashing the OS. Still waiting on a response from tarsnap. (EDIT: See below.)

If anyone knows of an online backup service that's compatible with Orbstack, please chime in.

from orbstack.

Heard back from Colin of tarsnap:

The storage usage will be efficient since all the implicit zeroes will be deduplicated down to a single block of zeroes; but tarsnap doesn't currently make use of the "SEEK_HOLE" extension so it will try to "read" the entire image, including file holes.

He didn't elaborate on what "reading" the whole disk image would entail, but I doubt it's good.

from orbstack.

There is an easy tool, Asimov, to exclude directories and files from a Time Machine backup. I've added .orbstack/data to my local configuration.

Sadly I I encountered the same issue as @pjv, all my old backups are gone except for the last.

from orbstack.

Time Machine does not seem to allow you to exclude ~/Orbstack.

> sudo tmutil addexclusion -v /Users/jscheel/OrbStack
/Users/jscheel/OrbStack: The operation couldn’t be completed. Invalid argument

But it seems to already exclude it anyways:

> sudo tmutil isexcluded /Users/jscheel/OrbStack
[Excluded]    /System/Volumes/Data/Users/jscheel/OrbStack

from orbstack.