Torrenting isn’t as big deal as it once used to be. Many vendors provide their big file downloads via Torrents. Why? I thought only pirates operate the Torrent seas? Not true. Torrents are great for downloads because they have the best performance & they have pretty much become a norm (you can download ubuntu from ubuntu.com via torrents). The more people that try to download the file, then eventually the faster everyone will download the file (because more end points will have the pieces of the file to download.) This is backwards of none P2P downloading which with more downloading users, the slower the connection.
The problem with torrenting and COW filesystems (BTRFS, ZFS), copy on write, filesystems is that torrenting picks random chunks of the file and writes it. This random write behaviour causes fragmentation (especially if you have other writes and processes going on at the same time). There is 1 thing you can do to help that:
1. On utorrent select: Preallocate Files (you will find it in the settings). That basically defines the files start and end point, so there is no room for fragmentation.
Bonus thing to do: On BTRFS systems
Disable COW on the folder that holds your downloads. That way any files written there are written just like legacy filesystems, so fragmentation wont be an issue.
chattr -R +C /home/torrentuser/Downloads
Above assuming that the /home/torrentuser/Downloads folder is where the torrents download.
Pre-allocating does not help at all on COW filesystems. Although the pre-allocated file(s) are in a more or less (best effort) sequential order on disk, every write that will follow ‘inside’ that file will occupy a new sector. That is just the property of the COW system, to prevent data loss and a guarantee a ‘previous state’ of the file in case of an error or power loss.
But indeed using chattr +C on BTRFS will remove the COW property, so I’d say it is ‘mandatory’ to do so on your download folder when running BTRFS.
I stumbled upon your post because I have used BTRFS and ZFS extensively, but my system is still quite sluggish when donwloading to my COW-disabled BTRFS filesystem 🙂 so I was looking for some insights on that and your site came up first. It feels like pre-allocation does not really work on BTFS somehow, or it is just very efficient. Allocating a torrent of 50 GB is done in seconds which is far above the transfer speed of the disk, but maybe allocating with zeroes can be done more efficiently in the filesystem allocation table.
Interesting to know. Thanks for insight.
I constantly run the “mostfragged.sh” (look for it in my articles) script on my storage system, to monitor fragmentation. My fragmentation isnt too bad (but then again I dont have numbers to compare with). My storage system is btrfs, running samba, so I map the share to my PC, and Windows Utorrent writes to it. To see how bad my system is fragmented. Check it out let me know what kind of fragmentation you get. I made the script spit out some good juicy information.
Most of my files have between 1 to 100 extents/fragments. Ive seen systems with 1000s (which is semi bad), but even worse, 100,000 of extents on files (lun files that were huge).
Here is mostfragged ran on my system:
# cat MF-EXTRA--_data--STATS-of-all-fragmented-files-more-than-1-extent.txtStats of fragmentation (Everything is average & totaled per file)
Average Number of Extents: 29.1196 (less is best)
Average Filesize: 2.19999e+08 Bytes, 214843 KiB, 209.807 MiB, 0.20489 GiB, 0.000200088 TiB
Average Extents Per Byte: 1.32362e-07 (this value does not mean much, bigger is better)
Average Size of an Extent: 7.55502e+06 Bytes, 7377.95 KiB, 7.20503 MiB, 0.00703616 GiB, 6.87125e-06 TiB (bigger is better)
Total Number of Files: 8580
Total Size of files: 1.88759e+12 Bytes, 1.84335e+09 KiB, 1.80015e+06 MiB, 1757.96 GiB, 1.71675 TiB
Total Number of Extents: 249846 (less is best)
# cat MF-EXTRA--_data--STATS-of-all-files.txtStats of fragmentation (Everything is average & totaled per file)
Average Number of Extents: 5.64756 (less is best)
Average Filesize: 3.96871e+07 Bytes, 38756.9 KiB, 37.8485 MiB, 0.0369615 GiB, 3.60952e-05 TiB
Average Extents Per Byte: 1.42302e-07 (this value does not mean much, bigger is better)
Average Size of an Extent: 7.0273e+06 Bytes, 6862.6 KiB, 6.70175 MiB, 0.00654468 GiB, 6.39129e-06 TiB (bigger is better)
Total Number of Files: 51892
Total Size of files: 2.05944e+12 Bytes, 2.01117e+09 KiB, 1.96404e+06 MiB, 1918 GiB, 1.87305 TiB
Total Number of Extents: 293063 (less is best)
I already ran uncow.py on my system before and I ran a defrag constantly (well once a week). I bet if I ran uncow.py it will make it all better.
So for your system I recommend “Btrfs defrag” followed by uncow.py on the whole thing (just make sure you have enough freespace). Let me know how your numbers do. Oh and make sure you have abackup before you do anything.
LINK1: http://ram.kossboss.com/uncow/
LINK2: http://ram.kossboss.com/mostfragged/
So I was looking for a way to re-enable cow on existing files. After a bit of Googling, I guess I can’t avoid copying the entire file. However, after copying the file to a directory with cow enabled, I’m still not sure whether I can replace No_COW version atomically. Using mv to overwrite it is fairly quick. So I’m just using it for the time being.