Web lists-archives.com

Re: Experiences with BTRFS -- is it mature enough for enterprise use?




Hi Rick,

On Tue, Dec 26, 2017 at 11:37:32AM -0800, Rick Thomas wrote:
> Is btrfs mature enough to use in enterprise applications?

Not in my opinion. I've dabbled with it at home and based on those
experiences I will not be using it professionally any time soon.

> If you are using it, I’d like to hear from you about your experiences — good or bad.

During the time of Debian wheezy I made use of btrfs for my home
fileserver, which is an HP Microserver with 4x 3.5" SATA drives and
an 8 bay disk chassis with 6 more 3.5" SATA HDDs in it, connected by
eSATA. It had previously been using LVM on top of Linux MD without
issue, but I'd become mindful of the amount of storage that was
being used without consistency checks (except for a weekly MD
scrub).

In order to do this I required a backports kernel and btrfs-tools
from git. I went for one of the more simple btrfs configurations
which is a raid1.

Over the next few years I didn't lose any data, but I did
experience:

- Out of space errors even when there was plenty of space

- Filesystems that went read-only on a device failure even though
  there was enough redundancy

- Filesystems that couldn't be remounted read-write after failed
  device replacement, even though the hardware is hot-swap, due to
  bugs in btrfs which required a kernel upgrade to fix (therefore a
  reboot, despite having the redundancy otherwise).

In summary, btrfs lowered the availability of the system due to
being buggy even in a relatively unexciting configuration. I could
not recommend it for serious use yet.

I am sure there will be plenty of people who've used it for years
without experiencing any issue. I am still subscribed to the btrfs
mailing list though, and unfortunately I still see people on there
reporting serious issues including data loss.

> My proposed application is for a small community radio station
> music library. We currently have about 5TB of data in a RAID10
> using four 3TB drives, with ext4 over the RAID.  So we’re about
> 75% full, growing at the rate of about 1TB/year, so we’ll run out
> of space by the end of 2018.

A year to solve this problem is nice to have, though. :)

> I’m proposing to go to three 6TB drives in a btrfs/RAID5 configuration.

If I were you I'd think very very carefully before using RAID-5 for
anything, btrfs or not.

- Four spindles to three means reduced performance.

- RAID-5 means parity means reduced performance.

- Lose one device and you're operating on just two spindles and
  recalculating parity. It will run like a dog and any further error
  means data loss, and array failure, which can be stressful and
  nerve-wracking to repair.

- I wouldn't like to chance my arm finding previously-unknown bad
  areas on two 6TB devices. When you have a three device RAID-5 and
  one device dies, you REQUIRE every sector on both the other
  devices to be readable in order to reconstruct the data onto the
  new device.

Personally I would not risk 3x6TB in any kind of RAID-5. HDDs are
pretty cheap so I really have to question the wisdom of cutting down
the number of spindles this far, and then using RAID-5.

RAID-10: okay, it "wastes" the most capacity in exchange for better
write performance. If this is a media library I get that maybe you
don't need the write performance (have you benchmarked your current
system??). RAID-10 is what you use when you can afford it, but if
you can't then compromises have to be made. In that case consider 4
or more spindles RAID-6. At least you stand a chance of being able
to replace a failed device before coming across a bad area on
another device.

Spread the extra cost of whatever you need to make it to four
devices in RAID-6 across the expected lifetime of your system (~6
years?) and does it still seem too much to pay?

Finally, the parity RAID levels in btrfs are newer than RAID-1 and
-10 and have seen a lot more bugs. Including really bad data loss
bugs.

One look at https://btrfs.wiki.kernel.org/index.php/RAID56 should be
enough.

> Would I be safer with ext4 over RAID5?

It's a bit of a frying pan / ground zero nuclear blast situation,
really.

Cheers,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

"I remember the first time I made love.  Perhaps it was not love exactly but I
 made it and it still works." — The League Against Tedium