Re: Fuzzing KDE software
- Date: Wed, 06 Mar 2019 20:35:44 +0100
- From: Albert Astals Cid <aacid@xxxxxxx>
- Subject: Re: Fuzzing KDE software
El dimecres, 6 de març de 2019, a les 20:20:03 CET, alcinos va escriure:
> As some of you might have heard, we have been rewriting significantly some
> key components of Kdenlive over the past couple years, in an effort to
> improve stability, testability, and separation of concerns between the view
> and the model.
> As part of this effort, on top of traditional unit-tests (none existed
> before the rewrite), I have been looking at fuzzing as a way to push the
> code to its limit. For those who are not familiar with it, fuzzing is a
> testing technique that essentially consists in generating a lot of random
> inputs and feeding them to a target program in an attempt to generate
> unintended behaviours (memoryleak, crash,...). This technique has an
> excellent track record of finding bugs and vulnerabilites, even in well
> tested softwares (see eg. https://llvm.org/docs/LibFuzzer.html#trophies)
> In the case of Kdenlive, we generate a lot of operations that the user
> could execute on the timeline, and simulate them in the model, to check for
> robustness over unexpected action sequences.
> Note that this kind of testing doesn't explicitly test for correctness (it
> doesn't check whether a clip is correctly moved after a move operation for
> example), but try to get the software to crash, if there is a way to make
> that happen. Also, we have some internal checking mechanisms that
> periodically ensure that the model is in a sane state, and the invariants
> are respected. So if things go wrong with respect to that, it will also be
> caught by the fuzzer.
> This approach has already led to the discovery of several bugs in kdenlive,
> although the parts covered by the fuzzer are still fairly limited.
> Are any other projects in KDE using that technique? In order for it to be
> useful, it pretty much needs to be run continuously (I've been running it a
> bit on my laptop, but it's not very practical). Some companies offer such
> service for free for OSS, under some conditions: see for eg:
> https://github.com/google/oss-fuzz and https://fuzzbuzz.io/pricing,
> although I'm not sure whether Kdenlive would meet the requirements for
> either of those. Is there any way we could do this inside the KDE
> infrastructure instead?
We (KDE Security team, but here basically just me) are running kcodecs and kimageformats in oss-fuzz.
I can help you with the setup if you want.
> Ideally, this are the requirements for such a
> - Continuous running
> - Git hook to get the latest code
> - Automatic bug deduplication and reporting
> - (optional) Coverage report to ensure the fuzzing is effective.
> Google open-sourced most of its tooling for fuzzing at scale on a cluster,
> so there is no need to start from scratch.
> What are your thoughts?
> Best regards,
> Nicolas Carion