Web lists-archives.com

Re: Review Request 129703: [baloo_file_extractor] Limit CPU usage

This is an automatically generated e-mail. To reply, visit: https://git.reviewboard.kde.org/r/129703/

On January 2nd, 2017, 10:51 p.m. UTC, Albert Astals Cid wrote:

Without knowing anything about baloo this looks totally wrong

QList<KFileMetaData::Extractor*> exList = m_extractorCollection.fetchExtractors(mimetype);

why would not you want to iterate over all the iterators that support a given mimetype?

On January 3rd, 2017, 5:01 a.m. UTC, Anthony Fieroni wrote:

It's a waste of time. Extractor should store file content in DB for fast access when file content search is performed, so if more than one extractor performs a file it will result in high cpu usage and huge transaction size in DB, basically file content * num of extractors, at least we loose time and disk size for nothing.

On January 3rd, 2017, 11:46 a.m. UTC, Jan Kundrát wrote:

Do you have some numbers as a result of profiling? Have you checked that the existing extractors are in fact redundant? Is the order of their presence in the returned list of extractors deterministic and is the most specific one returned first?

One small example, there is a generic plantext extractor which returns a number of lines in any file with the text/* MIME type. Your patch changes that.

On January 3rd, 2017, 11:58 a.m. UTC, Anthony Fieroni wrote:

  1. No
  2. Yes
  3. No About me it's better to make some flag, or whatever, to indicate a parser has done his work and we can safety stop iteration. At least this patch tries to reduce CPU usage, it's not a panacea
  1. You claim it is useful to reduce CPU usage, but fail to provide any data points.
  2. Please provide a list of redundant extractors
  3. An extractor knows if itself has extracted any data, it can not know if a different extractor may find any data. Extractors may be orthogonal and provide different data.

- Stefan

On January 3rd, 2017, 11:43 a.m. UTC, Anthony Fieroni wrote:

Review request for Baloo, Boudhayan Gupta, Pinak Ahuja, and Vishesh Handa.
By Anthony Fieroni.

Updated Jan. 3, 2017, 11:43 a.m.

Repository: baloo


Processing large directories, +5000 files, can be CPU eater. Large file, itself, can be another issue.


  • src/file/extractor/app.cpp (97332469)
  • src/tools/balooctl/indexer.cpp (45e42c1c)

View Diff