Web lists-archives.com

Re: Review Request 129703: [baloo_file_extractor] Limit CPU usage




This is an automatically generated e-mail. To reply, visit: https://git.reviewboard.kde.org/r/129703/

On January 2nd, 2017, 10:51 p.m. UTC, Albert Astals Cid wrote:

Without knowing anything about baloo this looks totally wrong

QList<KFileMetaData::Extractor*> exList = m_extractorCollection.fetchExtractors(mimetype);

why would not you want to iterate over all the iterators that support a given mimetype?

On January 3rd, 2017, 5:01 a.m. UTC, Anthony Fieroni wrote:

It's a waste of time. Extractor should store file content in DB for fast access when file content search is performed, so if more than one extractor performs a file it will result in high cpu usage and huge transaction size in DB, basically file content * num of extractors, at least we loose time and disk size for nothing.

Do you have some numbers as a result of profiling? Have you checked that the existing extractors are in fact redundant? Is the order of their presence in the returned list of extractors deterministic and is the most specific one returned first?

One small example, there is a generic plantext extractor which returns a number of lines in any file with the text/* MIME type. Your patch changes that.


- Jan


On January 3rd, 2017, 11:43 a.m. UTC, Anthony Fieroni wrote:

Review request for Baloo, Boudhayan Gupta, Pinak Ahuja, and Vishesh Handa.
By Anthony Fieroni.

Updated Jan. 3, 2017, 11:43 a.m.

Repository: baloo

Description

Processing large directories, +5000 files, can be CPU eater. Large file, itself, can be another issue.

Diffs

  • src/file/extractor/app.cpp (97332469)
  • src/tools/balooctl/indexer.cpp (45e42c1c)

View Diff