Web lists-archives.com

Re: looking for a file format for time series data other software can easily consume




On Mon, Oct 30, 2017 at 11:33:36PM -0700, Dan Hitt wrote:
I plan to produce some time series data and i would like it to be in a
form that makes it easy or effortless for pre-existing programs to
plot or even process further.

The data would consist of a few hundred to a few thousand time points,
and for each time point there would be a half-dozen to a dozen values.

So conceptually it would be something like a csv file with a few
hundred to a few thousand rows, and a half-dozen to a dozen columns.

However, a csv file doesn't seem exactly right, as i would like the
files to include some embedded metadata (ideally in some form that at
least in principle could be digested by another program).  I suppose i
could have some comment lines somehow, but that feels pretty ad hoc.
But maybe i'm being overly fastidious.

Something like a RIFF file format (like WAVE) might be better, but not
sure what it would be.

For the plotting or processing software i'd want to use something like
octave or python/matplotlib or a derivative.

I googled around for time series file formats, and i didn't see
exactly what i was looking for.

So i'd appreciate any advice or suggestions, and thanks in advance!

The lowest-common-denominator here is usually RRD [1], the Round-Robin Database. This will give you a fixed-size file containing various "buckets" for data at specific intervals (for example every 5 minutes for a day, every hour for a month and every day for a year). Lots of programs can read and write RRD because it's been stable for ages.

If you want something a bit more new/capable/interesting, there's whisper [2], the TSDB for carbon/graphite. Or OpenTSDB [3], or InfluxDB [4] or Prometheus [5].


[1] https://oss.oetiker.ch/rrdtool/index.en.html
[2] http://graphite.wikidot.com/whisper
[3] http://opentsdb.net
[4] https://www.influxdata.com
[5] https://prometheus.io


dan


--
For more information, please reread.

Attachment: signature.asc
Description: PGP signature