Re: Beowulf gone?
Am Montag, 25. Juni 2018, 10:13:21 CEST schrieb Mike Castle:
thank you very much for this very informative response. So it looks like, I
have to search for a solution which will exactly fit my needs.
The primary goal or interest will be, to render videos (I say some kind of
"Micro-Pixar-Studio" :) ), as I was aked for this by some friend.
All responses her in the debian grpup showed me, that things are not as easy
as I imagined and there is no siomple amnd easy solution. That is no problem,
as I am willing to learn and I alrerady found several documentations, whixch I
will read the next time.
For now, thank you all for yourt hints and informations, and I believe, for
now this thing can be marked as solved.
Best regards and have always fun!
> Down to it's basic, rendering videos is nothing more than a simple
> map-reduce, partioning a workload in a bunch of identical bits of
> processing. That could be done with N machines and a few simple shell
> scripts. Not really any need for anything fancy. What the fancier
> software gives you is stuff like automatic retries, fault tolerance,
> fancy ways of tracking the process of a job, and so on. But they
> likely require more management to maintain.
> Other reasons for a cluster might be web serving. You want to be able
> to handle some amount of queries per second with a reasonable latency
> for most of those queries. That may require you to scale in different
> ways. You need more machines to handle serving up the web content, a
> bit more to handle the various types of processing that needs to be
> done, some sort of scaling solution for your data store. Still
> parallelism, but each thing happening in parallel is likely a
> different type of task. A simple thing like Puppet or Chef could help
> you keep your machines in order.
> Maybe you don't need the computational power of a bunch of CPUs, but
> you need all of the memory for something? Maybe you need a bunch of
> machines with 128G of ram to look like a single machine with 1P of
> ram. So you need to have something specialized for moving bits around
> efficiently (either code to the data, or data to the code).
> Maybe you want to have different types of pipelines that do completely
> different things, and you want to make efficient use of the machinery
> you have. Say, you're doing both of the above, and during some hours
> of the day, you need more processing power to go towards web serving,
> you take away resources from you map-reduce. During off hours, you
> can let the MR have more processing power. Now you're dynamically
> trading off resources, and this is where some sort of clustering
> management might become useful.
> There is not likely to be any one solution for everyone. So no longer
> having anything exactly like Beowulf or some sort of direct
> replacement is not suprising. Likely stuff was learned from Beowulf
> and friends. Some things worked well, some things no, some things
> were never used, some things were needed. Folks take that experience
> and build new tools and systems. Old ones languish. And no direct
> replacement exists.
> Likely you need to break down what you really want to do, then look to
> see what solutions might work best for you, and experiment with the
> various ones, and see which one you like best.