Web lists-archives.com

Re: Beowulf gone?

Am Montag, 25. Juni 2018, 10:13:21 CEST schrieb Mike Castle:
Hi Mike, 

thank you very much for this very informative response. So it looks like, I 
have to search for a solution which will exactly fit my needs. 

The primary goal or interest will be, to render videos (I say some kind of 
"Micro-Pixar-Studio" :)   ), as I was aked for this by some friend.

All responses her in the debian grpup showed me, that things are not as easy 
as I imagined and there is no siomple amnd easy solution.  That is no problem, 
as I am willing to learn and I alrerady found several documentations, whixch I 
will read the next time.

For now, thank you all for yourt hints and informations, and I believe, for 
now this thing can be marked as solved.

Best regards and have always fun!

> Down to it's basic, rendering videos is nothing more than a simple
> map-reduce, partioning a workload in a bunch of identical bits of
> processing.  That could be done with N machines and a few simple shell
> scripts.  Not really any need for anything fancy.  What the fancier
> software gives you is stuff like automatic retries, fault tolerance,
> fancy ways of tracking the process of a job, and so on.  But they
> likely require more management to maintain.
> Other reasons for a cluster might be web serving.  You want to be able
> to handle some amount of queries per second with a reasonable latency
> for most of those queries.  That may require you to scale in different
> ways.  You need more machines to handle serving up the web content, a
> bit more to handle the various types of processing that needs to be
> done, some sort of scaling solution for your data store.  Still
> parallelism, but each thing happening in parallel is likely a
> different type of task.  A simple thing like Puppet or Chef could help
> you keep your machines in order.
> Maybe you don't need the computational power of a bunch of CPUs, but
> you need all of the memory for something?  Maybe you need a bunch of
> machines with 128G of ram to look like a single machine with 1P of
> ram.  So you need to have something specialized for moving bits around
> efficiently (either code to the data, or data to the code).
> Maybe you want to have different types of pipelines that do completely
> different things, and you want to make efficient use of the machinery
> you have.  Say, you're doing both of the above, and during some hours
> of the day, you need more processing power to go towards web serving,
> you take away resources from you map-reduce.  During off hours, you
> can let the MR have more processing power.  Now you're dynamically
> trading off resources, and this is where some sort of clustering
> management might become useful.
> There is not likely to be any one solution for everyone.  So no longer
> having anything exactly like Beowulf or some sort of direct
> replacement is not suprising.  Likely stuff was learned from Beowulf
> and friends.  Some things worked well, some things no, some things
> were never used, some things were needed.  Folks take that experience
> and build new tools and systems.  Old ones languish.  And no direct
> replacement exists.
> Likely you need to break down what you really want to do, then look to
> see what solutions might work best for you, and experiment with the
> various ones, and see which one you like best.
> mrc