LocalData

Significant engineering + QA investment.

Have to understand what the end-user is looking to "scale" There's trade-offs, nothing is free

Fundamentals, and a system architecture clear and clean enough that the whole team can understand it.

Anticipating and removing all logical limitations and/or bottlenecks in your design

Lots of cloud instances on which to test your scale-out design. Can't afford to strap together than much hardware in a data center.

@patrickrogersca I would build on this to say - an engineering culture that embraces finding the next bottleneck as a win.

Scale-out generally requires loosely-coupled, distributed architectures.

Often a custom file system, metadata/catalog work, locking mechanisms, some ACID characteristics, easy growth, masterless architecture, etc.

followup question what is the mix of hardware vs software? thoughs color commentary?

@fdmts #agree

this is why I'm skeptical by default if I see an existing product or company start to claim scale-out. Really doing is major under the covers work.

98% software

patience and emphasis rock solid internode communication, write plans, etc. before going after checkbox features that customers want to talk about

IMO, it should be all software for it to be successful. The hardware should be irrelevant.

tangent comment: #blockchain cyptocurrencies are showing both decentralized & distributed models to be very interesting.. thoughts

I'd call scale-out an interesting inflection point where products with older "design centers" will be overtaken overall by products that are built with scale out in their "design center".

I think that it starts with the software architecture - the algorithms - and that the architecture imposes constraints on the hardware. As Rickover said, "The devil is in the details, but so is salvation."

I'm not throwing shade on older products but overhauling an existing product to make it scale-out is HARD. Not impossible but seriously hard.

Agree. Blockchain potential goes way beyond cryptocurrency!

love @jhughes comment 98% software that makes HW vendors 2%ers :-)

Heavily, heavily software. You can't move fast enough with ASIC and/or hardware design to adapt.

@johna_white Disagree. Hardware is where the bytes live. I've seen too many designed that assume that a cow is a unit sphere filled with milk.

Oh God, Blockchain came up. Everybody DRINK.

@fdmts if #bitcoin drink twice

@fdmts Here you go then. Scale out = An innovative, game changing, and leading solution that leverages actionable synergies across unique, optimized paradigm shifts to deliver bottom line results that you can double click on.

@fdmts Yeah I think you are missing my point. I shouldn't have to buy specialty hardware to make a scale-out solution work. Commodity stuff all day.

Everyone now wins buzzword bingo. ;)

At the compute and storage layer, it's mostly software. You can build some massive scale out systems using standard white boxes. For networking, there are benefits to using custom ASICS to minimize latencies.

@fdmts Drinking is in order. Cryptocurrency is a distributed transaction model, not a distributed infrastructure model. 😀

Software has to assume there will always be hardware limitations. The challenge is living within the confines of the hardware units.

@johna_white Fair enough. I've seen good solutions tip over when the sales team agreed to try to run them on my legacy hardware.

So if a vendor tries to sell you on custom gear for scale out storage, run the other way. :)

@steve_pao We could use the BitCoin chain for file level locking. 10 minute lags FTW.

Clustering adds fault tolerance scale out does not necessarily add ft

I think that most scale out systems are "clustered" in one sense of the word. The distinction lies in scale out's ability to expand capacity in a balanced way without introducing bottlenecks.

cluster = quorum and more active/passive. Scale out provides all nodes to perform a function.

clustering often refers to a dual node system, though not always, in a sense, most traditional scale up storage architecture are actually 2 clustered controllers

Clustering is way to achieve scale-out.

begs the question "how do clusters scale out" :-)

historically clustering was about resiliency and removing single points of failure. Let's go back to Novell Cluster Services for instance. Scale-out is focused at very different thigns.

clustering usually means tightly coupled. all the resources are pooled together. scale-out many times just means dividing up the work

@csharney Clustering doesn't necessarily add fault tolerance. It's a big place.

Clustering may not add performance or capacity

some companies have sought to merge those though - aka NetApp with scaling out dual controller clusters for their Data Fabric. I'd argue that's not scale-out per se.

I don't view clustering as a way to increase horsepower just resiliency. Scale out is all about adding linear available resources.

@johna_white true that clusters require quorum, but most scale out systems do as well

It seems like some of us are using the word "clustering" to mean "High Availability Pairs." That's pretty old school, isn't it?

@johna_white definitely agree. clustering helps survive, not increase capacity

Clustering is a tech that can potentially be used for scaling out but it depends on the architecture. For example a master/slave architecture cannot scale out in the same way as a shared-nothing architecture.

@dinisco sure, a new age quorum typically exists across all nodes now... not just one place.

Clustering focus = keep my services online. Scale-out = grow forever.

Often, clustering can be used for scale-out, but also clustering might restrict scale-out to achieve redundancy. So, the terms are somewhat orthogonal.

@fdmts I think of a cluster as a logical collection of resources performing the same task...versus a set of discreet resources allocated to different things

@fdmts I'll agree that's how I think of it. In some cases cluster capabilities could be a subset of scale-out capabilities.

Didn't Isilon invent "scale out" ? :-)

pretty sure that was Al Gore

remember the vax cluster in the early 80's...that's old school

Yup. And they spent a decade explaining what the hell it meant too.

I know I always said scale-out to describe Isilon when at an EMC partner. People seemed to get it and loved the theory.

One example is Oracle RAC. High-availability without much scale.

@dvellante DecNet was a great net protocol stack killed SNA but then TCP took over #OpenWins

Isilon scales but also hits the wall at 144 nodes, so not limitless.

@andriven and @fdmts, agree with both, think one fundamental difference between cluster and scale out is that clusters can have passive resources, scale out generally has all active resources

@dvellante VMS (DEC's) OS is what microsoft copied with Windows NT (WNT) one letter after VMS

Spinnaker :-)

@andriven Something about terms like Kleenex and Post-it.

@dinisco Good distinction - in scale-out often all resources can be used and during failure scenarios the total resources available diminishes rather than inherent performance decrease.

love that story...Dave Cutler