LocalData

Right after your maintenance contract is due

Stood up a 400 node Isilon system. "It can be done"

when the back end interconnect has no port capacity

without being to product specific, there was a "scale-out" architecture (RAIN) that maxed at 16 nodes. Just a hard stop...

I'm a consultant, they don't call me to fix the scale out system when it's working right.

@jhughes #nope #hardpass

@andriven hahaha, always thought that architecture was a bit of joke

It's usually when the backplane gets saturated, so the system can't stay in sync anymore.

@jhughes Just to be clear, it didn't "scale", but it stood up

@jhughes And you backed away very slowly, both hands in clear view.

Network almost always defines the limits. There's a reason Google brags about their CLOS setup

We ran out of API calls per second on GCS - that was fun.

That moment when you realize that east-coast-2 is actually a finite resource.

@dinisco would usually see this with Infiniband before 10 GigE based architectures. Of course Infiniband could usually drive lower latency more predictably.

Any architecture that requires a locking mechanism or uses a one of more master nodes will have scaling limits.

@kenhuiny Excellent point - scalable locking mechanisms as well as masterless architecture are huge.

Something about split-brain.

customer want to know

node fails, service still online

Are there any software limits to the architecture.

I don't think it's a useful question. I would rather deal with technical features than buzzword compliance.

are there hard maximums which require major equipment upgrades? In simpler terms, no forklift upgrades.

When I can power on a new node and it auto joins a cluster

How they answer what the limit is

Ok Chris: @fdmts What technologies are driving the change to scale out

Is it a consistent namespace across all nodes or components in the architecture.

@andriven All systems have limits. When a vendor says "no limits" I hear "we didn't scale test it hard enough."

a 4 node cluster provides exactly 4X the performance of a single node

in defense of non-scaleout, if you can predict your growth and environment, a non-scaleout architecture might make the most sense financially. I'd argue that almost no companies can accurately predict that in a 3 year window though.

I think we handled the drivers to scale out in an earlier question. I mean ... containerization seems like a biggie.

we argued about this for years - just because something can scale, doesn't mean that it really is scale-out. Need to understand the limitations. Of course, not everything must be infinitely scalable to be useful.

@fdmts partly agree - sometimes I'll distinguish between theoretical limits (i.e. our filesystem can grow to 32 exabytes) and what they've tested/QA-ed especially.

I like where Chris @fdmts is going with this..what triggers the BS detector when vendors talk "scale out"

@fdmts agree about limits, but solid tech and qa rigor should lead to limits that 99% of customers can't reach

@dinisco #IBreakThings

It's also about minimum size and the linearity from there. I think if you have lots of upfront requirements to achieve scale, you are simply planning for capacity growth, not scaling out.

Non-scale out typically requires you to provision capacity ahead of when you actually need it which becomes very expensive.

good question, plenty of BS, anytime there's talk of a masters or slaves, any node more important than another means not true scale out

@dinisco More seriously, you never remove a bottleneck, you just move it around. Recall the happy days when the whole industry freaked out about how RAID rebuilds on 1TB disks would take longer than MTBF on another disk in the set.

workloads that are very latency sensitive simply don't need it, the cost of coordination between nodes isn't worth it

Adding the same hardware to a solution without being able to use it. Think DataDomain. You would buy a solution for the capacity you needed then when you needed more capacity you would have to buy a whole new DataDomain and use it as a new target

Most of EMCs current products.

Adding more disk shelves.

My network closet.

Oracle Databases... Most businesses simply throw TB's of RAM at them to make them run better and avoid licensing charges.

Legacy batch HPC approaches.

Anything where you say "I loved my first; I hated my ninth" - e.g., NetApp, Data Domain, etc.

Truly, most cloud services are not transparently scaleable. You still need to provision individual servers and services incrementally.

@dinisco going back to my EMC days, I do remember candid latency comparisons between Isilon, VNX, and VMAX with some solid technical explanation about the latency involved in any scale-out architecture.

the need for a large namespace warrants it, but there are cases where isolated islands of data make sense

Sounds like the argument about what is truely hyper-converged. :)

It's not scale out if your workload is has to be sharded and mapped to specific nodes so that each time a node fails or is added, you have to manually remap everything.

It's not true scale-out if adding nodes does not provide linear or near linear increase in performance.