RealDataStories

#ProductionReadyOpenStack
Join us to discuss use cases, considerations for product environments & things that are coming up
   9 years ago
#realdatastoriesInfrastructure at ScaleWe'll be talking about Infrastructure at Scale and the recent HP Apollo announcement.
   9 years ago
#RealDataStoriesInfrastructure at Scale HP & Scality discuss choices for building production-ready Web-scale applications & services
Leo Leung
http://www.via-cc.at... - Next topic - Availability

Leo Leung
Is there such a thing as downtime or maintenance windows anymore?
HPE Cloud
It would seem that data storage with no regular access needed would have a different model than needing instant access
Leo Leung
We think the answer is a firm "no"
Joseph B George (JBG)
I think the expectation is "all the time" - one of the expected values of cloud
Leo Leung
@hphelioncloud - that's the thing - people will access at any time - and they have no tolerance
Joseph B George (JBG)
in fact, we expect the cloud SLA to remain constant no matter what the network traffic
Leo Leung
"Wow, I'm happy to wait ten seconds for this Kanye video," said no one ever
Pete Brey
That goes back to the earlier conversation - you can abstract away the maintenance in the cloud and make it appear available. So even though a hard drive might fail, it won't impact the end user.
HPE Cloud
Is data redundancy the key to HA or is there another way?
Leo Leung
@cloudstorageguy - exactly - that is the actual infrastructure doing the work
Leo Leung
@hphelioncloud - that's part of it - another is "routing" to the data
Nicolas Trangez
In the long run I believe we'll require a notion of 'manageable inconsistencies' (relaxed consistency) for file-like storage similar to how it's already handled for database-style storage (eventual consistency and such)
Leo Leung
Going back to @eikke point - implementations need to route to the data around failures as well
HPE Cloud
Are SSD the key over disk drives or are they just a higher cost?
Leo Leung
@eikke - I'm waiting for a reworked CAP theorem Venn diagram...
Pete Brey
Totally agree @jbgeorge - people expect a certain level of availability - doesn't matter if a failure happens or if the system is under extreme demand.
Leo Leung
@hphelioncloud - SSD's can help with latency, but again you're looking at tradeoffs
Pete Brey
At the end of the day, does it really just boil down to a tradeoff between cost and availability?
HPE Cloud
What are those tradeoffs?
Leo Leung
.@scality uses SSD's for lookups, but HDD's for capacity
Joseph B George (JBG)
that's why your infrastructure should be able to support various HDD formats - SAS, SATA, SSD
Nicolas Trangez
Indeed, it's all about reducing the number of seeks and optimize read-ahead/caching.
Joseph B George (JBG)
its not just a cost or preference decision - different drive types facilitate different uses within object storage design
Joseph B George (JBG)
in most traditional cases, it is a cost vs availability discussion - HOWEVER, if your architecture is more purpose built for cloud storage / object storage, costs can be more effective
Pete Brey
What's even more interesting is building a tiered system: SSD for really fast access to hot data, SAS drives for middle-of-the-road access, SATA for less expensive storage, and even disk-to-disk and tape for cost optimized longer-term retention!
Leo Leung
@jbgeorge This really calls for a picture, and a blog
Joseph B George (JBG)
totally agree - let's get on it
Leo Leung
For HA, we also believe in software decoupled from hardware
Leo Leung
We have customers that have run five generations of HP hardware at the same time in the same system, with no down time.
Leo Leung
About to cut over to last topic - simplicity
Leo Leung
http://www.via-cc.at... - First topic - Scalability

Leo Leung
I think @cloudstorageguy is making up brontobyte
Leo Leung
Mega-huge byte
Pete Brey
Bigger than you can imagine
HPE Cloud
I assume that both scale and access are an issue with this type of massive data storage?
John Furrier
reliability is key first and then scale as upside for deployments
Leo Leung
They are separate issues
Pete Brey
@hphelioncloud Scale and access do go hand in hand. Building a brontobyte sized storage system requires new approaches. The old ways of doing things just don't work.
Leo Leung
Scale from capacity, number of objects, number of nodes perspective
Leo Leung
Access from a number of requests, parallelism, performance perspective
HPE Cloud
@cloudstorageguy At brontobyte level, is time to access the issue or finding the data?
Pete Brey
The bigger issue to achieving scalability is the ease of deployment and the ease of ongoing maintenance.
Leo Leung
Most systems cannot do both @hphelioncloud
Leo Leung
Most systems cannot even do all of the attributes within one of the categories
HPE Cloud
So, does the #OpenStack Swift object storage design solve the long term massive storage need?
Leo Leung
Talked to a bunch of prospects dealing with uncompressed 4K video yesterday - any guesses how big that is?
Leo Leung
Answer: 49MB per frame, 60 frames per second
Joseph B George (JBG)
This is where we need to be looking at emerging tech or new paradigms for design - most std tech was not designed for this type of scale, esp at the infra level
Pete Brey
Doing things at scale magnifies the challenges with traditional approaches to file storage. Simple operations can become time consuming when you're dealing with petabytes (and brontobytes) of storage. Object storage addresses these issues, and more
Leo Leung
ESG just did a study that object storage is still commonly misunderstood
HPE Cloud
What do people think it is?
Pete Brey
@hphelioncloud At brontobyte level, time to access can be a challenge with traditional file systems, But the real challenge is how you keep something that large up and running.
Joseph B George (JBG)
people understand what "cloud storage" is a bit better
Leo Leung
we @scality think of it as both an underlying architecture and an interface
HPE Cloud
@jbgeorge Thus, the confusion is how you actually do that. We need to educate better
Leo Leung
object scales better as an underlying architecture - that's indisputable now
Leo Leung
AWS has over 4 trillion objects in S3
Leo Leung
Azure Storage has over 40 trillion objects
Leo Leung
both are object based architectures
HPE Cloud
That means clouds are adding hundreds of millions of objects a day. That is true scale
Joseph B George (JBG)
agree @hphelioncloud - when we talk to customers, it usually starts with file and block to set context, then move to object, then to how it can be used
Leo Leung
The other aspect is interface
Jeff Denworth
Azure also hosts MSFT apps - like Hotmail (a lot of objects) - so not totally apples to apples.
Leo Leung
@JeffDenworth sure thing - point was just scale
Leo Leung
Good morning btw
Nicolas Trangez
@hphelioncloud It's important to make the distinction between Swift-the-API, which imposes no scaliablity limits inherently, and Swift-the-implementation, which can bump into scaling limitations.
Leo Leung
absolutely @eikke - that's where things get interesting
HPE Cloud
@eikke Agreed, I think having various implementations offer customers choice
Leo Leung
Hence the topic - what do you need to be production ready
Pete Brey
Good point @eikke - at the end of the day all systems are composed of components which will have inherent limitations. So even though the API may provide scalability, the underlying infrastructure may limit.
Joseph B George (JBG)
what are we going to agree on as "production ready" for this conversation?
Leo Leung
Ability to scale capacity and access certainly
Joseph B George (JBG)
that's what is interesting - the notion that infrastructure is less important in cloud is not accurate - the value is DIFFERENT than traditional arch, but measured differently in cloud arch
Leo Leung
Lots of lessons in the field on that one
Leo Leung
For example - can you handle a lot of churn in your infrastructure
Leo Leung
Tons of adds, deletes, etc.
Leo Leung
You don't usually account for things like that early on.
Leo Leung
Another example, how do you handle peak load - another thing that's hard to design for in the first pass
Leo Leung
Third - what happens when the cloud grows under the covers? You can hide that to a certain extent as a service, but not as a private deployment.
Leo Leung
@jbgeorge - love the comment about how infrastructure actually works - no matter what the "face" is
Pete Brey
It's a fair statement that you can abstract away some of the complexities with the cloud. But under the covers, it is being dealt with.