RealDataStories

Infrastructure at Scale
HP & Scality discuss choices for building production-ready Web-scale applications & services
   4 years ago
#RealDataStories#ProductionReadyOpenStackJoin us to discuss use cases, considerations for product environments & things that are coming up
Leo Leung
Q3: Can object storage systems be used for web-scale applications and services?
Pete Brey
Yes! In fact, they were designed with that specific use case in mind. That is one of the reasons a RESTful interface is so important.
Leo Leung
@furrier absolutely - this was the starting point
Lacee
Absolutely! We had a great HP/Scality webinar yesterday discussing the benefits of our joint solution specifically for web and cloud providers #realdatastories
Leo Leung
What people don't realize is that more and more apps fit the model.
John Furrier
yes it is for storing lots of data then layers of extraction are being built on top of it with traditional tools and visualization; things like SQL won't be going away
John Furrier
. @lacee_mcgee can you share the link to the webinar
Leo Leung
@furrier - yes and no - you can build apps in a way where transactions are not "locked" like in a relational database. More architects are building that way.
Leo Leung
Distributed/object systems help with capacity scale, performance scale, multi-site fault tolerance, multi-site data availability
Leo Leung
For Web apps, it's crazy to see the layering that has to happen when folks start with NAS / RAID / replication / protocol translation / etc.
Pete Brey
Most web applications may still use a tiered architecture - block for "hot" data for fast access and low latency, and scale out object for "warm" or "cold" data.
John Furrier
. @lleung what's ironic is that today we are doing a resizing of our elasticsearch clusters as we get "data full" this is the new normal
John Furrier
Q2: What problems does object storage solve that SAN and NAS don’t?
Leo Leung
like @chrismevans said a week ago in https://www.crowdcha... - a bit of it is architectural
Lacee
It allows customers to achieve a much more cost effective scalable solution using industry standard servers
Leo Leung
there's local/low latency (block) and distributed/high scale (object). To us, file is becoming a different organization scheme for high scale.
Pete Brey
Scale! Sorry, broken record... :)
Pete Brey
There are several sub-bullets to scale, though. There is just the ability to build a system large enough, the feasibility of building such a system. Then there is the affordability of building such a system.
Leo Leung
the cloud "fellas" were the first to prove the need for a massive "bucket" and they wrote their own (S3, GFS, Haystack)
John Furrier
Chris is awesome on that chat. thx for linking to that. scale is it and price is great bc value shifts and this enables #bigdata analytics
John Furrier
at #hadoopsummit the #1 think is the impact of scale for storage enabling value extraction from the data - object is it in this value proposition #nobrainer
Leo Leung
eliminating capacity limitations and changing the economics certainly changes the perspective on data value and analytics
John Furrier
Leo gr8 point my followup is: Can object storage systems be used for web-scale applications and services?
Leo Leung
and the cloud guys are openly posing the question, "what if storage were free" with their actions
Pete Brey
good point - in many ways object is becoming an "on-ramp" for big data applications
John Furrier
. @lleung at #hadoopsummit the question is the data should be free many see scale as table stakes but the bottleneck is data locked in which sucks for app devs
John Furrier
@cloudstorageguy totally and it's huge for Internet of Things #IoT last week at Mongoworld they are trying to pivot to #iot
John Furrier
Mongo groping for #IoT shows the old way is shifting fast
John Furrier
Q10: Is object storage an architecture or a type of product?
Pete Brey
Architecture, it's a way of doing storage that is simpler and more cost effective.
Leo Leung
Architecture. More and more SaaS and other storage products are backend-ed with an object architecture.
Leo Leung
Just like there is really no "flash market," I also don't think there is an "object" market.
Pete Brey
Totally agree @lleung - I don't think of "object" as a market. It's a technology or an approach, a way to solve the problem.
Mike Onsing
I think it's a product but it enables a set of new solutions so it's a tough call.
John Furrier
Q1: With the prevalence of block and file storage systems, why do we need object storage?
Leo Leung
a question i've been hearing for ten years. ;)
Scality
To us, it's more about offering a bigger horizon when it comes to capacity-driven capabilities
John Furrier
@lleung block is good for performance to make remote think its connected and object is for all the data in #bigdata
John Furrier
object store has no bottlenecks for #bigdata and is perfect for any big data app and most important for #IoT Internet of Things
Leo Leung
let's break it down - we believe the vectors are latency v. bandwidth / low capacity v. high capacity
Pete Brey
it's all about scale...when you're building petabyte size storage you have to do things very differently
Scality
- another way to think about it is local and low latency vs. distributed and high scale/bandwidth
Scality
@scality there will always be a need for low latency and "near" the application - we see a parallel need in unlimited scale
Pete Brey
block is good for structured data, especially where performance and latency are critical. file is good for smaller amounts of unstructured data, but classical file systems break down quickly with large volumes of data.
Scality
@cloudstorageguy agree - object is a great architecture for eliminating limitations in file counts, and applying additional context
John Furrier
Q11: What is the connection with cloud (or enteprise) app developers with Infrastructure at Scale? What should be the top 3 conversations?
Leo Leung
- "What is the app trying to solve?" - when the goal is set, take a look around at what is available now in terms of languages, frameworks, and infrastructure - there are better ways to accomplish the goal now than even five years ago.
Mike Onsing
conversation is how to eliminate the need to configure and tune infrastructure to meet the needs of the app and workload
Mike Onsing
I like what was said app developers don't want to write code that cost them more money that could be handled by someone else
Leo Leung
As always "What is the low hanging fruit?" - conversation with an enterprise yesterday indicated that more than 50% of internal apps had a GUI and were web-based - easy pickings for modern approaches
Pete Brey
How fast can you develop your app?
Leo Leung
@mikeonsing - i like that - infrastructure should "just work" and not need a whole lot of tuning. An architect said to me recently that "infrastructure shouldn't need to know what the app needs, but should just work."
John Furrier
Q4: Why don’t object storage systems use traditional RAID technology for data protection?
Leo Leung
too limited to a specific stripe set, typically limited to a server, rebuild times can be days for multi-terabyte drives - the last one is killer for system availability
Mike Onsing
isn't the object store not a file system so doesn't raid not apply? Am I missing something
Leo Leung
in #realdatastories - two drive failures can bring a whole site down
Leo Leung
@mikeonsing - basically yes, though some object stores are built on top of file systems, so that used to actually happen. i don't think anyone is that crazy anymore. ;)
Leo Leung
Need... ability... to... reply... with... pictures...
Pete Brey
RAID was great for small drives, but with modern drive sizes, it takes way too long to rebuild a RAID set. The problem with this is it leaves you exposed to another drive failure (which could be catastrophic) during the rebuild.