SparkFree

Please post your replies here

Waiting for your replies .... let us get this chat rolling!

hybrid cloud is when an entity has some data services hosted on premises and others hosted outside their company. The key is to have them work seamlessly together (e.g., one data view) in your analytics efforts.

A hybrid cloud data services architecture involves deploying fit-to-purpose data repositories and analytics platforms in various roles in a unified architecture.

Hybrid cloud data services leverage the resources of the private and public cloud to deliver services to the enterprise

A tiered data environment which bridges 2 services like #Hadoop and #Spark and can bridge public/private clouds too

A hybrid cloud data services architecture may use a Hadoop platform for info refinery, a Spark platform for exploration, NoSQL for mobile/IoT data analytics, streaming analytics platform for low-latency, etc.

Hybrid #cloud data services architecture encompasses a number of delivery options that match the needs of many today.

A hybrid cloud data services architecture may also combine public and private clouds into a unified environment.

Put another way: HCDSA is more inclusive vs exclusive.

Nowadays, 'hybrid' is the way to go with technology - from delivery to software stack

Essentially, a cloud data services architecture is hybrid when it has two or more zones, each with distinct roles within a common services model, but each zone is a different platform or is under different admin.

Time for question #2 - at the top of your screen!

Q1 Essentially, a mix of in-house, private cloud and a third party cloud provider...

Please post your replies here

Looking forward to your views on question #2 please @bobehayes @jameskobielus @jf

Benefits are inclusiveness, openness, more data, perhaps even better models (because you're using more data/metrics).

Theability to mix and match per use case, persist data, and support scaling are what come to mind for me.

The chief apps of a hybrid cloud data services architecture include processing/analyzing structured & unstructured data; supporting both batch and real-time data analytics in parallel; supporting both production and dev ops in parallel.

Great replies, requesting the participants to chime in with their views please

Since performance is always on my mind, the big benefit is right-sizing your app: match the app resource needs to the right service that can provide it

an important application would be one that allows you to integrate disparate data sources.

Benefits of hybrid cloud data services architecture include the flexibility to encompass diverse apps/data and leverage existing data/analytics investments while evolving to support new requirements.

Multi-tier data warehousing, Data Lakes and a Data hubs also come to mind as I think about it more.

@bobehayes Hey Bob, do you mind logging in from twitter? That way I can RT your replies through our handle?

The opportunity to provide data security and privacy by a delineation of what is stored and processed in the 3rd party cloud and what is kept in house

I did log in using Twitter. Let me try again.

One of the use cases is IoT where you may need batch processing to build models on historical data, on-premise service to capture and process the streaming data, and cloud services to store the data for reporting

I hope it works this time.

Time for question #3 a the top of your screen

To me, it would be more flexibility while the kinks are being worked out

Please post your replies here for question #4

We are half way through... please post your replies here :)

@furrier @bobehayes @tcrawford @davidtc

@DrZeydy

Deploy your data lake and data-science exploration platform in a fully managed and secured cloud data services environment.

First: Don't inhibit your thinking w/ traditional architectures. New methods and architectures are needed.

Second: There is not a prescriptive, one-size-fits-all approach. Understand the requirements well.

Realize a lot of data when you start out will be full of junk.

the questions is how do you fill the data lake at scale

Realize that moving from traditional DW to a Data Lake will require a lot of rethinking and breaking of old habits.

Use Spark for your adv analytics/data science dev tool and Hadoop/NoSQL as your data lakes platform(s). Make sure they share cloud-based bidirectional source connectors and data integration services to support flexible integration patterns.

@furrier but you don't want to fill the lake with bad data. You need to come up with a plan to ensure the right data is streaming into said data lake

@furrier Right! How you fill the lake, with what, how...and which lake.

@davidtc Old habits die hard. :)

@davidtc excellent point how to you scale meta data to manage the ability to pull data out of the lake to be used in value creation activities

@jameskobielus While I love #ApacheSpark it's not the right fit for some use cases and we all need to keep that in mind. They are making massive strides but one tool will not fit all

On that note - time for question #5 at the top of your screen

Make sure that the advanced analytics development team--i..e, data scientists--and data lake team--i.e, data engineers--share a common development environment (e.g, DSX), collab/workflow, and learning/community resources.

.@jameskobielus A common dev env for advanced analytics is very useful for enabling the collaboration needed