think2019

this is perhaps the core of organization's journey to AI or even to a successful Data Lake, Data Science

In area of Data Science, typically we include a Data Engineer who work side by side with Data Scientist and are critical to take findings and put into Catalog as well as provide key features needed to modeling phase.

there is an excellent session at THINK, hosted by Jay @jaylimburn - https://myibm.ibm.com/events/think/all-sessions/se...

it’s mainly done at LOB level in most of the companies I have worked with in advisory capacity. Central tools, policies and procedures need to be built for data governance. I believe the WHAT of data cleansing and cataloging must stay with LOB and HOW with IT.

as usual, there are multiple solutions too handle this, but ICP for Data is a platform that includes and enforces these capabilities by default .. Learn more @ this THINK session : https://myibm.ibm.com/events/think/all-sessions/se...

I have yet to see a organization that has this process streamlined. Most established companies have many, many meetings about how data set is going to be used internally and the logistics around it.

This an important requirement in the maturing of AI/advanced analytics. Solutions should support distributed and multi-cloud data, and ideally support orchestration and optimization of moving code to data or vice versa.

You need a combination of a cross frictional team, the right access to data and tools to build your AI foundation.

some organizations refer to this as Data Preparation or Data Curation..

one of the advantages of building cutting edge tech and creating new data products/services is that this is dealt with further down the line

Besides Profiling, cleansing, cataloging, Data classification is another critical attribute. Here is where Ml automation can go a long way. IBM Information Server provides complete solution

Model bias is something we are very focused on, especially from a dev ops perspective. Understanding this is important and critical to your organizations future as you incorporate key decisions using AI. So Trust AI but verify :)

Would agree if you think about IoT use cases with massive volumes of data points continuously produced. Aggregation and storage can happen at the edge. It's not just data warehousing though.

It's not clear to me how you can argue that DWs are a good fit with edge computing. DW is the heart of data governance, which tends to require centralized data storage/control. Please clarify.

E. g. an Edge Computing Device can outsource simple computing tasks to a cloud using a Function-As-A-Service concept. Here, the cloud does not store anything and no backend is set up on it. The cloud only offers computing power for any functions that are transmitted on the fly

Clean data in ---> great ML and AI; not clean data in --> lots of cleanup. Just say no to data pollution!!

Yes .. here is a 3rd party listing of vendors offering cleansing tools : https://www.analyticsindiamag.com/10-best-data-cle...

Yes we in IBM have good solutions around Data Quality, ML and rule enabled. This is very critical part of Tusting your data

Here's a good session at THINK, in case you are interested: https://myibm.ibm.com/events/think/all-sessions/se...

I see it as a never-ending journey. One is never done. There is a legacy to begin with, but every moment, new data (sources) may get added to your current landscape. Fun!

Here's what Scott Hebner, IBM, had to say recently about applying comprehensive policies, compliance, and protections to data: Three critical steps in making your data *#AI ready*

Data privacy regulations are coming whether we like them or not. GDPR is already here, CCPA is coming soon. Enterprises, small and large, have to starting thinking about the data being collected and shared.

@jameskobielus @SLHebner +1

#analytics systems typically have several layers of protection and compliance regimes. accessing the platform is at a system level whereas anonymizing data depends on the data set (as well any contracts associated with it)

Early days for establishing compliance and protection policies. It will probably need a company to have a Wall Street Journal disaster to focus minds on this issue!

To ensure unbiased AI models, policies on data analytics are more important than ever.

very little to be honest & I think this is huge issue given increased and diverse regulations , GDPR being the latest

How important are good policies if their ratification is not automated? Deep integration across the analytics 'stack' can solve for that

I would love to see poll on this one. Every CDO would want to say YES to this. Need ML/AI based solutions to automate these activities, and we in IBM analytics have solutions to make this EASY (a hard problem)

thee's always a policy, but the restrictions depend on the purpose associated with how the data is being used. When my #datascience team built models for negotiation purposes, even our internal status reports listed our work as confidential.

Policy driven will be a very important portion of a machine driven future. Getting policy down and having machines figure out new policies on the fly address both on demand AI and real time AI

ML and AI are next frontiers in Data Governance Platforms and these models will work in conjunction with policies! So it’s “policy driven ML enabled” approach which seems most practical with the tools we have today!

Here's what Daniel Hernandez, IBM VP, had to say recently about data virtualization on theCUBE:  https://video.cube365.net/c/911319 

One of the biggest I've seen is that companies think they are behind vs other companies.. What companies need to understand is that its a journey and they just need to start. most companies are learning and growing in this space.

it's not a silver bullet and it's only as good as the people, clean data (as input), and tools. Another issue is the notion that it's easy. It's hard

I recommend, pick the one use case, put small team on the project and start. If it fails, that is normal. just goto next one and for the wins, it will cancel out many failed projects.

@CAppugliese Fully agree, Carlos. One should have the end in mind. But getting started is the first - and most important - step. Then 'hack' yourself fast forward ;-)

Thanks, John.. do you see organizations putting emphasis on clean data?