DataAutomated

A Smart Data Lifecycle can massively reduce the cost of performing high quality data analytics. You can get to a single source of the truth much more easily, you'll have less duplicate data and you can deploy self-serve analytics to business users with confidence.

@dvella 60-70% of a Data Scientists efforts are often spent wrangling data rather than developing revenue generating algorithms, Io-Tahoe flips this on its head automating the discovery process, increasing the accuracy value of data, giving scientists and users time to make money

Data Analysts and Data Scientists will spend less time wrangling data and more time on value add analysis and building advanced models. You'll have less, better reporting - no longer multiple slightly different copies of the same report!

The Smart Data Lifecycle helps organisations understand their data; it’s key to driving analytics, making sure the datasets are optimal, delivering cost efficiency, improved time to insight and ultimately competitive differentiation.

How many times do you see businesses buy the next greatest MI tool only to be let down by the underlying data?!

is this also about recognizing that not all data is the same or has the same value? Moving the data that you need to the right platform to execute workloads is part of working smarter with data.

so true Yusef, and with skills shortage the last thing you want is wasting time of a top resource

Consolidation of data into a single platform where analytics can be run, or at least provisioned from, reduces cost significantly

By reducing the Total Cost of Ownership cross all elements – through automation reduce the amount of time Data Scientists need to spend chasing and understanding data.

By reducing infrastructure and application silos

By reducing the copies of data within, and across infrastructure and application silos

The need for automation and using ML algos across the data lifecycle is a must with the ever growing data challenges and massive data growth an enterprise organization faces doing business in the age of IoT and the growing need to migrate to the cloud to manage data te

Data integration, harmonization, customization and self-service are key to meet diverse requirements which otherwise impossible to perform manually.

Having a smart data lifecycle approach end to end, will enable organizations to quickly adapt and gain insights into their data assets and realize value much more quickly.

Smart Data Lifecycle – leverages all the metadata related to the data, in order to perform simple repetitive tasks that contribute to reduced fragmentation, . . . .

redundancy, duplicates, anomalies and also to improve data quality, data governance. So it's this valuable metadata that is used to 'inject' automation on these repetitive
tasks, and serves all Data Consumers to improve business agility and performance.

(edited)

I'm going to come back to reduction in end-to-end cycle times - creating an efficient data factory that works for the entire organization, not just one department or division

The data life cycle by itself is a sequence of stages that a particular unit of data goes through from its initial generation by Data Producers through to serving Data Consumers.

Otherwise it's duplication, duplication, duplication all the way!

@ajay_vohora The 'secret sauce' is employing these statistics to drive automation of repetitive tasks upon the underlying data files and records. . . . .

@ajay_vohora These repetitive tasks are commonly indexing, ML based classification, pattern recognition, anomaly detection, data quality scoring, sensitive data tagging etc.

(edited)

Isn't everyone tied of capturing it all in Excel? :)

"Death by Excel"

Because it’s a key area of competitive advantage (or disadvantage). A smarter approach to data directly and significantly reduces cost and risk whilst improving agility and creating an environment for innovation.

Tech and business are now inextricably linked; those that don’t recognise this and leverage the power of their data are going to be left behind...

And don't forget democratization. give more people more data with more tools to work the data, and authority to do something about it...... opens up innovation

*CMDB

Because it's difficult, too much friction.

There's definitely a manual overhead to be considered in trying to tie together the valuable data across different systems and application siloes

@tom_poole Sometimes the effort required to address the challenges seems so daunting it's easy to just do the basic 'superficial' things and keep going on the same old way.

Yes, and tagging/classifying can be limited to the perspective of the classifier/tagger. I.E., a silo'd view

@tom_poole Indeed. Breaking down the silo is letting the other businesses see what data you already have... maybe they can leverage it or supplement it

It certainly is an exciting journey. I learn something every day about the challenges various businesses face and its exciting to try and solve these challenges

@tom_poole agreed it is a difficult process but its something that should be done.....slowly. With increased compliance requirements & ever increasing security threats, its a choice between doing it now in your own time or rushing to do it when its most likely too late!

So many IT teams are having to do-more-with-less... #DataAutomation is no longer a want, it's a need.

There is more pressure now than ever to put data to work to help drive the customer experience.

(edited)

automation has become a mandate - what used to scare people is now compulsory - digital transformation is largely about leveraging data - no digital, no business in this COVID era

@ajay_vohora The demand and the need for working smarter with data is being driven by the customer, customer behaviours and expectations.

The pandemic has been a forcing function for change. We need to share knowledge faster and smarter among a more disparate group of poeple.

We've entered a major thrift economy now more than ever before and need to to keep the lights on, need to do more for less in triple quick time to keep things going. Automating data is a must now to ensure we make the most of the dollars available.

Hadn't heard that term "thrift economy" - definitely have to do more with less, but isn't this SOP for IT?

What are the "highest priorities" that you see with customers since mid march 2020 ? "House on Fire" concerns that need to be addresses ASAP ?

Smart Data is data from which metadata i.e. signals and patterns have been DISCOVERED (i.e. generated and extracted) and performed by software functions and algorithms.

Smart Data can be acted upon and sent to a downstream for further data consolidation and analysis.

@ajay_vohora and what's different today is this metadata generation and classification can be automated fully

Smart data is about data being "addressable" for both infrastructure and applications. To me lots of devops principles like horizontally scalable and vertically integrated architecture is key to success

By the way - we're not talking about scanning tables and columns to find table and column names - that's a very superficial technique and has limited value to being able to solve challenging problems to do with understanding the underlying data values, content, context

. . . . .and semantics e.g. reading the the actual email addresses, zip codes, addresses it where valuable statistics can be generated to then apply ML methods to these stats.

The rise of Smart Data has come from the challenges thrown up by “Big Data” i.e. massive data sets where automation married with metadata enhances the ability to perform data processing, advanced analytics or Machine Learning at scale.

optimized placement based on need/outcome/workload and at a reasonable approved cost.

I think Smart Data is an automatic capture and assignment of meaning, location, importance and movement associated to data that would not otherwise be found in metadata or people knowledge alone.

"Smart data" is obviously a broad term. Its more baout being smart about your data. We invest heavily in IT infrastructure management, when the real value is in our data. We need to understand what we have and how we can drive value from it.

A Smart Data Lifecycle prioritises simplicity, flexibility and efficiency whilst improving data visibility

@ajay_vohora > In some cases, the metadata becomes more valuable than the actual data

@JeffFrick and sometimes more voluminous!