EntArch_VS_BigData

Big Data promises to address the same areas: cost reduction, new opportunities, building unique capabilities which are difficult to repeat for competitors. So, there are should be some natural intersection of applying EA and Big Data

And from current interviews of #Hadoop and #Spark practitioners, we can clearly see that they talk about development to production time reduction, finding right messages to individuals (not market segments!), building processes involving #ML and #Streaming

Another popular item is simplifying and democratizing access to data, which is core element of Enterprise Architecture. Architecture Development Method (#ADM) is based on Business, Information and Technology architectural roadmaps.

#BigData as truly interruptive technology is promising to change all three architecture layers. Business is going to generate value new ways, available info will explode, technology will accommodate new platforms and ways to connect elements together.

So, why to put #EA vs #BigData instead #EA and #BigData? Problem is current vision of #BigData solutions and maturity of #EA in companies. Some way they live in parallel worlds and compete for the same resources.

Normal #EA development cycle takes around 5 years, which means that every company project follow roadmap leading to cost reduction, getting new opportunities or building unique capabilities.

#EA approach requires to follow company and industry standards, reuse architectural building blocks in Business, Information and Technology Architecture. And it requires to follow vendors reference architectures as well as common architectural patterns.

Now, we can clearly see that #BigData is not yet ready to provide value in this structured architectural approach. #Hadoop doers' presentations show one remarkable thing: success if happened is impossible to repeat without the same team and environment

It means that companies are just waiting for industry standard #BigData solutions from IT leaders. And of course, a lot of us are playing in #R&D trying to get new flavor, build custom solutions, overcoming limits of current technology stack.

Here I mean than different team will build totally different solution using different set of tools #Spark instead #MapReduce, #Kafka instead #Flink. And even integration with existing company tech is going to be different.

Is this flexibility helpful when we plan #TechArchitecture? Only after we have made extensive POC, have team in place, funds for internal development. In other cases, risk assessment is going to exclude #BigData from #EA building block for next iteration.

Specifically Big Data use case is based on user profile and transaction history. @ggilbert41 loves to use the example to talk about big data/smart data. In essence, it's calculation of one number (0-10) where 0 is clear, 10 is fraud, rest is in gray zone.

In accordance with @ggilbert41, time of calculation should be in sub-second to make sense for user experience. In general, it excludes using big datasets and limit calculation to rules' set, pre-calculated from Stat, ML, or some known cases.

This approach promises that we move from fraud detection after transaction done to prevention to not allow the transaction at all. For that point we are almost in field of authentication rather than authorization.

Authentication based on history and profile raises a lot of questions, tech and privacy based. Basically you are sort of punished for changing consumer behaviour or geography. It can be annoying SMS, phone calls and even blocked credit cards.

From the other side if we tighten the security, it can have some inconvenience to customer as well. It's much easier to tap card that use pin or sign the receipt. But security measured can be less intrusive and use more factors than credit card itself.

For example, face recognition module can add extra layer to authentication. Again, it only calculates number, but the same number can be stored in credit card for comparison. If it doesn't match, ask to enter pin. If pin doesn't match, block the card.

Image recognition can be done on client side; its cost including hardware goes down constantly (and economy on volume can be really huge). And prevention effect is going to be times more significant than Big Data fuzzy rules engine application.

This theoretical experiment is easy to estimate and it confirms that there are no substitutes for right Architecture. Right solution on design phase eliminates need of statistical analysis of deviations.

From that point of view, it means that Big Data Value should be discovered earlier during design phase. So, Big Data solutions should help with range uncertainty (rather than full uncertainty, "I don't know, what I don't know" @Crowdfather )

If you don't know what you don't know then the architecture will be as good as your knowledge

@Crowdfather It is not only company knowledge, but industry knowledge and vendor's reference architectures and best practices. Not much degrees of freedom after that to discover with Big Data.

this is a great conversation to have with the @wikibon community and analysts

getting the data gives you a baseline to work a new methodology like Agile vs waterfall development

Gartners BiModal IT is BS why would someone want to go slower

@furrier Can give you example from field: core telecom services should be reliable 99.999. So HA, DR, Fault Management should rely on architecture, not on "unpredicted" discovery in data

devops use cases use sandboxing to get agile into production its ops who need to enable speed

App development and Enterprise Architecture (it's company structure and processes, not only IT itself) have different life cycles, both can have benefits from Big Data. Healthcare provider with Data Lake is perfect example of changing EA with Hadoop.