agiledatascience

Agile Data Science
Discuss applicability of Agile methodologies to team-based development of data science deliverables.

What percentage of data science teams are using Agile methodologies?

jameskobielus
http://www.via-cc.at...

Dez Blanchfield
- Agile is a process that was designed to manage rapid, iterative work..
jameskobielus
Please reply in the scrolling display directly under the question you're addressing.
Dez Blanchfield
- I've seen many successful Data Teams are now moving to Agile and Cross-Functional operation
Joe Caserta
c. Team-based work, shared responsibilities, uber-collaboration iterating through incremental stages towards ultimate solution #agiledatascience
Dez Blanchfield
- The advantage of Agile is rapid iterative development & rapid feedback cycles from customers, very different to older Waterfall style project methodologies
jameskobielus
@dez_blanchfield Dez: Is agile usually a "designed" process, or is it usually more of an "emergent" process that naturally coalesces from iterative development ad-hocracy?
Dez Blanchfield
- Agile is everything that Waterfall methods like PMBOK & Prince2 are not
Joe Caserta
Typically prioritizing for the most valuable features or requirements and delivering small packages of those features first #agiledatascience
Steve Ardire
agile enables you to change the kind of analysis you're doing depending on what the data is telling you.
Bob E. Hayes
Agile #datascience is development through collaboration of cross-functional teams.
Joe Caserta
@dez_blanchfield continuously re-evaluating and re-defining business priorities in tight cycles
jameskobielus
@joe_caserta Joe: "Ultimate solution"? Sounds fairly final, cut-and-dried. How many real-world data science intiatives are that closed-ended vs. ongoing/exploratory?
8 Path Solutions
from a software/tech standpoint, #agile is an approach that allow faster development than traditional SDLC processes
Dez Blanchfield
- Agile can come naturally to some but it's generally in my experience a designed process, it borrows from the likes of Taiichi Ohno's "Kanban" for example
Dez Blanchfield
- Fail & Fail fast requires an agile approach, PMBOK & Prince2 don’t work for Fail & Fail fast
Bob E. Hayes
I like the idea of using data to inform you where your development needs to head. You can leverage behavioral analytics. How do customers use your product/solution? Build what improves usage.
jameskobielus
@sardire Steve: If Agile is about changing your analysis methodology mid-project, is there any point in doing upfront planning of the data-science project/workstream in iterative projects?
Dez Blanchfield
@joe_caserta - indeed, collaboration is a huge part of it but not exclusive to Agile, short sharp fast "sprints" are the key to Agile ;-)
Dez Blanchfield
- Reducing time to value is more easily realised through short sharp agile sprints
8 Path Solutions
Agile is great process for developing new features, but not a good fit for implementing changes on complex systems
Joe Caserta
#agiledatascience is continuously iterative
Dez Blanchfield
@sardire - yes yes yes.. far easier to turn the Titanic around with an Agile methodology than with PMBOK or PRINCE2 ;-)
jameskobielus
@8pathsolutions Jennifer: Does "waterfall" (which I'm assuming you're referrig to under "traditional SDLC processes) apply to data science projects? If so, how often? What sorts of projects?
Dez Blanchfield
@joe_caserta - totally agree, the epitome of "fail and fail fast" too ;-)
jameskobielus
Just about to drop question 2 into the crowdchat stream. Look at the top of your screens for it.
Joe Caserta
@dez_blanchfield agile formalizes the process to make collaboration most effective.
Dez Blanchfield
- I'm sure Jen will agree that the heart and soul of Analytics stems from how Agile allows iteration of Models and Data Sources & input from the business / user / customer..
jameskobielus
@dez_blanchfield The pre-chat poll response at the top of this page says no more than half of data science projects may use Agile. Are the rest "waterfall"? If so, is that "waterfall" percentage declining over time?
8 Path Solutions
waterfall can be applied to #datascience projects that need to be integrated into into existing systems and therefore require extensive testing
jameskobielus
http://www.via-cc.at...

Dez Blanchfield
- Yes but not everybody in the business made the transition smoothly
Dez Blanchfield
- Business folk found it tough shifting to Agile as an approach after a lifetime of Prince2
Dez Blanchfield
- I've successfully used an Agile approach on ALL size and scale of project when it's done right with a good Agile lead on the project.
Steve Ardire
Agile is becoming more and more common even in larger enterprises
Dez Blanchfield
@sardire - for many it's a slow learning process though don't you think? culture and behaviour changes they aren't always comfortalble making..
Dez Blanchfield
- Some of the Data Science team worked as long wolves and found hunting in packs a challenge
Dez Blanchfield
- Others jumped at it and loved it and make the shift quickly and painlessly
jameskobielus
@dez_blanchfield Is Agile your predominant methodology in your data science/app-dev team. How complete is the transition in your org to Agile?
Dez Blanchfield
- Results have been mixed but the overall outcome seems be positive if the shift to Agile is kept simple
Bob E. Hayes
Our team is using an agile methodology to build in new features to the data platform... and always informed by what our users want and use.
Dez Blanchfield
- In an existing environment Agile is often a tough idea and value proposition to convey and inject
Dez Blanchfield
- In new projects, Agile is usually in my experience far easier and more comfortable to inject or apply - esp. with younger team members without pre-set ideas about "how we do it here" ;-)
Dez Blanchfield
- Results have been mixed but the overall outcome seems be positive if the shift to Agile is kept simple
Dez Blanchfield
- Some find it time-consuming & distracting attending all the meetings and stand-ups
Dez Blanchfield
- Others also found short-duration iterations stressful and useless or couldn’t change pace
Bob E. Hayes
@dez_blanchfield I agree, Dez. Keep it simple.
jameskobielus
@sardire Steve: what are the factors that might keep larger orgs from bringing Agile into their data science initiatives? Less tolerance for un/under-documented development? Need for structured processes and audit trails for compliance?
Dez Blanchfield
- in every single project or programme of work in Data Science et al, It’s a balancing act between tech / dev / biz culture, people, personalities & experience
Dez Blanchfield
@sardire - for a long time it was tough, at least in APAC and Australia, to find good Agile coaches and Agile savvy Data folk, as it's been a DevOps / Dev "thing" for the most part..
jameskobielus
@bobehayes Bob: To what extent are users/customers directly engaged in your Agile data science collabs/workstreams?
Steve Ardire
if larger orgs have a more 'open world' culture and behavior agile will be adopted much easier
Joe Caserta
we create data environments tooled to foster agile collaboration
jameskobielus
Question 3 coming in just a sec. Look at the tops of your screens.
Dez Blanchfield
@bobehayes - that's the true value of Agile right, the short sharp sprints and then instant input and feedback from the user / consumer / customer ;-)
8 Path Solutions
Yes, for short term investigations #agiledatascience is useful and a good fit given the short timeline
Dez Blanchfield
@joe_caserta - where do you start to inject that sort of thinking Joe, how early do you get it into the conversation, Agile that is, as an idea, a value propositon?
Bob E. Hayes
We look at how they use the project (has usage gone up/down since we made the changes?). We also listen to them when we ask for feedback about how we can improve the product. So, both implicit and explicit feedback.
Dez Blanchfield
@sardire - amen to that, but at the same time IMHO we as "practitioners" and "thought leaders" have to be savvy to the fact that some times it's a huge leap of faith for orgs to "get Agile" ;-)
jameskobielus
@dez_blanchfield What's the primary data scientist habitual behavior they need to "unlearn" to do Agile effectively?
Joe Caserta
@dez_blanchfield agile coaching has become a common role to transition lone wolf mindset
Dez Blanchfield
@8pathsolutions - Jen how do you think Agile works in Medium to Long term projects, I'm finding that short term is ok, medium perhaps, but long term Agile is a tough challenge to "maintain" at times.
jameskobielus
@dez_blanchfield How do you measure the effectiveness/results from Agile? Do you report it up to a Chief Data Science Officer in your org? Or, are you that Officer?
Bob E. Hayes
@dez_blanchfield @sardire Yes... change is difficult, especially when you're old like me... but all I need to be convinced is some data that tells me what we are building is working (improving user sat and usage).
Dez Blanchfield
@joe_caserta - yes you've hit the proverbial nail on the head there mate, I'm finding it as tough to find good Agile Coaches as I am good Data Scientists & Analysts !!
Joe Caserta
@dez_blanchfield biggest challenge can be keeping 10-min stand-up to 10 min.
Dez Blanchfield
@joe_caserta - it's almost like Agile Coaches, good ones, are the new Unicorn ;-)
jameskobielus
@8pathsolutions Jennifer: Is Agile less well-suited to longer-running projects than quick-turnaround data-sci modeling/exploration?
8 Path Solutions
@dez_blanchfield Long term projects tend to have more extensive requirements, deadlines, and milestones - In my experience, agile isn't well suited to handle meeting all of these expectations