For those who are familiar with my previous Talend blogs perhaps you’ve noticed that I like to talk about building better solutions through design patterns and best practices. My blogs also tend to be a bit long. Yet you read them; many thanks for that! This blog is going to focus on methodologies as they apply to Talend solutions. My hope is that at the end of it we all agree that any successful Talend project should use proper methodologies, or face the proverbial project crash and burn.
A stool needs 3 legs upon which it can stand, right? I believe that any software development project is like that stool. It too needs 3 legs, which are defined as:
- USE CASE – Well-defined business data/workflow requirements of a solution
- TECHNOLOGY – The tools with which we craft, deploy, and run the solution
- METHODOLOGY – A given way to do things that everyone agrees with
Software projects tend to focus only on the use case and technology stack, often ignoring methodologies. This, in my opinion, is a mistake and leads to painful results, or lack thereof. When planning for successful Talend projects, incorporating best practices for Job Design Patterns (JDP&BP) and Data Model Designs (DMD&BP) is a fundamental objective. These disciplines fall into the methodology leg of our proverbial stool. The foundational precepts discussed in my JDP&BP blogs suggest that consistency is perhaps the most important precept to apply. The reason is simple: even if you are doing something the wrong way, or let’s be nicer, not doing it the best way, then if it was being done the same way it is much easier to correct. I like that. Successful methodologies should, therefore, encapsulate consistency.
3 Pillars of Success
Let’s take a closer look. Each leg of the stool is a pillar for success. Each pillar holds up its share of the load having a purpose to serve in the bigger picture. Software is not magic and understanding the evolution and life cycle from inception to deprecation is fundamental to any successful implementation. Hardly any serious developer ignores this, yet all too often they succumb to the business pressures and in most cases it is methodology that suffers the most. As methodologies are ignored, hidden long term costs skyrocket. The worst case scenario is a complete project failure; the best case is lower quality, loss of completive edge, and/or extended delivery dates.
So let’s get serious about this. Here is a starter outline to consider:
Modify these as needed; we all have different yet often similar requirements. Let’s continue.
Any successful methodology is one that is adopted and put into practice; nothing more, nothing less. When a software project team applies a methodology it then knows how it plans to achieve the desired results. Without a properly defined and adopted methodology, what we get is analogous to the ‘wild-wild west’, or ‘shoot-then-aim’ scenarios. Neither fosters success in any software project, IMHO! We must have that 3rd leg of the stool.
When we talk about methodologies we can easily cover a lot of different topics. Essentially, however, this discussion should focus initially on planning for the design, development, release, deployment, and maintenance of a software project. Commonly known as the ‘Software Development Life Cycle’ or SDLC, this should be the first step of many in setting up, managing, coordinating, and measuring the required resources, significant milestones, and actual deliverables. The two most common SDLC methodologies, many of you readers already know:
- WATERFALL – Strict, prescribed steps (phases) that scope an entire software result
- AGILE – Flexible, iterative approach (sprints) that optimize early results
- JEP+ – A hybrid, ‘best of’ approach encapsulating all and only what matters
Without an SDLC methodology in place I submit that any perceived savings for not doing this work (and it is work), is all too often dwarfed by the cost of misguided, misconfigured, mistaken results that miss the mark. How much do software blunders cost? I wonder… trust me, it’s a lot! So, choose an SDLC; choose one that is suitable. Implement it. This is a no-brainer! Agile appears to have become the de-facto standard across the software industry, so start there. While the essential tasks within both methods are similar, it’s their approach that is different. This blog is not going to dive in deep on these, but let’s cover two indispensable areas.
Originating from the manufacturing and construction industries, the Waterfall SDLC method prescribes a sequential, non-iterative process which flows downstream like a waterfall. Originating with an idea or concept, all requirements are gathered, analyzed, and clearly specified, a design is conceived, diagramed, and extensively vetted, code development ensues (sometimes for what seems like an eternity), followed by testing/debugging, production rollout, and follow-on maintenance of the code base.
The real benefit of this method everyone knowing exactly what they will get at the end of the project (supposedly). The downside is that it can take a long time before any results can be seen or used. This can become a costly problem if any of the upstream steps failed to capture the true intent of the initial concept. The further up-stream the misstep occurs the costlier it can be correcting. Well, at least we can often get beautiful documentation!
Principles in Agile SDLC focus on short development tasks and prompt delivery of their functionality to users. Promoting a highly-iterative process, this adaptive method reacts to and is flexible to an evolutionary event sequence: A continuous improvement and addition of features, delivering value incrementally over time.
The clear advantage of Agile over Waterfall is that it supports cross-functional teams to be involved in the process end-to-end, fostering better communication and the ability to make changes along the way. The letdown can be that while best practices should be in place to avoid such, feature creep can become a real irritation.
Oh, and don’t count on pretty documentation either.
Don’t get me wrong, Agile is definitely, in most cases the right way to go. In fact, don’t stop there. Agile does not envelop all aspects of broader SDLC best practices. Take for instance: data or better yet, big data. For our purposes, we are building software that pushes data around in one form or another. If you’ve followed my previous blogs, then you know I am talking about: DDLC, or the ‘Database Development Life Cycle’. Read more in my two-part series (link above) on Data Model Design & Best Practices.
Introducing JEP+: – Just Enough Process
My years in the software industry have resulted in many successfully projects and ,yes, painful failures. What I learned was that the SDLC/DDLC process is critical and necessary as it allows you to measure quality thus supporting better business decisions for project/product release. I also learned that too much process gets in the way and can bog down the creative deliverables reducing their value and increasing their cost.
My brother Drew Anderson and I, for the past 20 years or so have fashioned a hybrid SDLC process we call JEP+ or Just-Enough-Process-Plus; which means having the right amount of process in place ‘plus’ a little bit more to ensure that there is enough. Essentially an Agile/Scrum based method, we incorporated the Capability Maturity Model, Waterfall, and ISO 9000 methodologies where beneficial. JEP+ is where DDLC has its roots.
Often, when we presented our methods to customers, they almost always asked: how can we buy this? In the end, it was our competitive advantage which led to us getting hired for many consulting projects. Drew still does! Many have recommended we write a book. Scott Ambler – Mr. Agile, suggested the same. We may do it one day.
There are many elements, or processes in any successful methodology. For Talend developers, a key element we should discuss is Job Design Patterns. This allows me to expand upon some hints I in my JDP&BP series. Presuming we have a clearly defined use case, and we have nailed down the technology reference architecture, before we really dive in to writing code we should have a solid design pattern in mind. Design Patterns provide Talend developers a prescribed approach to building a successful job. Let’s look at some useful design patterns:
This is not an exhaustive list, yet I hope it gets you thinking about the possibilities.
Talend is a versatile technology and coupled with sound methodologies can deliver cost-effective, process efficient and highly productive business data solutions. SDLC processes and Job Design Patterns present important segments of successful methodologies. In my next blog I plan to augment these with additional segments you may find helpful.
Till next time…
Bigdata and data center