Investing in a custom application for your business is an important decision, and it’s even more important to choose the right development partner. With thousands of firms in the market and even more freelancers, it can be difficult to know which are legitimate and able to deliver the high-quality application your company needs for mission-critical operations… and which are lacking in what it takes to carry the day.

Thankfully, there are some criteria that you can use to evaluate the strength of your application development partner candidates. Having a set of guidelines can be helpful as you interview and weigh potential partners based on competency, business practices, and the ability to add strategic, expert value to your project. Here’s what you should look for.

1. Good Match to Your Tech Preferences (but Not a One-Trick Pony)

If your tech team prefers to work with a certain tech stack, you’ll want to find an application development partner that is well-versed in those technologies and has the expertise to fit your needs. Beware of potential partners who don’t specialize—it’s a good sign they’re outsourcing your project to whoever they can find to do it and may not have the depth of knowledge they need to execute efficiently or deliver a high-quality product.

But the other end of the spectrum is potentially problematic too: firms that offer only one solution as a “one size fits all” approach to every problem. A firm that’s the equivalent of a hammer looking for a nail may miss potential opportunities for optimizing performance and functionality by clinging to a single infrastructure or framework. Your partner should have enough diversification in technologies that it’s well equipped to make strategic suggestions that will improve the final product, even if they’re not currently present in your tech stack.

2. Proven Application Development Experience

Any potential application development partner should provide no doubt that they bring a wealth of experience to your project. It should be clear that they have done the type of engagement you are looking to do many, many times. While they likely haven’t built the exact application you’re looking to develop (after all, this is custom development), their work experience should reflect expert working knowledge of key components of success in your software. One way to vet that is to review case studies, testimonials, and links to previous projects. When evaluating the firm as a potential partner, you can glean a lot from the quality of the case study—including how well they were able to understand their client’s challenges and specific needs, as well as the quality of the finished product.

But to go even deeper, make sure that your potential development partner is willing to provide you with references. Speaking with past and current clients will give you the opportunity to ask questions. It’s in unfiltered, open conversation that will allow you to really find out what went smoothly and what didn’t and how both parties worked together to solve it. How flexible and adaptable was the firm? How communicative and responsive to their clients’ needs? You’ll gain a more accurate sense of the working relationship and what you can expect from the partner in the future.

3. Multidisciplinary Expertise

If the team is comprised of employees, it’s also good to ask about their knowledge and skill sets. Is it a team of generalists who know a little about a lot, or are there experts in specific areas across the organization? Are there dedicated front-end and back-end engineers? Infrastructure specialists? DevOps engineers? UX/UI designers? A team of specialists will possess the skill sets to develop creative solutions to your and your users’ problems and be better equipped to work through potential challenges that may arise during the development process. Expect a large difference in quality when you transition from working with generalists to experts in their field.

4. Developers as Employees, Not Subcontractors

It’s not uncommon for development firms to hire subcontractors to execute on their projects. It’s cheaper to hire ad hoc, and they can expand or reduce their staff according to their client load. Many even use offshore labor. But it’s generally not a great customer experience and it doesn’t lead to high quality work product.

In order to provide consistent, reliable outcomes, your application development partner should employ and invest in training developers on staff. This provides more continuity for projects throughout the lifecycle. It also allows the firm to set and hold employees to standards and best practices and invest in professional development. This enables their team to grow their skills over time and stay abreast of new trends, which benefits your development. Also, having the same team that built your original application building new features and functions down the line is very beneficial.

5. Frequent, Transparent Communication

An application development partner is a partner in the process, meaning you should expect collaboration and open communication around work progress, questions, and changes in the delivery schedule. Any firm who brokers in “black box” development without providing you insight into the product lifecycle is not a good fit. Too many clients have to find out the hard way, investing thousands of dollars into a project and getting burned in the process when it goes over time and over budget without a clear explanation. It becomes very hard to know who to trust.

At MercuryWorks, we’ve developed several ways to keep our clients apprised of their project’s status. First, they can see progress firsthand in demos completed at the end of every two-week sprint. But for even more transparency, we’ve also developed an extranet specifically for clients to login and see the progress of their project, work completed to date, work that still needs to be done, project spend to budget ratios, and more. It’s not common for many application development partners to offer this level of transparency, but we find that it brings value and peace of mind. Our clients can trust as much as they’d like and verify as much as they need.

6. Strong DevOps Culture

DevOps is a system of practices that makes application development more efficient and improves product quality. From best practices for maintaining code repositories to managing complicated projects, DevOps has evolved in the past decade from eliminating bottlenecks to creating a pipeline for continuous delivery and improvement.

What does this mean for clients? A development partner with a strong DevOps culture and proven experience tends to produce better, faster results up front, which ends up saving you time and money. In addition, a continuous pipeline for release management enables ongoing support after release, reducing the amount of technical debt that your team might otherwise incur from a firm that releases a product manually.

7. Investment in Your Product Vision

Product vision is what separates a strategic partner from a simple order taker. They will help you think through your users’ needs. A good application development team will begin by attempting to gain a deep understanding of your users and their needs, as well as yours. In fact, it’s not uncommon (and good business practice on both sides) for the work agreement to take the form of two independent contracts: one for discovery, planning, and assessment, and a second for the actual application development.

The discovery process should involve your partner gaining a deep understanding of your business and assessing your challenges before making a recommendation. Being able to view your product from a business perspective instead of just a tech perspective is absolutely vital to creating an application that meets your needs. It also enables your application and your relationship with your partner to contribute increasing value after launch—so that every activity from hosting, managing, and maintaining the application to investing in ongoing product development is accomplished with your product vision at top of mind.

8. Commitment to Knowledge Transfer

One of the greatest differences between a mediocre development partner and a stellar one is an expectation for a long-term relationship based on open sharing of knowledge. It makes no sense to pay for a firm to develop an application, only to have them “hold the keys” when it comes to knowing how it’s built and how to maintain it.

A good partner is more than a good developer—they’re a teacher and a guide who is committed to your success. They should work to empower your team and share how the application works inside and out. They should be willing to codevelop with your team, if desired. After release, your partner should be equipped to assist with onboarding and support for as long as you’d like and have the requisite organizational infrastructure for doing so professionally.

9. Should Feel Right

While there may be many firms out there that can handle your technical challenges, that doesn’t mean that they’re all a good fit for you and your organization. Pay attention to how the firm is engaging with you and your team. Are they professional in how they do so? Do they get back with you in a timely manner? Are their communications clear? These are important questions to ask yourself while engaging with a prospective partner. Having the right fit for your organization’s style and preferences will go a very long way towards a successful long-term partnership.

MercuryWorks has solved wicked problems for hundreds of clients and has developed many long-standing relationships based on trust, customer service, and quality. With more than 20 years of experience in software development, we consistently ship custom solutions that closely align with business needs and continue to work with clients for their long-term, ongoing, evolving support and development.

Interested in an application development partner for your organization?

For several years at MercuryWorks we wrestled with a means to tie together more than one Scrum team in the development of large software systems.  Sometimes software systems are just too large for a single team with the generally-accepted “7 plus or minus 2” size to handle.  Because the popular Scaled Agile frameworks tend to be more complex, I wanted to derive something just a little simpler to solve the problem.

Why Scaled Agile?

Agile in its various flavors came to the fore in the early 2000s as a reaction to classic waterfall development and, particularly with frameworks like Scrum, soon became the standard method to develop modern software.  With this success came growing pains – including difficulties with multiple small fully-functional Scrum teams working on large enterprise software implementations.

Scaled Agile frameworks soon sprung up to help solve this size problem – to unify the work of multiple Scrum “feature teams” to build large enterprise systems.  Most of the frameworks incorporate DevOps teams to form a cohesive process.

One of the core overarching strategies of the scaled frameworks is to align technology and business strategy at a more strategic level.  By coordinating the moving pieces of large Product releases, the scaled frameworks take a larger “portfolio” view of the enterprise and value chain.

Problems With Multiple Teams and No Scaled Framework

Mercury has made a few attempts at building large software systems using Scrum but without the use of a scaled framework.  The following are common problems we ran into:

  • Reduced productivity
  • Divergent Product visions
  • Divergent architectures
  • Divergent user experiences
  • Conflicting cross-team coding work

Universal Elements of All Scaled Frameworks

There are several popular Scaled Agile frameworks including SAFe, LeSS and Nexus.  While they all differ in their complexity and approach, all of the scaled frameworks attempt to unify software, operations and business teams as follows:

What Is SAFe and Why SAFe as a Starting Point?

SAFe is the most popular and long-lasting Scaled Agile framework.  SAFe aims to provide a knowledge base of integrated principles, practices and guidance to unite multiple modern bodies of knowledge around software development.  SAFe does not attempt to replace Scrum but rather it adds a set of additional roles and ceremonies around Scrum teams to unify their collaborative work.

Bodies of knowledge that SAFe embraces includes Lean, Agile, DevOps and Design Thinking.  All of those are winners and is why I find SAFe to be a particularly strong base.  However, I will admit that in full, SAFe is a very large methodology/undertaking and why I feel a “minimally viable” variation is called for in most cases.

The SAFe "House of Lean"

SAFe combines the core principles and tenets of both Lean and Agile to establish it’s own spin on the famous “house of Lean” established by Toyota.  For those of you versed in the Toyota Production System, you will note the heavily borrowed metaphors and mappings back to JIT, Continuous Improvement and foundation of Kaizen, Teamwork, etc.

© Scaled Agile, Inc.

SAFe Lean-Agile Principles

Building on the “House of Lean”, SAFe’s Lean-Agile principles form the operating system of the SAFe approach and in many ways mirror the Agile Manifesto values.  The following table enumerates SAFe’s Lean-Agile principles and some specific interpretations MercuryWorks applies in our minimally viable approach to scaled agile:

Lean-Agile Principle As applied in Minimally Viable Scaled Agile
Take an economic view Prioritize backlog in descending order of outcome value divided by size
Apply systems thinking Seek system maxima, not module maxima; pull work that results in a shipped system
Assume variability; preserve options Don’t lock into long-term decisions when a Last Responsible Moment can be sought
Build incrementally with fast, integrated learning cycles Tight vertical slice user stories, continuous deployment
Base milestones on objective evaluation of working systems Assessment of targets are based on shipped software and not proxy measures (finished stories, lines of code, etc.)
Visualize and limit WIP, reduce batch sizes and manage queue lengths Respect and monitor Scrum team capacity limits, visualize operations activities
Apply cadence, synchronize with cross-domain planning Set fixed-length sprints, lock start/stop across teams, Scrum of Scrums
Unlock the intrinsic motivation of knowledge workers Provide autonomy across teams, reinforce system goals, not local proxy measures
Decentralize decision-making Set the mission at the system level and empower each team to determine means and make ground-level decisions
Organize around value Organize teams around discrete areas of the value chain rather than all works on all

The Agile Release Train

The Agile Release Train (ART) is the collection of feature development teams, operations team(s), business stakeholder(s) and delivery pipelines centered around a value stream. Often when people refer to the ART they are referring to the delivery pipeline of the team that results in the continuous release of software so keep that in mind when talking Scaled Agile.

SAFe also makes a nice addition of the term “Continuous Exploration” (CE) to form a triumvirate with the CI/CD acronyms that Scrum and DevOps practitioners are used to.  The following diagram shows the evolution across the “train” from discovery/product management through released working software:

© Scaled Agile, Inc.
“Continuous Exploration” is another term for the typical Agile/product management function for roadmapping and backlog authoring

The Program Increment

SAFe introduces/borrows another nice concept called the Program Increment (PI).  A PI is the timebox during which an ART delivers incremental working, tested software.  Net-net a PI is a grouped number of sprints focused around specific backlog Feature(s).  In other words, a PI is to an ART as a Sprint is to the Scrum Team.

Program Increment planning ties back to backlog artifacts as follows:

  • User Stories are sized with the intent of fitting inside of one sprint
  • Features are sized with the intent to fit inside of one PI
  • Epics are sized to fit within 2-3 PIs

One of the things that I particularly like about the use of PIs is that it brings a resolution to the often-expressed Scrum team feeling of “I know we are sprinting away every two weeks but I’m not sure how and when it all adds up”.  The Program Increment gives the team (and business stakeholders) a clear goal to likely-released software and how multiple stories and sprints add up to delivered business value.

© Scaled Agile, Inc.

Let’s Break Down The New Roles

SAFe also introduces some new roles to the Agile/Scrum picture that helps frame up the Agile Release Train operations and Program Increment planning that I think are helpful even for a minimally viable implementation.  None of these are conceptually new but their specific names and duties are fairly new to those used to practicing straight Scrum:

Release Train Engineer
  • Servant leader and coach; major responsibilities are to facilitate ART events and processes and assist the teams in delivering value
  • RTEs communicate with stakeholders, escalate impediments, help manage risk and drive relentless improvement
  • RTEs guide the ART to hit PI goals and make decisions that meet the overall mission
System Engineer
  • Responsible for defining and communicating a shared technical and architectural vision
  • Help ensure the system under development is fit for its intended purpose
System Team
  • Specialized team that assists in building and supporting the Agile development environment, typically including the continuous delivery pipeline(s) toolchain
  • May also support the integration of assets from Agile teams and assist with deployment and release
  • Works against a continuous Kanban board as opposed to a bounded sprint cadence

Two New Ceremonies

At this point you may be asking yourself “so even in a minimally-implemented version of scaled Agile, how do you keep the teams from bumping into each other?”.  While I will admit that Scrum calls for a number of ceremonies, scaled Agile adds two more valuable-but-lightweight ceremonies to the picture to address coordination:

Scrum of Scrums

The Release Train Engineer (RTE) facilitates a weekly Scrum of Scrums (SoS) event.  The SoS helps coordinate the dependencies of the ARTs and provides visibility into progress and impediments.  The RTE and one representative from each team meet to review their progress toward PI objectives and dependencies among the teams.  Timeboxed for 30-60 minutes (there can be a ‘stay-behind’ for individuals needing a deeper dive into specific topics).

PO Sync

The RTE facilitates a weekly Product Owner Sync (PO Sync) event.  The purpose is to get visibility into how well the ART is progressing toward meeting its PI objectives and discuss problems or opportunities with Feature development.  The event may also be used to prepare for the next PI and may include backlog refinement and prioritization ahead of the next PI planning event.  Timeboxed for 30–60 minutes.

A Visualization of How the Pieces Fit Together

Map the Value Stream

Early in this post I made mention of the schools of thought that SAFe embraces and one of those was Lean.  A common practice in Lean planning is the creation of a value stream diagram – this is a solid place to start in creating your ART.  The following is a prototypical online shopping “job to be done” mapped from beginning to end:

Map Intended Subsystems to the Value Stream

Since we will ultimately need to assign portions of the eventual enterprise software system to individual teams within the ART, we next need to subdivide the overall system into modules/subsystems.  Naturally at this point the System Engineer will need a pretty good concept of the eventual system composition so sufficient discovery and ideation will have had to occur.

Assign System Users to Subsystems/Modules

Since at its core Agile, Scrum and Lean are customer-centered models we strive to map intended end users to each subsystem/module so we can identify the stakeholders of each eventual team within the ART.  With this final step we can make some clear choices on how and where to assign each team within the ART in a way that will result in the least dependencies and conflicts.

Identify Composition of the Subsystems

Once we have identified each of the subsystems and end users for each portion of the value stream we can begin to visually illustrate touchpoints across the subsystems.  The following is an example block diagram for what will become the eventual enterprise system for the example online shopping solution.  Note that it is worthwhile to map in third party systems – even though those will not be built by the ART (they will need to integrate into them).

Break Down the Major Modules

A final ART planning step is to break each of the main system blocks into the pieces that are likely to be major pieces of code/data/interface.  This will finally get us to the point where we can assign each team to a collection of code pieces that maps back to the value stream and thus end users.

Program and Dependency Board

Now we arrive at one of the key pragmatic pieces of the scaled plan – the Program/Dependency Board.  As you can see, this board maps back to the code segments/repos and is used during Program Increment planning to makes subsystem dependencies visual.  In the following diagram the green and blue rectangles represent intended PI User Stories and Features while the red rectangles indicate stories/Features that have dependencies and that will require team coordination.

So What Does Program Increment Planning Actually Look Like?

Thus far I’ve spent a lot of coverage deciding on how to compose the ART, the system and subsystems they will be building and how to begin to get individual Scrum teams rationally assigned to actual pieces of software they will be responsible for.  At this point you will have two to three Scrum teams to develop features and an enablement team to handle cloud infrastructure and pipelines.  We have also established that a PI will be made up of three sprints and will ship a small number of backlog Features.

So what actually happens during PI planning?  Take the following with a grain of salt and apply it to the particulars of your solution and ART but know that this is an example PI planning schedule that we have found to be effective:

Net "Minimal" Ceremonies

Now we come to the portion of ART planning wherein we plan and schedule the needed ceremonies.  The accompanying table calls out an example set of ceremonies/meetings for an ART for this example challenge (it covers both individual Scrum team and overall ART ceremonies) along with a practical frequency.

Ceremony Frequency Participants
PI Planning every 6 weeks All ART members
Sprint Planning every 2 weeks Scrum team members
Scrum of Scrums weekly RTE & one rep per Scrum team
PO Sync weekly RTE & each Scrum PO
Sprint Retrospective every 4 weeks Scrum team members
Executive Progress Briefing every 6 weeks RTE & business owners

Wrapping It All Up

At this point we have covered a lot of ground, generalized a lot of things and certainly left out a ton of details and rationale.  However, I think we have covered the absolute bare minimum composition and details related to an effective start towards scaling your Agile practice.  While this approach will not come close to doing justice for ARTs of more than 3-4 teams (that is what full-blown SAFe is for), I have found this to be an effective recipe here at MercuryWorks:

Unite 2+ Scrum feature teams and 1 system team into a single cohesive group called an Agile Release Train (ART)

Each Scrum team works independently in two-week sprints (we like to start Wednesday and end Tuesday)

Work in 3-sprint-long Program Increments (PI), fitting two PIs per quarter

Appoint one Release Train Engineer and one System Engineer

Each system/module build and deploy is automated and continuous

Systems/modules are released via feature flag at conclusion of each PI

SonarQube scans and cross-team peer reviews at the conclusion of each PI as a technical health and consistency mechanism

Your mileage, as they say, will surely vary but please do let me know your thoughts below!  Also, to dig deeper into some of the first principles, take a look at these resources: