At SadaPay, we’re just starting the journey towards our engineering vision. Here’s what we’ve already achieved and what we’re still working on.

We’ve just updated this post to reflect our progress over the last few months. I’m so proud of our team! We used a strikethrough effect for things that are now out of date and corrected them underneath, so you can see our journey.

‍

Changing Code with Confidence

Integrity is one of SadaPay’s values and we also apply it to our software. Sometimes we have to work fast, but we never work lazy. We don’t take shortcuts because we know maintaining the integrity of our software is good for everybody.

‍

Practicing Test-Driven Development

Every line of code we commit is test-driven, so we know what to write and that it works as documented. Working this way means we can change our code with confidence and develop software quickly. Front-loading testing front-loads our understanding of requirements, so we don’t waste time building the wrong things.

We test everything thoroughly across multiple layers, but most engineers are not practicing TDD yet. We’d love to have more experienced TDD practitioners join and share their skills.

Every developer at SadaPay practices TDD consistently and our testing practices continue to improve – we’re nailing it!

‍

Not Relying on End-To-End Testing

Everybody would love to have comprehensive end-to-end tests running in a production-like environment, but this approach is almost impossible to execute and often becomes a white whale. We prefer to verify the correctness of integrations at compile-time using asynchronous techniques like code generation from shared API specifications and per-service functional testing against mocks.

We use shared API specifications and code generation for our API providers, but we’re still investigating code generation for our API consumers.

We use code generation for our API providers and iOS consumers and are now rolling it out for Android consumers. JavaScript consumers will be next!

We use code generation for all API providers and consumers, including Spring, iOS, Android and NextJS.

‍

‍

Solving Problems Together

At SadaPay, empathy helps us serve our customers better. We also use it to improve our teamwork, skills and engineering outcomes. We work together closely and try to understand each other deeply, instead of just looking at each others’ code.

‍

Collaboration over Critique

We front-load collaboration using requests for comment, kick-offs, mobbing, walkthroughs and tech huddles to share context and solve problems together. These practices provide great learning opportunities – we get to learn each others’ ways of thinking and solving problems instead of just seeing the code at the end.

We collaborate closely before and during implementation, so code review is usually fast and routine.

Almost all codebases are now owned by a single squad and changed using mob programming and so that review is continuous and collaborative.

‍

Pairing with Rotation Relay Mobbing

With minimal traditional code review, we are serious about using pair rotation mobbing to get input from others and socialise code changes. We rotate pairs in the middle of our work and take the input of our new pairs seriously. The harder the work, the more pair rotations naturally take place – teamwork scales automatically with complexity.

Right now, we do not pair frequently, partly due to time zone challenges. We’ll work hard to improve this in the next few months. If you’re a skilled pair programmer, we could use your help normalising this practice.

We pair and mob every day, but we’d like to spend even more time pairing. The biggest challenge is still time zones.

We use mob programming consistently, with the whole squad focusing on their most important work.

‍

Integrating Continuously

We keep it Sada. Many companies employ complex development and release workflows like Gitflow to solve problems that have simpler, more efficient solutions. For example, many teams have a special hot-fix Git workflow, but this is only necessary because the cycle time in their normal workflow is too long.

‍

Working in Small Steps

Instead of checking out a branch, taking everything apart and then struggling to put it back together again, we make a series of small, production-ready changes using the Red Green Refactor cycle. Small change-sets are faster, safer, easier to reason about and easier to review.

Most back-end changes are executed in task branches that live for less than 24 hours. Front-end changes are typically made on longer-lived feature branches. We’d love to see branch lifetimes get even shorter and eventually practice trunk development across all platforms.

Mobile changes are also made on short-lived task branches and we practice trunk-based development when working on our NextJS apps.

Almost all backend, mobile and web changes are made in small steps, with commits pushed directly to trunk after each red green refactor cycle. There are some exceptions, either regulatory or where shared code hasn’t been properly extracted to separately owned services or modules yet.

‍

Decoupling Release from Deployment

Deploying code and releasing software are different things. We wrap changes to the behaviour of our software in feature flags so that each change can be released or unreleased safely at runtime. This also allows us to test new behaviour in production before releasing it to the public. Crucially, it helps us keep trunk in a releasable state without a complex Git workflow.

Many behaviour changes are now taking place behind feature flags, but back-end flag positions cannot be changed in real time or used for testing in production.

All behaviour changes in our mobile app are made behind feature flags and we’ve discovered that proper API versioning greatly reduces the need for feature flags in our backend.

‍

Practicing Trunk-Based Development

We collaborate closely with each other, have confidence in our testing, have runtime control over the release of our software and know how to work in small, production-ready steps. It is safe for us to push every commit directly to trunk, integrating continuously to minimise merge conflicts, integration failures and Git mistakes. It is an incredibly efficient way to work.

We’re just starting on this journey, having introduced feature flags and reduced the lifetime of task and feature branches. If you are an advocate of trunk-based development, we’d love to have your help.

We’re already working on trunk for our NextJS apps – up next will be back-end and then mobile!

Since we have strong ownership and use mob programming consistently, we rarely create branches at SadaPay. Most code changes are made using trunk-based development.

‍

‍

Being Full-Stack

‍

Understanding the Whole

We work on vertical slices that cut through multiple layers to deliver user value. Designing and implementing solutions that span multiple layers is much easier when you understand how each layer works. As product engineers, we care about every system that impacts our users.

Engineers working on different platforms collaborate regularly and effectively, but almost nobody is working across multiple platforms. We’ll extend this opportunity to our engineers and encourage them to try it out via cross-role pairing. If you’re interested in being a Full Stack Engineer, we’d love to hear from you.

We don’t have many single-platform specialists at SadaPay. While many people joined with experience in only one platform, most are now working in either a mobile platform and our Spring backend or NextJS and our Spring backend.

We still have some backend specialists, some of which will join our new Platform Engineering Team to build generic solutions to our most challenging backend problems. Others will stay in Product Engineering and grow their mobile and web skills to become fuller-stack.

‍

Learning Together

We recognise the value of full-stack engineers and encourage cross-role pairing, so that people can teach each other different technologies. It’s a great learning opportunity.

Right now we only cross-role pair for kick-offs, desk-checking and ad-hoc meetings. We don’t implement the solution together. This is mostly for time zone reasons and as the structure of our team changes, we’ll be able to improve it.

Some of our mobile engineers have begun their full-stack journey, working with other engineers across all layers to deliver stories end-to-end. We’re reorganising our squads around smaller groups of more flexible developers.

Our squads consistently have only 3 or 4 engineers, mobbing together for several hours every day. People with strong backend skills have done a great job of spreading them to former mobile specialists, but we need to invest more time in teaching mobile and web to our backend specialists.

‍

Fast Feedback

At SadaPay we believe in being transparent – with our customers and each other. Being honest about what’s working well and what isn’t helps us continuously improve ourselves, professionally and personally.

‍

Continuous Improvement

Complex systems work better with feedback. An air conditioner works better with a thermostat, an accelerator works better with a speedometer and an engineer works better with input from peers.

We seek 1:1 feedback regularly, reflect together on the effectiveness of our processes and review our experiences solving problems together.

We have great discussions during retrospectives and are quick to identify problems and solutions with our team processes. However, we don’t have much 1:1 feedback. We recently implemented an optional 1:1 feedback process and will explore ways to help engineers participate more.

‍

Tight Feedback Loops

Working quickly requires that we get fast feedback on our work. Imagine trying to drive a car where the speedometer lags 30 minutes behind your actual speed! We use continuous delivery to get fast feedback from our users and continuous integration to get fast feedback on our code changes.

Our test suite is fast and thorough, but our delivery is not as continuous as we’d like. There is also room to improve how we measure the impact of our work so that we can learn more from each release.

Our delivery is much more continuous than before. We regularly use showcases, analytics, internal production testing, phased rollouts and occasional AB testing to get feedback on our work.

We are starting to have trouble with build times. We’re keeping short timeouts in our CI pipelines and resisting the urge to increase them. When builds start timing out, we rush to optimise them. It’s a bit disruptive and we are exploring using more powerful runners and adjusting our test strategy.

‍

The Journey Continues

As you can see, we still have a long way to go on our journey. If you believe in our vision, we’d love for you to come join us at SadaPay so we can make it a reality together. We have multiple open Engineering positions on our Careers page at www.sadapay.pk/careers

‍

About SadaPay

SadaPay is bringing modern financial services to Pakistan in partnership with MasterCard. By removing the high costs of managing the physical infrastructure of traditional banks from the equation, we can pass those savings onto the customer to provide free financial services. Our mission is to eliminate the complexity of banking and make money so simple that any other way would become unthinkable.