Digital transformation is one of today’s biggest buzzwords. Everyone is talking about it; everyone wants it. We all know the role technology is playing in enabling businesses to innovate at an unprecedented pace. But making the digital transformation leap is easier said than done, especially if you are a large enterprise with huge monolithic legacy applications. Speed and agility may seem like a distant dream rather than attainable goals.
The “why now” is pretty clear. We all benefit from the convenience brought to us by the Amazons, Googles, Airbnbs, and Ubers of this word. As consumers, we get used to the new ways of buying, traveling, consuming and start demanding it from traditional service providers as well. With no technical debt, pure play digital players can push the boundaries faster, but traditional enterprises need to follow suit quickly or risk losing business to the newcomers.
What’s needed is not a secret: a strong partnership between business and technology, organizational-wide buy in, a futureproof, nimble system based on the principles of openness, interoperability, extensibility, and modularity. Getting there isn’t easy but it is worth the effort to tackle leadership, cultural, and technical challenges.
The Challenges of Digital Transformation
Technology brings efficiency gains to an entire organization, whether sales and marketing, purchasing, or even administrative roles—everyone is touched by and benefits from it some way or another. Systems are connected, disparate data is merged, a full picture of the business achieved. Horizontally aligned, the modern IT department has the potential to empower every single business unit. However, it is often treated like just another department. Only those enterprises that understand the need of becoming a ‘technology company that happens to do X’, will survive. Technology enabled efficiency gains are so dramatic, that not leveraging it is a secure path to demise.
Leadership Challenges
While it’s not difficult to make that case—just think of Blockbuster vs. Netflix, Hospitality vs. Airbnb, brick-and-mortar retail vs. Amazon—there are generational challenges. Some leadership teams may not understand or embrace new technology. Though the concept of well-rounded leaders isn’t new, those who have experience in IT departments are far and few in between. That leads to leadership teams that often lack the needed technical expertise to make the right strategic decisions. The leadership teams of disrupters, on the other hand, tend to be more technology savvy and visionary. Overcoming that gap is critical to succeed.
Another issue is the traditional, established reward systems which generally encourage short-term thinking. While a multi-year effort may be the right business decision, benefits may outlive a senior executives’ tenure. Decision-makers are often rewarded to maximize revenue for stakeholders and themselves today, not 10 years from now. We are dealing with a broken system that doesn’t incentivize long-term thinking. To counter that, organizations must completely reinvent their reward system.
Cultural Challenges
Let’s face it, technology departments have a bad reputation—their track record hasn’t been great. But was it really their fault? IT has often been tasked with “changing the world” but didn’t get the needed support from the business units they were supposed to serve. Additionally, the role of IT has shifted from “keeping the lights on” to strategic innovators who leverage technology developments to better serve customers (e.g. deliver new features faster).
Transformation is also up against human nature. Change means uncertainty and people are programmed to be risk averse, resulting in workforces that will potentially advocate for keeping processes and technologies. Forcing a new technology solution upon teams has proven a recipe for failure. You’ll need user buy-in through education and change management before the rollout of any new solution. Safety (including job security) is in the speed of change, not in stagnation. Once your team understands that, transforming your business will be much easier.
Uncertainty Challenges
Most organizations know digital transformation is needed, but it’s hard to figure out where to start. Any change bares risk—businesses have customers and revenue and change may disrupt the business flow. Complex entities, enterprises require complex solutions, yet the end game is often not clear. While the goal is flexibility, what the future will bring is uncertain.
Many organizations are trying to become technology companies. But what does the customer really value? It’s not technology per se. Before it was price, product, and proximity. Though this still holds true, the notion of convenience has changed. Convenience means serving the customer in any way they want quickly.
Slowly creating a monolithic monster
Implemented over time and often focused on achieving process improvements and efficiency gains vs a future proof, flexible design, enterprises tend to have complex architectures. These rigid platforms can’t be easily adapted to new technologies and may never have undergone a refactoring process.
The reason monolithic systems are so prevalent is due to the constant evolution of technology. A core system is developed, and new features are added as requirements evolve. This leads to increasingly larger and entangled systems, making it really difficult to update components without affecting other components (the ‘spaghetti code’ problem).
While old and clunky, legacy systems still work and create value. There is no need nor is it practical to get rid of them and replace them with new shiny technology right away. It’s possible to break down the elephant and create a nimbler path for the future.
Vendor lock-in—how we got here
Historically, big companies such as Microsoft, IBM, Oracle and others, have built software for very different companies and organizations covering diverse operational areas of businesses. They were pioneers of software development and have played a key role in the rise of the software development industry overall.
Knowledge that these big software companies collected in close collaboration with their customers has been translated into innovations and improved software products. While this has been beneficial to the industry, it has also created issues for customers in operations, maintenance and improvements of the software they acquired.
Many enterprise-ready, closed-source (proprietary) solutions have become too complex and heavy for customers to operate and customize without vendor support, costly upgrades, or switching to different solution without substantial costs. This is often referred to as “vendor lock-in", “customer lock-in" or “proprietary lock-in."
But how did we get to the vendor lock-in issue? The root causes include:
-
Poor Architecture. Sometimes it’s just due to bad architecture and development methodologies leading vendors unable to improve products in an easy, clean and efficient way. The solution’s complexity forced them to create additional layers of complexity, even just to add a new simple functionality.
-
Trust through name recognition also plays a big role in this. It’s easier for vendors to add ‘additional functionalities’ to well-established products instead of launching a separate product. This leads to even larger and more complex software that tries to be too many things.
-
Customized for the ‘average customer’, vendors want to maximize sales so functionality is often designed for the average customer. While this is fine if your needs are average, it’s a challenge for those who need a more customized solutions or have unique business needs.
-
Lock-In as a business model. The more dependent you are, the better for vendors. Locking you into partnership can be quite lucrative. Many vendors aren’t really interested in providing you with the flexibility to easily migrate to another solution. Sadly, this has become a business model and beat competition.
The Rise of Open Source and How it Changed the IT Landscape
The rise of open-source technology in late 90s, has dramatically shifted the status quo. While it started out as technologies used by developers for some pet projects, today, there are enterprise-ready open-source alternatives to many closed-source solutions. Enterprises have options now that weren’t readily available 5-10 years ago.
While these alternatives don’t usually come as a single, ready-to-use solution requiring additional development/customization and integration, the input the community provides in growing and improving these technologies easily outweighs that.
Open-source is certainly not a “silver bullet” and doesn’t necessarily replace closed-source solutions in any business area. However, if it fits the need, it provides:
-
Flexibility and agility
-
Speed
-
Cost-effectiveness
-
Replaceability
-
Security
-
Ability to innovate
The Perfect System
In a simplified view, a system is software surrounded by data and business logic. During the lifespan of a system, data and logic are constantly changing requiring an ongoing maintenance and support.
A perfect system (in the context of our discussion) consists of small self-sufficient, self-contained and loosely coupled pieces (e.g. microservices, backend for front-end, background process, front-end application, etc.) which:
-
Can be handled by a small engineering team
-
Can be easily customized, replaced, maintained, scaled, etc.
-
Provide problem isolation
-
Are easier to oversee and understand
These small pieces should be highly customizable, replaceable, and extensible. Well implemented, open-source lightweight solutions provide these benefits enabling organizations to adapt to the ever-changing IT landscape.
Drawbacks from traditional heavyweight solutions
So how does this compare to the more traditional, heavyweight solutions, and what’s the long-term impact on your organization?
Data is one of the most important assets companies own. However, many closed-source solutions come with proprietary data structures, often hard to understand and certainly difficult to customize. This becomes a problem when you decide to migrate to another vendor as different data structures aren’t compatible with one another. Updating the data for migration is a costly endeavor. Generally, business drives the data and, no matter what solution is used, it is preferred to keep your data consistent with business.
Business logic customization is also generally limited in the closed-source solutions. To achieve as specific outcome the solution doesn’t offer, you must wait for a newer version that does or attempt to customize the solution in collaboration with the vendor’s support team—a rather difficult endeavor.
There are two main customization approaches:
-
Big Bang. In some cases, the entire legacy system is recreated and then everything is moved to the new version with the new functionality. Developers may prefer this as they won’t need to consider legacy system related constraints (well, except the data migration) and have an absolute freedom of implementation approach. In practice, however, this approach only works for smaller systems. For larger and more complex ones it is usually not applicable as the legacy system continues to grow and change while the new system is being built. While not impossible, syncing old and new worlds may become very challenging.
-
Piece by piece. This safer approach migrates small pieces over a period of time. This also helps bringing a new culture to engineering teams. First, you take a small logical part of the legacy system and create a new version in parallel. This is basically the same approach as #1, but instead of recreating and migrating the entire thing at once, you break the system down, recreate a component, and migrate them piece by piece until all is in the new system.
We always advocated for the later approach. It has multiple strong advantages:
-
It is easier and faster to build a smaller piece
-
Huge legacy system is broken down to a better/granular architecture piece by piece
-
It is easier to rollback a smaller piece then the entire legacy system replacement
-
It is easier to identify the crucial parts of the legacy system that must be rebuilt first (hitting the pain-points first)
While the ‘what’ is clear—the industry at large clearly understands that a change is needed—the ‘how’ is the real challenge. As we’ve seen, cultural challenges start at the top, from conflicts of interest to a lack of understanding the technology. Changing the culture to a change-embracing one, is no easy task. On the technology-side a lot has changed. The way software was created a decade ago vastly differs from today’s modern approaches. Applications were built as monoliths and vendors likely exploited that with further locking customers in. That was the way things were done and we all (had to) accept it.
The open source and cloud-native movement, however, changed the dynamics. It’s almost a counter-movement: modular, flexible, compatible, mix and match technologies. This new trend has empowered enterprises to reject traditional lock-in alternatives and opt for solutions that can be adapted to the uncertainty of the future. We know that transitioning isn’t easy but since technology in general has become more modular and lightweight, the cultural shift can be accommodated as quickly or slowly as the humans’ speed allows.