WSO2 API Manager vs WSO2 ESB: Should the API Manager replace the ESB?

In the `good old days’ the aim of the Enterprise Service Bus (ESB) was to integrate applications inside the organisation. Many years later, the need for integration still continues to grow, and the integration scenarios tend to exponentially diversify. These days it’s absolutely critical to be able to connect with partners and customers outside the organisation in much more flexible and agile ways. This is where API Managers (or API Gateways as offered by some vendors) are predominantly used. Does this mean that the ESB is old-fashioned and should be replaced by an API Manager? 

There used to be only one way to integrate applications: exchanging files. Then application transaction calls were added followed by asynchronous exchanges using queuing. Now, mobile, Internet of Things, and machine-to-machine (M2M) technologies introduce multiple additional scenarios using APIs and streaming integration styles. Nevertheless, even with these new scenarios, older technologies remain in use. So, when all these integration scenarios are combined, it becomes hard for developers to decide which should be used over the other or when to use both with respect to Integration Solutions (ESB) and API Managers.

So, back to the question: Should ESBs be replaced by API Managers? In most cases, it will not be an either/or situation; ESBs and API Gateway Services can coexist and complement each other. However, depending on the integration requirement and cost implications (using both technologies is expensive), one might have to be chosen over the other. Let’s take a look at some examples of real-world integration requirements and see how they should be addressed by using either the ESB or the API Manager or both. 

Scenario 1 - A loosely coupled integration solution

In an organization, there are numerous heterogeneous systems (as a result of a merger between several companies), from database applications to legacy ERP systems to cloud applications like Salesforce. There are also some SOAP web services and some recently developed REST APIs. The need is to seamlessly connect them through a centralized integration layer so that tight coupling between the applications can be avoided.  The solution should be on-premise and provide application connectors, message queueing, transformation, service orchestration and monitoring. Ability to add security and other policies to enable security, throttling etc. are useful features. What can this organization use?

Solution: ESB 

An ESB can be defined as:

- An architectural construct that provides fundamental services to complex architectures

- An entity that acts as a hub connecting many diverse systems

- A central driver that facilitates Enterprise Application Integration (EAI)

Some of these basic services of an ESB:

- Message passing, routing and filtering

- Message transformation

- Protocol conversion

- QoS enforcement (security, reliable delivery etc)

- Logging, auditing and monitoring

The WSO2 ESB can be used to integrate a large number of heterogeneous systems in an enterprise setting. It is currently used in hundreds of production deployments all around the world to connect various applications, implemented using various technologies (both open source and proprietary) running on various platforms (Windows, Linux, .NET, J2EE, LAMP, cloud). In other words, it is used as a centralized driver that facilitates EAI. The configuration model of the WSO2 ESB is so agile and powerful that practically any EAI pattern can be implemented on top of them. It has a very flexible enterprise messaging model. This is achieved through its ability to support for any wire level protocol or any message format that can be easily implemented on top of this model and can be deployed as a separate pluggable module. 

. Figure 1 - WSO2 ESB as a centralized integration layer

Figure 1 shows how the ESB can act as a centralized integration layer which contains all integration logic to connect to heterogeneous applications and systems. It can be integrated with other WSO2 middleware products such as the WSO2 Identity Server to implement security scenarios; it can publish events to the WSO2 Data Analytics Server (DAS) to perform batch, real-time, predictive and interactive analytics. 

Scenario 2 - A centrally-owned but locally-managed integration solution

An IT services provider has a closed group of clients, who have identical business interests. The clients want to be able to make use of some common integration services as well as a few others that are specific to each client to be able to communicate over domain specific standards and protocols. In order to perform information retrieval and write their own applications, they want these services to be provided by the said IT services provider. For example, a national data registry or database needs to be accessed over a custom protocol by different municipalities in a country to write their apps. Both the clients and the service provider prefer a cloud service as opposed to an on-premise solution at each site due to ease of management and to have a single point of control - for example if there are global changes or updates, they can be easily applied at a single place. (Check  [5] for a real case study on a connected government system). What can the IT services provider use?

Solution: A multi-tenanted ESB in private/public cloud

WSO2 middleware products inherently support multi-tenancy. In this example, each client’s integration needs can be configured in separate tenants in the ESB. The service provider will have to create an adapter(s) for the ESB to integrate with the data registries and create integration services for each client. 

Scenario 3 - Basic API management

An organization already has ready-to-use REST APIs and SOAP services; they don’t want to change anything (no transformations required); they want to become API providers. Some APIs will be only for internal use to enable business agility and customer engagement. Some others will be B2B APIs to optimize value chain relationships and processes. And the rest will be public APIs for access by thousands of developers across the open Web.  It is important to make it easy for API users — developers, whether internal or external to the API provider’s organization, who write applications that leverage an API — to access and understand how to use the API; to be able to secure and throttle the access of and API, and know who is using an API, typically by having them register for an API key; and ensure that API users have the support necessary to solve any problems they may have, whether that support comes from the API provider or from other API users (e.g., through community discussion forums). What does this organization need?

Solution: API Manager

,

An API manager provides developer portal frameworks with pre-built capabilities for the above mentioned requirements and more. It can enforce agreements on API usage and security. An API key is often only the first element of an API provider’s control over API use. API management products enforce usage parameters as agreed to between an API provider and API users in a variety of ways: the use of a secure sockets layer (SSL) or digital signatures for added security; the use of OAuth to allow the API provider’s customers to authorize access to their data; or quotas and rate limits for how many API calls an API user can make. 

Figure 2 - WSO2 API management platform

The WSO2 API Manager is a complete solution for designing and publishing APIs, creating and managing a developer community, and for scalably routing API traffic. It leverages proven, production-ready integration, security, and governance components from the WSO2 Enterprise Service Bus, WSO2 Identity Server, and WSO2 Governance Registry. In addition, it leverages the WSO2 Data Analytics Server for Big Data analytics, giving you instant insight into APIs behavior.

Scenario 4 - API management with simple mediation

An organization has several REST and SOAP APIs already and needs the same capabilities as mentioned in the 3rd scenario. However, the APIs are not ready-to-use and there are ‘minor’ transformations required. For example, a mobile application can only send and receive JSON messages but the backend APIs/services process only XML. Should the organization use the ESB instead of the API Manager or should it use both?

Solution: API Manager

Apart from routing, enforcing security and throttling, the WSO2 API Manager's gateway includes in-built minor payload and protocol transformation capabilities. In fact, the API Manager's Gateway is a basic stripped-down version of the WSO2 ESB and is not intended to be used as a fully-fledged ESB. For minor transformations, dependance on the ESB can be reduced or avoided altogether. For this scenario the API Manager would suffice. 

Scenario 5 - API management with complex integrations

There is a mix of APIs, services and applications (including home-grown applications). In addition to the ready-made APIs and services, a service/API layer must be introduced to access the applications. Some applications communicate over proprietary protocols while some others support standard protocols. Service chaining/orchestration may also need to take place. All services must be exposed centrally to external and internal parties. Management and monitoring of all these APIs are absolutely necessary. Obviously this is far more complex than the previous scenario; so should both the API Manager and ESB be used in this case?

Solution: API Facade Pattern (The combination of the API Manager and the ESB)

API facade is an interface between API consumers (application developers) and the backend systems of the organization. API facade would hide the complexity of the backend services from the application developers and provide a unified layer that developers can use for their applications. 

Because of the heterogeneous nature and the complexity of the backend services, they cannot be exposed directly as APIs to application developers. These backend services require a certain level of mediation; this highlights the need to have a separate mediation layer behind the facade layer. Coupling these two layers increase the complexity in the facade layer, create architectural and scalability limitations, and would require a significant level of mediation logic to be outside the firewall if the APIs are to be exposed to external developers. The solution to this problem is the use of the API Facade pattern.

Figure 3 - API facade pattern 

Implementation of the API facade pattern can be done by using the WSO2 API Manager to build a facade layer and the WSO2 ESB to build the mediation layer (add in products like the WSO2 Data Analytics Server for analytics) and connect to the existing services/applications. 

Other Frequently Implemented Patterns with the API Manager

WSO2’s open source platform provides many opportunities to extend API management. Aside from being the only fully open source solution, the unique feature for WSO2 is the ability to extend API management features . 

Scenario 6 - External key manager

An organization already has an OAuth2 authorization server which issues and validates OAuth2 tokens and want to use that instead of the in-built Key Manager of the WSO2 API Manager. Can this be done?

Absolutely! With the revamped architecture of the API Manager (having started from version 1.9.0), all integration points with the key manager are made to be extensible, thereby enabling the plugging of an external OAuth2 authorization server. See [7] for more information. 

Scenario 7 - External API gateway

What if an organization already has an existing API gateway or ESB component (non-WSO2), which handles token generation and validation on its own or in combination with a Key Manager, and what they are looking for are API store and publisher portals, and their API management capabilities (such as versioning, lifecycle management, notifications etc.). Can they still use the WSO2 API Manager?

This pattern can be implemented using WSO2 Governance Registry 5.0.0 [8].  The Governance Registry can provide store and publisher interfaces. It can be used to write executors to push API definitions to the external gateway (in the required format) once an API’s lifecycle state changes from ‘created’ to ‘published’ state in the publisher (and removed/disabled when the state changes to ‘deleted’). 

Resources - 

[1] - https://www.forrester.com/TechRadar+Integration+Technologies+Q2+2015/fulltext/-/E-RES116227

[2] - http://techfeast-hiranya.blogspot.com/2012/09/busting-synapse-and-wso2-esb-myths.html

[3] - http://wso2.com/whitepapers/the-evolution-of-Integration-a-comprehensive-platform-for-a-connected-business/

[4] - http://wso2.com/blogs/architecture/2013/05/a-pragmatic-approach-to-the-api-faade-pattern/

[5] - http://wso2.com/whitepapers/connected-government-cloud-enabling-public-services/

[6] - http://wso2.com/library/blog-post/2014/10/blog-post-revamping-wso2-api-manager-key-management-architecture-around-open-standards/

[7] - https://docs.wso2.com/display/AM191/Configuring+a+Third-Party+Key+Manager

[8] - http://wso2.com/products/governance-registry/