Event Driven Architecture for Scalable Integration


Video Transcription

Um Hello everybody. So a little bit worried about myself. Uh So I am an enterprise architecture um And I'm working for Firmans. I'm helping my organization to have the firm footing in this era of digital transformation.So I started my career like 16 years ago with uh as ac developer, I moved on to C++ Java, I moved on to new technologies and started all kind of architectural work which I begin to really enjoy, you know, to understand the nitty gritty of design and I'm still loving it. And I really thank women tech network to give me this opportunity to share my passion with you. OK. So let's begin. Uh what I'm going to talk about today is uh the even driven architecture. So um when we talk about even driven architecture, it's nothing new, you know, it has been started, just started to have a lot of traction these days and lots and lots of companies are now adapting uh even of an architecture because the benefits are are visible, you know, they are showing up and more and more tools are available to support this kind of architecture.

So it is important to have this concept very clear. And uh before I detail out Ed A, um I would uh like to focus on why integration architecture is so important. Then we deep dive into eaed A how we differ from. So A and uh Ds and do so why integration architecture? So if you can look at picture, if you don't architect uh your integration successfully, you will end up into something like this, you know, with this fast moving world and everything with agile methodology, we often focus more on, you know, delivering more and more of business value fast and we want applications to be developed quickly with unique look and feel, you know, excellent user experience and having great functionalities.

Um but we often lose sight of the bigger picture, right? So we neglect the future possibilities. Uh What if we have to expand the solution? Um What if we have to integrate it with available solutions? What if you know you have to, how will my day the flow? Uh what are the new da data points that can be leveraged in future? And every how everything works together or fit together in an ecosystem? We have to think about all this from the beginning. And if you uh if the site is lost, then at one point in time, you will have to redevelop your application because scaling the application at a particular after a particular point in time would not be possible. So you really have to sit and think about your data and information, how it is flowing and how it is mutating around your applications, you know, and enterprise architecture will help you to define is the principles defined how to take care of your moving data, how to put on governance, how to integrate applications, how to reduce inconsistencies and so on.

So, um so you really have to think about all these architectural, all three spa points before you start developing a solution and integration architecture has also evolved over time. So when we started uh the journey of creating we, you know, when once upon a time we started building applications, uh an application was a concept where you know, we put everything together together in one monolithic application. Uh And then we keep on, we kept on adding features over features and in terms of infrastructure also we keep here over hardware. And then at a certain, at a certain point in time, there came a point where you cannot expand your application further otherwise it will collapse.

So what um we soon realized that what we build is really has to be scalable and you know, we really have to think differently. Then suddenly somebody has an aha moment and said, oh why am I dealing with this huge piece? Why don't I break it into small pieces? You know, if I break, break it into small pieces, maybe it will be easier for me to manage to carry it right. So then came the concept of so a which is service oriented architecture. So we divide the problems into uh problem domains and each domain has a separate app, you call it application or a system or a service. And it was all looking fine, you know, these small modular things, they are working, they're easier to maintain. But now there was a different problem because all these were solutions in itself and they have to communicate with each other. So what did they do? They started talking to each other and with each system exchanging data with another, there was this whole spagettii you know, all system needs to understand each other's language protocols. The technologies have to be assigned et cetera. So there were a lot of other problems that came into picture and now we need to make change.

Uh If we need to make change to one system, we are breaking a lot of more things, you know, it is a nightmare to fix something because um uh if you touch anything, there are so many dependencies around it, you have to figure out what else you will break. So uh in terms of architecture, it was a scalable architecture, but it was a nightmare in terms of an integration architecture. Then somebody thought, oh but I have another solution. Why don't we put like a service bus and this service bus was a revolution. It is still being used to a very, very large extent and it's a very, very nice uh uh and good integration pattern. So uh the idea was why don't we use a service bus? Uh We put the service bus in the middle and all the complexity of integrations is handled by it. So we add remove exchange systems as and when we please, there is no problem. There is no dependency that each and every system individually has to maintain. All the transformation was supposed to be handled by this uh service bus and you can integrate system with different technologies. Also, you don't need to align on protocols and technology. So it almost seems perfect and you know, um we thought it is like this bridge which looks so perfect uh which will bridge the gap and nobody will drown.

Um But uh in reality, what happened is that every bridge had its own shape and form and, and there were different problems associated with it. So sometimes it was not placed correctly, sometimes, you know, you build a bridge just to um connect to monoliths, sometimes uh it was overloaded. Some times there are flexibilities issue. Sometimes you can't just extend your bridge to uh to expand to further integration, you know, so there are all different kind of problems that came to the different kind of implementation of the service bus. So although it is working, it is still working in the industry a lot.

It is not as perfect as it seemed to be. And then came some um um some contemporary ideas around. So a uh with microservices, which is still a lot of buzzing uh technology wise, um I in terms of having these uh separate systems broken down further into uh into services and communicating. So uh we have broken these uh these uh so a with these individual systems further into small microservices. But in the long term, we are actually not reaching our integration vision, which we need to be, right? So, so microservice is, is is a very debatable concept. We I will not go uh or spend too much time into it. But uh the idea is that so a is all about domain logic being separated into system and exposing it as a services, you use whatever protocol. So press et cetera but basically it's it's client driven by client here. I mean the front end, you know, there is a request to response interaction. So you request something and you need a response and then you wait, unless you get your response, your resources are blocked and now you're yy, you know, uh until you get a response, you don't know what happened to your request, whether it is successful, whether it is not successful.

So there are various kind of frameworks you have to put around it in order to just handle this request response scenarios. So now we are saying that OK, the integration is working fine but we still don't have you know this, this uh this sense of that my integration is perfect then um came in a life philosophy, you know, that, that the life is a series of events and sensations and everything else is just interpretation.

So, you know, uh what, what, what we mean by this is, you know, something or, or many things keep on happening around us and all we do is react so we have control over a few things and that is all we should bother about rest, everything keeps moving, changing. And you know, we try to engage with knowing what is happening around us, but we don't need to be bothered by everything. And similarly, no one else should be bothered by what I am doing internally. So basically the philosophy of live and let live. And if the same philosophy has been applied in integration and and this is called an ED A. So Ed A is basically an architectural uh pattern and it is based on the core concept that all the data and information around us is changing. And there are times when the request response model will not work as things are changing too fast and you cannot wait for the response every time. So all we can do is understand in time now how things are changing and replay the same thing on our side if you are interested. So you know, everything else around us is just good to know. And uh mm every event I know about all the events that is happening around me. But it is my choice whether I react to that event or not.

So nobody is forcing me to give a response, neither I am or neither the calling system is waiting for a response from anybody. So Ed A is everything which is just revolving around events. So we have to understand first what is an event in true sense. So event represents a fact, it represents an information. It is just like, you know, you give somebody oh you know fy I something like that. So event is kind of a self contained, unique and a time relevant object event can propose a problem, an opportunity, a threshold or a deviation. And um uh what do you do? You can invoke a service or trigger a business process or you can have another um uh event in response to an event, but it is your choice. You don't really have to do it. You can just discard the event. And one very important concept here is that events are not necessarily a message, message contains data. Event is just an information. So event is something that has, that has happened. It depends on how event is relevant to you. A lot of people get confused with, you know, messages and events. You know, there are several patterns floating around. There is message pattern, there is even pattern, there's command pattern. So the difference between this basically lies in the intent.

While message has no special intent, it is data even to inform you about something what has happened and it has happened in the past that is very important to understand. Whereas command is something that triggers something. So it will tell you what you are supposed to do in future, right? So that is basically how you decide which pattern to use when. So and and one more thing that I just want to clarify here is messaging is not about events, events just represent one possible messaging pattern. So now um you know that we have understood um what is Ed A and what is events? Uh let's uh deep dive into like uh the Ed A architecture. So Ed architecture is very simple and very clean and that's why it's the best integration uh architecture around. It basically consists of an event store and this event store is linked to event publisher and subscriber.

So there are many publishers, what do publishers do? They publish their information into their event store? And what do subscriber do they listen to this event? And they say I have subscribed to this event to listen to e listen to it and you don't have to worry about what I do with this event. So just like it is just like in the social media that, you know, I post something on, on Facebook, I want people to know about it. But you know, I don't care who is seeing my post um uh or who is not, you know, if somebody acknowledges if somebody comments good, but I can't force them to act and same for whoever is reading. You know, you have all this information but you're not forced to do anything if it interests you. Um uh you engage with it, if not, your resources are not blocked. Another example that I really, really like to give is the newspaper or, you know, any new site. So I know that every morning I will get something at my doorstep. You know, I have made a routine that I will make uh take a look at the important news to just make myself aware.

It's just events happening around me, then it depends on what I'm interested in today. If I'm looking for a job, I will go to the job portal and probably get engaged there. If, if I, I am, I'm looking to get married, I will go and look at the matrimony, right? But otherwise I'm not concerned with it. Let it be. But I know that is happening. Plus uh what is also uh good with events is that I store these stacks of information, right? And so at one point in time, if somebody tells me, oh, but have you seen that newspaper on that day? There was an important thing there, I can still go back and replay and I can look at that information, right? So uh uh what is good about Ed A is I'm getting real time notification. It comes to my doorstep. I don't need to go and fetch it is fire and forget. And if consumer is interested, they take an immediate action. So this kind of setup, it helps us to better support the endless demand of business, right? Because there's no point to point communication. Um We choose what to subscribe to. We choose what to do with that information and we choose also the priority.

As I told you, maybe we want to collect all this information and replace it just once a month because I'm not necessarily interested with that information, but I still want to keep a watch, right? So with all these things now, uh there are, there are several questions raising in our mind, you know, uh we say that, you know, I is should we not use request response model uh anymore or should we stop using batch process or should we, you know, build a synchronous batch process or asynchronous batch process?

Or should the batch process be fired and forget there can be any number of combinations, you know, the synchronous call calls can be blocking, asynchronous calls can be blocking as well. Um Should I use API S uh is ep api even processing? And my last question is uh why do we have voice mails, uh voice mails if we um have a mobile phone? And if you have understood the answer to this question, you have understood the concept of Edf for sure. So um to answer these questions, we just want to look at uh these different three styles of interactions which happens during interact uh during integration. So first is the time driven wherein uh uh let's take example of a payment system. So there is a scheduled job which runs in the payment system and runs uh every scheduled um our seconds, days, weeks, months. So it is scheduled and it is time bound and it is time driven. Second model is request driven. So I ask system to do something, system response. Um One system ask another systems to do something and you respond, the U I asked back in to do something and the backend response and so on and so forth. So it is client. Uh so for uh in in context of payment system, you can say I want to withdraw X amount. Whereas what even driven will do is even driven will just give you information.

So it will say, OK, the payment uh has run um mm uh last time at this time stamp, it's run previously at this. This, this this time X amount have been withdrawn one hour before. And hey, you, you, you know your system is less on uh balance. This is the analytics which is provided on events. So good thing about this is you can analyze your event and then you can build intelligence on top of your event. So that is why with lot of suggestions, recommendations even driven architecture is really really gaining popularity. All right.

So um uh we should keep these styles of interaction in mind in order to decide what we want to use. Right? Also, apart from that, we also have to ask ourselves uh relevant questions. So you know, in order to choose an integration pattern, what about uh your data is your data static? Um So if you know, client makes a request to the service and expect a response um And uh you know, it is great for the persistent static data. So if your data is static and it has to be persistent, perfect you go for. So a but it, it gets a little hard when the data keeps changing around you, you have to, you know, either uh build a mechanism to pull these uh data changes, you have to detect these changes. In that case, it would be better to go for a um a subscri subscription model, right? Which uh which pushes even to the consumers. Um Do you care what happens to uh what happens after your actions? So if you don't care uh what someone is doing after your actions, Ed A is best, you know, the producer and the consumers don't have to engage producers have no knowledge of uh you know what consumers are there or what they are doing with their data.

There is still, you know, a minimal uh uh coupling in terms of uh name of the cues and topics and even formats, but it is extremely lightweight. Right? Third is what about availability? You know, can you guarantee availability of your service in soa a service must be available at the time of the request. Um You know, if a client make a request service has to be available. If even if you're doing an asynchronous response handling, your service still needs to be available. Events do not require a reply. So inherently, it is asynchronous events can be persisted for future consumption. So you don't have to ensure that all your systems are connected to Ed A uh uh in an Ed A fashions are available at all point of time in all point of time. And that basically helps you to also um uh make use of your infrastructure of your cloud infrastructure. You still want to go to uh you know, a time driven fashion but still on an event where you stole or your event and your uh your server comes up once a day, does all its action on its events information.

So you don't have to engage your infrastructure throughout uh the life cycle as well. Then the next question you should ask yourself is what about flexibility? You know, uh the processing logic in the request response API is hardwired to the service at point. So you know whether uh whether your service uh I is with or without service discovery, you know, if the logic needs to be changed or extended, um you have to change the systems, you know, the definition of the co has to change, the service must be up to date updated. So in it introduces like change management and regression, like risk and all those things. Whereas in ed a additional events, uh producers and consumers can be added to a system at any time without any explicit process definition. So you have like immense flexibility in that in that context, right? And um what about consistency? So synchronous communication makes it a bit easier. I am talking synchronous communication makes it a bit easier to have a consistent state. And this is where Ed A really have to, you have to make a choice. Are you OK with eventual consistency? Because Ed A does not offer you um consistency by default, it is eventually consistent but not now. So there is also saying that something can be either eventual intelligent or eventual consistent. So soa A is um uh so a is immediate consistent but eventual intelligent. Um Ed A is immediate intelligent but uh but eventually consistent, right?

So um um now with uh uh for every integration, you have to think about these questions, right? And uh see whether it is Ed A or son and what, what is our um verdict on that, that both are valid architecture, right? But um we have to understand that they complement each other and not discard each other. They both have to fit in the overall architecture on the need basis. And you know, for each integration you build, you have to ask yourself all those questions about scalability, flexibility, whether your data is static or dynamic and what kind of consistency you're looking for, et cetera, et cetera, right? So uh mm uh again, you have to build your overall integration architecture, keeping all these things in mind. But uh the thing is that you have to think whether you need so a if you don't need so a probably Ed A is your choice because Ed A enhances your your ability to, you know, be more modular to have a number of generators and consumers, you can um add and remove it as as you please and so on.

So it brings this whole asynchronous nature into picture so you can get rid of all the linear flows. It also, you know, it empowers your application to react rather than being commanded. And uh you can, you can really, really program how do you want to react to an event and it empowers applications to uh mm mm to really build their separate logic uh in uh to and uh to and scale. So all these machine learning quantum computing logic, you can actually segregate it from your main logic and you can just have, you know, the real intelligence plugged into your application. It is extremely valuable for a lot of cross cutting concerns like logging, monitoring, event processing and so on.

And it provides a capability to rebuild and replay the series of event. And um, and do you understand that, how much do we gain from here? That means you can build any state at any point in time? That is we are persisting the state of a data object. If you understand, in terms of object oriented programming, it is very, very difficult to understand the state of an object. And here with Ed A, we are actually preserving the entire state. So it gives you immense power of rebuilding a system from scratch or from whichever point to whichever point at any point in time. So, you know, uh Ed A is good choice to go if uh if your application does not demand. So a so now we understood that Ed A is a good way to go and we are all excited and we say, OK, we'll build this perfect Ed A system and you know, what we wanted was uh a perfectly choreographed performance and what we ended up implementing is, is no different than a spaghetti.

You know, we have all the events scattered everywhere and everyone is doing whatever they want with them. So idea is that although there is an autonomy, there should still be some discipline and to bring that discipline, we have to start thinking, you know, in event first. So it is called event first. Thinking it is an approach where uh we do not think of request response model but rely on the events. The best example here I can give you is uh the uh Amazon call service. So I'm sure all of us must have called a call center at any point in time. So what do we do? We pick up our phone, we call the call center and we are listening this nice music. Um um it, it appears nice for the first few seconds. Then it is really horrible. So we are hanging on with the call and um it goes for hours and hours. We we eventually shut it off. We try again. But if you have ever called Amazon uh call center, what do they do? Uh They will tell you, ok, your position in the queue is so and so and you will receive a call from us in, in one minute, two minute. So what have they done? They have reacted to an event, ok. They reacted to an event that I called and what did it, what did it bring it bring an amazing user experience? So what have, what has happened is really my responsibility. Responsibility have been shifted. So I'm no longer responsible to chase them, you know.

Um um meanwhile, they are receiving a call from me and they can also analyze that. Ok. So and so person is calling, let's look at the order, what what could be wrong? So, they can also prepare themselves to answer my call and prepare themselves to offer me something, which will further help me. So, um this is how the whole event first thinking can shift your approach to your solutions. So start with that, start with uh you know, event, first thinking and uh start thinking about your business processes and context of event. And you know, there are lots and lots of things to uh take care of here. But I'm just giving you a few warnings and the few mistakes that people generally do. So, I'm listing down some uh some uh do s and don't here. So don't think of your uh first event as the only event. When you build something, you should build something as we only keeping in mind, event, don't think about what data is passed in the event, what what uh customer will do uh with the event and so on. You have to build a holistic picture. So you have to correlate and interconnect your event across multiple layers of event architecture and not just start with one event and start building your, you have to have the business processes aligned with the event and so on.

Also don't make even driven your uh uh dominant programming model. So if everything goes even driven, then uh uh what will happen to request response. And most of the U I BA is based on request response. So you have to basically use everything on the platter, you have to use API S, you have to use ETLS, you have to use whatever fits your business solution. So don't only think even so, again, we go too much use of anything is again, not good. Uh Also do not start thinking technology first. So a lot of uh uh I have seen a lot of industries, a lot of companies that people start saying, oh K AFC A is coming up as a technology. What can we do with K AFC A? You know, all let's implement something with K AFC A. No, that's, that's the wrong thinking. You have to first think of what you want to build, what is um uh what are the events? What do you want to exchange? And then so the best technology you have to plan your events which can span across multiple clouds and technologies and not stick to a technology, right? And also don't confuse event architecture with event patterns. There are many events floating around, there is command query pattern, there is command query responsibility pattern, there are messages, uh messaging patterns so on and so forth. So the architecture is different and the patterns you use within that architecture is different.

So there is a clear distinction and do not get confused by uh creating an architecture around it. And the most important thing is that you cannot go ahead with the ED A strategy um by being in silos. So you cannot have one application building their own ed A another application building it because Ed A is the whole um concept which work in unison. So you have to make a uniform um Ed A strategy, right? And um uh that's also just if I have to recap um we have understood here that uh you know, integration architecture is really, really important, you know, it has evolved slowly but it has become the most talked about architecture for success. And do think about your integration architecture along with your application architecture, we have understood the difference between, so A Ed A, we have also understood that they work um um they complement each other. Um We have also understood that event is not a message. Uh We understood what Ed A brings to the table, both in terms of good and bad. You can't uh uh you know, if you're looking for consistency of data across the organization and you propose Ed A model that is going to be a disaster, right? And uh uh we also discussed that how to go about building your uh Ed A strategy. So um uh do you know um everything works in unison, use all the patterns available.

And um um um Ed A is a great way to start when you are building an um an enterprise which is not only talking to itself but talking, you know, talking to everything outside like IOT devices. Like right now you have uh more and more applications coming around your work place, giving you uh you know, these kinds of notifications saying OK, don't start for office today because you know there is a traffic here and there all these applications are built on the event driven architecture.

There are many, many solutions coming up. There is solace, there is confident which will help you to start building your ed A uh very quickly. So um thank you. Um that's all I have. Thank you so much for listening to me and investing your time here. I wish you all encounter very happy events in future. Ok? Um If there are any questions, I can probably take few. If not, then uh then thanks a lot people for listening to me and I am looking um uh forward to the rest of the talks. Thank you guys. Thank you very much.