We are all well aware of how integration patterns are used for applications: applications composed of multiple services are integrated together using a variety of patterns. However, there is another paradigm that is a requirement for many organizations, known as data integration. This has happened especially during the last decade, when the generation and availability of data has been incredibly high. The velocity, variety, and volume of data being generated has increased drastically, and there is data almost everywhere.
Every organization has many different types of applications, and they all generate data in their own proprietary format. Often, data is also purchased from the marketplace. Even during mergers and amalgamations of organizations, data needs to be migrated and combined.
Data integration refers to the process of bringing data from multiple sources and generating new output that has more meaning and usability.
There is a need for data integration in the following scenarios:
- For migrating data from a source or group of sources to target destinations. There are a variety of reasons for doing so.
- With the rapid availability of data, organizations want insights and to create data warehouses. To build these data warehouses, data should be in a format that is consumable by data warehousing tools.
- For generating real-time dashboards and reports.
- For creating analytics solutions.
Application integration has a runtime behavior when users are consuming the application; for example, in the case of credit card validation and integration. On the other hand, data integration happens as a backend exercise and is not directly linked to user activity.