Today’s industry standards of Service Oriented Architecture (SOA) and microservices split data into “”micro-databases””. Domains are hard to fracture and many enterprises face the challenge of “”joining”” data across databases typically resulting in cross-service chatter for certain types of workloads. This chatter scales with request and data volumes, and number of domains. Industry solutions such as event logs and monolithic databases have varied shortcomings such as data duplication, scalability and the lack of a rich query language to interact with data across services.
In response to this problem, we present our design of a scale-out, just-in-time data virtualization solution using PostgreSQL and Foreign Data Wrappers. Our design addresses most shortcomings of the common industry standard solutions while providing fault-tolerant high-speed data access across database boundaries. We also share some 9.6/pg_fdw tips and tricks to deal with certain limitations, different data access patterns and a demo of our implementation with real-world industry data & results.