Enterprise Resource Planning

ERP Journal on Ulitzer

Subscribe to ERP Journal on Ulitzer: eMailAlertsEmail Alerts newslettersWeekly Newsletters
Get ERP Journal on Ulitzer: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn

ERP Journal Authors: Progress Blog, Automic Blog, Janakiram MSV, Louis Nauges, Jason Bloomberg

Related Topics: Java EE Journal, ERP Journal on Ulitzer

J2EE Journal: Article

Liquid Data: XQuery-Based Enterprise Information Integration

Liquid Data: XQuery-Based Enterprise Information Integration

Modern enterprises are drowning in a sea of information. Despite owning an ever-increasing volume of information, most enterprises cannot exploit it to even a fraction of its full potential.

This is because the information is strewn across many systems with diverse data formats and interfaces - systems that are largely unaware of one another and of the relationships their content has with information contained elsewhere. Therein lies the Enterprise Information Integration (EII) problem: enterprises need the ability to easily access information about a given business entity, such as a customer or an order, from a varied and distributed collection of information sources.

The EII problem isn't new - the database world has been struggling with it for over two decades - but today's business trends bring a new urgency to finding a solution. Luckily, we are also at a point today where an alignment of emerging technologies is finally enabling a solution. This article looks at why this is so and how BEA's Liquid Data for WebLogic exploits this alignment to offer a commercial EII solution today.

An Example Integration Scenario
Picture a large telecommunications vendor with a Wireless Division and a Broadband Division. Each division has its own order management system and its own customer support system. The vendor also has a centralized CRM system that maintains information about customer purchasing histories and product promotions. A closer look at the vendor's IT environment might reveal the following mix of back-end systems:

Wireless Division:
Order Management System: SAP ERP using Oracle as the RDBMS
Customer Support System: Clarify using Oracle as the RDBMS

Broadband Division:
Order Management System: PeopleSoft ERP using Sybase as the RDBMS
Customer Support System: Homegrown system with SQL Server as the RDBMS

Corporate CRM System:
Siebel using DB2 as the RDBMS

As you can see, customer-related information is spread across five different information systems. (In real situations, the overall number of systems in an enterprise's IT environment would likely be closer to the 10-100 range.) These systems use a mix of technologies from different independent software vendors, and the five systems are unaware of one another. Some allow direct database access, while others allow information access only through their business-level APIs.

The vendor wants to improve its customer responsiveness and operating efficiency by giving customers up-to-the-minute access to all of their account information via several channels - a phone tree system, live telephone customer support representatives, the customer service desk at its sales outlets, and a self-service Web portal. The vendor also hopes to boost sales by adding recommendation features to all four channels. To accomplish this, the vendor wants to create a comprehensive 360º "single view of customer" that can be shared across the channels. To meet the up-to-the-minute goal, the view's data must be obtained directly from the vendor's operational systems. This is a clear instance of the EII problem!

Current EII "Solutions"
To solve this problem, application developers face three basic engineering challenges:

  • The Metadata Challenge: The metadata describing the information contained in different back-end systems is captured in various forms, including RDBMS system catalogs, WSDL files, application views (for J2EE-CA adaptors), and XML Schemas.
  • The Data Access API Challenge: Information contained in different back-end systems must be accessed using the appropriate protocols, such as JDBC/SQL, ODBC/SQL, OCI/SQL (for Oracle data), SOAP, JCA, etc.
  • The Data Model Challenge: The basic format of the data returned from a given back-end system depends on the nature of that system. Possible formats include relational rowsets, Java objects, and XML documents.

    Table 1 lists four common approaches to tackling the EII problem today and briefly analyzes their main strengths and weaknesses. None of these approaches offers a full EII solution.

    Liquid Data: An XML-Based EII Solution
    In November of 2002, BEA introduced Liquid Data for WebLogic. Liquid Data provides real-time access and aggregation of data from disparate data sources, improving visibility for front-office applications. "Real-time" refers here to using the most up-to-date information available. Unlike a data warehouse, which houses a preintegrated copy of selected enterprise data that is days or weeks old, Liquid Data provides access to the current state of the enterprise by directly integrating and providing access to the data in the enterprise's various operational systems. Liquid Data's focus is on data visibility, or integrated read access. BEA's WebLogic Integration complements Liquid Data for applications that require update support. It targets front-office applications, which typically require fast access to an integrated view of one or a few business entities at a time. Our 360º view-of-customer scenario is a classic front-office application.

    Liquid Data distinguishes itself from past EII approaches by taking an XML data integration approach. It combines many of the strengths of the EAI and relational data integration approaches without inheriting their weaknesses. Liquid Data provides an XML standards-based view of all enterprise data - enterprise data sources appear either as virtual XML documents or as a collection of functions that take XML parameters and produce XML results. The W3C XML Schema standard is the enterprise data model, and the W3C XML Query (XQuery) draft standard is the declarative language for specifying integrated reusable views and queries over data distributed across the enterprise. XQuery functions model the functional data sources. EAI-style adaptors are used to "XML-ify" packaged applications that don't natively expose XML APIs. Using XML rather than flat tables makes Liquid Data well suited to handling information from complex data sources, and using XML rather than a proprietary EAI object model positions Liquid Data to interact with applications that support standards-based XML and Web service APIs.

    The Liquid Data Architecture
    Figure 1 depicts the architecture of Liquid Data. At the bottom are the various flavors of data sources supported out of the box. Liquid Data natively supports JDBC/SQL sources (Oracle, DB2, Sybase, SQL Server, and PointBase), Web services (both simple RPC-style and complex document-style), XML data files, and application views of packaged or legacy applications (SAP, PeopleSoft, etc.) using BEA WebLogic Integration adaptors. For JDBC/SQL sources, Liquid Data automatically introspects their relational catalogs to produce default XML views of their content. In the case of Web services, it automatically introspects their WSDL descriptions. In addition to these sources, Liquid Data supports custom Java functions to enable developers to write XML plug-ins for data sources that are not natively supported by Liquid Data. Data views (discussed later), once defined, can also be used as Liquid Data data sources.

    The top of Figure 1 shows applications making data query calls involving data views. The Liquid Data world has the following key players and corresponding responsibilities:

  • System administrator: Configures the data sources for the Liquid Data Server.
  • Data architect: Subject matter expert who understands the relevant business entities and data sources. Defines one or more layers of reusable integrated views (data views) of the business entities of interest. Defines stored queries on top of the data views.
  • Application developer: Calls stored queries, which are usually parameterized, from within application code.

    For the system administrator, Liquid Data extends BEA WebLogic Server's Web-based console. For the data architect, Liquid Data provides a tool, DataView Builder, for use in graphically defining data views and stored queries. Liquid Data focuses on parameterized stored queries because they provide a way to simplify application developers' lives while simultaneously protecting the enterprise's operational data stores from ad hoc query traffic. For the application developer, Liquid Data offers an EJB API, a JSP tag library API, and a Web services API for invoking stored queries.

    At the center of Figure 1 is a distributed query execution engine. This engine processes XML queries by calling the underlying data sources, doing the required data transformations and computations, and returning integrated XML results to the application. Caching is also supported. The data architect can indicate that a given query's results may be cached for reuse when the same query or view is reaccessed (with identical parameters) within a specified time period. Not shown in Figure 1, but important, is that Liquid Data supports security policies on stored queries, data views, and base data sources. The Liquid Data server has been implemented as a J2EE application that can be deployed on a BEA WebLogic Server cluster.

    XQuery: Liquid Data's Secret Sauce
    Liquid Data uses XQuery, which is rapidly gaining broad industry support as the declarative language for defining integrated views of an enterprise's many data sources. We give a brief example here to illustrate how an XML query can stitch together data from multiple sources. The XQuery language is detailed at www.w3.org/TR/xquery; Liquid Data's use of XQuery is covered in http://edocs.bea.com/liquiddata/docs10/index.html. The most heavily used XQuery expression in Liquid Data is the FLWR expression, which is analogous to SELECT-FROM-WHERE queries in SQL. A FLWR expression has the following parts:

  • A FOR clause that generates one or more value sequences, binding the values to query variables. The FOR clause in XQuery is similar to the FROM clause in SQL.
  • A LET clause that binds a temporary variable to the result of an expression. The LET clause is similar to the provision of temporary views in some vendors' dialects of SQL.
  • A WHERE clause that contains Boolean predicates that act to restrict the FOR clause's variable bindings. The WHERE clause in XQuery is directly analogous to the WHERE clause in SQL.
  • A RETURN clause that specifies the query's desired XML output. The XQuery RETURN clause is roughly analogous to the SELECT clause in SQL, though the structures that it can specify are a good deal richer than those of SQL.

    Consider our initial EII scenario. Suppose we want to get contact information for a given Wireless Division customer together with information about their orders for Broadband Division products. Listing 1 shows how XQuery can do this. The topmost FOR clause binds a variable to each wireless customer. The LET clause that follows concatenates the customer's first and last names for output formatting. The WHERE clause filters customers by only keeping those whose ID matches a query parameter. Finally, the large RETURN clause specifies the query's output, which consists of customer data from the Wireless Division's database plus the result of a nested query. The nested query pulls order data from the Broadband Division database; its WHERE clause ensures that only the orders for the customer of interest are queried, and its RETURN clause retrieves the desired order information. Listing 2 shows a sample of this query's output.

    DataView Builder UI
    To minimize the need for the data architect to write XQuery views and stored queries by hand, Liquid Data includes a tool called DataView Builder. DataView Builder supports a pictorial, drag-and-drop, mapping-based approach to query design and construction for XQuery. Figure 2 shows an example of the design view, where input data sources are mapped to target data views to graphically construct an XQuery query. DataView Builder also provides a test-view window where the query can be inspected, run, and optionally hand-tuned.

    Programmatic APIs
    Liquid Data provides three APIs that application developers can use to call Liquid Data queries from within application programs. They are:

  • EJB API: This provides a Stateless Session Bean (SLSB) interface to the Liquid Data server. It is a JDBC-like interface that has methods to execute either stored queries (the norm) or ad hoc queries, with or without query parameters, against the integrated enterprise data known to the Liquid Data server.
  • JSP Tag Library: For JSP and portal developers, Liquid Data provides a JSP tag library facade over its SLSB interface. Each SLSB method has a counterpart in the tag library.
  • Web Service Interface: Developers can ask Liquid Data to convert stored queries into Web services. A corresponding WSDL file is produced for each such stored query. Any application that talks to Web services can utilize Liquid Data services through this API.

    Other Features
    The features of LiquidData are too numerous to be covered here. For more detailed information, we encourage you to browse the online documentation at http://edocs.bea.com/liquiddata/docs10/index.html and download and try the product itself, but we will highlight a few of the key additional features here.

    Administration Console
    For system management, Liquid Data extends WebLogic Server's existing administration console. New console-supported activities include:

  • Managing basic Liquid Data server settings
  • Configuring new Liquid Data data sources
  • Configuring the Liquid Data caching subsystem
  • Defining Liquid Data security policies for data sources and data views
  • Monitoring the activity of the Liquid Data server

    Liquid Data includes support for security policies and supports access control lists (ACLs) for each stored query. Liquid Data's ideas of users and groups are identical to those of BEA WebLogic Server. For ad hoc queries, Liquid Data supports ACLs at the data source level as well.

    To manage EII assets, Liquid Data uses a lightweight, server-based repository to store source and target XML schemas, stored queries, WSDL files, code and metadata for custom functions, and so on. The repository is file-based and provides a simple notion of folders.

    Product Combinations
    Liquid Data is an integral part of the overall BEA Platform family. It can be used in combination with any of the existing BEA products. Interesting BEA product combinations include:

  • Liquid Data plus WebLogic Portal: Portal handles Web presentation and personalization while Liquid Data handles enterprise data integration and aggregation.
  • Liquid Data plus WebLogic Integration Adaptors: This enables Liquid Data to integrate data from packaged applications together with relational and Web service data.
  • Liquid Data plus WebLogic Integration: Liquid Data gives workflows an integrated view of enterprise data while the workflows support acting on (e.g., updating) the data.
  • Liquid Data plus WebLogic Workshop: Liquid Data provides an integrated view of enterprise data to Web service-based applications.

    This article has presented a brief technical overview of BEA Liquid Data for WebLogic, which takes a fresh, XML-oriented approach to attacking the enterprise information integration problem. It provides a virtual XML document-based view of the enterprise's disparate data sources, and a declarative XQuery-based paradigm is used to create integrated views of the enterprise's key business entities and to define stored queries over those views.


  • Landers, T., and Rosenberg, R., "An Overview of Multibase," Proceedings of the 2nd International Symposium on Distributed Data Bases, Berlin, Germany. North-Holland Publishing Co., September 1982.
  • "BEA Liquid Data for Weblogic: Increasing Visibility in the Distributed Enterprise," BEA Systems White Paper, October 2002.
  • XQuery 1.0: An XML Query Language. www.w3.org/TR/xquery/. W3C Working Draft, August 2002.
  • Curbera, F., et al. "Unraveling the Web Services Web: An Introduction to SOAP, WSDL, and UDDI," IEEE Internet Computing 6(2), March-April 2002.
  • Liquid Data for WebLogic Documentation: http://edocs.bea.com/liquiddata/docs10/index.html
  • Liquid Data for WebLogic Download: http://commerce.bea.com/downloads/liquid_data.jsp
  • Comments (0)

    Share your thoughts on this story.

    Add your comment
    You must be signed in to add a comment. Sign-in | Register

    In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.