Datapower XB60 Large File Size Handling

Datapower has become a proxy to receive large files from customers but also to send large file to enterprise back end. These files are processed, transformed, validated, authenticated, authorized and routed.

If we are handling large file we need to make sure the auxilary storage on the appliance is configured correctly: http://publib.boulder.ibm.com/infocenter/wsdatap/v3r8m1/index.jsp?topic=/xb60/administratorsguide210.htm as well as how to manage files on the appliance: http://publib.boulder.ibm.com/infocenter/wsdatap/v3r8m1/index.jsp?topic=/xb60/administratorsguide190.htm

Using the XB60 for received files (payment files/invoices files/transaction files) it is better to create a Multi protocol gateway to handle the transfer of the and the B2B gateway to process, transform, validate, authenticate, authorize and route the file content.

The reason for that is the Datapower XB60 appliance has an Internal hard drives that are 140GB of usable space, 70 GB of AES encrypted space for B2B payloads, should you choose to store them on the device, and 70GB for the metadata store. Below is some options to consider when handling large files through an Datapower appliance:

Option 1: File larger than 100 MB cannot be sent through the B2B Gateway as there is not streaming functionality to stream a file. File Streaming functionality is only available in the Multi Protocol Gateway. So the solution is to enable streaming on a Multi Protocol Gateway in the return action point the file to the B2B gateway for processing. The files can be stored off device to either a NFS mount point or an ISCSI Drive subsystem.

Option 2: Best practice for large files is to pass files into a downstream system for processing and use the XB60 as just a gateway. This system can be an dedicated FTP server or WebSphere MQ FTE to securely and efficiently handle the transfer of the file into the intranet to be processed by the Datapower XB60. Datapower XB60 can also call a WebSphere MQ FTE agent to transfer a file from a source to an destination using an XML Transfer file.

Option 3: Chucking files into smaller sizes and sent to Datapower for processing. This can be done using MQ FTE, see example link below: http://publib.boulder.ibm.com/infocenter/wmqfte/v7r0/index.jsp?topic=/com.ibm.wmqfte.doc/f2m_split_length.htm also split a file into same length message and PUT to Websphere MQ Queue.

Option 4: Proving the size of the document and the average TPS per appliance add more appliance to handle the file size and TPS requirements. Before choosing which option is the best for your organization first go through the Document and Metadata exercise below and replace the values with your organizations current or estimated values:

Document Storage on Datapower XB60 Exercise: For illustrative purposes, assume 10,000 EDI messages per day with an average size of 10 KB. Therefore, you are expecting 100 MB of content per day. This capacity requires 360 MB of document storage per day, which equates to approximately 84 days. This corresponds to:

• 200 MB of stored data for these messages. 2 copies of each message — one copy for “off the wire” and one for the processed request.
• 40 MB (4 KB per message) for protocol responses
• 120 MB for the raw MDN (4 KB per message), HTTP request containing MDN (4 KB per message), and HTTP response containing MDN (4 KB per message)

Metadata Storage on Datapower XB60 Exercise: If the messages per day are split evenly between inbound and outbound and that every outbound message requests an asynchronous MDN, the space for metadata storage adds up to 15,000 transactions per day, which equates to 60 MB (15000 * 4 KB) of metadata storage per day. At the default 1 GB persistence store size, this will fill in approximately 16 days. If the persistence store is increased to 64 GB, over 1,000 days of message metadata can be persisted. Using the default persistence store size in this example, setting the “Archive Document Age” property to 7 days will remove document metadata from the persistence store before it fills. See the following article on Capacity planning for Datapower XB60 Appliance: http://www-01.ibm.com/support/docview.wss?uid=swg21329746