file transfer – Thys Michels Blog

MQ FTE V7.0.2: The Enterprise File Bridge

File Transfer Scenarios with WebSphere Datapower

Below is some scenarios for using Datapower S/FTP frontside handler/poller to solve enterprise file transfer requirements.

Scenario 1: Enterprise customers want to bridge legacy S/FTP-based messaging flows with newer HTTP Web Services.

The following tasks are performed by Datapower:

1. A file is retrieved by polling a remote FTP Server

2. Routes the file content to an enterprise HTTP Server for processing

3. Captures the response by the HTTP Server application and places a response file on the remote FTP Server. This is done by creating a predetermined naming pattern.

Scenario 2: Datapower extends the ESB to include an FTP-based client requiring additional security

The following task are performed by Datapower:

1. Datapower retrieves file by polling a remote SFTP server

2. It passes binary file through to an enterprise Websphere MQ Queue

3. And captures the response from WebSphere MQ system and places the response file on the remote SFTP sever. This is done by creating a predetermined naming pattern.

Scenario 3: Datapower protects existing enterprise FTP servers.

The following tasks are performed by Datapower:

1. Datapower access FTP connection from remote FTP Clients which allow client to connect to a inbound FTP server transparently and securely.

2. Authentication using username and password is done from incoming FTP Client.

3. Datapower supports the streaming of large file through the appliance.

Scenario 4: Datapower provides additional security for SFTP-based message exchange patterns

The following tasks are performed by Datapower:

1. Datapower acts as an SFTP server to remote client connections.

2. Datapower presents a virtual files system that exists only on the device, proxing the actual back end servers.

3. Datapower dynamically routes files to the desired location on the back end SFTP Server

4. Datapower performs public key authentication on the inbound FTP Clients.

MQ File Transfer Edition Error and Trace Handling

MQ File Transfer Edition Error and Trace Handling

The following are the trace levels that can be specified:

1. off

2. flow

3. moderate

4. verbose

5. all

Step 1: Set Agent Trace Level

Run the following command on Agent AGT1WIN, select your specific trace level:

fteSetAgentTraceLevel -traceLevel all AGT1WIN

Step 2: View Trace file

Navigate to: *\WMQFTE\config\QM_COORD1\agents\AGT1WIN\logs

Optional is to enable trace only for specific criteria:

Only enable trace for a specific transfer:

fteCreateTransfer -trace “com.ibm.wmqfte=all” -sa AGT1WIN -da AGT1LIN_XXX -dm QM_AGENTHOST2 -sd leave -t binary -de overwrite -dd “/tmp” “C:\student\*.jpg”

View the new trace by navigating to: *\WMQFTE\config\ QM_COORD1\agents\AGT1WIN\logs

Disable trace by setting trace level to off:

fteSetAgentTraceLevel -traceLevel off AGT1WIN

Setup Websphere MQ File Transfer Ant Script using Eclipse

Setup MQ File Transfer with Ant

Prerequistes confirm that an agent has been created and is running.

Step 1: Open Websphere MQ Explorer

Select Window à Preferences

Select the WebSphere MQ Explorer tab.

Select the radio button next to “in an Eclipse Workbench” under Startup. This will enable the WebSphere MQ Explorer to be used for other perspectives, such as Resource, Java, and the debug perspective.

Step 2: Now click on General Editors File Associations.

Step 3: Add New File Type

Press Default button

Press OK

Step 4: Change MQ Explorer Perspective -> Resource

Step 5: Modify Ant Runtime environment

Navigate to Windows -> Perference menu item

Expand the Ant folder.

Select the Runtime folder.

Select the Global Entries folder.

Press the Add External JAR’s… button.

Navigate to Navigate to *\WMQFTE\lib folder and add all the jars.

Step 6: Set Ant Home

(Follow on Step 5)Click the Ant Home button and navigate to *\WMQFTE\ant

Click OK to close the Preference window

Step 7: Import Ant Project

Select File->New->Other->Java Program from Exisiting Ant Build File

Step 8: Execute Ant File

<?xml version=’1.0′?>

<description> Simple FTE Transfer </description>

<!– set global properties for this FTE JOB –>

<fte:uuid length=”8″ property=”jobNumber” />

</target>

<fte:filecopy cmdqm=”${cmdqm}”

src=”${SNODE}” dst=”${DNODE}”

outcome=”await”

idproperty=”uuid.property_1″

jobname=”${jobName}”

rcproperty=”step1.result”>

<fte:metadata>

<fte:entry name=”departId” value=”${departId}” />

<fte:entry name=”jobNumber” value=”${jobNumber}” />

</fte:metadata>

<fte:filespec srcfilespec=”${srcfile}” dstfile=”${dstfile}” overwrite=”true”/>

</fte:filecopy>

</target>

<not>

</not>

</condition>

<fail if=”step1-failed”

message=”step1 transfer failed MAXRC=${step1.result}.”>

</fail>

</target>

</project>

Step 9: Run Ant File

Right click on ant xml file and select Run As -> Ant Build

An sucessful transfer output:

Setup Websphere MQ File Transfer Database Logger

Step 1. Run the following db2 command:

— drop if necessary and then create FTELOGDB

echo *******************************************************

echo the following command may fail because the database did

echo NOT previously exist, this is an acceptable error

echo *******************************************************

drop database FTELOGDB

echo — Creating database FTELOGDB

create database FTELOGDB

echo — Connect to FTELOGDB database

connect to FTELOGDB user esbuser using wesb4soa

— catalog FTELOGDB as a odbc data source

catalog system odbc data source FTELOGDB

list system odbc data sources

echo — now create the schemae, tables and views

echo — use this db2cmd command console

echo — it is already connected to the database

Step 2: Create the DB2 Schema Tables

Navigate to: C:\Program Files\IBM\WMQFTE\tools\sql

Execute the following command:

db2 –v -t –f ftelog_tables_db2.sql

Step 3: Configure the Database Logger Properties File

Navigate to: *\WMQFTE\config

Open file in Notepad: databaselogger.properties

Edit the following values:

wmqfte.queue.manager=QM_COORD1

(Coordinator Queue Manager Name)

wmqfte.database.name=FTELOGDB

(Database Name created in Step 1)

wmqfte.database.driver=D:/Applications/IBM/SQLLIB/java/db2jcc.jar;D:/Applications/IBM/SQLLIB/java/db2jcc_license_cu.jar

(DB2 Driver Path)

wmqfte.database.native.library.path=C:/Applications/IBM/SQLLIB/lib

(Library Path to DB2)

Step 4: Setup the transaction manager

Open MQ Explorer right click on the Coordinator Queue Manager and select Properties. Under XA Resource Manager click the ‘Add’ button.

Enter FTELOGDB as the Name.

Enter jdbcdb2.dll as the SwitchFile.

Enter db=FTELOGDB,uid=db2admin,pwd=db8admin,toc=p,tpm=mq as the XAOpenString.

Make sure the ThreadOfControl is set to Process.

Press the OK button to add the XA resource manager to the queue manager.

Click OK

Step 5: Copy the jdbcdb2.jar to the existing folder

Navigate to *\IBM\WebSphere MQ

execute the following command to copy the jdbcdb2.jar file: copy java\lib\jdbc\jdbcdb2.dll exits

Step 6: Execute a Controlled Stop of the Coordinator Queue Manager

Step 7: Start the Database Logger

Execute the following command in *\MQFTE\tools\bin: fteStartDatabaseLogger –F

Step 8: Run a new Transfer

Navigate to ‘Manage File Transfer’ and connect the Coordinator Queue Manager and right click on Transfer Log and select ‘New Transfer’

Select a Agent, file to transfer and Click Next till you see the following window:

Add two metadata attribute value pairs:

Click Next and Finish to start file transfer

Step 9: Open DB2 Control Panel to View the Database Logger Table

Navigate to your Database Logger Database -> Views -> Basic_Transfer_Details.

Navigate to your Database Logger Database -> Tables -> Metadata.

Optional Stop Database Logger:

Navigate to *\MQFTE\tools\bin\fteStopDatabaseLogger.cmd

Database Logger will stop succesfully else see result of error.

WebSphere MQ File Transfer Edition – Definition File

FTE Definition File Transfer

Transfer definition files enable the creation of complex transfer requests. The transfer definition file is a XML file that describes the source and destination directories or files or datasets.

Step 1: Create a Definition Transfer File (below is an example):

<?xml version=”1.0″ encoding=”UTF-8″?>

<transferSpecifications

xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”

xsi:noNamespaceSchemaLocation=”FileTransfer.xsd”>

<file>E:\student\*.txt</file>

</source>

</destination>

</item>

<file>E:\student\*.zip</file>

</source>

</destination>

</item>

</transferSpecifications>

Step 2: fteCreateTransfer -sa AGT1WIN -da AGT1LIN -dm QM_AGENTHOST2 -td E:\student\GroupTransferSpecification.xml

– sa is the source agent name

-da is the destination agent name

-dm is the destination Queue Manager

-td is the XML Definition File

Install and Configure Websphere MQ File Transfer Edition

First Configuration of Websphere MQ File Transfer Edition:

Step 1: Create a Coordination Queue Manager in Websphere MQ Explorer, in my case my coordination Queue Manager is called FTE_COORD

Step 2: Create the Command Queue Manager

Step 3: Start Websphere MQ FTE Installation

Complete all the information for the FTE_COORD Manager in the installation window:

Complete all the information for the FTE_COMMAND Manager in the installation window:

Step 4: After installation you will receive the following message:

C:\Documents and Settings\All Users\Application Data\IBM\WMQFTE\config\FTE_COORD\FTE_COORD.mqsc has been created.
Ensure this file has been run against the coordination queue manager FTE_COORD before attempting file transfers.This can be done by redirecting the contents of the file FTE_COORD.mqsc into a runmqsc session.

Navigate to the path above in the command window and enter the following command:

C:\Documents and Settings\All Users\Application Data\IBM\WMQFTE\config\FTE_COORD>runmqsc FTE_COORD < FTE_COORD.mqsc

The mqsc script will create the FTE System Queues, Topics, namelists,

DEFINE TOPIC(‘SYSTEM.FTE’) TOPICSTR(‘SYSTEM.FTE’) REPLACEALTER TOPIC(‘SYSTEM.FTE’) NPMSGDLV(ALLAVAIL) PMSGDLV(ALLAVAIL)DEFINE QLOCAL(SYSTEM.FTE) LIKE(SYSTEM.BROKER.DEFAULT.STREAM) REPLACEALTER QLOCAL(SYSTEM.FTE) DESCR(‘Stream for WMQFTE Pub/Sub interface’)* Altering namelist: SYSTEM.QPUBSUB.QUEUE.NAMELIST* Value prior to alteration:DISPLAY NAMELIST(SYSTEM.QPUBSUB.QUEUE.NAMELIST)ALTER NAMELIST(SYSTEM.QPUBSUB.QUEUE.NAMELIST) + NAMES(SYSTEM.BROKER.DEFAULT.STREAM+ ,SYSTEM.BROKER.ADMIN.STREAM,SYSTEM.FTE)* Altering PSMODE. Value prior to alteration:DISPLAY QMGR PSMODEALTER QMGR PSMODE(ENABLED)DEFINE QLOCAL(SYSTEM.FTE.DATABASELOGGER.REJECT) + DESCR(‘Messages rejected by the FTE database logger.’) + DEFPRTY(0) + DEFSOPT(SHARED) + GET(ENABLED) + MAXDEPTH(999999999) + MAXMSGL(4194304) + MSGDLVSQ(PRIORITY) + PUT(ENABLED) + RETINTVL(999999999) + SHARE + NOTRIGGER + USAGE(NORMAL) + REPLACEDEFINE QLOCAL(SYSTEM.FTE.DATABASELOGGER.COMMAND) + DESCR(‘Command messages to control the FTE database logger.’) + DEFPRTY(0) + DEFSOPT(SHARED) + GET(ENABLED) + MAXDEPTH(999999999) + MAXMSGL(4194304) + MSGDLVSQ(PRIORITY) + PUT(ENABLED) + RETINTVL(5000) + SHARE + NOTRIGGER + USAGE(NORMAL) + REPLACE

Step 5: Verify that the following system Queues has been created: