Welcome!

IBM Cloud Authors: Yeshim Deniz, Elizabeth White, Pat Romanski, Liz McMillan, Stefan Bernbo

Related Topics: Java IoT, IBM Cloud, Machine Learning , Apache

Java IoT: Article

WebSphere Datapower FTP Polling Functionality Use Cases

FTP Polling, sFTP over SSL & sFTP over SSH

2- Create a FTP Poller Front side handler with the following values

Input

Values

Name

FTPserviceFTPPoller

Target Directory

ftp://172.20.5.247:21/get

Delay Between Polls

10000

Input File Match Pattern

.*(\.xml$|.XML$)

Delete Input File on Success

on

Delete file on processing error

on

Generate Result File

off

3- Create a new XML Manager (front end) with the following values

Input

Values

Name

FTPserviceXMLManager

 

4- Create a new User Agent Configuration (front end) with the following values

Input

Values

Name

FTPserviceUserAgent

 

 

5- Click on “Basic-Auth Policy” tab menu and add ftp server’s username and password

 

6- Click on “Apply” until you reached to the MPGW screen. Click on “Apply” at the Multi-Protocol Gateway screen

7- Create the Ftpserver.xsl file and type the following. Upload the file on the Datapower

<?xml version="1.0" encoding="UTF-8"?>

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"

xmlns:dp="http://www.datapower.com/extensions"

xmlns:regexp="http://exslt.org/regular-expressions" extension-element-prefixes="dp regexp"

exclude-result-prefixes="dp regexp" version="1.0">

<xsl:template match="/">

<xsl:variable name="FrontsideFTP" select="dp:variable('var://service/URL-in')"/>

<xsl:variable name="Filename" select="regexp:match($FrontsideFTP, '([^\/]*)$')"/>

<xsl:variable name="Filename2" select="substring-before($Filename,'?Rename')"/>

<xsl:variable name="BacksideFTP" select="concat('ftp://172.20.5.247:21/put/',$Filename2)"/>

<dp:set-variable name="'var://service/routing-url'" value="$BacksideFTP"/>

<xsl:message dp:priority="debug">OId URL: <xsl:value-of select="$FrontsideFTP"/></xsl:message>

<xsl:message dp:priority="debug">Filename: <xsl:value-of select="$Filename"/></xsl:message>

<xsl:message dp:priority="debug">Filename2: <xsl:value-of select="$Filename2"/></xsl:message>

<xsl:message dp:priority="debug">New URL: <xsl:value-of select="$BacksideFTP"/></xsl:message>

<xsl:copy-of select ="*"/>

</xsl:template>

</xsl:stylesheet>

 

8- On FTPserviceMPGW, create a new Policy and named it “FTPservice2Policy”

9- Create a new rule and configure Match action icon for “* ” for URL

10- On the same rule, assign action icon Transform and assign the “FTPserver.xsl” file

 

11- Click “Apply Policy” and then “Close Window”. Click on “Apply” at the Multi-Protocol Gateway screen

12- On the “FTPserverMPGW”, create a new XML Manager (backend) with the following values

Input

Values

Name

FTPserviceBEXMLManager

13- Create a new User Agent Configuration ( backend) with the following values

Input

Values

Name

FTPServerBEUserAgent

 

 

14- Click on “Basic-Auth Policy” tab menu and add ftp server’s username and password

15- Click on “Apply” until you reached to the MPGW screen. Click on “Apply” at the Multi-Protocol Gateway screen

16- The overall FTPserviceMPGW should look like this

 

TESTING:

1- On the FTP server, drop the few XML files in the “/get” folder

 

2- After 10 seconds the files should be moved in “/put” folder of the FTP SERVER

3- And the GET folder should be empty

 

 

2- FTP Poller (Forward Request to another Service)

In this scenario, the Datapower device performs the following tasks:

a- Polling files from FTP Server and forward them to another service

b- A loopback XML firewall service is created for the demonstration

 

PROCESS:

Perform the following steps:

1- On the existing FTPServiceMPGW create a new Policy and named it “FTPservicePolicy”

2- Create a new rule with Match ALL action icon

3- Drop a Route action icon and set the value to the loopback XML firewall http://192.168.137.99:3033

4- Click on “Apply Policy” and close the window.

5- Apply the settings on the MPGW

 

TESTING:

1- For testing, enabled the probe on the loopback XML Firewall, i.e. “loopbackFW”

2- Drop a XML file in the “/get” folder on the FTP Server

3- After 10 seconds, you should see the request on the probe

4- Disable the FTPserviceFTPPoller.

 

3- FTP Poller (sFTP – FTP over SSL)

In this scenario, the Datapower device performs the following tasks

a- Polling files from FTP Server over SSL (sFTP – FTP over SSL, it is not SSH) and forward them to another service

b- A loopback XML firewall service is created for the demonstration

PROCESS:

In order to understand the sFTP – FTP over SSL, first we have to see the configuration on a standard FTP client. For the demonstration, a FileZilla ftp client is used

STANDARD FTP CLIENT SETTINGS:

1- On the FileZilla ftp client, create a new connection with the following settings

Input

Values

Host

<ftp hostname>

Port

21

Protocol

FTP – File Transfer Protocol

Encryption

Require explicit FTP over TLS

User

<ftp username>

Password

<ftp username’s password>

 

 

2- Click on “Transfer Settings” tab menu and select the following settings

Input

Values

Transfer mode

Passive

 

 

3- Click on “Connect”.

4- When connecting first time, the Filezilla will show FTP Server’s certificate. Accept it.

5- From the FTP logs, you should see that FileZilla is connected the FTP server over SSL

 

DATAPOWER SETTINGS:

Perform the following steps on Datapower to connect to sFTP (FTP over SSL) Server

1- First create a SSL Proxy Profile with

a. SSL direction Forward (client)

b. Configure a Crypto Validation Credentials

c. Configure a Crypto Certificate with the FTP server’s certificate

 

2- On the existing FTPServiceMPGW create a new FTP Poller Front Side handler “FTPService_SSLexp_FTPPoller” with the same values

 

3- Create a new XML Manager with the following values

Input

Values

Name

FTPservice_SSLexp_XMLManager

 

4- Create a new User Agent Configuration with the following values

Input

Values

Name

FTPService_SSLexp_UserAgent

 

5- Click on “Basic-Auth Policy” tab menu and add ftp server’s username and password

6- Click on “SSL Proxy Profile” tab menu and assign the FTPServerSSLProxyProfile

7- Click on “FTP Client Policy” tab menu and assign the following values

8- Click on “Apply” until you reached to the MPGW screen. Click on “Apply” at the Multi-Protocol Gateway screen

 

TESTING:

1- For testing, enabled the probe on the loopback XML Firewall, i.e. “loopbackFW”

2- Drop a XML file in the “/get” folder on the FTP Server

3- After 10 seconds, you should see the request on the probe

4- In the Datapower logs, you should see the SSL handshake with Datapower and the FTP server

5- Disable the FTPservice_SSLexp_FTPPoller.


4- FTP Poller (sFTP – FTP over SSH)

In this scenario, the Datapower device performs the following tasks

a- Polling files from FTP Server over SSL (sFTP – FTP over SSH) and forward them to another service

b- A loopback XML firewall service is created for the demonstration

 

PROCESS:

In order to understand the sFTP – FTP over SSH, first we have to see the configuration on a standard FTP client. For the demonstration, a FileZilla ftp client is used

STANDARD FTP CLIENT SETTINGS:

1- On the FileZilla ftp client, create a new connection with the following settings

Input

Values

Host

<ftp hostname>

Port

22

Protocol

SFTP – SSH File Transfer Protocol

Logon Type

Normal

User

<ftp username>

Password

<ftp username’s password>

 

 

2- Click on “Transfer Settings” tab menu and select the following settings

Input

Values

Transfer mode

Passive

 

 

3- Click on “Connect”.

4- Validate the security Warning from the SSH Server. Accept it.

 

5- From the FTP logs, you should see that FileZilla is connected the FTP server over SSL

DATAPOWER SETTINGS:

Perform the following steps on Datapower to connect to sFTP (FTP over SSH) Server

1- On the existing FTPServiceMPGW create a new FTP Poller Front Side handler “sFTPservicesFTPPoller” with the same values with the exception of Target Directory

Input

Values

Target Directory

Sftp://172.20.5.247/get

 

 

2- Create SSL Client Connection with the following values

Input

Values

User name

<ftp username>

Profile Usage

SFTP

User Authentication

Password

Password

<ftp username’s password>

 

3- The overall FTPserviceMPGW should look like this

 

TESTING:

1- For testing, enabled the probe on the loopback XML Firewall, i.e. “loopbackFW”

2- Drop a XML file in the “/get” folder on the FTP Server

3- After 10 seconds, you should see the request on the probe

4- In the Datapower logs, you should see the SSH Connection between Datapower and the sFTP/SSH server

 

CONCLUSION:

Datapower offers strong integration capabilities in using the FTP Polling protocol. This is accomplished with FTP Poller FSH, SFTP Poller over SSL and SFTP Poller over SSH.

When combining these capabilities with custom XSLT stylesheets, there is an endless realm of possibilities.

 

RESOURCES:

In this article the Wing FTP Server (www.wftpserver.com) was used.

The following FTP ports were enabled

 

 

More Stories By Asim Saddal

Asim Saddal works in the Middleware (WebSphere Application Server, WebSphere Datapower, WebSphere Process Server, WebSphere VE) practice of IBM Software Services for WebSphere.

IoT & Smart Cities Stories
@DevOpsSummit at Cloud Expo, taking place November 12-13 in New York City, NY, is co-located with 22nd international CloudEXPO | first international DXWorldEXPO and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time t...
CloudEXPO New York 2018, colocated with DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City and will bring together Cloud Computing, FinTech and Blockchain, Digital Transformation, Big Data, Internet of Things, DevOps, AI, Machine Learning and WebRTC to one location.
DXWorldEXPO | CloudEXPO are the world's most influential, independent events where Cloud Computing was coined and where technology buyers and vendors meet to experience and discuss the big picture of Digital Transformation and all of the strategies, tactics, and tools they need to realize their goals. Sponsors of DXWorldEXPO | CloudEXPO benefit from unmatched branding, profile building and lead generation opportunities.
In this Women in Technology Power Panel at 15th Cloud Expo, moderated by Anne Plese, Senior Consultant, Cloud Product Marketing at Verizon Enterprise, Esmeralda Swartz, CMO at MetraTech; Evelyn de Souza, Data Privacy and Compliance Strategy Leader at Cisco Systems; Seema Jethani, Director of Product Management at Basho Technologies; Victoria Livschitz, CEO of Qubell Inc.; Anne Hungate, Senior Director of Software Quality at DIRECTV, discussed what path they took to find their spot within the tec...
The deluge of IoT sensor data collected from connected devices and the powerful AI required to make that data actionable are giving rise to a hybrid ecosystem in which cloud, on-prem and edge processes become interweaved. Attendees will learn how emerging composable infrastructure solutions deliver the adaptive architecture needed to manage this new data reality. Machine learning algorithms can better anticipate data storms and automate resources to support surges, including fully scalable GPU-c...
Disruption, Innovation, Artificial Intelligence and Machine Learning, Leadership and Management hear these words all day every day... lofty goals but how do we make it real? Add to that, that simply put, people don't like change. But what if we could implement and utilize these enterprise tools in a fast and "Non-Disruptive" way, enabling us to glean insights about our business, identify and reduce exposure, risk and liability, and secure business continuity?
Nicolas Fierro is CEO of MIMIR Blockchain Solutions. He is a programmer, technologist, and operations dev who has worked with Ethereum and blockchain since 2014. His knowledge in blockchain dates to when he performed dev ops services to the Ethereum Foundation as one the privileged few developers to work with the original core team in Switzerland.
DXWorldEXPO LLC announced today that Telecom Reseller has been named "Media Sponsor" of CloudEXPO | DXWorldEXPO 2018 New York, which will take place on November 11-13, 2018 in New York City, NY. Telecom Reseller reports on Unified Communications, UCaaS, BPaaS for enterprise and SMBs. They report extensively on both customer premises based solutions such as IP-PBX as well as cloud based and hosted platforms.
"Akvelon is a software development company and we also provide consultancy services to folks who are looking to scale or accelerate their engineering roadmaps," explained Jeremiah Mothersell, Marketing Manager at Akvelon, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"Space Monkey by Vivent Smart Home is a product that is a distributed cloud-based edge storage network. Vivent Smart Home, our parent company, is a smart home provider that places a lot of hard drives across homes in North America," explained JT Olds, Director of Engineering, and Brandon Crowfeather, Product Manager, at Vivint Smart Home, in this SYS-CON.tv interview at @ThingsExpo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.