Often, the express purpose of a web page is to deliver data to a webbot, another website, or a stand-alone desktop application. These web pages aren’t concerned about how people will read them in a browser. Rather, they are optimized for efficiency and ease of use by other computer programs. For example, you might need to design a web page that provides real-time sales information from an e-commerce site.
Today, the eXtensible Markup Language (XML) is considered the de facto standard for transferring online data. XML describes data by wrapping it in HTML-like tags. For example, consider the sample sales data from an e-commerce site, shown in Table 29-1.
Table 29-1. Sample Sales Information
Brand | Style | Color | Size | Price |
|---|---|---|---|---|
Gordon LLC | Cotton T | Red | XXL | 19.95 |
Ava St | Girlie T | Blue | S | 19.95 |
When converted to XML, the data in Table 29-1 looks like Example 29-7.
Example 29-7. An XML version of the data in Table 29-1
<ORDER>
<SHIRT>
<BRAND>Gordon LLC</BRAND>
<STYLE>Cotton T</STYLE >
<COLOR>Red</COLOR>
<SIZE>XXL</SIZE>
<PRICE>19.95</PRICE>
</SHIRT>
<SHIRT>
<BRAND>Ava St</BRAND>
<STYLE>Girlie T</STYLE >
<COLOR>Blue</COLOR>
<SIZE>S</SIZE>
<PRICE>19.95</PRICE>
</SHIRT>
</ORDER>XML presents data in a format that is not only easy to parse, but, in some applications, it may also tell the client computer what to do with the data. The actual tags used to describe the data are not terribly important, as long as the XML server and client agree to their meaning. The script in Example 29-8 downloads and parses the XML represented in the previous listing.
Example 29-8. A script that parses XML data
# Include libraries
include("LIB_http.php");
include("LIB_parse.php");
# Download the order
$url = "http://www.WebbotsSpidersScreenScrapers.com/29_7.php";
$download = http_get($url, "");
# Parse the orders
$order_array = return_between($download ['FILE'], "<ORDER>", "</ORDER>", $type=EXCL);
# Parse shirts from order array
$shirts = parse_array($order_array, $open_tag="<SHIRT>", $close_tag="</SHIRT>");
for($xx=0; $xx<count($shirts); $xx++)
{
$brand[$xx] = return_between($shirts[$xx], "<BRAND>", "</BRAND>", $type=EXCL);
$color[$xx] = return_between($shirts[$xx], "<COLOR>", "</COLOR>", $type=EXCL);
$size[$xx] = return_between($shirts[$xx], "<SIZE>", "</SIZE>", $type=EXCL);
$price[$xx] = return_between($shirts[$xx], "<PRICE>", "</PRICE>", $type=EXCL);
}
# Echo data to validate the download and parse
for($xx=0; $xx<count($color); $xx++)
echo "BRAND=".$brand[$xx]."<br>
COLOR=".$color[$xx]."<br>
SIZE=".$size[$xx]."<br>
PRICE=".$price[$xx]."<hr>";As useful as XML is, it suffers from overhead because it delivers much more protocol than data. While this isn’t important with small amounts of XML, the problem of overhead grows along with the size of the XML file. For example, it may take a 30KB XML file to present 10KB of data. Excess overhead needlessly consumes bandwidth and CPU cycles, and it can become expensive on extremely popular websites. In order to reduce overhead, you may consider designing lightweight interfaces. Lightweight interfaces deliver data more efficiently by presenting data in variables or arrays that can be used directly by the webbot. Granted, this is only possible when you define both the web page delivering the data and the client interpreting the data.
Before we explore proper methods for passing data to webbots, let’s explore what can happen if your design doesn’t take the proper security measures. For example, consider the order data from Table 29-1, reformatted as variable/value pairs, as shown in Example 29-9.
Example 29-9. Data sample available at http://www.WebbotsSpidersScreenScrapers.com/29_9.php
$brand[0]="Gordon LLC"; $style[0]="Cotton T"; $color[0]="red"; $size[0]="XXL"; $price[0]=19.95; $brand[1]="Ava LLC"; $style[0]="Girlie T"; $color[1]="blue"; $size[1]="S"; $price[1]=19.95;
The webbot receiving this data could convert this string directly into variables with PHP’s eval() function, as shown in Example 29-10.
Example 29-10. Incorrectly interpreting variable/value pairs
# Include libraries
include("LIB_http.php");
$url = "http://www.WebbotsSpidersScreenScrapers.com/29_9.php";
$download = http_get($url, "");
# Convert string received into variables
eval($download['FILE']);
# Show imported variables and values
for($xx=0; $xx<count($color); $xx++)
echo "BRAND=".$brand[$xx]."<br>
COLOR=".$color[$xx]."<br>
SIZE=".$size[$xx]."<br>
PRICE=".$price[$xx]."<hr>";While this seems very efficient, there is a severe security problem associated with this technique. The eval() function, which interprets the variable settings in Example 29-10, is also capable of interpreting any PHP command. This opens the door for malicious code that can run directly on your webbot!
An improvement on the previous example would verify that only data variables are interpreted by the webbot. We can accomplish this by slightly modifying the variable/value pairs sent to the webbot (shown in Example 29-11) and adjusting how the webbot processes the data (shown in Example 29-12). Example 29-11 shows a new lightweight test interface that will deliver information directly in variables for use by a webbot.
Example 29-11. Data sample used by the script in Example 29-12
brand[0]="Gordon LLC"; style[0]="Cotton T"; color[0]="red"; size[0]="XXL"; price[0]=19.95; brand[1]="Ava LLC"; style[0]="Girlie T"; color[1]="blue"; size[1]="S"; price[1]=19.95;
The script in Example 29-12 shows how the lightweight interface in Example 29-11 is interpreted.
Example 29-12. A safe method for directly transferring values from a website to a webbot
# Get http library
include("LIB_http.php");
# Define and download lightweight test interface
$url = "http://www.WebbotsSpidersScreenScrapers.com/29_11.php";
$download = http_get($url, "");
# Convert the received lines into array elements
$raw_vars_array = explode(";", $download['FILE']);
# Convert each of the array elements into a variable declaration
for($xx=0; $xx<count($raw_vars_array)-1; $xx++)
{
list($variable, $value)=explode("=", $raw_vars_array[$xx]);
$eval_string="$".trim($variable)."="."\"".trim($value)."\"".";";
eval($eval_string);
}
# Echo imported variables
for($xx=0; $xx<count($color); $xx++)
{
echo "BRAND=".$brand[$xx]."<br>
COLOR=".$color[$xx]."<br>
SIZE=".$size[$xx]."<br>
PRICE=".$price[$xx]."<hr>";
}The technique shown in Figure 29-12 safely imports the variable/data pairs from Example 29-11 because the eval() command is explicitly directed to only set a variable to a value and not to execute arbitrary code.
This lightweight interface actually has another advantage over XML, in that the data does not have to appear in any particular order. For example, if you rearranged the data in Example 29-11, the webbot would still interpret it correctly. The same could not be said for the XML data. And while the protocol is slightly less platform independent than XML, most computer programs are still capable of interpreting the data, as done in the example PHP script in Example 29-12.
No discussion of machine-readable interfaces is complete without mentioning the Simple Object Access Protocol (SOAP). SOAP is designed to pass instructions and data between specific types of web pages (known as web services) and scripts run by webbots, webservers, or desktop applications. SOAP is the successor of earlier protocols that make remote application calls, like Remote Procedure Call (RPC), Distributed Component Object Model (DCOM), and Common Object Request Broker Architecture (CORBA).
SOAP is a web protocol that uses HTTP and XML as the primary protocols for passing data between computers. In addition, SOAP also provides a layer (or two) of abstraction between the functions that make the request and receive the data. In contrast to XML, where the client needs to make a fetch and parse the results, SOAP facilitates functions that (appear to) directly execute functions on remote services, which return data in easy-to-use variables. An example of a SOAP call is shown in Example 29-13.
In typical SOAP calls, the SOAP interface and client are created and the parameters describing requested web services are passed in an array. With SOAP, using a web service is much like calling a local function.
If you’d like to experiment with SOAP, consider creating a free account at Amazon Web Services. Amazon provides SOAP interfaces that allow you to access large volumes of data at both Amazon and Alexa, a web-monitoring service (http://www.alexa.com). Along with Amazon Web Services, you should also review the PHP-specific Amazon SOAP tutorial at Dev Shed, a PHP developers’ site (http://www.devshed.com).
PHP 5 has built-in support for SOAP. If you’re using PHP 4, however, you will need to use the appropriate PHP Extension and Application Repository (PEAR, http://www.pear.php.net) libraries, included in most PHP distributions. The PHP 5 SOAP client is faster than the PEAR libraries, because SOAP support in PHP 5 is compiled into the language; otherwise both versions are identical.
Example 29-13. A SOAP call
include("inc/PEAR/SOAP"); // Import SOAP client
# Define the request
$params = array(
'manufacturer' => "XYZ CORP",
'mode' => 'development',
'sort' => '+product',
'type' => 'heavy',
'userkey' => $ACCESS_KEY
)
# Create the SOAP object
$WSDL = new SOAP_WSDL($ADDRESS_OF_SOAP_INTERFACE);
# Instantiate the SOAP client
$client = $WSDL->getProxy();
# Make the request
$result_array = $client->SomeGenericSOAPRequest($params);SOAP interfaces to web services provide a common protocol for requesting and receiving data. This means that web services running on one operating system can communicate with a variety of computers, tablets, or cell phones using any operating system, as long as they have a SOAP client.
SOAP is a very heavy interface. Unlike the interfaces explored earlier, SOAP requires many layers of protocols. In traffic-heavy applications, all this overhead can result in sluggish performance. SOAP applications can also suffer from a steep learning curve, especially for developers accustomed to lighter data interfaces. That being said, SOAP and web services are the standard for exchanging online data, and SOAP instructions are something all webbot developers should know how to use. The best way to learn SOAP is to use it. In that respect, if you’d like to explore SOAP further, you should read the previously mentioned Dev Shed tutorial on using PHP to access the Amazon SOAP interface. This will provide a gradual introduction that should make complex interfaces (like eBay’s SOAP API) easier to understand.
An interface that has been gaining popularity lately is Representational State Transfer (REST). While books (and even doctoral papers) have described the protocol, REST is essentially just a form submission to the appropriate URI. Sometimes APIs that use REST are called RESTful.
REST gets its name from the fact that the client—or in our case a webbot—is at rest for most of the time and requests information from a RESTful server only on an as-needed basis. This configuration is designed to minimize the traffic load on the server. In reality, this is how nearly every system works, whether referred to as RESTful or not.
The format of REST request is dictated by the resource you’re using, so it’s important to know the format of the REST request before you write a REST interface. For our example, let’s assume that we have access to an API that returns registration, accident, and other history information about cars, based on the VIN that is provided. The REST request might look something like Example 29-14.
Example 29-14. Sample REST request
http://www.someurl.com/vin_reports?VIN=JH4KB2F56BC000000&dealerCode=324
As you can see in Example 29-14, the REST request is basically a GET method form submission. In most cases, the data is returned as an XML document, but that’s not always the case. Depending on the need, data could be returned as images, PDF documents, spreadsheets, or any other MIME type.
There are two downsides to the REST request in Example 29-14:
The most obvious problem is that the request is sent in cleartext. If privacy is a concern, the host server could be configured to require that the REST request is sent to an SSL-encrypted web page that accepts POST method requests.
While not an issue for the REST request in Example 29-14, GET method form submissions are limited by the maximum number of characters that the host server will accept. POST method submission, however, has no (practical) limit to the number of characters in the request.