Since it is hard to accurately analyze an HTML form by hand, and since the importance of submitting a form correctly is critical, you may prefer to use a tool to analyze the format of forms. This book’s website has a form handler that provides this service. The form analyzer works by substituting the form’s original form handler with the URL of the form analyzer. When the analyzer receives form data, it creates a web page that describes the method, data variables, and cookies sent by the form exactly as they are seen by the original form handler, even if the web page uses JavaScript.
To use the emulator, you must first create a copy of the web page that contains the form you want to analyze, and place that copy on your hard drive. Then you must replace the form handler on the web page with a form handler that will analyze the form structure. For example, if the form you want to analyze has a <form> tag like the one in Example 6-9, you must substitute the original form handler with the address of my form analyzer, as shown in Example 6-10.
Example 6-9. Original form handler
<form
method="POST"
action="https://panel.schrenk.com/keywords/search/"
>Example 6-10. Substituting the original form handler with a handler that analyzes the form
<form
method="POST"
action="http://www.WebbotsSpidersScreenScrapers.com/form_analyzer.php"
>To analyze the form, save your changes to your hard drive and load the modified web page into a browser. Once you fill out the form (by hand) and submit it, the form analyzer will provide an analysis similar to the one in Figure 6-3.
This simple diagnosis isn’t perfect—use it at your own risk. However, it does allow a webbot developer to verify the form method, agent name, and GET and POST variables as they are presented to the actual form handler. For example, in this particular exercise, it is evident that the form handler expects a POST method with the variables sessionid, email, message, status, gender, and vol.
Forms with a session ID point out the importance of downloading and analyzing the form before emulating it. In this typical case, the session ID is assigned by the server and cannot be predicted. The webbot can accurately use session IDs only by first downloading and parsing the web page containing the form.
If you were to write a script that emulates the form submitted and analyzed in Figure 6-3, it would look something like Example 6-11.
Example 6-11. Using LIB_http to emulate the form analysis in Figure 6-3
include("LIB_http.php");
# Initiate addresses
$action="http://www.WebbotsSpidersScreenScrapers.com/form_analyzer.php";
$ref = "" ;
# Set submission method
$method="POST";
# Set form data and values
$data_array['sessionid'] = "sdfg73453845";
$data_array['email'] = "sales@schrenk.com";
$data_array['message'] = "This is a test message";
$data_array['status'] = "in school";
$data_array['gender'] = "M";
$data_array['vol'] = "on";
$response = http($target=$action, $ref, $method, $data_array, EXCL_HEAD);After you write a form-emulation script, it’s a good idea to use the analyzer to verify that the form method and variables match the original form you are attempting to emulate. If you’re feeling ambitious, you could improve on this simple form analyzer by designing one that accepts both the submitted and emulated forms and compares them for you.
The script in Example 6-12 is similar to the one running at http://www.WebbotsSpidersScreenScrapers.com/form_analyzer.php. This script is for reference only. You can download the latest copy from this book’s website. Note that the PHP sections of this script appear in bold.
Example 6-12. A simple form analyzer
<? setcookie("SET BY THIS PAGE", "This is a diagnostic cookie."); ?> <head> <title>HTTP Request Diagnostic Page</title> <style type="text/css"> p { color: black; font-weight: bold; font-size: 110%; font-family: arial} .title { color: black; font-weight: bold; font-size: 110%; font-family: arial} .text {font-weight: normal; font-size: 90%;} TD { color: black; font-size: 100%; font-family: courier; vertical-align: top;} .column_title { color: black; font-size: 100%; background-color: eeeeee; font-weight: bold; font-family: arial} </style> </head> <p class="title">Webbot Diagnostic Page</p> <p class="text">This web page is a tool to diagnose webbot functionality by examining what the webbot sends to webservers. <table border="1" cellspacing="0" cellpadding="3" width="800"> <tr class="column_title"> <th width="25%">Variable</th> <th width="75%">Value sent to server</th> </tr> <tr> <td>HTTP Request Method</td><td><?echo $_SERVER["REQUEST_METHOD"];?></td> </tr> <tr> <td>Your IP Address</td><td><?echo $_SERVER["REMOTE_ADDR"];?></td> </tr> <tr> <td>Server Port</td><td><?echo $_SERVER["SERVER_PORT"];?></td> </tr> <tr> <td>Referer</td> <td><? if(isset($_SERVER['HTTP_REFERER'])) echo $_SERVER['HTTP_REFERER']; else echo "Null<br>"; ?> </td> </tr> <tr> <td>Agent Name</td> <td><? if(isset($_SERVER['HTTP_USER_AGENT'])) echo $_SERVER['HTTP_USER_AGENT']; else echo "Null<br>"; ?> </td> </tr> <tr> <td>Get Variables</td> <td><? if(count($_GET)>0) var_dump($_GET); else echo "Null"; ?> </td> </tr> <tr> <td>Post Variables</td> <td><? if(count($_POST)>0) var_dump($_POST); else echo "Null"; ?> </td> </tr> <tr> <td>Cookies</td> <td><? if(count($_COOKIE)>0) var_dump($_COOKIE); else echo "Null"; ?> </td> </tr> </table> <p class="text">This web page also sets a diagnostic cookie, which should be visible the second time you access this page.