You learned how to download files from the Internet in Chapter 3. In this chapter, you’ll learn how to fill out forms and upload information to websites. When your webbots have the ability to exchange information with target websites, as opposed to just asking for information, they become capable of acting on your behalf. Tasks that you may want webbots to do for you may include:
Transferring funds between your online bank accounts when an account balance drops below a predetermined limit
Buying items in online auctions when an item and its price meet preset criteria
Autonomously uploading files to a photo sharing website
Advising a distributor to refill a vending machine when product inventory is low
Webbots send data to webservers by mimicking what people do when they fill out standard HTML forms on websites. This process is called form emulation. Form emulation is not an easy task, since there are many ways to submit form information. In addition, it’s important to submit forms exactly as the webserver expects them to be submitted, or the server may generate errors in its log files. People using browsers don’t have to worry about the format of the data they submit in a form. Webbot designers, however, must reverse engineer the form interface to learn about the data format the server is expecting. When the form interface is properly debugged, the form data from a webbot appears exactly as if it were submitted by a person using a browser.
If done poorly, form emulation can get webbot designers into trouble because poorly designed form emulators may make errors that are impossible for a person to make using a standard browser. This is especially troublesome when creating an application that delivers a competitive advantage for a client and you want to conceal the fact that you are using a webbot. A number of things could happen if your webbot gets into trouble, ranging from leaking (to your competitors) that you’re gaining an advantage through the use of a webbot to having your website privileges revoked by the owner of the target website.
The first rule of form emulation is staying legal: Represent yourself truthfully, and don’t violate a website’s user agreement. The second rule is to send form data to the server exactly as the server expects to receive it. If your emulated form data deviates from the format that is expected, you may generate suspicious-looking errors in the server’s log. In either case, the server’s administrator will easily figure out that you are using a webbot. Even though your webbot is legitimate, the server log files your webbot creates may not resemble browser activity. They may indicate to the website’s administrator that you are a hacker and lead to a blocked IP address or termination of your account. It is best to be both stealthy and legal. For these reasons, you may want to read Chapter 26 and Chapter 31 before you venture out on your own.
Webbot developers need to look at online forms differently than people using the same forms in a browser. Typically, when people use browsers to fill out online forms, performing some task like paying a bill or checking an account balance, they see various fields that need to be selected or otherwise completed. Webbot designers, in contrast, need to view HTML forms as interfaces or specifications that tell a webbot how a server expects to see form data after it is submitted. A webbot designer needs to have the same perspective on forms as the server that receives the form. For example, a person filling out the form in Figure 6-1 would complete a variety of form elements—text boxes, text areas, select lists, radio controls, checkboxes, or hidden elements—that are identified by text labels.
While a human associates the text labels shown in Figure 6-1 with the form elements, a webbot designer knows that the text labels and types of form elements are immaterial. All the form needs to do is send the correct name/data pairs that represent these data fields to the correct server page, with the expected protocol. This isn’t nearly as complicated as it sounds, but before we can go further, it’s important that you understand the various parts of HTML forms.
