Here are a few additional things to remember when writing webbots that access password-protected websites.
For clarity, the examples in this chapter use a minimal amount of code to perform a task. In actual use, you’ll want to follow the comprehensive practices mentioned elsewhere in this book for downloading pages, parsing results, emulating forms, using PHP/CURL, and writing fault-tolerant webbots.
It’s important to note that no form of online authentication is effective unless it is accompanied by encryption. After all, it does little good to authenticate users if sensitive information is sent across the network in cleartext, which can be read by anyone with a packet sniffer.[60] In most cases, authentication will be combined with encryption. For more information about webbots and encryption, revisit Chapter 19.
If your webbot communicates with more than one domain, you need to be careful not to broadcast your login criteria when writing webbots that use basic authentication. For example, if you hard-code your username and password into a PHP/CURL routine, make sure that you don’t use the same function when fetching pages from other domains. This sounds silly, but I’ve seen it happen, resulting in cleartext login credentials in server log files.
Websites may use a combination of two or more authentication types. For example, an authenticated site might use both query and cookie sessions. Make sure that you account for all potential authentication schemes before releasing your webbots.
The latest versions of all the scripts used in this chapter are available for download at this book’s website.