The Complete Reference™: Information Security

Applications, the software that ultimately represents the interface between the computational environment of the computer and the real world, are the raison d’être of computers. A major portion of the job of protecting the security of information on the network boils down to controlling what applications can run, and what they are allowed to do. As such, this is one of the most important areas of information security.

Applications can be controlled on the network, by allowing or denying the network connections required for the applications to communicate; and they can also be controlled on the computers on which they run, by restricting which applications can be run on computers and controlling what functions each application is allowed to perform through policy templates. This chapter covers both network and computer-based control of applications.

During the first couple of decades in the history of computer security, security professionals had to rely on network security devices, such as firewalls, intrusion detection systems (IDSs), and intrusion prevention systems (IPSs), to protect their network. This worked fine in the old days, when each application used a single, well-known, unique TCP or UDP port to communicate. Figure 30-1 demonstrates a traditional firewall approach to blocking ports associated with application communications—the ports are so well known, they can be identified by name instead of number.

But with the rapid growth in the number of applications, the network traffic landscape changed as well. Application developers discovered that security administrators could disable their software by blocking ports, but since everybody used the Web, they could design their applications to use port 80 to get past security controls. These days, as Web 2.0 becomes more prevalent, applications are increasingly taking the form of web applications. For example, a typical company employee may start his daily work by logging in to Salesforce.com to prioritize his workday, writing a letter to a customer via a cloud-based document service, and using a web-based email program to send it. So far, he has only used remote, web-based services. He may then go to a document repository to review the latest report and marketing newsletters, or visit a bug-tracking system to see whether a product defect has been fixed. Or, for that matter, he may decide instead to outsource his work to China and spend his day on a social networking site playing games. All of these appear to be completely different applications, but behind the scenes, they all go through the same port—TCP port 80. Everything this hypothetical employee has done today has appeared on the network as HTTP web traffic.

Network administrators now find themselves having a hard time even understanding what’s going on in their organization’s network, let alone managing it. They see everything flowing through HTTP and HTTPS, but which traffic is work related and which traffic is game playing or movie watching? Intelligent network traffic fingerprinting is now required to identify applications. Gone are the days when network administrators could run a report breaking down traffic by ports.

Sometimes, a desktop application, in order to ensure robust communication with external servers, adopts a technique called port hopping. It first tries to connect to its server on port 21 (FTP), and if it fails, it will try port 22 (SSH), then 80 (HTTP), then 443 (HTTPS), and so on, until a successful connection is established. None of these communications is compliant with the actual protocol specification of the port. This is a very effective way to evade the traditional port-based access control.

Traditionally, network applications were assigned their own unique ports by IANA, the Internet Assigned Numbers Authority, via the Service Name and Transport Protocol Port Number Registry. An RFC (Request for Comments) was issued to define the application’s network protocol, and then a fixed TCP or UDP port was assigned, or perhaps multiple ports if needed. Because the port is represented as a word integer, there can be only 65,535 TCP or UDP ports in total. The pool of ports available is limited and cannot serve the explosive growth of Internet-based new services. Over time, new applications started switching over to HTTP and HTTPS (TCP ports 80 and 443) as a backup method of communication when their assigned port was being blocked. At some point in time, applications gave up entirely on using ports other than web ports. Why should they bother, when communication over HTTP and HTTPS is practically guaranteed?

Have you noticed all that unknown traffic floating around in your network, consuming all your bandwidth? That traffic is probably caused by peer-to-peer (P2P) applications, which use distributed network architecture and share the load of file transferring. The ports they use are not standard, and they keep changing intentionally in order to evade detection. We will discuss these applications in more detail shortly.

In Figure 30-2, notice the significant growth over the past few years of Real-Time Entertainment, notably the streaming of movies and broadcasts, as a major category of Internet traffic, as described in the Sandvine Global Internet Phenomena Report for the first half of 2012. By comparison, traditional web browsing is in decline. And peer-to-peer applications are consuming a significant fraction of the total usage of the Internet.

Your business depends on fast and secure information sharing. To ensure 24×7×365 availability of your data and applications, you need to identify and resolve network problems before they impact your business. If the network access is slow, is it because someone is watching a streaming movie or uploading a pirated movie on a peer-to-peer network? If these activities are deemed illegal by the copyright owners, your business might even get in trouble for what your users are doing.

To accurately identify performance issues, you need clear visibility into the application traffic on your network. Network analysis software can help with this. Many network devices support the NetFlow protocol (developed by Cisco Systems), which is used for network traffic monitoring. Network devices send NetFlow information containing ports, protocols, byte counts, and other useful information to any software-based management console capable of processing the information. Many software options are available. Figure 30-3 shows an example of application identification and tracking.

Network-based application-control devices can be used to enforce policy with a variety of response options. These devices can block an application completely, allow all communication, or restrict communication with rate limits and other granular actions. They can also enforce policy based on time, geolocation, and user identity. Figure 30-4 shows what a sample policy might look like in a modern application control firewall.

Advanced application firewalls are now available that can identify application communications on the network based on the contents of the network packets (fingerprinting), and limit what they can do based on policy settings, just like old-fashioned network firewalls. They have various granular controls that restrict the network traffic of applications, even if that traffic is running on port 80 or 443.

Geolocation is the detection of the physical location of an Internet-connected computing device. This type of policy can be used to block communication with hostile countries or web sites, and it can also be used to provide different policy settings based on where the user is in your network.

By tracking to the user, rules are granular enough for modern business operation. And identity-based rules make good operational sense. Enterprises increasingly rely on unified use of user directories and identity management to support access controls. User changes happen once and propagate out. Security policies stay up to date as the user community changes.

This feature, sometimes referred to as data filtering, is used to control files based on the content, instead of basing control on the transferring protocols. For example, if you wanted to block all Microsoft Word template files (which have a .DOT extension), once you set the rule, the same policy will be applied whether the file is transferred via FTP, e-mail, or a web download.

When an application is known to be vulnerable to certain attacks, or when a worm outbreak is spreading via that application, the risk of seeing malicious content in that application will be much higher than the risk of seeing it in other applications. A reputation-aware device will be able to dynamically recognize the applications in danger, and provide control adjustment suggestions to you or dynamically block the application’s traffic. Later, when a specific security solution is implemented or the vulnerability is patched, the reputation is restored to green and the application is allowed to communicate again.

Suppose your organization’s executives insist on using a social application on the Web despite your organization’s security policy against that, and you can’t talk them out of it. With sub-application control, you can compromise by giving the VIPs an exception to the policy while denying access to all other employees due to the risk of potential information leakage via this uninspected channel. Or, as another example, you can permit the use of instant messaging applications for the purpose of chat but not for the purpose of file transfers. Individual functions such as file transfer can be blocked, while allowing other functions such as chat.

Another way to control applications is directly on the computer itself, where the applications originate. Unlike controlling network communications, which are very dynamic and hard to keep up with, preventing unwanted applications from running at all is another approach, which is very effective and remarkably easy to do with modern technologies. The following sections describe some methods to restrict applications.

Application whitelisting software relies on defined groups to allow and deny software execution. This type of control can be applied not only to software programs installed and run by the end users, but also to any kind of code such as executable files, DLLs, libraries, drivers, Java applications and scripts, ActiveX controls and scripts, registry settings, and other application components. Figure 30-5 shows a sample of how an application whitelisting software policy configuration screen might look, with authorized and unauthorized applications configured in the policy.

Application whitelisting software is typically configured initially with a whitelist of known good applications based on some kind of initial profile of a clean system built with an approved software set, such as one from a standard image. Figure 30-6 shows a sample application whitelisting policy that allows Windows programs to run but does not allow Microsoft games to run. Once the whitelist is locked in, no other code can run. However, software can be allowed a certain amount of latitude for updates as necessary. Modern application whitelisting software is smart enough to distinguish between unauthorized changes and known good software patch installation.

This highlights another advantage of application whitelisting software—the ability to defend against malware. Generally speaking, malware is unwanted code execution that exploits vulnerabilities in the host system to perform malicious behaviors. But with application whitelisting software in place, malware can’t run. In addition, some application whitelisting software products also include restrictions on known unwanted behaviors such as buffer overflows and privilege escalation, which malware takes advantage of in order to exploit vulnerabilities. As a result, patching becomes somewhat less critical than in environments with no restrictions on applications.

Change management is also facilitated by application whitelisting software. The administrator of the application whitelisting platform can control which changes are allowed, and when. This helps enforce change management policies in environments where administrators and support staff may not always see the need to follow process.

Application whitelisting software can be challenging to implement in a typical enterprise end-user IT environment, where user behaviors and software requirements are very dynamic, highly adaptable, and ever-changing. It’s best suited for static, well-defined systems such as kiosks, specialty computers such as ATMs, check-in computers, and manufacturing systems (referred to as fixed-function devices). Any computer that needs to remain in a known good state without risk of changes or unauthorized software execution is a good candidate for application whitelisting. However, with careful consideration and an up-front investment of time in profiling all the possible system behaviors your end users want, you can use application whitelisting software to lock in your end-user computing configurations to comply with your security policy.

Some of the more mature, legitimate software applications often enable you to make security choices. These can take the form of profiles, which are built-in configuration choices that collectively set various options inside of a particular application. Other applications and operating systems have templates that you can apply externally to configure security settings. These are both useful tools to control and limit what applications are allowed to do.

Some applications have built-in security profiles that can be selected in their options. Usually, these profiles range from Low (which generally includes a warning that it’s not recommended) to High (which is often too strict to allow needed functionality). Figure 30-7 shows an example of Internet Explorer security settings on a Microsoft Windows system, where the security level for the Internet zone is set to medium-high. In an enterprise environment, it’s considered a best practice to define a security profile setting for each application, and apply that profile across the organization. End users should not have to choose their own security levels; that should be left in the hands of seasoned administrators under the guidance of the security manager.

Built-in security profiles can be helpful in setting various configuration items within an application. Just make sure your users don’t change them—instruct users to disregard messages from web sites that encourage them to lower their security level to allow cookies, scripting, and ActiveX controls. Better yet, configure the profiles for them, and lock down the profiles so users can’t change them.

Security templates are also often available for operating systems and applications. These templates are made available by the manufacturer, or by third parties with an interest in security. When applied, security templates configure potentially large numbers of settings that are available in the underlying software, such as browser behaviors, USB, DVD and peripheral options, and access and behaviors of system components.

Microsoft has freely available security templates for its operating systems, and independent security organizations also publish downloadable templates that can be run on end-user systems to enhance security. These templates provide a shortcut for the security administrator who doesn’t have the time or inclination to go through every possible setting and define the best security options. Figure 30-8 shows Microsoft security templates that can be imported into Windows to establish best-practice security settings for various roles.

Figure 30-9 shows some of the settings that one of Microsoft’s security templates establishes on a local system. These settings are all configured at once when the template is imported, saving time and energy for the system administrator.

Security templates offer a good shortcut to better security levels than those with which the applications ship. For the security administrator who doesn’t have time to walk through every possible setting, templates can serve as a temporary stopgap. But they’re even better as a starting point for defining your own security levels.

From your perspective, some applications are good and some are bad. In any organization’s IT environment, there are business and productivity applications required to support the operation of the business. These are usually predefined, and you want to ensure that they are the only applications that can be used. This can be accomplished at the network level by controlling the applications’ network communications (blocking some or all of the network traffic associated with particular applications, such as with an application firewall) or, at the computer level, restricting applications from running on end-user computers (such as with application whitelisting software). A combination of both network and computer controls can provide good control over application behaviors.

Out of the box, applications rarely have the settings you want them to have in order to behave in accordance with your security policy. Controlling the functionality of each individual application can be accomplished by manually setting each option or, more quickly, by using predefined built-in profiles and external templates.

Beechey, Jim. “Application Whitelisting: Panacea or Propaganda?” SANS, 2011. www.sans.org/reading_room/whitepapers/application/application-whitelisting-panacea-propaganda_33599

Nahari, Hadi, and Ronald Krutz. Web Commerce Security: Design and Development. Wiley, 2011.

Orchilles, Jorge. Microsoft Windows 7 Administrator’s Reference: Upgrading, Deploying, Managing, and Securing Windows 7. Syngress, 2010.

Schneier, Bruce. “Whitelisting vs. Blacklisting.” 2011. www.schneier.com/blog/archives/2011/01/whitelisting_vs.html

Shackleford, Dave. “Application Whitelisting: Enhancing Host Security.” SANS, 2009. www.sans.org/reading_room/analysts_program/McAfee_09_App_Whitelisting.pdf

Tipton, Harold, and Micki Krause, eds. Information Security Management Handbook. 6th ed. Auerbach Publications, 2012.

CHAPTER
30	Controlling Application Behavior