the TCP/IP protocols were not designed with security in mind. One of the most comprehensive attempts to rectify this is IETF's IPSec. which comprises a number of core security standards for use with IPv4 and the new IPv6. Unlike firewalls which protect the perimeter of a network, IPSec is an end-to-end defence which secures data for the entire length of its trip from source to destination. It operates at the IP layer.
IPSec offers two alternative security services: the authentication header, integrity and resistance to replay attacks; in addition to these features the encapsulating security payload (ESP) offers privacy as well. In addition IPSec includes a facility for exchange of encryption keys, known as Internet Key Exchange (IKE).
IKE is used to establish two security associations (SAs), one for each direction, at the beginning of an IPSec conversation. An SA is negotiated by the two participants and includes: mutually agreed keys; security protocols (one for integrity and authentication and the other, in ESP, for encryption); and a 32-bit security parameter index (SPI), chosen to distinguish amongst multiple SAs existing at the receiver.
In IPv4 AH goes between the IP header and the payload using a protocol ID of 51. It contains three important elements.
an SPI;
authentication data which is in fact a message digest, generated either by MD5 (Message Digest 5) or SHA (Secure Hash Algorithm) applied to the packet (but excluding fields like TTL which may change as it crosses the internetwork);
a 32-bit sequence number which is used to prevent replay attacks and must be unique on any given SA.
ESP provides for privacy as well. In transport mode ESP, it is used simply to encrypt the contents of an IP datagram (a TCP or UDP segment) while in tunnel mode ESP, the entire IP datagram, including the IP header is encrypted. In addition to an ESP header, a trailer is also appended to the packet. As with AH, the ESP header includes an SPI and a sequence number. The trailer is used to pad out the IP packet data to make the encrypted section (IP data + ESP trailer) the correct length (the encryption algorithms used in ESP operate on data in multiples of some unit block), while the message digest is appended after the trailer.
Figure 1 ESP transport mode encryption: shaded area is encrypted
In transport mode ESP (figure 1), the contents of the IP datagram plus the ESP trailer are encrypted. however, the header of the original IP packet is unencrypted, and is therefore susceptible to snooping. In tunnel mode (Figure 2), on the other hand, the entire IP packet is placed in the payload of a new packet and is then encrypted. The new IP header, prepended to the encrypted information, may be used to send the packet, not to the destination specified by the original datagram, but to a router or firewall protecting a secure local network at the destination. At this router, the decryption is performed and the original datagram is then routed to the final destination. Using this technique, the hosts on the secure network do not themselves need to be able to handle ESP decryption, but there must be a high degree of confidence that attacks are not likely within that network itself. It is also impossible for an external eavesdropper to determine much about the actual traffic since the original IP headers are themselves encrypted. It is even possible to use tunnel mode ESP to tunnel protocols other than IP, such as, for example, IPX.
Figure 2 ESP tunnel mode encryption: shaded area is encrypted
Windows 2000 supports IPSec for all IP communications but it is optional as to whether it is actually used or not. For IPSec to operate both ends of a conversation must support it (this automatically holds if both are Windows 2000 computers) and have activated it. In Windows 2000 IPSec operates via policies which are managed using the Microsoft Management Console (MMC) "IP Security Policies" snap-in shown in Figure 3 (you need to have administrative access to do this).
Figure 3: Setting IPSec Policies in Windows 2000
Three predefined policies are provided: one (shown) for a client (workstation) that will not normally use IPSec unless asked to do so by a server; one for a server (all Windows 2000 systems have some server capabilities) which will use IPSec with any client which is willing to do so; and one for a secure server which will only interact with clients using IPSec. The three policies are intended to be used in Windows 2000 domains and use Kerberos to establish trust between the communicating machines. However, users can define their own policies for non-domain using certificates or pre-shared keys.