Smart socket Essays

Smart socket Essays Smart socket Essay Smart socket Essay Although ad-hoc solutions exist for some of these problems, it is usually up to the applications user to discover the cause of the connectivity problems and find a solution. In this paper we describe Smokestacks, a communication library that lifts this burden by automatically discovering the connectivity problems and solving them with as little support from the user as possible. Categories and Subject Descriptors: C. 2. 4 [Distributed Systems]: Distributed applications General Terms: Algorithms, Design, Reliability Keywords: Connectivity Problems, Grids, Networking, Parallel Applications 1 . Parallel applications are increasingly run in Grid environments. Unfortunately, on many Grid sites the ability of machines to create or accept network connections is severely limited by network address translation (NAT) [14, 26] or firewalls [15]. There are even sites that completely disallow any direct communication between the compute nodes and the rest of the world (e. G. , the French Grinding system [3]). In addition, multi homing (machines with multiple network addresses) can further complicate connection setup. For parallel applications that require direct communication between their components, these limitations have hampered the transition from traditional multi receptors or cluster systems to Grids. When a combination of Grid sites is used, serious connectivity problems are often encountered. Smokestacks is part of the Ibis project, and can be found at CSS. Vi. Ml/ibis Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or the full citation on the first page. Many problems, however, can only be solved by adapting the application or the communication library it uses. To make matters worse, as soon as the set of Grid systems being used changes, large part of this process needs to be repeated. As a result, running a parallel application on multiple Grid sites can be a strenuous task [34]. In this paper we will describe a solution to this problem: the Smokestacks communication library. The primary focus of Smokestacks is on ease of use. It automatically discovers a wide range of connectivity problems and attempts to solve them with little or no support from the user. Smokestacks combines many known solutions, such as port forwarding, TCP splicing and SSH tunneling, and introduces several new ones that resolve problems with multi homing and machine identification. In 30 connection setup experiments, using 6 different sites worldwide, Smokestacks was always able to establish a connection, while conventional sockets only worked in 6 experiments. Using heuristics and caching, Smokestacks is able to significantly improve the connection setup performance. Smokestacks offers a single integrated solution that hides the complexity of connection setup in Grids behind a simple interface that closely resembles sockets. We will show that it is relatively straightforward to port an existing application to Smokestacks, provided that certain programming guidelines are followed. Smokestacks is not specifically intended for use in parallel applications or Grids. It can also be applied to other distributed applications, such as visualization, cooperative environments, or even consumer applications such as instant messaging, file sharing, or online gaming. However, many of these applications only require a very limited degree of connectivity. Often, clients simply connect to a server in a well-known location, making it relatively easy to apply an ad-hoc solution when a connectivity problem occurs. Parallel applications, however, can be much more challenging. They often require a large number of connections between the participating machines, and each machine must such applications in a Grid environment with limited connectivity is difficult. Therefore, this paper will focus on this domain. N Section 2 we describe the connectivity related problems encountered while running applications on multiple Grid sites. Section 3 describes how these problems are solved in Smokestacks and briefly looks at the programming interface. Section 4 evaluates the performance of Smokestacks, Section 5 describes related work, and Section 6 concludes. 2. CONNECTIVITY PROBLEMS In this section we will give a description of the network related problems that can occur when running a single parallel or distributed application on multiple Grid sites. . 1 Firewalls As described in [1 5], A firewall is an agent which screens network traffic in some way, blocking traffic it believes to be inappropriate, dangerous, or both. . Many sites use firewalls to protect their network from unauthorized access. Firewalls usually allow outbound connections, but block incoming connections, often with the exception of a few welkin ports (e. G. , port 22 for SSH). It is obvious that this inactivity restriction can cause severe problems when running a parallel application on multiple sites. When only a single participating site uses firewall, the connectivity problems can sometimes be solved by ensuring that the connections setups are in the right direction, I. E. , that all required connections between open and firewall machines are initiated at the firewall site. This solution may require changes to the applications or communication libraries, however. Also, if both sites use a firewall, this approach can no longer be used. In this case, a firewall will always be encountered regardless of he connection setup direction. One way to solve the problems is to request an open port range in the firewall. Connectivity can then be restored by adapting the application to only use ports in this range. Besides requiring reconfiguration of the firewall, open ports are also seen as a threat to site security. When both machines are behind a firewall it may still be possible to establish a direct connection using a mechanism called TCP splicing [6, 10, 13, 20]. Simply put, this mechanism works by simultaneously performing a connection setup from both sides. Since this approach requires explicit cooperation teens the machines, some alternative communication channel must be available. . 2 Network Address Translation As described in [21], Network Address Translation is a method by which IP addresses are mapped from one address realm to another, providing transparent routing to end hosts. . NAT was introduced in [12] as a temporary solution to the problem of IPPP address depletion. Although the intended solution for this problem, IPPP, has been available for some time, NAT is still widely used to day. Frequently used [21, 29]. This type of NAT allows outbound connections from sites using riveter addresses, but does not allow incoming connections. Both the IP address (and related fields) and the transport identifier (e. G. , TCP and JODI port numbers) of packets are translated, thereby preventing port number collisions when a set of hosts share a single external address. As mentioned above, NAT only allows outbound network connections. Incoming connections are rejected, since the connection request does not contain enough information to find the destination machine (I. E. , only the external IP address is provided, but that may be shared by many machines). This restriction leads to connectivity problems hat are very similar to those caused by firewalls. Therefore, the solution described in Section 2. 1 (connecting in the right direction) also applies too NAT setup, and fails in a similar way when multiple NAT sites try to interconnect. Although the TCP splicing mechanism can also be used to connect two NAT sites, a more complex algorithm is required to compensate for the port translation performed by NAT [6, 20]. Some NAT implementations have support for port forwarding, where all incoming connections on a certain port can be automatically forwarded to a certain host inside the NAT site. Using mechanisms such as Upon [5], DIF [28], or MEDICO [30], applications can contact the NAT implementation and change the port forwarding rules on demand. Port forwarding lifts many of the restrictions on incoming connections. Unfortunately, Upon is mostly found in consumer devices, MEDICO is still under development, and DIF only supports NAT (and firewall) implementations based on Interfile [1]. As a result, these mechanisms are not (yet) generally usable in Grid applications. Currently, Smokestacks only supports Upon. In addition to causing connection setup problems, NAT also complicates machine identification. Machines in a NAT tit generally use IP addresses in the private range [26]. These addresses are only usable within a local network and are not globally unique. Unfortunately, parallel applications often use a machines IP address to create a unique identifier for that machine. When multiple NAT sites participate in a single parallel run, however, this approach can not be used, since the machine addresses are no longer guaranteed to be unique. 2. 3 Non-routed networks On some sites no direct communication between the compute nodes and the outside world is possible due to a strict separation between the internal and external networks. No aching is accessible, and the connectivity of this machine may be limited by a firewall or NAT. Two of the sites used in Section 4 use such a setup. It is clear that this is a major limitation when the site is used in a parallel application. The only possibility for the compute nodes to communicate with other sites is to use front-end machine as a bridge to the outside world, using, for example, an SSH tunnel or a SOCKS [24] proxy. These are non-trivial to set up, however. 2. 4 Multi Homing When multi-homed machines (I. E. , machines with multiple network addresses) participate in a parallel application, another interesting problem occurs. When creating a connection to such a machine, a choice must be made on which of the possible target addresses to use. The outcome of this choice may depend on the location of the machine that initiates the connection. For example, the front-end machine of a site has two addresses, a public one, reachable over the internet, and a private one used to communicate with the sites compute nodes. As a result, a different address must be used to reach teachings depending on whether the connection originates inside or outside of the site. In [34] we called this the Reverse Routing Problem. Normally, when a multi-homed machine is trying to connect to a single IP address, a routing table on the machine decides which network is used for the outgoing connection. In the example described above the reverse problem is encountered. Instead of having to decide how to exit a multi-homed machine, we must decide on how to enter it. This problem is non-trivial, since the source machine generally does not have enough information available to select the correct target address. As a result, several connection attempts to different addresses of the target may be necessary before a connection can be established. In Section 3. 2 we will describe heuristics that can be used to speed up this process. Multi homing can have a major effect on the implementation of parallel programming libraries. The example above shows that it is not sufficient to use a single address to represent a multi-homed machine. Instead, all addresses must be made available to the other participants of the parallel application. In addition, some of the addresses may be in a private range and refer to a different machine when used in a different site. Therefore, it is also essential to check if a connection was established to the correct machine. 3. SMOKESTACKS In this section we will give an overview of the design, implementation and programming interface of the Smokestacks library, and describe how it solves the problems described in the previous section. 3. 1 Overview Currently, Smokestacks offers four different connection setup mechanisms, Direct, Reverse, Splicing, and Routed. An overview of how these mechanisms solve the connectivity problems described in Section 2. As the table shows, each problem is solved by at least one mechanism. Table 1 : Overview of connectivity problems and their solutions. Connection Setup Mechanism Problems Direct Reverse Splicing Routed Identification X Multi Homing X Single FEW/NAT pox xx Dual MINOT (X) XX No Routing X The machine identification and multi-homing problems are solved by the direct connection setup. As will be explained below, this approach also has limited firewall traversal capabilities (using SSH tunneling), so in certain situations it may succeed in establishing a connection in a single or even a dual firewall setting. In the table these entries are shown between brackets. A reverse connection setup is only capable of creating a connection when a single firewall or NAT limits the connectivity. Splicing is capable of middling both single and dual firewall/NAT configurations. However, this approach is significantly more complex than a reverse connection setup (especially with dual NAT) and may not always succeed. Therefore, reverse connection setup is preferred for single firewall/NAT configurations. A routed connection setup can be used in any situation where the connectivity is limited. Unlike the previous two approaches it does not result in a direct connection. Instead all network traffic is routed via external processes called hubs (explained in Section 3. 3), which may degrade both latency and throughput of the connection. Therefore, the previous mechanisms are preferred. When connecting to or from a machine on a non-routed network, however, a routed connection is the only choice. The Smokestacks implementation is divided into two layers, a low-level Direct Connection Layer, responsible for all actions that can be initiated on a single machine, and a high-level Virtual Connection Layer that uses side-channel communication to implement actions that require cooperation of multiple machines. The direct connection layer is implemented using the standard socket library. The virtual connection layer is implemented using the direct connection layer. Both layers will be explained in more detail below. Currently, Smokestacks is implemented using Java [2]. 3. 2 Direct Connection Layer do not require explicit cooperation between machines, such as determining the local addresses or creating a direct connection. It also supports a limited form of SSH tunneling. 3. 2. 1 Machine Identification During initialization, the direct connection layer starts by scanning all available network interfaces to determine which IP addresses are available to the machine. It then generates a unique machine identifier that contains these addresses, and that can be used to contact the machine. This identifier will automatically be unique if it contains at least one public address. If all addresses are private, however, additional work must be done. A machine that only has private addresses is either in a NAT site or uses a non-routed network. In the first case, a unique identifier can still be generated for the machine by acquiring the external address of the NAT. Provided that this address is public, the combination of external and machine addresses should also be unique, since other machines in the same NAT site should have a different set of private addresses, and all other NAT sites should have different external address. The Smokestacks library will use Upon to discover the external address of the NAT site. If this discovery fails, or if the returned address is not public, a Universally Unique Identifier (JUDI) [23], will be generated and included in the machine identifier, thereby making it unique. 3. 2. 2 Connection Setup Once initialized, the direct connection layer can be used to set up connections to other machines. The identifier of the target machine may contain multiple network addresses, some of which may not be reachable from the current location. The private dresses in the identifier may even refer to a completely different machine, so it is important that the identity of the machine is checked during connection setup. As a result, several connection attempts may be necessary before the correct connection can be established. When multiple target addresses are available, a choice must be made in which order the connection attempts will be performed. Although simply using the addresses in an arbitrary order should always result in a connection (pro-vided that a direct connection is possible), this may not be the most efficient approach. Many Grid sites offer historiographer networks such as Merriment [7] or Infinite [4] in addition to a regular Ethernet network. Using such a network for inter-site communication may significantly improve the applications performance. In general, these fast networks are not routed and use addresses in the private range, while the regular Ethernet networks (often) use public addresses. Therefore, by sorting the target addresses and trying all private ones first, the fast local networks will automatically be selected in sites with such a setup. The drawback of if the connection originates on a different site. This may cause a significant overhead. Therefore, Smokestacks uses a heuristic that sorts the target addresses in relation to the addresses that are available locally. For example, if only a public address is available on the local machine, it is unlikely that it will be able to create a direct connection to a private address of a target. As a result, the connection order public before private is used. This order is also used if both machines have public and private addresses, but the private addresses refer to a different network (e. G. , 10. 0. 0. 10 vs 192. 168. 1. 20). The order private before public is only used if both machines have private addresses in the same range. Section 4 will illustrate the performance benefits of this heuristic. Unfortunately, it is impossible to make a distinction between addresses of the same class. For example, if a target has multiple private addresses, we can not automatically determine which address is best. Therefore, if a certain network is preferred, the user must specify this explicitly. Without this explicit configuration, Smokestacks will still create a direct connection (if possible), and the parallel application will run, but its performance may be suboptimal. When a connection has been established, an identity check s performed to ensure that the correct machine has been reached. This would be a simple comparison if the complete identifier of the target is available, but unfortunately this is not always the case. User provided addresses are often used to bootstrap a parallel application. These addresses are often limited to a single hosannas or IP address, which may only be part of the addresses available to the target machine. Therefore, the identity check used by Smokestacks also allows the use of partial identifiers. Whenever a connection is created, the target machine provides its complete identity o the machine initiating the connection. This machine then checks if both the public and private addresses in the partial identity are a subset of the ones in the complete identity. If so, the partial identity is accepted as a subset of the complete identity, and the connection is established. Note that although the connection is created to a machine that matches the address specified by the user, it is not necessarily the correct machine from the viewpoint of the parallel application. Unfortunately, in such cases it is up to the user to provide an address that contains enough information to reach the correct machine. . 2. 3 Open Port Ranges and Port Forwarding When a firewall has an open port range available, Smokestacks can ensure that all sockets used for incoming connections are bound to a port in this range. There is no way of discovering this range automatically, however, so it must be specified explicitly by the user. In addition, Smokestacks can use the Upon protocol to configure a NAT to do port forwarding, I. E. Automatically forward all incoming connections on a certain external port to a specified internal address. However, as explained before, this protocol is mainly used in consumer devices. In addition to regular network connections, the direct connection layer also has limited support for SSH tunneling. This feature is useful for connecting to machines behind a firewall that allows SSH connections to pass through. It does, however, require a suitabl e SSH setup (I. E. , public key authentication must be enabled). Creating an SSH tunnel is similar to a regular connection setup. The target addresses are sorted and tried consecutively. Instead of using the port specified in the connection setup, however, the default SSH port (I. E. , 22) is used. When a connection is established and the authentication is successful, the receiving SSH daemon is instructed to forward all traffic to the original destination port on the same machine. If this succeeds, the regular identity check will be performed to ensure that the right machine has been reached. Although this approach is useful, it can only be used to set up a tunnel to a different process on the target machine. Using this approach to forward traffic to different machines requires extra information. For example, setting up an SSH tunnel to a compute node of a site through the sites fronted, can only be done if it is clear that the fronted must be contacted in order to reach the target machine. Although this approach is used in some projects [8], the necessary information cannot be obtained automatically and must be provided by the user. Therefore, Smokestacks uses a different approach which will be described in detail in Section 3. 3. 3. 2. Limitations The direct connection layer offers several types of connection setup which have in common that they can be initiated by a single machine. No explicit cooperation between machines is necessary to establish the connection. There are many cases, however, where connectivity is too limited and the direct connection layer cannot be used. In general, direct connections to sites that use NAT or a firewall are not possible. Although SSH tunneling and o pen port ranges alleviate the firewall problems, they require a suitable SSH setup or extra information from the user. Port forwarding reduces the problems with NAT, but is rarely supported in Grid systems. Therefore, these features are of limited use. In the next section we will give a detailed description of the virtual connection layer, which solves these problems. 3. 3 Virtual Connection Layer Like the direct connection layer, the virtual connection layer implements several types of connection setup. It offers a simple, socket-like API and has a modular design, making it easy to extend. Besides a direct module that uses the direct connection layer described above, it contains several modules that offer more advanced types of connection setup. These modules have in common that they cooperation (and thus communication) between the source and target machines in order to establish a connection. As a result, side-channel communication is required to implement these modules. 3. 3. 1 Side-Channel Communication In Smokestacks, side-channel communication is implemented by creating a network f interconnected processes called hubs. These hubs are typically started on the fronted machines of each participating site, so their number is usually small. When a hub is started, the location of one or more other hubs must be provided. Each hub will attempt to setup a connection to the others using the direct connection layer. Although many of these connections may fail to be established, this is not a problem as long as a spanning tree is created that connects all hubs. The hubs use a gossiping protocol to exchange information about themselves and the hubs they know, with the hubs that they are connected to. This way information about each hub quickly spreads to all hubs in the network. Whenever a hub receives information about a hub it has not seen before, it will attempt to set up a connection to this hub. This way, new connections will be discovered automatically. All gossiped information contains a state number indicating the state of the originating machine when the information was sent. Since information from a hub may reach another hub through multiple paths, the state number allows the receiver to decide which information is most recent. By recording the length of the path traversed thus far in he gossiped information, hubs can determine the distance to the sites that they can not reach directly. Whenever a hub receives a piece of information about another hub containing a shorter distance than it has seen so far, it will remember both the distance and the hub from which the information was obtained. This way, we automatically create a distributed routing table with the shortest paths between each pair of hubs. This table is later used to forward application information (as will be described below). When an application is started, the virtual layer on each machine creates a single connection to the hub local to its tit. The location of this hub can either be explicitly species- fled or discovered automatically using JODI multicast. 3. 3. 2 Virtual Addresses The connection to the hub can now be used as a side channel to forward requests to otherwise unreachable machines. To ensure that the target machines can be found, addresses are used, consisting of the machine identifier (see Section 3. 2), a port number, and the identifier of the hub the machine is connected to. All requests for the target machine can then be sent to the local hub, which forwards it in the direction of the target