This is commonly referred to as Network Layer 1. This layer is responsible for buffering data for presentation to the lower level transports. It interacts with the generic NeL serialization and streaming system to send and receive data to the higher layers. It also contains the multi-threaded listening system for services. This layer should not typically be used directly by developers using the NeL framework.
Layer 1 includes the following classes:
CBufNetBase: Buffer functionality common to client and serverCBufClient: Implements client-specific buffer functionalityCBufServer: Implements server-specific buffer functionality
¶ Server Structure
The server, for example CBufServer, provides a single receive queue (see "Receive FIFO Buffer" on the object diagram below), and one send queue per connection ("Send FIFO Buffer"). Internally each connection is associated with a receive buffer to handle non-blocking receiving of incomplete data blocks (in CServerBufSock). The actual receives and sends are done by in layer 0 by a socket, such as CTcpSock.
Every connection is managed by a receive thread such as CServerReceiveTask. Instead of having one thread per connection, which would slow the system down and limit the maximum number of connections, there is a pool of threads that handle several connections. For example 30 threads handling 30 connections allow 900 simultaneous connections. If only one thread was in charge of all the connections the select() operation would take too much overhead for a large number of sockets.
¶ Launching The Server
The server CBufServer starts a listening socket CListenSock which is handled by a particular thread CListenTask. It can then accept incoming connections.
¶ Accepting Connections
When a connection is accepted the server advertises the connection by pushing a connection event into the receive queue. Then it dispatches the associated socket to a receive thread from the pool (or a new one).
¶ Reading Data
The user of this layer, which is usually a higher NeL Net layer, makes a call_CBufServer::dataAvailable()_ to check for incoming data. If a connection or disconnection event is found at the top of the receive queue the associated system callback is called. Then the socket is logically connected (CBufSock::connectedState() is true) in case of a connection event.
If dataAvailable() returned true, the user calls_CBufServer::receive(data,&sockid)_. The second argument tells which connection the data is coming from. It can reply to it directly by calling CBufServer::send(replydata,sockid).
The user must call CBufServer::update() for the system to work properly.
¶ Sending Data
Sending data is buffered as well. The moment when the data is actually sent depends on the triggers specified. By default the time trigger is enabled with a value of 20ms. It means that the data will actually be sent after 20 milliseconds, providing that the update() method is called evenly. The user can also specify a size trigger. The size trigger is when the size in the send buffer exceeds the specified size the data is sent. Eventually the user can force sending by calling flush().
¶ How the Non-Blocking Receiving Works
Each receiving thread, CServerReceiveTask, performs a select() on its sockets. When incoming data is reported the method_CServerBufSock::receivePart()_ tries to read a block, which is made up of a length prefix and a payload buffer. If the actual data received is smaller than expected it is retained in the receive buffer for later completion. When it is complete it is pushed into the main receive queue.
¶ How the Non-Blocking Sending Works
All data from a send queue is copied into a buffer and then sent. If the sending was not entirely done or it would block part of the buffer is kept for later sending.
¶ Handling a Disconnection
When a receive thread detects that a socket is disconnected it pushes a a disconnection event into the receive queue by the next_CBufServer::update()_. When the disconnection event is processed, at the top of the receive queue, the socket is added to the synchronized set of connections to remove of the thread. It will be effectively removed before its next select.
¶ Wake-up Pipes
A Unix pipe is added in every select set. This is so that the select can be stopped when there is a new connection to add to its set or when the server is required to exit. Similarly, the listen thread performs a select on the listen socket and the wake-up pipe.
Under Windows, the wake-up mechanism waits on an Event object.
¶ Thread Synchronization
Because of the sharing of data among different threads some mutexes are used to synchronize data access/modification on items:
- The receive queue
- The thread pool
- The connections
- The set of connections to remove
- The connected property of sockets
To ensure a fair access to the variables the GNU/Linux implementation uses a semaphore instead of a pthread_mutex.
¶ Gathering Statistics
Several methods are provided in CBufServer to know how many bytes have been read and written.
¶ Client Structure
The client provides one receive queue and one send queue. The receiving is done is a separate thread, CClientReceiveTask, but the actual sending, flush(), is done in CBufClient::update(). The socket is in blocking mode.
Unlike in the server there is no connection advertisement. A disconnection event is pushed into the receive queue when the receive thread or the sending detects the disconnection. It is also important to note that the client does not remove the socket after disconnecting. It removes it at destruction time so the user can reuse the same CBufClient object by calling again connect() after having disconnected.