async, non blocking tcp server

[Chris Knipe]

Hi All,



I've looked at Net::Server (and have my "new" server with some 20K lines of
code written in it that we're currently finalising), POE, and a few other
things, but I cannot find what I am looking for. I'm hoping for some
assistance / pointers please. I am looking for the following simple
scenario.



On the server side, read data through the socket (for simplicity, let's say
"commands" are terminated by \n)

On the client side, send numerous commands to the server, WITHOUT waiting
for a response - that's simple enough.



Now, let's say the server sleep($randomtime) between commands to simulate a
work load. I effectively want the following



C> Do 1

C> Do 2

S> Done 1

C> Do 3

C> Do 4

S> Done 4

S> Done 2

S> Done 3



I know this sounds simple, but can anyone give a rough example of the server
code (even if it's just an echo with a random sleep). For the love of. I
cannot seem to stop the synchronous nature of POE, or Net::Server. The
whole wait for input, deal with input, write output thing is really blocking
my application's performance, seriously, at this point in time. I need to
be truly asynchronous.



I need to also preferably need to use PreForks. The application needs to
deal with some 500Mbps in terms of bandwidth on the sockets, and a good
amount of thousands of concurrent connections. Again, I'm not sure whether
Net::Server will scale (we're developing / debugging currently, whilst in
production we currently use xinetd).



Many thanks,

Chris.

[Sam]

Post by Chris Knipe
Hi All,
I’ve looked at Net::Server (and have my “new” server with some 20K lines
of code written in it that we’re currently finalising), POE, and a few
other things, but I cannot find what I am looking for… I’m hoping for
some assistance / pointers please. I am looking for the following
simple scenario…
On the server side, read data through the socket (for simplicity, let’s
say “commands” are terminated by \n)
On the client side, send numerous commands to the server, WITHOUT
waiting for a response – that’s simple enough.
Now, let’s say the server sleep($randomtime) between commands to
simulate a work load. I effectively want the following
C> Do 1
C> Do 2
S> Done 1
C> Do 3
C> Do 4
S> Done 4
S> Done 2
S> Done 3
I know this sounds simple, but can anyone give a rough example of the
server code (even if it’s just an echo with a random sleep). For the
love of… I cannot seem to stop the synchronous nature of POE, or
Net::Server. The whole wait for input, deal with input, write output
thing is really blocking my application’s performance, seriously, at
this point in time. I need to be truly asynchronous.
I need to also preferably need to use PreForks. The application needs
to deal with some 500Mbps in terms of bandwidth on the sockets, and a
good amount of thousands of concurrent connections. Again, I’m not sure
whether Net::Server will scale (we’re developing / debugging currently,
whilst in production we currently use xinetd).
Many thanks,
Chris.

Are you saying the normal 'unix' way won't work? (ie. listen on socket,
fork on an accepted connection, do the work, close)


$server = IO::Socket::INET->new(LocalPort => 8801,
Type => SOCK_STREAM,
Reuse => 1,
Listen => 10)
or die "Server open failed: $!\n";


while(1){
#accept() will block until connection received
$client = $server->accept();
#create new process for client
my $pid = fork();

if($pid == 0){
processClient($client);
}
else{
close($client);
waitpid(-1,WNOHANG);
}


}



--Sam

[Chris Knipe]

Are you saying the normal 'unix' way won't work? (ie. listen on socket, fork
on an accepted connection, do the work, close)

Not at all. My problem is not related to creating, accepting, or
forking at all.

The block / waiting occurs whilst the socket is already connected. I
think a better way to describe this from terms I've seen whilst
googling... I need to be able to read & write simultaneously on the
socket after the client connected.

Again, as per my example given, that is one single client socket
connected to the server. The asynchronous part comes in to the effect
that the client must be able to CONTINUOUSLY (non blocking) supply
commands to the server, whilst the server MAY take a large time before
responding to those commands. Now, every single (almost) perl
application I've looked at, does this to some form of degree in terms
of reading/writing on sockets...

while (1) {
# read $_ from socket
# do stuff with $_
# respond to client with some form of print
}

In that example, whilst you are doing stuff with $_ (the workload,
sleep($randomtime) in my example, the client is blocked and the client
cannot write to the socket (or technically it can, but the server
can't process it rather). I am thus referring to the communication
stream, needing to be asynchronous.

I don't even know how to accurately present this in text, but I'm
basically after something to the tune of the below. And the *real*
crux of the matter is going to be that the writing will indeed
sometimes happen at the *same time* as what the reading will happen.
Or at least, it needs to.

while (1) {
# read $_ from the socket and pass it to some non blocking sub / event handler
}

sub processit {
# do stuff with $_
# respond to the client with some form of print
}

SMPP is a protocol which can be seen as a very good example. Whilst
it CAN work synchronously (receive data, process it, respond), it
operates immensely better when it's written as a asynchronous
application. THe same with certain aspects of NNTP for example (RFC
4644), and I'm sure that there is allot of other examples I can give
where asynchronous communication is absolutely vital.

I'm most certainly also not saying that it isn't possible with Perl
either. All that I am saying is that, to date, I have not been able
to find one single example, recipe, or cookbook entry on HOW to do
this. Not wanting to step on any toes, but I almost get the feeling
that the examples of code being given is rather outdated.

[Carl Inglis]

I'm not sure if there's a misunderstanding somewhere (which may be mine).

As I understand it what you want to do is launch a command towards the
server, and then another, and then another, all the time waiting for a
response to one or more of your commands.

I suspect you're looking for something like this:
http://www.perlmonks.org/?node_id=66135

In fact, it's specifically mentioned in the Perl Cookbook:
http://docstore.mik.ua/orelly/perl/cookbook/ch07_15.htm

Hope that helps put you on the right track.

Regards,

Carl

Post by Chris Knipe

Post by Sam
Are you saying the normal 'unix' way won't work? (ie. listen on socket,

fork

Post by Sam
on an accepted connection, do the work, close)

Not at all. My problem is not related to creating, accepting, or
forking at all.
The block / waiting occurs whilst the socket is already connected. I
think a better way to describe this from terms I've seen whilst
googling... I need to be able to read & write simultaneously on the
socket after the client connected.
Again, as per my example given, that is one single client socket
connected to the server. The asynchronous part comes in to the effect
that the client must be able to CONTINUOUSLY (non blocking) supply
commands to the server, whilst the server MAY take a large time before
responding to those commands. Now, every single (almost) perl
application I've looked at, does this to some form of degree in terms
of reading/writing on sockets...
while (1) {
# read $_ from socket
# do stuff with $_
# respond to client with some form of print
}
In that example, whilst you are doing stuff with $_ (the workload,
sleep($randomtime) in my example, the client is blocked and the client
cannot write to the socket (or technically it can, but the server
can't process it rather). I am thus referring to the communication
stream, needing to be asynchronous.
I don't even know how to accurately present this in text, but I'm
basically after something to the tune of the below. And the *real*
crux of the matter is going to be that the writing will indeed
sometimes happen at the *same time* as what the reading will happen.
Or at least, it needs to.
while (1) {
# read $_ from the socket and pass it to some non blocking sub / event handler
}
sub processit {
# do stuff with $_
# respond to the client with some form of print
}
SMPP is a protocol which can be seen as a very good example. Whilst
it CAN work synchronously (receive data, process it, respond), it
operates immensely better when it's written as a asynchronous
application. THe same with certain aspects of NNTP for example (RFC
4644), and I'm sure that there is allot of other examples I can give
where asynchronous communication is absolutely vital.
I'm most certainly also not saying that it isn't possible with Perl
either. All that I am saying is that, to date, I have not been able
to find one single example, recipe, or cookbook entry on HOW to do
this. Not wanting to step on any toes, but I almost get the feeling
that the examples of code being given is rather outdated.
--
http://learn.perl.org/

[Chris Knipe]

Post by Carl Inglis
http://www.perlmonks.org/?node_id=66135
http://docstore.mik.ua/orelly/perl/cookbook/ch07_15.htm

The comments on the first URL is worrying, but I don't believe that it
would solve my issue. Lets take the following out of RFC4664 as an
example. This is an actual example of a communication stream between
a client and the perl server, the socket is already established, and
communication is flowing between the two parties. [C] indicates what
the client is sending to the server, and [S] indicates the responses
the server sends to the client. My comments added with # and thus
does not form part of the communication stream.

[C] TAKETHIS <***@example.com>
[C] Path: pathost!demo!somewhere!not-for-mail
[C] From: "Demo User" <***@example.com>
[C] Newsgroups: misc.test
[C] Subject: I am just a test article
[C] Date: 6 Oct 1998 04:38:40 -0500
[C] Organization: An Example Com, San Jose, CA
[C] Message-ID: <***@example.com>
[C]
[C] This is just a test article.
[C] .
# . indicates the end of the article. Perl now starts to do work,
processing the article it received. As this process takes time, the
main while (1) { loop reading from the socket is now blocked, causing
the server not to read any more from the socket until after the
article has been dealt with.
[C] TAKETHIS <***@example.com>
[C] Path: pathost!demo!somewhere!not-for-mail
[C] From: "Demo User" <***@example.com>
[C] Newsgroups: misc.test
[C] Subject: I am just a test article
[C] Date: 6 Oct 1998 04:38:40 -0500
[C] Organization: An Example Com, San Jose, CA
[C] Message-ID: <***@example.com>
[C]
[C] This is just a test article.
[C] .
# Perl server only NOW responds with an acceptance code for the first article.
[S] 239 <***@example.com>

We have effectively now COMPLETELY missed the second article in our
while (1) loop, as perl did not read from the socket WHILST processing
the article.

When you multiply the above example and think that you can effectively
be receiving up to 100 (if not more) TAKETHIS commands per second, and
that it takes (on average) say 100ms to "deal" with an article
received... I'm sure you can understand the economics at scale here as
to why this is important, and what I mean by non blocking...

The whole while (1) { read from socket; do work; respond; }; isn't
going to work. The moment you "do work", you block reading from the
socket until such time that the work has been completed. I don't
know how to be clearer about this.

[Chris Knipe]

And this is effectively what I WANT to happen...

[C] TAKETHIS <***@example.com>
[C] Path: pathost!demo!somewhere!not-for-mail
[C] From: "Demo User" <***@example.com>
[C] Newsgroups: misc.test
[C] Subject: I am just a test article
[C] Date: 6 Oct 1998 04:38:40 -0500
[C] Organization: An Example Com, San Jose, CA
[C] Message-ID: <***@example.com>
[C]
[C] This is just a test article.
[C] .
# . indicates the end of the article. Perl now starts to do work,
processing the article it received. As this process takes time, the
main while (1) { loop reading from the socket is now blocked, causing
the server not to read any more from the socket until after the
article has been dealt with.
[C] TAKETHIS <***@example.com>
[C] Path: pathost!demo!somewhere!not-for-mail
[C] From: "Demo User" <***@example.com>
[C] Newsgroups: misc.test
# Perl server only NOW responds with an acceptance code for the first article.
[S] 239 <***@example.com>
[C] Subject: I am just a test article
[C] Date: 6 Oct 1998 04:38:40 -0500
[C] Organization: An Example Com, San Jose, CA
[C] Message-ID: <***@example.com>
[C]
[C] This is just a test article.
[C] .
# Perl server only NOW responds with an acceptance code for the second article.
[S] 239 <***@example.com>

I am now presuming that we've read both articles completely (all
lines) from the socket.

[Carl Inglis]

Hi Chris,

The only way this is going to work as far as I can see is some form of
multi-threading - how about you have a thread which issues the commands, a
thread which reads from the sockets and shoves the results into a queue,
and another thread which processes that queue.

I think you're looking for a single threaded solution to an implicitly
multi-threaded problem. I can't see any way for a single thread to do what
you want.

Regards,

Carl

Post by Chris Knipe

Post by Carl Inglis
http://www.perlmonks.org/?node_id=66135
http://docstore.mik.ua/orelly/perl/cookbook/ch07_15.htm

The comments on the first URL is worrying, but I don't believe that it
would solve my issue. Lets take the following out of RFC4664 as an
example. This is an actual example of a communication stream between
a client and the perl server, the socket is already established, and
communication is flowing between the two parties. [C] indicates what
the client is sending to the server, and [S] indicates the responses
the server sends to the client. My comments added with # and thus
does not form part of the communication stream.
[C] Path: pathost!demo!somewhere!not-for-mail
[C] Newsgroups: misc.test
[C] Subject: I am just a test article
[C] Date: 6 Oct 1998 04:38:40 -0500
[C] Organization: An Example Com, San Jose, CA
[C]
[C] This is just a test article.
[C] .
# . indicates the end of the article. Perl now starts to do work,
processing the article it received. As this process takes time, the
main while (1) { loop reading from the socket is now blocked, causing
the server not to read any more from the socket until after the
article has been dealt with.
[C] Path: pathost!demo!somewhere!not-for-mail
[C] Newsgroups: misc.test
[C] Subject: I am just a test article
[C] Date: 6 Oct 1998 04:38:40 -0500
[C] Organization: An Example Com, San Jose, CA
[C]
[C] This is just a test article.
[C] .
# Perl server only NOW responds with an acceptance code for the first article.
We have effectively now COMPLETELY missed the second article in our
while (1) loop, as perl did not read from the socket WHILST processing
the article.
When you multiply the above example and think that you can effectively
be receiving up to 100 (if not more) TAKETHIS commands per second, and
that it takes (on average) say 100ms to "deal" with an article
received... I'm sure you can understand the economics at scale here as
to why this is important, and what I mean by non blocking...
The whole while (1) { read from socket; do work; respond; }; isn't
going to work. The moment you "do work", you block reading from the
socket until such time that the work has been completed. I don't
know how to be clearer about this.

[Chris Knipe]

Post by Carl Inglis
Hi Chris,
The only way this is going to work as far as I can see is some form of
multi-threading - how about you have a thread which issues the commands, a
thread which reads from the sockets and shoves the results into a queue, and
another thread which processes that queue.
I think you're looking for a single threaded solution to an implicitly
multi-threaded problem. I can't see any way for a single thread to do what
you want.

Not specifically looking for a single threaded solution - looking for
*any* solution. This is why I am saying, there really isn't anything
in terms of examples, cookbooks, etc. dealing with something like
this. This is really simple asynchronous communication at the end of
the day - in my books at least. Through my searching I've stumbled
onto something on perlmonks (I think) basically where there was a
thread to read from the socket, and a thread to write to the socket,
but it was advised against doing that for some reason or the other -
I'll see whether I can get the link again in the browser history or
something.

At the end of the day I suppose, see this as an opportunity (for the
mainstream developers). Whilst it will most certainly complicate
things, native support for things like this in POE, or Net::Server, or
Net::IO::Socket would really be appreciated. We are in the 21st
century afterall. IMHO the days synchronous communication are long
gone. And again, I mean this with the utmost respect :-) In the case
of POE or Any::Event (if my memory serves me correctly), one should
surely be able to "offload" the "work" through an event handler in a
non blocking way, and write back to the socket at any time? That was
my understanding of it at least - and guess my surprise when that was
not the case. Who knows, perhaps it is possible and I'm just
overlooking it too.

Multi-Threading and queueing is a option. The first thing that crops
up in my head again now though is that *IF* you receive data faster
than what you can process it, you're now looking at OOM issues again.
Secondly sitting with say, 1K forks from Net::Server and each fork
sitting with 2 or 3 Threads - the poor server is all that I'm going to
say...

Perhaps the simplest (and what I didn't want to do), is to just dump
it to a spool directory on disk and have a completely separate daemon
pickup and process the files from disk... At least then you can read
the list of files, spawn say 30 or 40 threads, and deal with the files
- quickly - say every minute or two. The server can also if needs be
I suppose fork through a non blocking sub to write the files to disk,
so the server doesn't even need to wait for the IO operation.

[Carl Inglis]

Hi Chris,

I see your point, but everything I've found, both historically in my own
work, and on the web, has been multi-thread, or even multi-process.

For example, where I work at the moment we've got an application which runs
on a series of monitoring equipment and dumps status to a disk file every 5
seconds. There's a sweeper process which runs totally separately on a
server which sweeps through those files and reads them into a database.
That sounds very much like your final paragraph. It does have the advantage
(for us) of separating the insert into the database from the work that the
monitoring equipment is doing and it means that if the database server
isn't available the equipment can at least continue to run and record
locally.

Does your communication protocol not include an option to back off the rate
of transmission? Or could you do something at a lower level and not ACK the
packets to force retransmission at the TCP level? (Clutching at straws a
little here).

Anyway, hope some of the thoughts and ideas does help you, and I'd be
interested to know what you end up doing.

Regards,

Carl

[big snip]

[Chris Knipe]

Post by Carl Inglis
Does your communication protocol not include an option to back off the rate
of transmission? Or could you do something at a lower level and not ACK the
packets to force retransmission at the TCP level? (Clutching at straws a
little here).

Unfortunately not (we are talking NNTP specifically here if it isn't
obvious yet) - and the data rates on the network in terms of the
stream we're looking at over 1Gbps (generated with the TAKETHIS
commands). The other seemingly obvious problem that I am going to
have by handing the work off to a separate process (or even thread for
that matter) is that I need to inform the remote client on the socket
in real time whether I want / do not want / if an error occurred to
the article in question issued by the command from the client.

It's times like this that I *really* wish I knew how to code in C - no
offence to anyone, again. Even in .NET doing something like this is
almost trivial by using separate event handlers for receiving data
from the socket - again which is why I firmly believed that something
like this could have been done via POE (yes, this would be handled via
threads - but it's done and managed by the .NET modules, and thus
require -very- little code, or understanding for that matter, from the
programmer's point of view).

Oh well - it's back to the drawing board for this one then.

[Gary Stainburn]

Post by Chris Knipe
Unfortunately not (we are talking NNTP specifically here if it isn't
obvious yet) - and the data rates on the network in terms of the
stream we're looking at over 1Gbps (generated with the TAKETHIS
commands). The other seemingly obvious problem that I am going to
have by handing the work off to a separate process (or even thread for
that matter) is that I need to inform the remote client on the socket
in real time whether I want / do not want / if an error occurred to
the article in question issued by the command from the client.
It's times like this that I *really* wish I knew how to code in C - no
offence to anyone, again. Even in .NET doing something like this is
almost trivial by using separate event handlers for receiving data
from the socket - again which is why I firmly believed that something
like this could have been done via POE (yes, this would be handled via
threads - but it's done and managed by the .NET modules, and thus
require -very- little code, or understanding for that matter, from the
programmer's point of view).
Oh well - it's back to the drawing board for this one then.
--
Regards,
Chris Knipe

Hi Chris,

I've not read the whole thread in details but what you need seems fairly
simple. I think you're wrong when you said about implementing it within POE
or any NET:: module because what you are after is definitely application
layer and not module layer.

I think what you want has been covered by Carl. Put very abtractly you need

* one task (process, thread, whatever) to handle communications

This can either be one task per connection which would be easiest to implement
using the traditional process of forking on each incoming socket connection
described in the first post, or can be one process handling all concurrent
connections, possibly using POE.

I believe that the notes your read about avoiding having one task reading a
socket and another task writing a socket is to do with the actual TCP
conversation, and I can see how that could cause issues. By having one
dispatcher per connection, you can syncronise the sending / receiving per IP
connection.

* one task per work unit (each article). Have one process accept the job from
the communications task, work on the article asyncronously, and return the
status back to the communications task.

There are many different ways within unix / Perl for inter-process comms for
this part of the job. Depending on the workload, a model similar to Apache
httpd where there are maximum X processes running, which I believe
periodicallty die and get replaced to prevent them going stale.


Gary

[Robert Wohlfarth]

Post by Chris Knipe
It's times like this that I *really* wish I knew how to code in C - no
offence to anyone, again. Even in .NET doing something like this is
almost trivial by using separate event handlers for receiving data
from the socket - again which is why I firmly believed that something
like this could have been done via POE (yes, this would be handled via
threads - but it's done and managed by the .NET modules, and thus
require -very- little code, or understanding for that matter, from the
programmer's point of view).

It sounds like you want an event loop that triggers on socket activity.
Would something like AnyEvent::Socket
<http://search.cpan.org/~mlehmann/AnyEvent-7.07/lib/AnyEvent/Socket.pm>
work for you? I came across it by googling "perl socket event handler".

[lee]

This is an actual example of a communication stream between a client
and the perl server, the socket is already established, and
communication is flowing between the two parties. [C] indicates what
the client is sending to the server, and [S] indicates the responses
the server sends to the client. My comments added with # and thus
does not form part of the communication stream.
[C] Path: pathost!demo!somewhere!not-for-mail
[C] Newsgroups: misc.test
[C] Subject: I am just a test article
[C] Date: 6 Oct 1998 04:38:40 -0500
[C] Organization: An Example Com, San Jose, CA
[C]
[C] This is just a test article.
[C] .
# . indicates the end of the article. Perl now starts to do work,
processing the article it received. As this process takes time, the
main while (1) { loop reading from the socket is now blocked, causing
the server not to read any more from the socket until after the
article has been dealt with.
[C] Path: pathost!demo!somewhere!not-for-mail
[C] Newsgroups: misc.test
[C] Subject: I am just a test article
[C] Date: 6 Oct 1998 04:38:40 -0500
[C] Organization: An Example Com, San Jose, CA
[C]
[C] This is just a test article.
[C] .
# Perl server only NOW responds with an acceptance code for the first article.
We have effectively now COMPLETELY missed the second article in our
while (1) loop, as perl did not read from the socket WHILST processing
the article.

1.) What you are trying to achieve requires that the server side is
capable of fully processing every connection within a limited amount
of time. This amount of time must be so small that you do not have
to queue up anything, unless the workload is unsteady, allowing you
to process a queue during times of lower loads.

You basically have a bucket here, with an unlimited amount of garden
hoses filling water into the bucket. The bucket has a drain to let
out the water. That drain must always be large enough to let so
much water run out that the bucket never gets full. The level of
water in the bucket may change over time, and it must never flow
over.

Unless you can guarantee that the drain is always large enough, you
*must* limit the amount of water flowing into the bucket --- like by
limiting the number of concurrent connections.

2.) Number 1.), limiting the amount of incoming data, kinda already
solves the problem.

3.) With the problem solved, you can look into ways of increasing the
performance. Having one thread queuing up the incoming data and
another thread process it may be a worthwhile approach. It could
help with 1.) because you could simply look at the size of the queue
and deny all new connections when the queue has reached a given
size.

That allows you to have one queue-thread per incoming connection,
with a given maximum number N of queue-threads. That number N
already sets a reasonable measure for the number of spool-threads
processing the queue because you will never have more than N
connections with clients waiting for an answer simultaneously.

You can have another thread that monitors the queue size and decides
whether new connections are to be accepted or not.

This approach might scale well because you can distribute the
threads across several servers until you get the required
performance.

4.) You probably have an issue with the design of the protocol. No
matter how many threads you use, it always takes time to set up
another thread to queue up the incoming data. Your clients *must*
wait for an acknowledgement that data they want to sent can now be
accepted. Otherwise you may miss data during the time which it
takes to get ready to receive it. Alternatively, the clients *must*
time out after a while when they do not receive an acknowledgement
that their data has been accepted, in which case they *must* act
accordingly on their side, like trying again.


I'm not sure if this makes sense to you; it's only how I would probably
go about this problem. If you cannot solve number 4.), I don't see any
possible solution to the problem other than not handling more than one
connection at a time. However, even when handling only one connection
at a time, it is bad design when clients just send data before being
told to do so, and/or when they don't act reasonably when they don't
receive an acknowledgement that their data has been received.
Inevitably, such design will result in data being lost, no matter what
you do.

[Chris Knipe]

Are you saying the normal 'unix' way won't work? (ie. listen on socket, fork
on an accepted connection, do the work, close)

Oh - and yes, if that is the 'unix' way, then yes, it's unacceptable.
The socket CANNOT be torn down after each and every single command.
Once the client is connected to the socket, the client can send to the
tune of hundreds of commands per second. Tearing down and then
re-establishing a TCP session each time, will be just as devastating
(if not more) to performance as dealing with requests synchronously.