<section id="softwarehistory" xreflabel="Usenet software: a historical perspective">
<title>Usenet software: a historical perspective</title>

<para>This section comprises excerpts from a well-known Usenet Periodic
Posting document which was last changed in Feb 1998. Our copy of that old
document was picked up from</para>

<para><literal>ftp://rtfm.mit.edu/pub/usenet-by-hierarchy/news/software/b/Usenet_Software:_History_and_Sources</literal></para>

<para>We suspect other copies will also be found elsewhere. The physical
file on the FTP server appears to have been touched last on 29 Dec
1999. The first few lines of the archived file provide information about
the origin of this document and its authors:</para>

<programlisting>
<![CDATA[
Date: Tue, 28 Dec 1999 09:00:19 GMT
Supersedes: <FMMECL.58s@tac.nyc.ny.us>
Expires: Fri, 28 Jan 2000 09:00:19 GMT
Message-ID: <FnG10J.HAo@tac.nyc.ny.us>
From: netannounce@deshaw.com (Mark Moraes)
Subject: Usenet Software: History and Sources
Newsgroups: news.admin.misc,news.announce.newusers,news.software.readers,news.software.b,news.answers
Followup-To: news.newusers.questions
Approved: netannounce@deshaw.com (Mark Moraes)

Archive-name: usenet/software/part1
Original-from: spaf@cs.purdue.edu (Gene Spafford)
Comment: edited until 5/93 by spaf@cs.purdue.edu (Gene Spafford)
Last-change: 9 Feb 1998 by netannounce@deshaw.com (Mark Moraes)
Changes-posted-to: news.admin.misc,news.misc,news.software.readers,news.software.b,news.answers
]]>
</programlisting>

<para>We have been seeing this document as a periodic posting in
<literal>news.announce.newusers</literal> since the early nineties, and
it has always been our final reference on the history of Usenet
server software. We reproduce excerpts below, retaining the portions
which discuss server software, and removing discussions of client
software, newsreaders, software for non-Unix operating systems,
<emphasis>etc.</emphasis> All quoted portions are reproduced unedited
other than changing FTP file paths to the modern URL format.  We have
added our comments emphasised, in separate paragraphs.  We feel the
information captured here is essential reading for anyone interested in
Usenet server software.
</para>

<para>If anyone can point us to a fresher version of this document, in
case it is still maintained, we will be happy to refer to that version
instead of this one, though we suspect the reader will not suffer due to
the four-year gap; most of the information reproduced below is
historical anyway.
</para>

<section><title>The quoted excerpts</title>

<para>Currently, Usenet readers interact with the news using a number of
software packages and programs.  This article mentions the important
ones and a little of their history, gives pointers where you can look
for more information and ends with some special notes about ``foreign''
and ``obsolete'' software.  At the very end is a list of sites from which
current versions of the Usenet software may be obtained.
</para>

<para> ... </para>


<section><title>History</title>

<para>Usenet came into being in late 1979, shortly after the release of V7
Unix with UUCP.  Two Duke University grad students in North Carolina,
Tom Truscott and Jim Ellis, thought of hooking computers together to
exchange information with the Unix community.  Steve Bellovin, a grad
student at the University of North Carolina, put together the first
version of the news software using shell scripts and installed it on the
first two sites: <literal>unc</literal> and <literal>duke</literal>. At
the beginning of 1980 the network consisted of those two sites and
<literal>phs</literal> (another machine at Duke), and was described
at the January Usenix conference.  Steve Bellovin later rewrote
the scripts into C programs, but they were never released beyond
<literal>unc</literal> and <literal>duke</literal>. Shortly thereafter,
Steve Daniel did another implementation in C for public distribution.
Tom Truscott made further modifications, and this became the ``A'' news
release.</para>

<para>In 1981 at U. C. Berkeley, grad student Mark Horton and high school
student Matt Glickman rewrote the news software to add functionality
and to cope with the ever increasing volume of news -- ``A'' News was
intended for only a few articles per group per day.  This rewrite was
the ``B'' News version.  The first public release was version 2.1 in
1982; the 1.* versions were all beta test.  As the net grew, the news
software was expanded and modified.  The last version maintained and
released primarily by Mark was 2.10.1.</para>

<para>Rick Adams, at the Center for Seismic Studies, took over
coordination of the maintenance and enhancement of the B News software
with the 2.10.2 release in 1984.  By this time, the increasing volume
of news was becoming a concern, and the mechanism for moderated groups
was added to the software at 2.10.2.  Moderated groups were inspired by
ARPA mailing lists and experience with other bulletin board systems.
In late 1986, version 2.11 of B News was released, including a number
of changes to support a new naming structure for newsgroups, enhanced
batching and compression, enhanced <literal>ihave/sendme</literal>
control messages, and other features.</para>

<para>The final release of B News was 2.11, patchlevel 19.  B News has
been declared ``dead'' by a number of people, including Rick Adams, and
is unlikely to be upgraded further; most Usenet sites are using C News
or INN (see next paragraphs).</para>

<para>In March 1986 a package was released implementing news transmission,
posting, and reading using the Network News Transfer Protocol (NNTP)
(as specified in RFC 977).  This protocol allows hosts to exchange
articles via TCP/IP connections rather than using the traditional UUCP.
It also permits users to read and post news (using a modified news user
agent) from machines which cannot or choose not to install the Usenet
news software.  Reading and posting are done using TCP/IP messages to a
server host which does run the Usenet software.  Sites which have many
workstations like the Sun and SGI, and HP products find this a convenient
way to allow workstation users to read news without having to store
articles on each system.  Many of the Usenet hosts that are also on the
Internet exchange news articles using NNTP because the load impact of NNTP
is much lower than UUCP (and NNTP ensures much faster propagation).</para>

    <para><emphasis>Our comments: This remark about relative loadings
    of UUCP and NNTP is no longer applicable with faster machines and
    networks, and with hugely increased traffic volumes. Today's desktop
    computers, let alone servers, can all handle both NNTP and UUCP loads
    effortlessly, if traffic volumes can be restricted. This is partly
    due to performance enhancements to UUCP as embodied in Taylor UUCP,
    and partly due to vastly faster processors.</emphasis></para>

<para>NNTP grew out of independent work in 1984-1985 by Brian Kantor
at U.  C.  San Diego and Phil Lapsley at U. C. Berkeley.  Primary
development was done at U. C. Berkeley by Phil Lapsley with help from
Erik Fair, Steven Grady, and Mike Meyer, among others.  The NNTP package
(now called the reference implementation) was distributed on the 4.3BSD
release tape (although that was version 1.2a and out-of-date) and is
also available on many major hosts by anonymous FTP.  The current
version is 1.5.12.2.  It includes NOV (News Overview -- see below)
support and runs on a wide variety of systems.  It is available from
<literal>ftp.academ.com:/pub/nntp1.5/nntp.1.5.12.2.tar.gz</literal>.
For those with access to the World-Wide Web on the Internet, the
WWW page <literal>http://www.academ.com/academ/nntp.html</literal>
contains a description and news about NNTP.  A different
variant, called nntp-t5, implements many of the extensions
provided by INN (including NOV support).  It is available from
<literal>ftp.uu.net:/networking/news/nntp/nntp-t5.tar.gz</literal>.
</para>

<para>One widely-used version of news, known as C News, was developed
at the University of Toronto by Geoff Collyer and Henry Spencer.  This
version is a rewrite of the lowest levels of news to increase article
processing speed, decrease article expiration processing and improve the
reliability of the news system through better locking, etc.  The package
was released to the net in the autumn of 1987.  For more information,
see the paper ``News Need Not Be Slow,'' published in The Winter 1987
Usenix Technical Conference proceedings.  This paper is also available
from <literal>ftp://ftp.cs.toronto.edu/doc/programming/c-news.*,</literal>
and is recommended reading for all news software programmers.  The most
recent version of C News is the Sept 1994 ``Cleanup Release.''  C News
can be obtained by anonymous ftp from its official archive site,
<literal>ftp.cs.toronto.edu:pub/c-news/c-news.tar.Z</literal>.  </para>

    <para><emphasis>Our comments: C News is no longer maintained by
    anyone that we know, other than ourselves. However, after fixing
    the remaining bugs in the source, we have not found the need for
    further maintenance.  NNTPd from Brian Kantor and Phil Lapsley is
    in the same state, but we are working on enhancements to the source
    for access control and other functionality.</emphasis></para>

<para>Another Usenet system, known as InterNetNews, or INN, was written by
Rich Salz <literal>(rsalz@uunet.uu.net)</literal>.  INN is designed to run
on Unix hosts that have a socket interface.  It is optimized for larger
hosts where most traffic uses NNTP, but it does provide full UUCP support.
INN is very fast, and since it integrates NNTP many people find it easier
to administer only one package.  The package was publicly released on
August 20, 1992.  For more information, see the paper ``InterNetNews:
Usenet Transport for Internet Sites'' published in the June 1992
Usenix Technical Conference Proceedings.  INN can be obtained
from many places, including the 4.4BSD tape; its official
archive site is <literal>ftp.uu.net</literal> in the directory
<literal>/networking/news/nntp/inn</literal>.  Rich's last official
release was 1.4sec in Dec 1993.</para>

    <para><emphasis>Our comments: The original paper by Rich Salz about
    INN, where he proposed the design of an alternate Usenet server
    software, is a must-read for readers interested in Usenet server
    software. So is the paper by C News authors, cited before it. Most of
    the issues that Rich Salz had with C News, as stated in his paper,
    were very relevant at that time. Today, with the current version of
    NNTPd and the incorporation of the message ID daemon and NOV, these
    issues are no longer relevant, and the choice of C News+NNTPd versus
    INN is now based more on the level of maintenance of source code,
    familiarity and personal preferences than on core design factors.
    </emphasis></para>

<para>In June 1995, David Barr began a series of unoffical releases
of INN based on 1.4sec, integrating various bug-fixes, enhancements
and security patches.  His last release was 1.4unoff4, found in
<literal>ftp://ftp.math.psu.edu:/pub/INN</literal>.  This site is also the
home of contributed software for INN and other news administration
tools.</para>

<para>INN is now maintained by the Internet Software Consortium
<literal>(inn@isc.org)</literal>.  The official INN home is now
<literal>http://www.isc.org/isc/</literal> and the latest version (1.7.2)
can be obtained from <literal>ftp://ftp.isc.org/isc/inn/</literal>.</para>

    <para><emphasis>Our comments: The URL for the INN home page above
    is probably incorrect. Try http://www.isc.org/products/INN/.
    </emphasis></para>

<para>Towards the end of 1992, Geoff Collyer implemented NOV (News
Overview): a database that stores the important headers of all news
articles as they arrive.  This is intended for use by the implementors
of news readers to provide fast article presentation by sorting and
``threading'' the article headers.  (Before NOV, newsreaders like
<literal>trn</literal>, <literal>tin</literal> and <literal>nn</literal>
came with their own daemons and databases that used a nontrivial amount
of system resources).  NOV is fully supported by C News, INN and NNTP-t5.
Most modern news readers use NOV to get information for their threading
and article menu presentation; use of NOV by a newsreader is fairly easy,
since NOV comes with sample client-side threading code.
</para>

<para> ... </para>

<para>Details on many other mail and news readers for MSDOS, Windows and
OS/2 systems can be found in the FAQ posted to
<literal>comp.os.msdos.mail-news</literal>.</para>

<programlisting>
<![CDATA[
  <ftp://rtfm.mit.edu/pub/usenet/comp.os.msdos.mail-news/intro>
  <ftp://rtfm.mit.edu/pub/usenet/comp.os.msdos.mail-news/software>
]]>
</programlisting>

</section>

<section><title>Newsfeed management software</title>

<para><emphasis role=bold>Gup</emphasis>, the Group Update Program is a Unix
mail-server program that lets a remote site change their newsgroups
subscription on their news feed without requiring the intervention of
the news administrator at the feed site.  Gup operates with the INN
(and likely the C News) batching mechanisms.  The news administrators
at the remote sites simply mail commands to gup to make changes to
their own site's subscription list.  The mail/interface is password
protected.  Gup checks the requests for valid newsgroup names,
patterns that have no effect and so on. Gup's authors are Mark
Delany (<literal>markd@mira.net.au</literal>) and Andrew Herbert
(<literal>andrew@mira.net.au</literal>).  Its official FTP location
is <literal>ftp.mira.net.au:/unix/news/gup-0.4.tar.gz</literal>,
but since that's not as well connected as UUNET, people are strongly
advised to obtain it from a mirror site, <emphasis>e.g.</emphasis>
<literal>ftp.uu.net:/networking/news/misc/gup-0.4.tar.gz</literal>.</para>

<para><emphasis role=bold><literal>dynafeed</literal></emphasis> is
a package from Looking Glass Software Limited that maintains a
<literal>.newsrc</literal> for every remote site and generates the batches
for them.  Remote sites can use UUCP or run a program to change their
<literal>.newsrc</literal> dynamically.  It comes with a program that the
remote site can run to monitor readership in newsgroups and dynamically
update the feed list to match reader interest.  The goal of this is
to get a feed that sends only exactly the groups currently being read.
<literal>dynafeed</literal> can be obtained from
<literal>ftp://ftp.clarinet.com/sources/dynafeed.tar.Z</literal>.</para>

</section>

<section><title>News processing software</title>

<para>Software also exists to automatically archive Usenet newsgroups.
The package <literal>rkive</literal>, written by Kent Landfield
(<literal>kent@sterling.com</literal>) can be configured to archive
news automatically based on different headers -- Archive-Name,
Volume-Issue, Chronological, Subject and External-Command to
name a few.  It can be run in batch mode from the command line or
from cron.  It can also be installed in the <literal>sys</literal>
or <literal>newsfeeds</literal> file to process articles as they are
received.  <literal>rkive</literal> supports local spool directories as
well as NNTP based access.  <literal>rkive</literal> is available via
FTP from <literal>ftp://ftp.sterling.com/rkive</literal>.  </para>

<para>Newsclip is a programming language for writing news
filtering programs, from Looking Glass Software Limited, marketed
by ClariNet Communications Corp.  It is C-like, and translates to
C, so a C compiler is required.  It has data-types to represent
the kinds of things found in article headers and bodies.  It can
maintain databases of users, message-ids, patterns, subjects, etc.
These can be used to decide whether to ignore or select an article.
Newsclip can either operate as a standalone program or as part
of rn. It is free for non-commercial use and is available from
<literal>ftp://ftp.clarinet.com/sources/nc.tar.Z</literal>.  Contact
<literal>clari-info@clarinet.com</literal> with a subject line of
``newsclip'' for more info.
</para>

</section>

<section><title>Commercial software</title>

<para>DNEWS is a commercial product from NetWin.  DNEWS licenses
are provided free to educational institutions for non profit
use. With DNEWS, the news is stored in a database so as not to
overload the raw file system.  DNEWS supports 'sucking' where only
groups which users read are pulled over from the feeder site. DNEWS
is currently known to run on VMS, Windows NT, Solaris, SunOS,
Unixware, HP/UX.  DNEWS binaries are available by anonymous FTP from
<literal>ftp://ftp.std.com/ftp/vendors/netwin/dnews</literal> or from
<literal>http://world.std.com/~netwin/</literal> DNEWS sources can be
obtained on request, see the file <literal>source.txt</literal> in the
FTP area for more information.</para>

<para><emphasis>Our comments: The information on DNEWS may be dated. We
have been seeing DNEWS on their own Website for quite a few years now.
Check <literal>www.netwinsite.com</literal>. Moreover, there are other
commercial Usenet server software systems available, including the one
bundled with the Internet Information Server of Microsoft Windows NT and
the ones from iPlanet. And for carrier class systems, there are many
commercial Usenet routers available.
</emphasis></para>

</section>

<section><title>Special note on ``notes'' and old versions of news</title>

<para> ... </para>

<para>``B'' news software is currently considered obsolete.  Unix sites
joining the Usenet should install C news or INN to ensure proper
behavior and good performance.  Most old B news software had
compiled-in limits on the number of newsgroups and the number of
articles per newsgroup; the increasing volume of news means that B
news software cannot reliably cope with a moderately-full newsfeed.
</para>
</section>

</section>

</section>
