Date : Sat, 02 Jan 1988 10:13:57 est
From : marwood@dmc-crc.arpa (G. J. Marwood)
Subject: PCPI Applicard character loss with MEX
Thanks to everyone who replied to my message regarding character
loss with MEX1.65 capture files, viz:
>> The CP/M system that I use is an Apple ][+ and PCPI
>> Applicard. While developing a new overlay and PCPI driver
>> using interrupts, to avoid the usual character loss at high
>> baud rates, I have come across a problem which seems to
>> afflict the Apple/Applicard even without interrupts and at
>> 1200 baud. At this baud rate, I experience no character loss
>> on screen during normal operation. However, if I invoke a
>> capture file, I do experience character loss at the place
>> where the capture buffer is being written to disk (in this
>> case it is the Applicard add-on Ramdisk). This loss occurs
>> both on screen and in the captured file and, typically may
>> be half a dozen or so characters. It seems that this bug has
>> always been present in the MEX software that I have used,
>> though I had not noticed it because I don't usually capture
>> files as large as the capture buffer (16-17K). Another local
>> user has done a similar test with MEX1.65B and a non-Apple
>> CP/M machine and the loss does not occur. It does not,
>> therefore, seem to be a basic problem with MEX. If any
>> Apple/Applicard user has noticed this before and has any
>> clues about solving the problem, I would be interested to
>> know.
This original message was obviously not totally clear, particularly
to those not familiar with the Apple/Applicard.
I should, therefore, make it clear that the Applicard is running
as a Z80 co-processor in the Apple. The modem is connected into
the Apple part of the system and a Driver in the Apple directly
handles the interface with the modem (a Super Serial Card). In
this Driver, which is entered following an Apple IRQ interrupt
when a modem character arrives, there is a 4K buffer. This
buffer is of a sufficient size (by a long way) that it should be
unnecessary to use x-on/x-off protocol.
The Ramdisk, which I use for the capture file, is the PCPI (2 in
fact) Ramdisk which plugs directly onto the Applicard. It,
therefore, has nothing to do with the Apple bus and is not
controlled in the same way as drives which are on the Apple bus.
I am aware that Apple interrupts are a problem when Apple 5 1/4
drives and standard drive controller are used, but they are not
involved in this problem.
The lost character problem, when the capture buffer is flushed to
the disk file, occurs at 1200 baud, not just at the higher baud
rates that I am really interested in.
The following are some of the responses that I have received and
some of my comments relating to them:
> From: gatech!killer!bdav@hao.UCAR.EDU (Bob Davenport)
>
> Gordon, I have the same problem. But not with the same
> hardware. The problem seems to be that an xoff is not sent
> while the buffer is being written to disk. My OS(TurboDOS)
> performs a read after write to verify all data being written.
> With a large capture buffer, this takes about 1.5 seconds, long
> enough to lose 10 to 15 characters. I have tried decreasing
> the capture buffer size, but this does not seem to help. The
> only method I have found that works, is to manually send an
> xoff (^S) to stop the transmission, go to command mode, write
> the buffer, re-open the buffer and then continue with the
> capture. Not elegant, or problem solving, but it works.
>
> bob Davenport
Bob - What system are you using ? Regarding
the use of x-on/x-off, perhaps this is
necessary with your system, though another
MEX user local to me has done a test with a
PMC 101 Micromate and hard disk, with STAT
XON and STAT XOFF both OFF and has observed
no character loss. I had thought that
setting STAT QUEUE ON would ensure that there
was no character loss during the disk write,
but tests that I have done do not confirm
this. I am not sure how often the modem
status is being checked during the disk write
operation. In the Driver which I have on the
Apple side of the system, I do detect when
the (4K) buffer is filling up and
automatically send out a ^S. When MEX has
reduced the backlog in the buffer a ^Q is
automatically sent to restart the remote.
However, this only seems to be necessary for
my system at 9600 baud. At 1200 baud the
buffer never seems to be filled. Despite
this, I do lose 2 characters during the disk
write.
____________________________________
> From: "Richard_F_Koehler.WBST128"@Xerox.COM
>
> Gordon,
> I use an Apple II+ with a PCPI card also. I believe the reason
> for the character loss is that Apple controls and writes to the
> disk drive under software control rather than by using a
> hardware disk controller. Since the timing is very critical,
> all interrupts must be disabled while data is being written (or
> read) to the disk drive. I also run MEX on my system but I have
> not had occasion to exceed the capture buffer and I run at 1200
> baud or less, so I have not experienced the problem you
> described.
>
> Rick Koehler
Rick - As I have noted above the disk I am
using for the capture file is the PCPI add-on
Ramdisk. This has no connection with the
Apple interrupts. Also, as noted, I am
seeing this character loss at 1200 baud
during the disk write. The original
character loss problem, which started all
this off for me was the loss that occurs at
the begining of each line (this is nothing to
do with capture buffers) at 1200 or 2400 baud
and higher (depending on whether you are
using a ][+ or ][e and which 80-col card).
This loss is a result of the time taken by
the 80-col card to scroll, which required re-
writing the whole screen, whenever there is
a CR and you are at the bottom of the screen.
I think that it is also a result of the
Apple/Applicard interface protocol, though I
have not totally sorted this out yet. I am
speculating that even with the modem port
QUEUE ON, character loss occurs because the
burst of characters which comes out of the
queue once the disk write is complete is also
too fast (faster than the normal 1200 baud
character arrival rate) for the 80-col system
to handle. In some way this may be linked to
getting the characters into the capture file.
This explanation may be nonsense, but it is
the best that I can come up with at the
moment. My lack of understanding of what the
modem port queue is doing, and when it does
it, is a limitation here. The above comments
on character loss apply to MEX running with
Apple interrupts off and, therefore, no
buffering in the Apple Driver.
____________________________________
> From: "Richard_F_Koehler.WBST128"@Xerox.COM
>
> I suspect that even though you are writing to the RAMdisk, the
> disk driver software still goes away completely from listening
> to incoming characters for a while. You might want to check
> there.
>
> Rick
Rick - You are right, but I am not sure with
STAT QUEUE ON, how often the modem is checked
for new characters. In any case, in my
arrangement the buffer in the Apple should
cope with any wait while the disk is written.
In fact my MEX overlay does not actually
check the modem status (for new characters),
it check for a difference in the 4K buffer
input and output pointers.
____________________________________
> From: Michael Neary <MNeary.es@Xerox.COM>
>
> Gordon,
>
> I don't use MEX, but some general comments may help you:
>
> Your MEX has to be set so that it sends flow control characters
> (usually X-ON/X-OFF) when writing to 'disk'. And the other end
> of your conversation has to be willing to pause when it
> receives flow control characters.
>
> If either of these is not true, then you will lose characters
> when you go to disk, no matter what communications program you
> are using. One possible exception is if your system can go to
> 'disk' without disabling interrupts (rare).
>
> I would troubleshoot in this order:
> 1) Is your MEX set to enable flow control?
> 2) Are you using the same host as your friend is using? Does
> that host support flow control? Is it the same flow
> control as you have enabled on your MEX?
> 3) Does your friend's RAM-Disk leave interrupts enabled in
> RAM-Disk Writes?
>
> Cheers,
> ~ Mike
Mike - As noted above, I think that my
comments regarding the buffer in the Apple
answer your item 1. Item 2 - All the hosts
involved support flow control. Item 3 - My
friend is not using a RAM disk and he is not
using interrupt driven MEX I/O.
____________________________________
> From MEDIN-T@SHARK.NOSC.MIL Wed Dec 30 12:27:09 1987
>
> Well its not a bug its the design of the diskette driver. Apple
> must have complete control of the cpu while they write the
> diskette so they turn off interupts while they write the
> diskette and vola! you lose characters coming down the line. I
> suspect the ram disk tho quicker still locks out interupts. The
> only solution we could come up with on kermit-65 was flow
> control. So we stop flow before we write the diskette. If you
> cant do flow control then i think you are stuck.
Ted - As noted above the Apple disks are not
involved. The PCPI Ramdisk has no influence
on the Apple interrupts.
____________________________________
> From: Bridger Mitchell <bridger%rcc@rand-unix.ARPA>
>
> If mex is "smart", it should be polling the modem input status
> between each sector-write call in the flushing routine, and
> buffering any received characters for "catch-up" processing
> when the buffer-flushing is completed. [Ron Fowler: does mex do
> this?] But if not, it's likely that characters are arriving at
> the modem port before the buffer-flushing operation completes.
> Unless the arriving bytes are buffered before they reach mex,
> the first ones will be overwritten by the later ones, creating
> a gap in the input stream received by the mex software.
Bridger - As you will have noted above, the
arriving bytes are buffered before they reach
MEX and this buffer does exercise automatic
flow control. However, despite this, I am
losing characters when the disk is written.
> The real time required to flush a buffer is system-dependent,
> probably dominated by the drive's step rate and the track
> number of the capture file, and also affected by the CPU clock
> rate. Another system with faster disks, a faster cpu, or even a
> differently located file, could have no lossage. On your
> system, if you capture to a ram disk, for example, there may be
> no lossage.
As noted, I am using a Ramdisk, operating
directly in the Z80 environment. The CPU
clock rate is 6 MHz at the moment (it can be
increased to 10MHz).
>
> Many systems have 1-3 byte buffering in the UART chip that
> receives the bytes from the modem; different systems use
> different UARTS. More buffering can be obtained by writing an
> interrupt-service routine for the modem ports; only a few
> bioses have such support, but it may be possible to write the
> mex overlay that way.
>
> --bridger
I have implemented interrupts for both
receive and transmit (on a 6551 ACIA) in the
Apple modem-port Driver. In fact I know that
the characters are coming from the modem into
the 4K Apple buffer as I separately send them
to the 80-col display directly from the
Apple (the display characters do not pass
through MEX). There is no character loss on
the screen under this arrangement, only from
the capture file. (In case there is some
confusion here, there are two interrupt
modes. In one, characters do not go via MEX
to the screen in the other they do. The
first scheme will work at 9600 baud, the
second will not. I have included the latter
to allow operation at lower baud rates and to
retain the possibility of using the MEX
terminal emulation modules).
___________________________________
> From: jimd%slacbbs.Berkeley.EDU@ucbvax.Berkeley.EDU
>
> I may be telling you something you already know, but MEX is
> supposed to send an XOFF, save the disk buffer, then send an
> XON. Possibly whatever system your calling either doesn't
> respond to XON/XOFF or the modem may have a buffer that suspends
> the result of the XOFF (so you lose stuff). I do not have an
> Apple computer, but I use MEX (version 1.14). If you have the
> source, check the area that saves the buffer, and make sure it
> sends the XON/XOFF. Oh, and another thing I just thought of: MEX
> is supposed to have a 'backup buffer' where stuff can go after
> an XOFF is sent. Check this out too; it may be disabled or very
> small on your version. BTW, is MEX1.65 the commercial version?
> Or is the version number higher because of the port to Apple?
>
> Jim Dumser
Jim - I think that I have covered most of
your points above. I suspect that the 'back-
up' buffer that you are referrring to is the
modem port queue (defined by PQSIZE in
MEXPAT). With no interrupts on (in the
Apple) the character loss occurs with STAT
QUEUE ON or OFF. MEX1.65 is also known as
MEXPLUS of which it is the latest major
revision. It has many features which do not
exist in the PD MEX1.14. The features are
described in the file MEX16SUM.WQ which is
available by FTP from SIMTEL20 in
PD1:<CPM.MEX>. There is also a file on GEnie
called MEX16.AZN which provides a
description.
____________________________________
Well, I have found this exchange of information quite useful and
thought provoking. I am now beginning to draw some tentative
conclusions. I think that I have been observing two different
loss mechanisms here. With recent 1200 baud tests that I have
done with the earlier MXH-AP55.ASM overlay and RINGBUF.A65 Apple
Driver, I have found that I get character loss with and without
STAT QUEUE ON and OFF (Note: this Driver will not run at 9600
baud, this is why I have been developing a new one!). I am not
sure why the QUEUE is not preventing this loss, but I think it is
to do with the Apple/Applicard interface and the time taken to
write characters to the screen. With interrupts ON, this earlier
version with its 256 byte buffer, does prevent character loss in
the capture file. With my revised Driver I always lose 2
characters at the time the capture buffer is written to disk (the
loss in the non-interrupt mode referred to above is always
several more characters than this). I think, therefore, that I
still have a bug to sort out in the Driver. (I should also point
out that I cannot in the end use the modem port QUEUE as, at 9600
baud, the capture process gets so far behind that the queue is
eventually overwritten).
Thank you again for all of the responses. I still have some work
to do!
Gordon