Date   : Mon, 22 Nov 2010 19:12:39 +0000
From   : jgh@... (J.G.Harston)
Subject: Hints'n'Tips: Detokenising BASIC code

Printing BASIC Tokens

BBC BASIC programs are tokenised, that is, BASIC keywords are stored as
one-byte values. This results in programs which execute faster and are more
compact. BASIC has code to detokenise program code when listing, but it is
unwise to try to call it directly as it is in different locations in
different versions, and also depends on workspace locations containing
values that BASIC expects, which will not be the case if trying to use the
routines when BASIC is not the current language.

The following code, which must run in the I/O processor, will search for
the token table in any version of BBC BASIC - even in non-6502 BASIC ROMs.

\ Detokenise BASIC tokens
\ ======================
\ needs, eg tbase=&70:tptr=&72 if calling from BASIC
\        eg tbase=&A8:tptr=&AA if calling as a *command
\
\ ----------------------------------
\ TokenInit - Find BASIC token table
\ ----------------------------------
\ On entry, BASIC ROM must be paged in
\ On exit,  tbase=>Start of token table
\           A,Y,tptr corrupted, X preserved
\
.TokenInit
LDY #0:STY tptr:LDA #&80:STA tptr+1    :\ tptr=>ROM start
.TokInitLp
LDA #1:CLC:ADC tptr:STA tptr           :\ Inc. tptr
LDA #0:TAY:ADC tptr+1:STA tptr+1
LDA (tptr),Y:CMP #&80:BEQ TokInit2:DEY :\ Found &80
.TokInit2
INY:LDA (tptr),Y:CMP #ASC"A":BNE TokInitLp :\ Not "A"
INY:LDA (tptr),Y:CMP #ASC"N":BNE TokInitLp :\ Not "AN"
INY:LDA (tptr),Y:CMP #ASC"D":BNE TokInitLp :\ Not "AND"
.TokInitOk
LDA tptr:STA tbase:LDA tptr+1:STA tbase+1
RTS
:
:
\ -----------------------------
\ PrToken - Print a BASIC token
\ -----------------------------
\ On entry, A=token byte
\           BASIC ROM must be paged in
\           tbase=>Start of token table, set by TokenInit
\ On exit,  A,Y,tptr corrupted, X preserved
\
.TokenPrint
PHA:LDA tbase:STA tptr           :\ Point to start token table
LDA tbase+1:STA tptr+1
.TokPrLp1
LDY #&FF
.TokPrLp2
INY:LDA (tptr),Y:BPL TokPrLp2    :\ Loop until token byte found
PLA:CMP (tptr),Y:BEQ TokPrFound  :\ Found matching token
PHA:TYA:BNE TokPrStep            :\ Step to next token
.TokPrLp3
INY:LDA (tptr),Y:BPL TokPrLp3    :\ Find next token byte
DEY:DEY
.TokPrStep
INY:TYA:SEC:ADC tptr:STA tptr    :\ Step past this token string
LDA #0:ADC tptr+1:STA tptr+1
BNE TokPrLp1                     :\ Loop to keep searching
.TokPrFound
TYA:BEQ TokPrNxt:LDY #0          :\ Skip past leading token
.TokPrLp3
LDA (tptr),Y:BMI TokPrEnd        :\ Token byte, end
CMP #32:BCC TokPrEnd:JSR OSWRCH  :\ Flag byte, end
.TokPrNxt
INY:BNE TokPrLp3                 :\ Loop back for next character
.TokPrEnd
RTS

You can test the code with the following:

   CALL TokenInit
   FOR A%=&80 TO 255:PRINT ;A%;": ";:CALL TokenPrint:PRINT:NEXT

Note that the code requires the BASIC ROM to be paged in, and also needs to
run in the I/O processor - where the ROMs are. This would normally be done
in a *command utility with code such as the following:

LDA &F4:PHA                      :\ Save current ROM
LDA &24B:CMP #&FF:BEQ errNoBASIC :\ Get BASIC ROM, error if no BASIC
AND #127:JSR SelectROM           :\ Page in BASIC ROM
...
JSR TokenInit
...
JSR TokenPrint
...
PLA                              :\ Restore previous ROM
.SelectROM
STA &F4:STA &FE30:RTS            :\ Page in ROM

This code has been tested with 6502 BASIC 1.00-4.00, 4.32, 4.40, 4.86, Z80
BASIC and PDP-11 BASIC.

See also:
http://mdfs.net/Info/Comp/BBC/ProgTips
http://beebwiki.jonripley.com/Detokenising

-- 
J.G.Harston - jgh@...