Tokenise - Tokenise BBC BASIC code
                   ==================================
                   File: BLib.Tokenise - Update: 1.03
                Author: J.G.Harston - Date: 20-Nov-2010

  The Tokenise library provides functions for tokenising BBC BASIC program
code. It runs on 6502, Z80, PDP-11, 32000, RISC OS, DOS and Windows BBC
BASIC.

Convert line of text to BASIC tokenised format
==============================================
DEFFNTokenise_(text$)

On entry: text$=plain text 
On exit:  returns the tokenised version of the supplied text

  The text is tokenised according to the tokeniser in the BASIC being run.
For example, extended tokens will not be recognised when running on 6502
BASIC, they will be tokenised as two-byte tokens on RISC OS BASIC, and as
one-byte tokens in the range &01-&1F on Windows BASIC.

  If the supplied text has a leading line number, the returned string
starts with the line number in plain text.

Notes
=====
  If the supplied string includes G. and a 1-digit or 2-digit line number
(eg G.10) the tokeniser buffer may overflow as the line number is converted
into the four-byte tokenised format.

Version History
===============
1.00 28-Jun-2006 First version written for 6502, ARM and Windows.
1.01 08-Feb-2009 Extra word on stack in ARM BASIC 1.3x.
1.02 28-Feb-2009 Added PDP-11, DOS and 32000 functions.
1.03 20-Nov-2010 Extra word on stack in 6502 BASIC 4.