Base Object of a lexer with efficient buffer handling. In fact I believe that this is the most efficient method of buffer handling that exists! Only at line endings checks are necessary if the buffer needs refilling.This diagram illustrates how the buffer, length, position, sentinel, and newline characters are considered:
"Example Text\n ha!" bufLen = 17 ^pos = 0 ^ sentinel = 12
NB: 'n' is one character/byte
Types
TBaseLexer = object of RootObj bufpos*: int buf*: cstring bufStorage: string bufLen: int stream*: PLLStream ## we read from this stream lineNumber*: int ## the current line number sentinel*: int lineStart*: int ## index of last line start in buffer offsetBase*: int ## use ``offsetBase + bufpos`` to get the offset
- Source Edit
Procs
proc closeBaseLexer(L: var TBaseLexer) {....raises: [], tags: [].}
- Source Edit
proc getColNumber(L: TBaseLexer; pos: int): int {....raises: [], tags: [].}
- Source Edit
proc getCurrentLine(L: TBaseLexer; marker: bool = true): string {....raises: [], tags: [].}
- Source Edit
proc handleCR(L: var TBaseLexer; pos: int): int {....raises: [IOError, Exception], tags: [ReadIOEffect, RootEffect].}
- Call this if you scanned over CR in the buffer; it returns the position to continue the scanning from. pos must be the position of the CR. Source Edit
proc handleLF(L: var TBaseLexer; pos: int): int {....raises: [IOError, Exception], tags: [ReadIOEffect, RootEffect].}
- Call this if you scanned over LF in the buffer; it returns the position to continue the scanning from. pos must be the position of the LF. Source Edit
proc openBaseLexer(L: var TBaseLexer; inputstream: PLLStream; bufLen: int = 8192) {. ...raises: [IOError, Exception], tags: [ReadIOEffect, RootEffect].}
- Source Edit