alt

This is of a series on parsing with Kotlin which starts here

Let’s look at these expressions:

   font
   "some string"
   (font 12.5 12.5)

Now consider the plight of our parser: it can only read one character at a time; it cannot put it back. More precisely, this is the interface we are consuming:

   interface Iterator<T> {
      fun next(): T
      fun hasNext(): Boolean
   }

(more, more precisely we are dealing with Iterator<Char>)

When we look at our examples, and we are at the beginning of any of our examples, hasNext() will tell us we can proceed, and read() will gives us one of 'f', '"', or '('.

We see an 'f' and we know that we have a symbol in our hands, and now we have to read the rest of the symbol, checking hasNext() before we read(), and when we get to the end of the symbol hasNext() returns false, and we know we are done.

In the case of the string, once we read the second '"', we also know we are done. For numbers is more of the same (read digits, dots and minus signs…). Alas, in the case of both numbers and symbols, we only know that we are done once we have read one character past the chunk that we are interested in: we need to implement takebacksies.

That is where Scanner<T> comes in:

   interface Scanner<T> {
      fun get(): T?
      fun peek(): T?
   }

get() gets us the next T available, and peek() reads it, shows it to us, and puts it back where it came from-ish. Well, excepts for I said we cannot put anything back with iterators, so when we peek() we just read() the next T from the iterator, then hide it behind our backs (unless we have already hidden it behind our backs, etc., etc.). get() is an exercise of calling next() judiciously or passing forward the T that we had previously peeked(). E.g.

CharScanner⌗ 

 class CharScanner(private val s: Iterator<Char>) : Scanner<Char> {
     var next: Char? = null
     override fun get(): Char? = 
         if (next != null) { 
             val ret = next
             next = null
             ret
         } else {
            if (!s.hasNext()) null else s.next()
         }

    override fun peek(): Char? =
        if (next != null) 
            next
        else {
            if (!s.hasNext()) null else { 
                val ret = s.next()
                next = ret
                ret
            }
         }
}

I really love the way in which get() and peek() exhibit symmetry, one producing next and one consuming it, from opposite sides of mirror if statements.

This code exhibits another key pillar of Kotlin: nullable types allow the embrace of null in a safe way by enabling the developer to specify whether a variable can be null, and enabling the compiler to stop us from allowing a null where it don’t belong.

Next we’ll see how we can go from Scanner<Char> to Scanner<Token>.