f starting (row/line, column). The token position is used only for error reporting and can be None or empty. Raise ParseError on errors. The ParseError.args is a tuple of: (token_string, position, error message) You can use this tokenizer as a base to create specialized tokenizers for your custom algebra by subclassing BooleanAlgebra. See also the tests for other examples of alternative tokenizers. This tokenizer has these characteristics: - The `expr` string can span multiple lines, - Whitespace is not significant. - The returned position is the starting character offset of a token. - A TOKEN_SYMBOL is returned for valid identifiers which is a string without spaces. - These are valid identifiers: - Python identifiers. - a string even if starting with digits - digits (except for 0 and 1). - dotted names : foo.bar consist of one token. - names with colons: foo:bar consist of one token. - These are not identifiers: - quoted strings. - any punctuation which is not an operation - Recognized operators are (in any upper/lower case combinations): - for and: '*', '&', 'and' - for or: '+', '|', 'or' - for not: '~', '!', 'not' - Recognized special symbols are (in any upper/lower case combinations): - True symbols: 1 and True - False symbols: 0, False and None z