Problem handing escaped identifiers in the Repl.

[pvillads]

Please consider this set of actions in the REPL where a variable of name '$id' is defined. This variable name is escaped with the single quotes to cater for the otherwise illegal $ character.

a = 2
a: 2

2 + a
4

'$id' = 2
'$id': 2

From there it would appear that there are three variables defined: (a, b, '$id'). However, when I attempt to get the value of the '$id' variable I get into trouble:

'$id'
Error 0-1: Unexpected characters....
... many more messages ...

However, you can get the variable by escaping the escapes:

'''$id'''
2

[pvillads]

Here is a simpler repro in C#, that strips away a lot of the overhead:

namespace PowerFxQuotedNamesProblem
{
using Microsoft.PowerFx;
using Microsoft.PowerFx.Types;
using System.Diagnostics;

internal class Program
{
    static void Main(string[] args)
    {
        var engine = new RecalcEngine();

        // Put some variables into the engine scope.
        engine.UpdateVariable("a", 1);
        engine.UpdateVariable("b", 2);

        // Now check that the engine can evaluate expressions with these names.
        var result = engine.Eval("a + b");
        Debug.Assert(result.AsDouble() == 3.0, "Incorrect result when using normal variables");

        // Now add an escaped variable
        engine.UpdateVariable("'$id'", 3);

        // All the names are accounted for.
        Debug.Assert(engine.EngineSymbols.SymbolNames.Count() == 3, "Wrong number of variables");
        Debug.Assert(engine.EngineSymbols.SymbolNames.Select(n => n.Name).ToArray()[0].ToString() == "a", "Incorrect variable name");
        Debug.Assert(engine.EngineSymbols.SymbolNames.Select(n => n.Name).ToArray()[1].ToString() == "b", "Incorrect variable name");
        Debug.Assert(engine.EngineSymbols.SymbolNames.Select(n => n.Name).ToArray()[2].ToString() == "'$id'", "Incorrect variable name");

        // The 3rd variable (with the escaped name) is not found.
        try
        {
            result = engine.Eval("'$id'");
            Debug.Assert(result.AsDouble() == 3.0, "Incorrect result when using escaped variable");
        }
        catch(Exception e)
        {
            Debug.Assert(false, e.Message);
        }

        // There is no variable with this name, yet it returns a result.
        try
        {
            result = engine.Eval("'''$id'''");
            Debug.Assert(false, "Reference to unknown variable yielded a value");
        }
        catch (Exception e)
        {
        }
    }
}

}

It seems that the issue is that when a string (i.e. '$id') is parsed (through the Eval call) to be an identifier, the start and end quotes are stripped. Look in LexIdent (TexlLexer @ 1361). Here the string designating the variable is returned as $id, not '$id'. This is what causes all the mayhem going forward. There is special code to handle the '''$id'' case:

                    if (IsIdentDelimiter(PeekChar(1)))
                    {
                        // Escaped delimiter.
                        _sb.Append(CurrentChar);
                        NextChar();
                        NextChar();
                    }

which causes the name to be left as '$id' and then everything snaps into place. But the name should not be escaped. I want the name to be '$id', I do not want to have understand the name as containing quotes.

[MikeStall]

Note that the single quotes ' are not part of the identifier's name - they're just lexical hints. Much like the quotes in "hello" are not part of the string literal.