[mayankdhingra02]

Native Unicode in a JSON Pointer is currently converted into UTF-8 bytes
and then read as Latin-1 by the escape codec. For example, /café resolves
as /café. If both keys exist, a JSON Patch operation can silently change
the wrong member.

Preserve native characters while decoding the existing compatibility
escape spellings separately. from_parts uses the same decoder. Unicode
normalization, the unicode_escape option, URI percent-decoding and
surrogate validation retain their existing roles.

The regressions cover pointer builders, mixed native/escaped characters,
astral and combining characters, backslashes, tilde and percent handling,
and all JSON Patch operations from Python objects and JSON text. The
wrong-key cases fail on the unchanged base. An additional bounded oracle
checks ASCII compatibility against the old decoder and native keys
against Python's JSON encoder.

Validation on macOS:

• All current CI Python versions, 3.10–3.15: 3,057 tests pass with one
  existing skip each.
• hatch run lint, hatch run typing, hatch run cov and
  hatch run docs:build pass. The changed executable lines and branch
  arcs are covered.
• hatch build and strict Twine checks pass.

Linux and Windows runs remain for CI.

[jg-rp]

Hi @mayankdhingra02,

I agree that the unicode_escape feature does have some issues, and it should never have been enabled by default.

But I don't think your solution matches the intended behaviour (I have not managed to communicate that well), and there's a lot of noise in test_json_pointer_unicode.py that's obscuring the bug we need to test.

These tests should go some way to demonstrating the intended behaviour.

@pytest.mark.parametrize(
    ("pointer", "data", "want", "want_unescaped"),
    [
        ("/café", {"café": 7}, 7, 7),
        ("/漢字", {"漢字": 7}, 7, 7),
        ("/😀", {"😀": 7}, 7, 7),
        ("/e\N{COMBINING ACUTE ACCENT}", {"é": 7}, 7, 7),
        ("/café~1part~0name", {"café/part~name": 7}, 7, 7),
        ("/café%2F", {"café%2F": 7}, 7, 7),
        ('/café"quoted', {'café"quoted': 7}, 7, 7),
        ("/café\n\t\0", {"café\n\t\0": 7}, 7, 7),
        ("/café\\😀", {"café\\😀": 7}, 7, 7),
        ("/café\\\\😀", {"café\\\\😀": 7}, 7, 7),
        ("/\\uD83D\\uDE00", {"😀": 7, "\\uD83D\\uDE00": 42}, 42, 7),
    ],
)
def test_unicode_pointer_tokens(
    pointer: str,
    data: dict[str, object],
    want: object,
    want_unescaped: object,
) -> None:
    p = JSONPointer(pointer, unicode_escape=False)
    assert p.resolve(data) == want
    assert str(p) == pointer

    p_unescaped = JSONPointer(pointer, unicode_escape=True)
    assert p_unescaped.resolve(data) == want_unescaped
    assert str(p_unescaped) == _unescape(pointer)

Where _unescape() replaces JSONPointer._unicode_escape(). It would need to be a simpler, more tolerant version of unescape_string():

python-jsonpath/jsonpath/unescape.py

Line 10 in bd91a15

def unescape_string(value: str, token: Token, quote: str) -> str: