Skip to content

Fetch Unicode columns as wide characters - #416

Open
quinnj-bot[bot] wants to merge 4 commits into
mainfrom
claude/wide-character-data
Open

quinnj-bot[bot] wants to merge 4 commits into
mainfrom
claude/wide-character-data

Conversation

@quinnj-bot

@quinnj-bot quinnj-bot Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Why

On Windows, text from nvarchar columns comes back as invalid strings, with characters outside the ANSI code page replaced by ? (#379). ODBC.jl fetched every character column as SQL_C_CHAR, and drivers convert SQL_C_CHAR text to the client character set: the ANSI code page on Windows, and the locale's character set on Linux and macOS. ODBC.jl then read those bytes as UTF-8. When the client character set is not UTF-8, columns the driver reports as Unicode are now fetched as SQL_C_WCHAR and transcoded to UTF-8.

Fixes #379. #310 (cp1252 text from Oracle and Access) has the same cause for Unicode columns on a Windows ANSI code page.

What changed

  • fetchtypes maps SQL_WCHAR, SQL_WVARCHAR, and SQL_WLONGVARCHAR to SQL_C_WCHAR when ODBC.utf8client() is false: a Windows ANSI code page other than 65001, or a non-UTF-8 LC_CTYPE codeset elsewhere. UTF-8 clients still fetch those columns as SQL_C_CHAR, as before.
  • Both fetch paths decode wide values from UTF-16 (UTF-32 under iODBC).
  • Columnar buffers for wide columns hold 4 bytes per character plus a wide terminator.
  • The docs section on Unicode issues says when columns are fetched wide and what narrow columns, driver character sets, and parameters still need.

Scope

  • UTF-8 clients fetch exactly as before. Julia starts with a UTF-8 LC_CTYPE on Linux and macOS, so there the wide path runs only if a driver resets the locale to a non-UTF-8 one (psqlODBC does this from LANG).
  • A non-UTF-8 character set configured in the driver itself, on a UTF-8 client, still returns narrow text that is not UTF-8. Examples are MariaDB CHARSET=latin1 and an Oracle NLS_LANG without UTF-8. The docs say to configure those drivers for UTF-8. An earlier version of this PR fetched every Unicode column wide, which covered this case too, but it made text fetches about 2x slower on MariaDB (76.5 to 157.7 ms per 100k rows, Linux) and 28 to 39% slower on psqlODBC for every user.
  • Narrow columns (SQL_CHAR, SQL_VARCHAR, SQL_LONGVARCHAR, such as SQL Server varchar) still arrive in the client character set. Casting to a Unicode type in the query works around that.
  • Parameters are unchanged. Non-ASCII String parameters are still sent as UTF-8 SQL_C_CHAR bytes, so a non-UTF-8 client stores them double-encoded. Binding them as SQL_C_WCHAR fixes SQL Server and MariaDB but breaks ANSI-only drivers: psqlodbca rejects it with HYC00: Unrecognized C_parameter type. That needs its own change.

Blast Radius

  • Only non-UTF-8 clients change, which in practice means Windows with an ANSI code page. There, every column the driver reports as Unicode takes the wide path: SQL Server nvarchar, and all text from MariaDB Connector/ODBC and psqlODBC's Unicode driver. Measured with the earlier always-wide version on Linux: MariaDB text fetches about 2x slower, SQL Server nvarchar 8% slower (0.045 s to 0.049 s per 100k rows), varchar unchanged.
  • On the wide path, embedded NUL characters are cut at the first NUL by Connector/ODBC before 3.1.22, and by iODBC's conversion of columnar results.
  • On a non-UTF-8 client, non-ASCII values that ODBC.jl wrote through String parameters were already stored double-encoded. Before this change, the read path reversed the error, so those values looked correct in Julia. They now read back as stored, which is how other tools already showed them.

Verification

  • New test: forces the C locale (ASCII codeset, like a Windows ANSI code page), checks ODBC.utf8client() is false, and reads a utf8mb4 value over a CHARSET=latin1 MariaDB connection in both fetch modes. It fails on main and passes here. It then checks that utf8client() is true again after the locale is restored.
  • CI (Linux, unixODBC, UTF-16 SQLWCHAR): Connector/ODBC 3.1.22 on Julia 1.6, 1, and nightly, and 3.1.20 on Julia 1.
  • macOS (iODBC, UTF-32 SQLWCHAR), Julia 1.12.6, MariaDB 13.0, branch merged with current main: full Pkg.test() passes, including the new test on the UTF-32 wide path.
  • With the earlier always-wide version: SQL Server 2022 (Azure SQL Edge) with ODBC Driver 18 and an ISO-8859-1 client locale returned "Bjørnefjært i går 望" for nvarchar(max) and nvarchar(50), where main returned "Bj\xf8rnefj\xe6rt i g\xe5r ?". The optional test/sqlserver.jl checks passed (15/15). psqlODBC (Unicode and ANSI) and SQLite ODBC round trips passed. These were not rerun on the current version. Windows is not covered by CI.

I reviewed every caller of jlcast and fetchtypes, along with the row and columnar paths and the GetData chunking, which already handled SQL_C_WCHAR terminators.

🤖 Generated with Claude Code

quinnj and others added 2 commits November 18, 2026 01:06
The driver converts text fetched as SQL_C_CHAR to the connection's
character set, so a latin1 connection returns invalid UTF-8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fetched as SQL_C_CHAR, SQL_WCHAR, SQL_WVARCHAR and SQL_WLONGVARCHAR
columns arrive in the client character set: the ANSI code page on
Windows, or the connection charset for MariaDB. Any code page other
than UTF-8 produced invalid strings and replaced characters outside
it with '?'. Fetch these columns as SQL_C_WCHAR and transcode to UTF-8.

MariaDB Connector/ODBC before 3.1.22, and iODBC for columnar results,
end wide values at an embedded NUL, so those test checks are marked
broken there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.59%. Comparing base (afe9085) to head (a300208).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #416      +/-   ##
==========================================
+ Coverage   84.47%   84.59%   +0.12%     
==========================================
  Files           7        7              
  Lines        1024     1032       +8     
==========================================
+ Hits          865      873       +8     
  Misses        159      159              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Fetching every Unicode column as SQL_C_WCHAR made text fetches about 2x
slower on MariaDB and psqlODBC, because the driver converts each value to
UTF-16. Clients whose character set is already UTF-8 get correct narrow
text, so only non-UTF-8 clients (a Windows ANSI code page, or a non-UTF-8
LC_CTYPE codeset) take the wide path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@quinnj-bot

quinnj-bot Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

Pushed 5c12a79 after an independent review. Fetching every Unicode column as wide characters made text fetches about 2x slower on MariaDB (76.5 to 157.7 ms per 100k rows on Linux) and 28 to 39% slower on psqlODBC, because the driver converts each value to UTF-16. SQL Server showed no measurable change. The new commit takes the wide path only when the client character set isn't UTF-8: a Windows code page other than 65001, or a non-UTF-8 locale codeset. That covers #379. With it, MariaDB measured 73.6 ms and psqlODBC 75.9 ms, back at main's speed.

Trade-off: a non-UTF-8 character set configured in the driver itself, on macOS or Linux (MariaDB CHARSET=latin1, or an Oracle NLS_LANG without UTF-8), still gets narrow text that isn't UTF-8. The docs now say to configure those drivers for UTF-8. The previous commit covered that case too, but every MariaDB and psqlODBC user paid the slowdown. Leaving this for a maintainer to choose before merging.

[comment generated via AI]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ODBC.jl does not process latin1 strings correctly on Windows

1 participant