Repository navigation
Fetch Unicode columns as wide characters - #416
quinnj-bot[bot] wants to merge 4 commits into
Conversation
The driver converts text fetched as SQL_C_CHAR to the connection's character set, so a latin1 connection returns invalid UTF-8. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fetched as SQL_C_CHAR, SQL_WCHAR, SQL_WVARCHAR and SQL_WLONGVARCHAR columns arrive in the client character set: the ANSI code page on Windows, or the connection charset for MariaDB. Any code page other than UTF-8 produced invalid strings and replaced characters outside it with '?'. Fetch these columns as SQL_C_WCHAR and transcode to UTF-8. MariaDB Connector/ODBC before 3.1.22, and iODBC for columnar results, end wide values at an embedded NUL, so those test checks are marked broken there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #416 +/- ##
==========================================
+ Coverage 84.47% 84.59% +0.12%
==========================================
Files 7 7
Lines 1024 1032 +8
==========================================
+ Hits 865 873 +8
Misses 159 159 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Fetching every Unicode column as SQL_C_WCHAR made text fetches about 2x slower on MariaDB and psqlODBC, because the driver converts each value to UTF-16. Clients whose character set is already UTF-8 get correct narrow text, so only non-UTF-8 clients (a Windows ANSI code page, or a non-UTF-8 LC_CTYPE codeset) take the wide path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Pushed 5c12a79 after an independent review. Fetching every Unicode column as wide characters made text fetches about 2x slower on MariaDB (76.5 to 157.7 ms per 100k rows on Linux) and 28 to 39% slower on psqlODBC, because the driver converts each value to UTF-16. SQL Server showed no measurable change. The new commit takes the wide path only when the client character set isn't UTF-8: a Windows code page other than 65001, or a non-UTF-8 locale codeset. That covers #379. With it, MariaDB measured 73.6 ms and psqlODBC 75.9 ms, back at main's speed. Trade-off: a non-UTF-8 character set configured in the driver itself, on macOS or Linux (MariaDB [comment generated via AI] |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Why
On Windows, text from
nvarcharcolumns comes back as invalid strings, with characters outside the ANSI code page replaced by?(#379). ODBC.jl fetched every character column asSQL_C_CHAR, and drivers convertSQL_C_CHARtext to the client character set: the ANSI code page on Windows, and the locale's character set on Linux and macOS. ODBC.jl then read those bytes as UTF-8. When the client character set is not UTF-8, columns the driver reports as Unicode are now fetched asSQL_C_WCHARand transcoded to UTF-8.Fixes #379. #310 (cp1252 text from Oracle and Access) has the same cause for Unicode columns on a Windows ANSI code page.
What changed
fetchtypesmapsSQL_WCHAR,SQL_WVARCHAR, andSQL_WLONGVARCHARtoSQL_C_WCHARwhenODBC.utf8client()is false: a Windows ANSI code page other than 65001, or a non-UTF-8LC_CTYPEcodeset elsewhere. UTF-8 clients still fetch those columns asSQL_C_CHAR, as before.Scope
LC_CTYPEon Linux and macOS, so there the wide path runs only if a driver resets the locale to a non-UTF-8 one (psqlODBC does this fromLANG).CHARSET=latin1and an OracleNLS_LANGwithout UTF-8. The docs say to configure those drivers for UTF-8. An earlier version of this PR fetched every Unicode column wide, which covered this case too, but it made text fetches about 2x slower on MariaDB (76.5 to 157.7 ms per 100k rows, Linux) and 28 to 39% slower on psqlODBC for every user.SQL_CHAR,SQL_VARCHAR,SQL_LONGVARCHAR, such as SQL Servervarchar) still arrive in the client character set. Casting to a Unicode type in the query works around that.Stringparameters are still sent as UTF-8SQL_C_CHARbytes, so a non-UTF-8 client stores them double-encoded. Binding them asSQL_C_WCHARfixes SQL Server and MariaDB but breaks ANSI-only drivers:psqlodbcarejects it withHYC00: Unrecognized C_parameter type. That needs its own change.Blast Radius
nvarchar, and all text from MariaDB Connector/ODBC and psqlODBC's Unicode driver. Measured with the earlier always-wide version on Linux: MariaDB text fetches about 2x slower, SQL Servernvarchar8% slower (0.045 s to 0.049 s per 100k rows),varcharunchanged.Stringparameters were already stored double-encoded. Before this change, the read path reversed the error, so those values looked correct in Julia. They now read back as stored, which is how other tools already showed them.Verification
ODBC.utf8client()is false, and reads a utf8mb4 value over aCHARSET=latin1MariaDB connection in both fetch modes. It fails on main and passes here. It then checks thatutf8client()is true again after the locale is restored.SQLWCHAR): Connector/ODBC 3.1.22 on Julia 1.6, 1, and nightly, and 3.1.20 on Julia 1.SQLWCHAR), Julia 1.12.6, MariaDB 13.0, branch merged with current main: fullPkg.test()passes, including the new test on the UTF-32 wide path."Bjørnefjært i går 望"fornvarchar(max)andnvarchar(50), where main returned"Bj\xf8rnefj\xe6rt i g\xe5r ?". The optionaltest/sqlserver.jlchecks passed (15/15). psqlODBC (Unicode and ANSI) and SQLite ODBC round trips passed. These were not rerun on the current version. Windows is not covered by CI.I reviewed every caller of
jlcastandfetchtypes, along with the row and columnar paths and the GetData chunking, which already handledSQL_C_WCHARterminators.🤖 Generated with Claude Code