Convert the two UTF-16LE Live_36 raw scripts to UTF-8 (#385) - #389
Open
meketreve wants to merge 1 commit into
Open
Convert the two UTF-16LE Live_36 raw scripts to UTF-8 (#385)#389meketreve wants to merge 1 commit into
meketreve wants to merge 1 commit into
Conversation
Fixes the two files reported in issue OpenPerpetuum#385: - Patches/Live_36/Raw_SQL/17_consolidate_statistics.sql - Patches/Live_36/Raw_SQL/37_Market_decoupling_4.sql Both were UTF-16LE with no BOM, which is the combination sqlcmd cannot detect: apply_all.bat invokes it without -f, so a file with no BOM is read using the ANSI codepage. Encoding only. Verified rather than assumed: - decoding each original from UTF-16LE and diffing against the converted file reports no difference, so no character changed - no BOM was added; both files now start with the ASCII bytes 55 53 45 - line endings are untouched -- both files were LF-only before and after, measured as 0 CR bytes on each side - both files are pure ASCII once decoded, so what sqlcmd reads is byte-identical to what it read before File sizes halve exactly, which is what a pure-ASCII UTF-16 to UTF-8 conversion produces: 3808 to 1904 and 8220 to 4110. CONTRIBUTING.md states that Patches are never to be altered. This change is made on an explicit instruction from the maintainers, given on Discord on 2026-08-15 in answer to issue OpenPerpetuum#385. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #385.
Converts the two reported files from UTF-16LE to UTF-8:
Patches/Live_36/Raw_SQL/17_consolidate_statistics.sqlPatches/Live_36/Raw_SQL/37_Market_decoupling_4.sqlDone on the instruction given on Discord on 2026-08-15 —
CONTRIBUTING.mdsaysPatchesare never to be altered, so this is deliberate and asked for rather than an oversight.Why these two broke and the others did not
apply_all.batinvokes sqlcmd without-f, so an input file with no byte-order mark is read using the ANSI codepage. UTF-16LE with a BOM is detected; UTF-16LE without one is not. These two had no BOM, which is the one combination that goes wrong.Verification
Encoding only. Each claim below was measured, not assumed:
55 53 45, notef bb bfOne thing the issue did not cover
Scanning all 58 files in
Live_36/Raw_SQLturned up two more UTF-16LE files, which issue #385 does not mention:16_sp_RecordResourceGathered.sql20_usp_RefreshAutoMarketOrders.sqlBoth carry a BOM, so sqlcmd reads them correctly today and nothing is broken. But they are not UTF-8 either, and the remark that prompted this PR was that everything had been switched back to UTF-8 and these were missed — which would apply to these two as well. Both are pure ASCII once decoded, so converting them carries the same zero risk.
They are not in this PR, because the issue named two files and
CLAUDE.mdasks for minimal, focused changes. Say the word and they are a one-line addition to this branch.