Skip to content

[v3-3-test] Stop variables export/import from silently corrupting values (#70944) - #71791

Draft
github-actions[bot] wants to merge 1 commit into
v3-3-testfrom
backport-f549b34-v3-3-test
Draft

[v3-3-test] Stop variables export/import from silently corrupting values (#70944)#71791
github-actions[bot] wants to merge 1 commit into
v3-3-testfrom
backport-f549b34-v3-3-test

Conversation

@github-actions

Copy link
Copy Markdown
Contributor
  • Stop variables export/import from silently corrupting values

airflow variables export writes a bare value for any variable without a
description, while airflow variables import treats every dict carrying a
"value" key as a {"value": ..., "description": ...} envelope. A variable whose
own value happens to have that shape is therefore unwrapped on the way back in:
its value is replaced by the inner "value" and a description is invented from
the inner "description". Nothing warns the operator, so a round-trip through a
file - the documented way to migrate variables between environments - quietly
rewrites their data.

Export also decodes the stored JSON before writing it out, which erases the
difference between a variable holding raw text and one holding a JSON-encoded
string. Re-importing flattens the latter, and any Dag reading it with
deserialize_json=True starts failing on a value that is no longer valid JSON.

Import's reconstruction is deterministic - strings are stored verbatim,
everything else is JSON-encoded, and one envelope layer is unwrapped - so export
alone can be made lossless. Fixing it there leaves import untouched and keeps
hand-written import files working exactly as before.

  • apply suggestion

(cherry picked from commit f549b34)

Co-authored-by: Y-C easoneason0905@gmail.com
Co-authored-by: Eason09053360 185830721+Eason09053360@users.noreply.github.com

…ues (#70944)

* Stop variables export/import from silently corrupting values

`airflow variables export` writes a bare value for any variable without a
description, while `airflow variables import` treats every dict carrying a
"value" key as a {"value": ..., "description": ...} envelope. A variable whose
own value happens to have that shape is therefore unwrapped on the way back in:
its value is replaced by the inner "value" and a description is invented from
the inner "description". Nothing warns the operator, so a round-trip through a
file - the documented way to migrate variables between environments - quietly
rewrites their data.

Export also decodes the stored JSON before writing it out, which erases the
difference between a variable holding raw text and one holding a JSON-encoded
string. Re-importing flattens the latter, and any Dag reading it with
deserialize_json=True starts failing on a value that is no longer valid JSON.

Import's reconstruction is deterministic - strings are stored verbatim,
everything else is JSON-encoded, and one envelope layer is unwrapped - so export
alone can be made lossless. Fixing it there leaves import untouched and keeps
hand-written import files working exactly as before.

* apply suggestion

---------
(cherry picked from commit f549b34)

Co-authored-by: Y-C <easoneason0905@gmail.com>
Co-authored-by: Eason09053360 <185830721+Eason09053360@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant