Skip to content

fix(jira): add missing _raw_data_* columns to _tool_jira_sprint_reports - #9015

Open
DoDiODev wants to merge 1 commit into
apache:mainfrom
DoDiODev:pr/jira-sprint-report-raw-data-columns
Open

fix(jira): add missing _raw_data_* columns to _tool_jira_sprint_reports#9015
DoDiODev wants to merge 1 commit into
apache:mainfrom
DoDiODev:pr/jira-sprint-report-raw-data-columns

Conversation

@DoDiODev

Copy link
Copy Markdown
Contributor

Starting point is the Jira Sprint Report bug below. The regression guard added
for it turned out to catch the same class of bug in three more plugins
(taiga, teambition, testmo), which are fixed here as well — see
§3.

Problem

Collecting Jira data fails in the extractSprintReport subtask with:

subtask extractSprintReport ended unexpectedly
  error getting batch from result (500)
  Error 1054 (42S22): Unknown column '_raw_data_table' in 'where clause'

Root cause

The Sprint Report feature (PR #8967 / #9010) added the table
_tool_jira_sprint_reports. Its migration
(20260722_add_sprint_report_table.go) creates the table from a struct that
does not embed common.NoPKModel:

type jiraSprintReport20260722 struct {
    ConnectionId uint64 `gorm:"primaryKey"`
    // ... no common.NoPKModel ...
}

The runtime model models.JiraSprintReport does embed common.NoPKModel
(→ common.RawDataOrigin), so it expects the columns _raw_data_params,
_raw_data_table, _raw_data_id, _raw_data_remark (plus created_at,
updated_at).

During extraction, api.NewApiExtractor deletes outdated rows via
WHERE _raw_data_table = ? AND _raw_data_params = ?
(helpers/pluginhelper/api/batch_save_divider.go). Because the migration never
created those columns, MySQL rejects the query with error 1054.

What changed

1. Fix for the reported bug (jira)

  • New migration plugins/jira/models/migrationscripts/20260727_add_raw_data_columns_to_sprint_report.go
    re-runs AutoMigrateTables on a struct that embeds the raw-data columns,
    adding them to existing _tool_jira_sprint_reports tables without data
    loss
    (GORM AutoMigrate only adds missing columns). Registered in
    models/migrationscripts/register.go.
  • The original 20260722_add_sprint_report_table.go is left unchanged:
    migration scripts are append-only, and editing it would not repair databases
    that already ran it (its version is already recorded in
    _devlake_migration_history).

2. Regression tests (schema-drift guards)

  • plugins/jira/e2e/migration_schema_test.go — runs the real Jira migration
    chain (framework + jira) and asserts every column each Jira model declares
    exists in the migrated table. Directly reproduces and guards the bug above.
  • plugins/schema_e2e/migration_schema_test.go — cross-plugin generalization:
    applies framework + all plugin migrations and validates model-vs-table
    column parity for every built-in Go plugin. Includes TestAllGoPluginsListed,
    which fails if a new plugin directory with an impl package is added but not
    registered, so the guard stays complete automatically.

Both deliberately run the real migration scripts instead of AutoMigrate-ing
the runtime model — an AutoMigrate-based check could never detect this class of
drift. Both live in e2e packages, so they run under make e2e-test-go-plugins
(require E2E_DB_URL) and are excluded from the DB-less unit-test run.

Implementation details worth noting:

  • Both tests call dalgorm.Init(...) to register the encdec GORM serializer.
    runner.CreateBasicRes does not do this (only CreateAppBasicRes does),
    so without it the migrations abort with invalid serializer type encdec.
  • Both fall back to a deterministic ENCRYPTION_SECRET if none is configured;
    some migrations (e.g. jira 20220716) refuse to run without one
    (jira v0.11 invalid encKey), and CI does not provide a value.
  • Models that no migration materializes (pure API-response models such as
    _tool_jira_server_infos) are skipped — the check targets drift between an
    existing table and its model.

3. Additional schema drifts found by the new cross-plugin guard

The guard immediately uncovered three pre-existing bugs of exactly the same
class. Each is fixed with a new, additive migration (registered in the
respective register.go):

Table Missing column(s)
_tool_taiga_scope_configs type_mappings
_tool_teambition_scope_configs id, created_at, updated_at
_tool_testmo_scope_configs connection_id, name

Files: plugins/{taiga,teambition,testmo}/models/migrationscripts/20260727_add_missing_scope_config_columns.go.

All new migration scripts use core/models/migrationscripts/archived — importing
core/models/common from a migration script is rejected by
make migration-script-lint.

Why these repairs are safe on populated tables

The scope-config repairs re-add columns that carry constraints in
common.ScopeConfig (name has a uniqueIndex) and, for teambition, an
AUTO_INCREMENT primary key. Both were verified against live engines rather than
assumed:

  • uniqueIndex on name — GORM adds new columns as nullable
    (ALTER TABLE ... ADD COLUMN name VARCHAR(255), no NOT NULL DEFAULT ''), so
    pre-existing rows are backfilled with NULL, and both MySQL and PostgreSQL
    permit duplicate NULLs in a unique index. Creating the index on a table with
    two pre-existing rows succeeded on MySQL 8.4.10 and PostgreSQL 17.2.
    The counter-test confirms the distinction: with an explicit
    NOT NULL DEFAULT '' column the same index fails
    (Error 1062 / pq: … (23505)) — which is exactly the case that does not
    occur here.
  • AUTO_INCREMENT id on the primary-key-less _tool_teambition_scope_configs
    GORM issues a plain ADD COLUMN, and MySQL backfills consecutive ids for the
    existing rows (verified: two pre-existing rows received id 1 and 2). No
    Error 1075 ("there can be only one auto column and it must be defined as a
    key").

4. Build script

scripts/build-plugins.sh builds every directory under plugins/ with
-buildmode=plugin. plugins/schema_e2e/ is not a plugin (it contains only the
cross-plugin test and has no main package), which made make build-plugin fail
with "-buildmode=plugin requires exactly one main package". The directory is now
excluded, alongside the existing core / helper / logs exclusions.

Why a new migration (not editing the old one)

  • The buggy migration 20260722 is already on main/upstream/main and has
    already been applied to real databases; its version is recorded, so it will
    never re-run. Only a new migration can repair those databases.
  • Append-only migrations keep the migration history reproducible across
    environments.

Testing

  • make migration-script-lint — OK.
  • gofmt -l on all added/changed files — clean.
  • go vet ./plugins/{jira,taiga,teambition,testmo,schema_e2e}/... — OK.
  • MySQL 8.4.10, fresh database:
    • go test ./plugins/schema_e2e/ — OK, 42/42 plugins PASS
      (TestAllGoPluginsListed included).
    • go test -run TestMigrationSchema ./plugins/jira/e2e/ — OK.
  • PostgreSQL 17.2, fresh database: both tests OK; the repaired scope-config
    tables contain the added columns.
  • Upgrade path on a populated database (not just a fresh one): all four
    migrations were applied to a real, long-running DevLake instance. They are
    recorded in _devlake_migration_history and the target tables carry the
    expected columns afterwards — e.g. _tool_teambition_scope_configs gained
    id, created_at, updated_at, and _tool_jira_sprint_reports (661 rows)
    gained the _raw_data_* columns without data loss.
  • Constraint safety on populated tables (unique index / AUTO_INCREMENT PK)
    verified separately against MySQL 8.4.10 and PostgreSQL 17.2 — see
    §3 "Why these repairs are safe on populated tables".
  • Negative test: dropping _raw_data_table from _tool_jira_sprint_reports
    makes the cross-plugin guard fail with
    [jira] table "_tool_jira_sprint_reports" is missing column "_raw_data_table"
    — i.e. the guard genuinely detects the original regression.

Known limitations of the guard (intentional)

  • It checks column presence only — not column type, length, nullability,
    primary keys or indexes.
  • It runs against the shared E2E database. Other plugin E2E tests use
    FlushTabler (drop + AutoMigrate of the runtime model), so a table touched by
    an earlier test can appear "repaired". On a fresh database (as in CI) this does
    not apply.
  • Tables that no migration creates are skipped, so "model listed in
    GetTablesInfo() but no table at all" is not flagged.

The Sprint Report migration 20260722_add_sprint_report_table.go creates
_tool_jira_sprint_reports from a struct that does not embed
common.NoPKModel, while the runtime model models.JiraSprintReport does.
The columns _raw_data_params / _raw_data_table / _raw_data_id /
_raw_data_remark (plus created_at, updated_at) were therefore never
created, so the ApiExtractor cleanup query

    WHERE _raw_data_table = ? AND _raw_data_params = ?

made the extractSprintReport subtask fail with
"Error 1054 (42S22): Unknown column '_raw_data_table' in 'where clause'".

Add a new, additive migration that re-runs AutoMigrateTables on a struct
embedding archived.NoPKModel. The original migration is left untouched:
migration scripts are append-only, and editing it would not repair
databases that already recorded its version.

Add two schema-drift regression guards that run the REAL migration
scripts instead of AutoMigrate-ing the runtime model, which would hide
this class of drift:

  * plugins/jira/e2e/migration_schema_test.go - Jira-specific guard.
  * plugins/schema_e2e/migration_schema_test.go - cross-plugin guard for
    every built-in Go plugin, including TestAllGoPluginsListed so the
    guard stays complete when a new plugin is added.

The cross-plugin guard immediately uncovered three pre-existing drifts
of the same class, each fixed with its own additive migration:

  * _tool_taiga_scope_configs      - missing type_mappings
  * _tool_teambition_scope_configs - missing id, created_at, updated_at
  * _tool_testmo_scope_configs     - missing connection_id, name

Finally, exclude plugins/schema_e2e from scripts/build-plugins.sh: it is
not a plugin and has no main package, which broke `make build-plugin`
with "-buildmode=plugin requires exactly one main package".

Signed-off-by: DoDiODev <DoDiDev@proton.me>
DoDiODev added a commit to DoDiODev/devlake that referenced this pull request Jul 28, 2026
Integriert den Inhalt von PR apache#9015 in den lokalen
Integrations-Branch.

Die Sprint-Report-Migration 20260722_add_sprint_report_table.go legt
_tool_jira_sprint_reports aus einem Struct an, das common.NoPKModel
nicht einbettet, waehrend das Laufzeitmodell models.JiraSprintReport es
einbettet. Die Spalten _raw_data_params / _raw_data_table /
_raw_data_id / _raw_data_remark (plus created_at, updated_at) fehlten
daher, sodass die Cleanup-Query des ApiExtractors

    WHERE _raw_data_table = ? AND _raw_data_params = ?

den Subtask extractSprintReport mit "Error 1054 (42S22): Unknown column
'_raw_data_table' in 'where clause'" abbrechen liess.

Neue, additive Migration statt Aenderung der alten: Migrationsskripte
sind append-only, und ein Edit wuerde Datenbanken nicht reparieren, die
die Version bereits protokolliert haben.

Zwei Schema-Drift-Guards, die die ECHTEN Migrationsskripte ausfuehren
(statt das Laufzeitmodell zu AutoMigrate-n, was genau diese Drift
verdecken wuerde):

  * plugins/jira/e2e/migration_schema_test.go - Jira-spezifisch.
  * plugins/schema_e2e/migration_schema_test.go - plugin-uebergreifend
    fuer alle Go-Plugins, inkl. TestAllGoPluginsListed.

Der uebergreifende Guard deckte drei bestehende Drifts derselben Klasse
auf, je mit eigener additiver Migration behoben:

  * _tool_taiga_scope_configs      - type_mappings fehlte
  * _tool_teambition_scope_configs - id, created_at, updated_at fehlten
  * _tool_testmo_scope_configs     - connection_id, name fehlten

Verifiziert gegen MySQL 8.4.10 und PostgreSQL 17.2: neue Spalten werden
nullable angelegt, doppelte NULLs sind im uniqueIndex zulaessig, und die
AUTO_INCREMENT-PK laesst sich auf der PK-losen Teambition-Tabelle
nachruesten (kein Error 1075). Alle vier Migrationen wurden zusaetzlich
auf der bestehenden lokalen Datenbank angewendet.

Ausserdem: plugins/schema_e2e in scripts/build-plugins.sh ausgeschlossen
(kein Plugin, kein main-Package -> "make build-plugin" scheiterte mit
"-buildmode=plugin requires exactly one main package").

AGENTS.md um die Schema-Drift-Guards und die zugehoerigen Fallstricke
ergaenzt (fork-only, nicht Teil des Upstream-PRs).

Signed-off-by: DoDiODev <DoDiDev@proton.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant