[Bug][org] All org plugin API endpoints panic with nil pointer — root cause: routes bind handlers before InitPlugins runs

[pballester]

What happened

Every plugins/org API endpoint (teams.csv, users.csv, user_account_mapping.csv, GET and PUT) returns HTTP 500 with a nil-pointer panic. Verified on v1.0.2 and v1.0.3-beta15 (helm deployment, MySQL backend). GET with ?fake_data=true works (it never touches the store).

Panic (v1.0.3-beta15, PUT /plugins/org/teams.csv):

runtime error: invalid memory address or nil pointer dereference
/app/plugins/org/api/team.go:87   <- h.store.deleteAll(&crossdomain.Team{})
/app/server/api/router.go:149

GET without fake_data panics at team.go:47 (h.store.findAllTeams()).

Root cause (with evidence)

It's an initialization-order race between route registration and plugin initialization:

1. runner.LoadGoPlugins only does Lookup("PluginEntry") + plugin.RegisterPlugin(...) — it does not call PluginInit.Init() (loader.go).
2. plugin.InitPlugins(basicRes) — the only place Init() runs in server mode — is invoked from pipelineServiceInit() (pipeline.go:83).
3. registerPluginEndpoints builds the router from each plugin's ApiResources(). For org, that returns bound method values on instance state: p.handlers.GetTeam, etc. (impl.go).
4. In Go, a method value captures its receiver at evaluation time. If ApiResources() is evaluated before Init() set p.handlers, the router permanently holds handlers bound to a nil *Handlers — running Init() later doesn't rebind them.

Startup timeline from our pod logs proving the order (a DB migration was pending at boot, which delays pipelineServiceInit while the HTTP server must start to serve /proceed-db-migration):

12:13:58  plugin loaded org
12:14:11  endpoint PUT /plugins/org/teams.csv ... registerPluginEndpoints   <- ApiResources() evaluated, p.handlers == nil
12:14:23  plugin: <X> doesn't implement 'PluginInit', it will be skipped.   <- InitPlugins runs, 12s too late

org is the only plugin whose ApiResources() binds per-instance state created in Init() — other plugins read package-level vars at call time — which is why only org breaks, and why it breaks non-deterministically across deployments (whoever wins the race). This likely explains the long trail of closed-as-stale reports: #8590, #8271, #7957, #7219 (different symptoms of the same race — reporters couldn't pin it because it depends on startup timing; a pending DB migration at boot makes it deterministic).

Note the v1.0.3-beta fix (&team{}).toDomainLayer(tt) (team.go:84) resolved a second, independent nil deref, which made the remaining one harder to spot.

What do you expect to happen

plugins/org CSV endpoints work regardless of startup timing.

How to reproduce

1. Deploy v1.0.2 or v1.0.3-beta15 (helm chart, MySQL) with a pending DB migration at boot (e.g. any version upgrade), or any startup where route registration beats pipelineServiceInit.
2. Confirm the migration, then curl -X PUT <endpoint>/plugins/org/teams.csv -F 'file=@teams.csv' → HTTP 500, panic above. GET /plugins/org/teams.csv → same at team.go:47.

Proposed fix (happy to submit a PR)

Two options, smallest first:

A (org-scoped, minimal): make org's ApiResources() resolve handlers at call time instead of bind time — closures over p (pointer receiver) rather than method values over p.handlers:

func (p *Org) ApiResources() map[string]map[string]plugin.ApiResourceHandler {
    wrap := func(f func(*api.Handlers) plugin.ApiResourceHandler) plugin.ApiResourceHandler {
        return func(in *plugin.ApiResourceInput) (*plugin.ApiResourceOutput, errors.Error) {
            if p.handlers == nil {
                return nil, errors.Internal.New("org plugin not initialized yet")
            }
            return f(p.handlers)(in)
        }
    }
    ...
}

B (framework): guarantee plugin.InitPlugins() runs before registerPluginEndpoints (e.g. right after runner.LoadPlugins in services.Init), decoupling it from pipelineServiceInit. Fixes the class of bug for any future plugin with instance state, but touches startup ordering for everything.

Version

v1.0.3-beta15 (also reproduced on v1.0.2). Helm chart on Kubernetes (EKS), external MySQL 8.0.

Are you willing to submit a PR?

Yes.