Skip to content

resync credentials in memory #52

Open

Description

@msathaye

Problem: bw has a bug in the auth layer where it doesnt properly resync the credentials in memory post a successful deploy.

1. At daemon startup (line 133): ss, err = commandutils.Generatecredentials(config, ns) - credentials are generated once
2. Dialer is created (line 169): grpc.WithPerRPCCredentials(ss) - the dialer uses these startup credentials permanently
3. Post-deployment in syncAuthorizationsPostDeploy():

  • Authorization files are updated via CloneAuthorizationFile()
  • Peers are synced via SyncAuthorizations()
  • Daemon's dialer uses ss credentials

New credentials are generated deterministically from cluster tokens, but the dialer never picks up changes to authorization policies because syncAuthorizationsPostDeploy() updates files but not daemon dialer.

Activity

  1. msathaye commented on Sep 24, 2025

    @msathaye
    Author

    Previous diagnosis incorrect. Generatecredentials just setups the nodes key. but since its the new users permissions i'd expect the issue to be in the notary structure. and the only thing the notary is used for in generate credentials is to add the node key to the notary. you dont need to sync the node key. its determined on server boot.

    better place to look:

        // copy the new authorization file from the deploy to the auth key file
    	errorsx.MaybeLog(notary.CloneAuthorizationFile(filepath.Join(dr.Root, bw.DirArchive, bw.AuthKeysFile), filepath.Join(dctx.NotaryStorage.Root, bw.AuthKeysFile)))
        // syncs authorizations between peers.
    	daemons.SyncAuthorizations(dctx)

    CloneAuthorizationFile should cause the file watcher to trigger for the notary/file.go#background function

    func (t *file) background() *file {
    	ts := time.Now()
    	log.Printf("authorization load initiated %s\n", t.source)
    	if err := LoadAuthorizedKeys(t.storage, t.source); err != nil {
    		log.Println("failed to load keys", err)
    	}
    	log.Printf("authorization load completed %s %s\n", t.source, time.Since(ts))
    
    	go func() {
    		for {
    			select {
    			case evt := <-t.w.Events:
    				var (
    					err error
    				)
    
    				if evt.Op == fsnotify.Chmod {
    					continue
    				}
    
    				log.Println("change detected", t.source, evt.Op)
    				m := NewMem()
    				if err = LoadAuthorizedKeys(m, t.source); err != nil {
    					log.Println("failed to load keys", err)
    					continue
    				}
    
    				t.storage = m
    			case err := <-t.w.Errors:
    				log.Println("watch error", err)
    			}
    		}
    	}()
    
    	return t
    }
    
    func (w *fsnotify.Watcher) Add(path string) error
    Add starts monitoring the path for changes.
    
    A path can only be watched once; watching it more than once is a no-op and will not return an error. Paths that do not yet exist on the filesystem cannot be watched.
    
    A watch will be automatically removed if the watched path is deleted or renamed. The exception is the Windows backend, which doesn't remove the watcher on renames.
    
    Notifications on network filesystems (NFS, SMB, FUSE, etc.) or special filesystems (/proc, /sys, etc.) generally don't work.
    
    Returns [ErrClosed] if [Watcher.Close] was called.
    
    See [Watcher.AddWith] for a version that allows adding options.
    
    Watching directories
    All files in a directory are monitored, including new files that are created after the watcher is started. Subdirectories are not watched (i.e. it's non-recursive).
    

    A watch will be automatically removed if the watched path is deleted or renamed.

    // CloneAuthorizationFile copies the authorization from one location to another.
    func CloneAuthorizationFile(from, to string) (err error) {
    	var (
    		ff *os.File
    		tf *os.File
    	)
    
    	debugx.Println("synchronizing authorization", from, "->", to)
    	if ff, err = os.Open(from); err != nil {
    		return err
    	}
    	defer ff.Close()
    
    	if tf, err = os.CreateTemp(filepath.Dir(to), fmt.Sprintf("%s.sync", filepath.Base(to))); err != nil {
    		return err
    	}
    	defer tf.Close()
    
    	if err = os.Chmod(tf.Name(), 0600); err != nil {
    		return err
    	}
    
    	if _, err = io.Copy(tf, ff); err != nil {
    		return err
    	}
    
    	debugx.Println("renaming", tf.Name(), "->", to)
    	return os.Rename(tf.Name(), to)
    }
    
    func CloneAuthorizationFile(from, to string) (err error) {
    	updated, err := os.ReadFile(from)
    	if err != nil {
    		return err
    	}
    
    	return os.WriteFile(to, updated, 0600)
    }
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions