Milestones
List view
Top-level milestone for the ALBS performance OKR (carried over from Q3). ## Objective Make ALBS measurably faster and make its distributed workers observable and manageable. ## Key results ### KR1 — Migrate build and sign nodes to Celery (0% -> 100%) Build nodes and sign nodes currently run their own polling loops against the web server, with no queue visibility: we cannot see what is queued, what a given node is working on, how deep the backlog is, or manage/drain workers without touching the machines. Moving them onto Celery gives us: * a real task queue with inspectable state (queued / active / reserved / failed per node) * worker management (ping, inspect, graceful shutdown, concurrency control, drain before maintenance) * retries, time limits and acknowledgement semantics handled by the framework instead of bespoke code * queue metrics that plug into the existing Prometheus/Grafana stack Scope: albs-node (build nodes), albs-sign-node (sign nodes), the dispatching side in albs-web-server, plus deployment (broker, systemd units, Ansible) and monitoring. Done when both node types take their work exclusively via Celery, queue and worker state is visible in monitoring, and the old polling path is removed. ### KR2 — Speed up the slow parts (1.5–2x on the targeted flows) Measure first, then optimize. Candidate areas already identified: * slow API endpoints (builds listing/feed, releases, errata) * release and sign flows * repo sync / bootstrapping * Pulp interaction and N+1 query patterns Done when the targeted flows are 1.5–2x faster than the baseline recorded at the start of the work, with before/after numbers from Prometheus. ## Related issues * #553 — Analyze Prometheus metrics (and add missing ones) to find and optimize slow code paths * #478 — Refactor query in build feed endpoint * #527 — Enable shared yum cache for mock builds * #467 — Gather performance metrics in Pulp * #453 — Deduplicate download/build of src.rpms when the build source is a remote src.rpm
Due by December 30, 2026We have several automations/pipelines around ALBS and other services that are using some developers' personal access tokens. This is not right. Every pipeline/automation that needs a token to work must use an integration-specific, non-personal token. Some affected services are: * ALBS * Github * Gitea In order for this to happen, we need to: * Collect all the automations that we have around and the tokens they use * Create users/tokens according to the different automations in the different affected services indicated above (some tokens could be shared depending on its purpose) * Update all the automations with their tokens
No due dateFix tests in the ALTS (AlmaLinux test system) Make adjustments in ansible playbook for dev environments Make automation to ensure it’s in a working state Speed up repo sync and initial bootstrapping (Also required for dev environments) Investigate which part of ALBS causes the slowness
No due date•1/5 issues closed* Update the guides * Move it to https://wiki.almalinux.org/#contributions-to-the-wiki
No due date•3/6 issues closed- No due date•0/1 issues closed
Red hat is working around evolving errata as explained in [The future of Red Hat security data](https://www.redhat.com/en/blog/future-red-hat-security-data). To summarize: * OVAL v1 was deprecated in mid-2023 * OVAL v2 will continue until the end of 2024 - but el8 and el9 ovals will continue receiving advisories until EOL * CVRF was replaced with CSAF (VEX formatted files), see [this blog post](https://www.redhat.com/en/blog/csaf-vex-documents-now-generally-available) * Introduction of VEX formatted files per CVE, see [this blog post](https://www.redhat.com/en/blog/vulnerability-exploitability-exchange-vex-beta-files-now-available) Our (short) tentative plan, and assuming that we're not generating CSAFs for now, we need to: * Start pulling information from Red Hat errata website * Since OVAL cacher is passing away, our new service needs to ensure that: * The payload data is sent over to API in the same (or similar) way * The DB/API should not need any adjustments (only `errata_type` if we're also bringing non-security advisories) * If we need to pull things from both sides (website and security data API), all the logic must live in the service * Given that we'll no longer getting OVALs, generate required OVAL-specific data before sending over to web_server. Take special attention to "modular erratas" * When we're sending everything right from the new service, and before OVAL is finally deprecated, we must ensure that inserting erratas from both OVAL cacher and the new service results in the same data in DB. This way, we ensure that all our errata data is as consistent as it is now, this is: db, pulp, resulting errata generated data (updateinfo, OVAL, errata.json, etc) When security advisories processing is already working, we also want to start pulling non-security advisories from red hat errata website. This involves: * Adding a switch in public-errata-parser so non-security erratas are also processed and sent to web server * Adding a `is_security` field to `NewErrataParser` * Non-security erratas should be treated exactly the same as security ones, with the exception that non-security erratas won't be included in AlmaLinux OVAL. We can make use of the `is_security` field to filter non-security records
Overdue by 1 year(s)•Due by December 31, 2024•15/17 issues closedWe want achieve following: 1. There are exists tests to validate 60% of code (by classes) in all components Tests should reflect readiness for production environment. Essential functions need to be covered. 2. Tests are run over PR and again over master after merge There should be an environment and automation for it that allow any engineer to run tests against his PR, without a need to wait for someone with access rights. The same tests should be run regularly or after changes against master brunch of codebase.
No due date•23/23 issues closed- Add ability to configure testing: - The repository where tests reside; - The directory where tests are located in the repository (or some rules|regexes); - Pre-test hook (optional); - Environment variables (necessary + custom if some test needs it); - Some order in which tests should be executed (a list of files, or everything in the folder, etc.); - Test execution timeouts (on package basis, with some meaningful defaults if nothing is provided by user); - General requirements: - Detect if VM didn't respond in some time via SSH, drop it, report it as a failure (required); - Be able to reboot VM if test posted a specialized string in its output (like ">>>VMREBOOT<<<"); - Possibly add the ability to configure a need for reboot on the testing configuration phase (optional, not necessary in the initial implementation); - DO NOT USE paramiko/fabric for SSH commands execution (required); - Any abnormalities during the test execution should result a VM termination/deletion and report to main node as test failure with as much explanation as possible (required); - Cache the test repositories in mirror mode to reduce cloning times for each particular environment (required); - Clean up package manager cache before installing anything on the system (required); - Do not expect the environment to have any tools installed (ansible, bats, testinfra, etc.) (optional); - Do not expect the environment to have a repository with tests on it (do some kind of copy to the remote environment) (required); - Be able to run test scripts in detached mode on the remote environment unless a particular runner does not require such approach (provisioners like Ansible, Chef, Puppet, etc.) (required); - Timeouts for tests execution on per-script basis (or per-runner, TBD) (required); - Report back the environment details the tests were running in (VM image name, installed packages, network details; - Add specialized runners for different tests (file formats): - Single command execution on the remote environment; - Shell script executor (with ability to run in detached mode); - BATS executor (with ability to run in detached mode); - Ansible playbooks executor; - Ability to easily add new executors if needed (with ability to run in detached mode); - Dynamic quota allocation for test tasks - Quota managed by user group - Director for overquoting that helps utilize full system while still allowing new urgent builds to be tested - Visualize queues and quotas for users in UI to address lack of transparency
No due date•15/17 issues closedWe need to adapt our packages building flow to all the recent changes in AlmaLinux development. That means adding different automated solutions to get the sources, create security advisories, test the changes, etc. This milestone will contain all the tasks that are related to AlmaLinux workflow change.
No due date•21/24 issues closedRight now it's counter-intuitive to build anything other than AlmaLinux with Build System. We want to remediate that. We reach this milestone when: - User can create a custom product :heavy_check_mark: - User can set up one or many repositories for it (#20) - Several repos in one product - User can set up certificate to sign packages with (#21) :heavy_check_mark: - User can sign built packages with proper certificate (#21) :heavy_check_mark: - Troubles in creating product sign keys - need to verify if it works (#288) - User can update certificate. (Packages signed with old certificate can NOT be validated) (#36) - User can set up their own sign node and use it to sign product packages - This is a long-term idea - In a short-term, we need to be able to limit some sign nodes to only certain products and vice versa - MUST HAVE: limit some sign nodes to only certain products (#289) - NICE TO HAVE: allow products to indicate which sign node they want to use for package signing - User can assign `product_maintainer` role for this product to (him/her) self or to another user without external help (#37) - Assign user roles in product is already possible (see [my comment](https://github.com/AlmaLinux/build-system/issues/37#issuecomment-2124556852)). I'd say that the rest of the AC can be considered low priority. - User can re-release package(s) into another one of their repositories (e.g. from "beta" to "stable") - Need to figure out how to implement this (#290) - Need an endpoint (or any machinery) in which we can create new distros on beholder side to monitor, so beholder helps on dealing with these and match packages in product from several repos (#291) - User can read straightforward, publicly available documentation on how to build custom products in ALBS (#292) - Toggle in product "requires sign", overrides in release the "need to sign" requirement, if marked as such (#293) - COPR - gpg check on vs off (#294) - Access rights for new product are not sufficient for releases - Bug when trying to add new members to product (#295) - Manager should be able to assign roles to team members (It works) :heavy_check_mark: - Prevent builds scheduler to not be overwhelmed by community builds (with configurable rate limits) (#296) - This is about ensuring that there are always available builders for AlmaLinux - Filter by product in build list (save to cookie) - "only my products"? (#297) - This is about having a user preference (like `filterByProductId`) stored in the browser and is taken into account when listing builds - UX: make it clear when you release something in product that is not built/added in product. You need to do it through "releases" menu (its about Add to product button). - The only way to release stuff should be using the release interface - Warn the person doing a release that what you're about to release does not belong to the product it was built for (if it's the case) (#299) - UX: Creating new product is not really intuitive - After we make the improvements around products, we need to update the product creation accordingly. - Needs to be discussed, scoped and described in an issue after most of the items on this milestone are done - UX: Creating release is not intuitive (need to hit "+" before hitting "submit") - Andrew suggests that, when there's only one build, clicking submit should work. When several builds are involved, you need to click +. - Vasily suggests to allow to just hit enter Limitations: * No rollout release capabilities for custom products - but we don't really think we need them in ALBS Things to check: * Ensure we sign COPR repodata - if not do it * if we're re-signing the repo in case a COPR repo gets a new key, we can't forget repodata
No due date•18/28 issues closed