Community Feed Moderation

The community feed, which is found at https://chocolatey.org/packages, is a moderated feed. That means all new versions of packages are human reviewed prior to approval to check for safety, quality, and correctness. See What is moderation for more details. There are also trusted packages, which only go through automated moderation review and bypass human review as they are coming from trusted sources and/or the software vendors themselves.

By safety - we check that the package scripts do not do anything devious and that you get the software that the package indicates you are getting. Please note that the underlying software may contain crapware/malware (although it is usually not installed when allowing Chocolatey to install silently). This is not checked for currently, but we have plans for checking this in licensed versions of Chocolatey because a feature doing that is not free for us to provide.

Definitions

Requirements and Guidelines

While probably the most comprehensive, this list may not be fully up-to-date. This should serve as a most general understanding, knowing that the validator may be checking for newer things than are written here and that reviewers/moderators may find newer things to check from time to time.

NOTE: Moderators tend to get somewhat picky about properly stating the license, authors (software vendors), and copyright attributions. They are very important to protect both maintainers and the software vendors.

Note: This is still written based on a reviewer reading it, this will get cleaned up more over time to better explain it from a non-reviewer perspective.

Existing Packages

This section provides the requirements for packages that have had at least one released version approved or exempted. This includes any packages that existed prior to moderation being turned on (possibly an Unknown status).

Requirements

Requirements represent the minimum quality of a package that is acceptable. When a package version has failed requirements, the package version requires fixing and/or response by the maintainer. Provided a Requirement has flagged correctly, it must be fixed before the package version can be approved. The exact same version should be uploaded during moderation review.

Guidelines

Guidelines are strong suggestions that improve the quality of a package version. These are considered something to fix for next time to increase the quality of the package. Over time Guidelines can become Requirements. A package version can be approved without addressing Guideline comments but will reduce the quality of the package.

Package Review Process

When reviewing new and existing packages, a reviewer/moderator will have a few things left for review after the verifier and validator have verified a package.

Moderation Workflow

First Time Go Workflow

When a good package is submitted, the normal flow of moderation works roughly like this:

  1. A maintainer submits a package. That puts the package in a "Pending" status (Pending automated review checks).
  2. If automated reviews don't require changes, the package moves to a "Ready" status. (Ready for Reviewer)
  3. If a moderator doesn't find any required changes, they move the package to an "Approved" status.

Full Workflow

The full normal workflow is like this:

  1. A maintainer submits a package. That puts the package in a "Pending" status (Pending automated review checks).
  2. If automated reviews don't require changes, the package moves to a "Ready" status. (Ready for Reviewer)
  3. If any of the automated review checks flag a package, the package moves to a "Waiting" status. (Waiting for maintainer to take corrective action)
  4. The package will sit in the Waiting status until a maintainer resubmits the package (starts the process from step one) or responds ("Responded"). Responses are typically questions, comments or requests for exempting from the verifier. (Maintainer responded, waiting for review/Maintainer update)
  5. If the package is in "Responded", it moves up the queue and waits for a reviewer to go over the response and process it accordingly.
  6. If a package is resubmitted, it doesn't go into a Ready status. It moves to an "Updated" status at the top of the queue. (Maintainer updated, waiting for reviewer)
  7. If a moderator asks for required changes, the package moves to a "Waiting" status. (back to step 4)
  8. If a moderator doesn't find any required changes, they move the package to an "Approved" status.

Trusted Package Workflow

This is the trusted package workflow:

  1. A maintainer submits a trusted package. That puts the package in a "Pending" status (Pending automated review checks).
  2. If automated reviews don't require changes, the package moves to an "Approved" status.
  3. If any of the automated review checks flag a package, the package moves to a "Waiting" status. (Waiting for maintainer to take corrective action)
  4. The package will sit in the Waiting status until a maintainer resubmits the package (starts the process from step one) or responds ("Responded"). Responses are typically questions, comments or requests for exempting from the verifier. (Maintainer responded, waiting for review/Maintainer update)
  5. If the package is in "Responded", it moves up the queue and waits for a reviewer to go over the response and process it accordingly.
  6. If a package is resubmitted, it doesn't go into a Ready status. It moves to an "Updated" status at the top of the queue. (Maintainer updated, waiting for reviewer)
  7. If the package passes automated review, the package moves to an "Approved" status.
  8. If a moderator asks for required changes, the package moves to a "Waiting" status. (back to step 4)
  9. If a moderator must manually override the approval, they move the package to an "Approved" status.

Maintainer Process

FYI: Ensure that you can receive emails from Chocolatey.org so that you will receive email notifications when a package review is updated.

The process of moderation review is an interactive process for both maintainers and moderators. As a maintainer you submit packages and they are reviewed to be sure they meet a minimum quality and correctness to be published on Chocolatey.org. It's an important distinction that while almost all valid packages are approved, a package can be rejected for a variety of reasons.

Packages go through three automated checks: validation, verification, and cleanup. There is about a 30 minute lag time from submission until automatic review kicks off - this allows the CDN to recheck and pull a newer version of the package up (in the case of resubmission), so that the package version being verified is the one you submitted and not a stale copy.

When you receive emails that require you to take action, you should review what is requested and make the changes. If a package is flagged and needs changes based on requirements, the process is for you to make the required changes and resubmit the exact same version. The faster you respond to the review process, the faster your package can get approved.

The cleanup automated check, aka the cleaner, checks packages that have been in a 'waiting' (waiting for maintainer to take action) status with no action/response within 20 days and follows up with a final reminder. If after 15 more days nothing has been done, the package will automatically be rejected on non-response. We feel that 35 days prior to automatic close is ample time for a maintainer to move the ball forward (even one going on holiday). If a package gets rejected, it doesn't mean that we don't value your contributions, just that we can not continue to hold packages versions in a waiting status that have possibly been abandoned. The rejected status is also reversible in case a maintainer wants to pick it back up within a year.

Moderators give you the benefit of the doubt and will work with you to help you get a package to an approved status. (This also includes the older review process based on email before the site allowed you to comment).

Reviewer / Moderator Process

Typically a package goes into the moderation queue when submitted.You can get to that by signing in and going to the packages page like you normally would.

  1. You should see a new drop down near the top that allows you to change your view. This is the moderation queue. Moderation Queue Dropdown
  2. You will see items arranged in order based on reviewed and resubmitted at the top, items ready for review in order based on when they were submitted, and at the end of the queue, you will see items that are waiting for maintainer response. Moderation Queue
  3. You grab a package and head in and review it based on the following items in the requirements and guidelines.
  4. Ensure the verifier has run. It will have both comments in the review and a colored ball up next to the title of the package (see image below). The ball should be:
    - Green if it is ready for review and approval.
    - Orange if still pending verification (has not yet run).
    - Red if it failed verification. The maintainer needs to fix or respond. If you find a package needs to skip verification, please contact an admin to do so. If you see a network issue from the log, you can rerun verification (see how in the next step).
    - Grey if a package skips verification for some reason (which will be listed by the admin that flagged the package to skip verification). If possible, you will need to run the install/uninstall yourself.
    Passed Verifier
  5. Check over the verifier logs to be sure everything looks good (follow the link from the button). If necessary, you can rerun the verifier.
    Rerun
  6. Go over the review log - shows history and review information so far. Note that when the validator runs it leaves comments. Look for it to have done the automated part of the requirements/guidelines checks. If it has not, you are responsible to check all requirements/guidelines (see Requirements and Guidelines above).
    Review Notes
  7. Look at the notes section from the latest run of the validator to see if there are additional flagging follow ups from the validator.
  8. Check over the package based on moderator review (below).
  9. Review the previous comments if there are any. image
  10. Look through the package files image
  11. Leave comments in the review box ("Add to Review Comments" section) if you have any. Note that you can use markdown here.
    Review box
  12. If you are approving the package, change Package Status to Approved. If you are Rejecting a package, change the status to Rejected. Otherwise leave the status as is (likely in Submitted).
  13. If you are making a comment or doing another action, but don't want to flag/hold the package for the maintainer to take action, uncheck the "require maintainer to make changes?" box. This is not required to be unchecked if you are approving the package.
    Require maintainers to make changes?
  14. If you are doing an action that doesn't need to notify the maintainer, uncheck "Send Maintainer email?".
  15. Click Save. You should get a message that the message was sent successfully.
  16. The maintainers receive an email noting the comments. They will follow up on the package page with their comments.
  17. Once the package is updated, it will show up in the top of the queue. At that time, please review it and make sure the maintainers made all changes requested.

Moderator Review

You can only ever require a maintainer to make changes if there are findings from the requirements section. Guidelines are strong suggestions that will improve the quality of the package, but consider that a quality over time. A maintainer is NOT required to make changes based on guidelines/suggestions. This deserves to be said twice: "A moderator cannot hold up a package based on guidelines/suggestions alone.

The validator checks quite a few items (https://github.com/chocolatey/package-validator/wiki) and leaves a few for you to check. Ensure you have looked over the notes that it has left.

With the exception of included binaries, a review that doesn't flag should take under a minute. If you are holding a package, you can refer the maintainer to this link to save time: https://github.com/chocolatey/choco/wiki/Moderation

Requirements

Always be explicit that you are waiting on the maintainer to fix and resubmit the same version of the package so you can move the review process along.

Guidelines

If a package is only flagging on guidelines, be sure to move forward on approval (this means no requirements flagged by you or the validator checks).

Roles

Becoming a Maintainer

To become a package maintainer, you must have an account on https://chocolatey.org and have at least one package on the site.

Becoming a Reviewer

TBD

Becoming a Moderator

There is no set process for becoming a moderator yet. Usually it is having many approved packages and understanding the process of creating Chocolatey packages. Eventually it will be something you earn through your reputation on the site.

Becoming an Admin

This is not an achievable status.

New Reviewers / Moderators