Policy Rules

Available in v1.31.2+

Policy Rules allow you to automatically attach custom GPU policies to workloads based on defined matching criteria. Note that all matching is evaluated against pod labels, annotations, and environment variables — not those set on the workload owner object (e.g. Deployment or StatefulSet).

Overview

GPU Policy Rules provide a powerful way to automatically apply specific GPU policies to workloads whose pods match certain criteria. This enables you to:

• Automatically apply appropriate GPU policies to workloads based on their pod characteristics
• Reduce manual GPU policy management overhead
• Ensure consistent policy application across similar GPU workloads
• Scale GPU policy management across large clusters

Creating Policy Rules

To create a GPU policy rule:

1. Navigate to the GPU Optimization → Policies page
2. Click on the Rules tab
3. Click Create new policy rule

Rules Definition

Define one or more rules to identify workloads that should be associated with the GPU policy. The policy is applied to all workloads whose pods match any of the defined rules.

Detected Workload Tag

A tag automatically added to workloads that match the rule criteria, indicating which rule was applied. The tag can be used for:

• Filtering workloads in the GPU optimization page
• Visibility in the workload overview dialog

Define Detected Workload Identifiers

The identifiers are used to match workloads with the GPU policy rule. All matching is evaluated against pod labels, annotations, and environment variables — not the labels or annotations set on the workload owner object (e.g. Deployment or StatefulSet). Multiple identifiers within a rule are combined with an AND operator, while multiple rules are combined with an OR operator.

Match Options

Logic Operators

• Within a rule: Multiple identifiers use AND logic (all conditions must match)
• Between rules: Multiple rules use OR logic (any rule can match)

Enforcing Policy Rules

Policy rules are enforced automatically for any workload matching the defined criteria.

Default Behavior

• If no rules match, ScaleOps applies the recommended auto-detected GPU policy
• If the workload’s GPU policy is managed manually (via UI actions or YAMLs), rules will not apply

Overriding Manual Policies

To apply custom rules instead of manual GPU policies:

1. Use the Restore Detected Policy actions at the cluster, namespace, or workload level in the GPU Optimization page
2. Remove any annotations or YAMLs that override GPU policy behavior

Examples

Example 1: Pods with Annotation and Environment Variable

Configuration:

• Rule: Workloads whose pods have annotation team/ml AND environment variable MODEL_ENV=production
• Logic: Both conditions must be met (AND operator)

Use Case: Apply a Real-time GPU policy to production inference workloads from a specific ML team.

Example 2: Pods with Label or Annotation

Configuration:

• Rule 1: Workloads whose pods have label workload-type=inference
• Rule 2: Workloads whose pods have annotation gpu-profile=latency-sensitive
• Logic: Either condition can match (OR operator between rules)

Use Case: Apply a Real-time policy to both explicitly labeled inference workloads and any workload annotated as latency-sensitive.

Best Practices

When to Use GPU Policy Rules

• Workload-type policies: Different GPU policies for inference, training, and batch workloads
• Environment-based policies: More conservative policies for production vs. development GPU workloads
• Team-based policies: Specific GPU policies for different ML teams
• Cost optimization: Apply efficiency-focused policies (e.g., Batch) to non-latency-sensitive GPU workloads

Configuration Tips

1. Start simple: Begin with a single rule and expand as needed
2. Use specific identifiers: Prefer key-value pairs over keys-only for more precise matching
3. Test thoroughly: Validate that your rules match the intended workloads
4. Monitor effectiveness: Regularly review which workloads are being matched by your rules
5. Document your rules: Keep clear documentation of what each rule is intended to match

Common Patterns

Troubleshooting

Rule not applying? Check for:

• Manual GPU policy overrides (UI actions or YAMLs)
• Incorrect identifier syntax
• Missing labels/annotations on target pods (rules match pod labels/annotations, not the workload owner)
• Conflicting rules with higher precedence