| 1 | # Linux Environments Playbook v1 |
| 2 | |
| 3 | ## Purpose |
| 4 | Project-independent playbook for reliable engineering and troubleshooting on Linux hosts. |
| 5 | |
| 6 | ## Scope |
| 7 | - Linux process/service runtime behavior |
| 8 | - Permissions and ownership model |
| 9 | - Resource limits and isolation boundaries |
| 10 | - Operational diagnostics baseline |
| 11 | |
| 12 | ## Evidence-Based Working Format |
| 13 | - Fact: capture concrete host symptom (service crash loop, permission issue, limit exhaustion). |
| 14 | - Hypothesis: define smallest plausible Linux-level cause. |
| 15 | - Check: run minimal reproducible verification command set. |
| 16 | - Decision criterion: predefine close/escalate threshold. |
| 17 | - Confidence mark: explicit certainty and residual risk. |
| 18 | |
| 19 | ## Core Linux Contracts |
| 20 | - Service model and lifecycle semantics must be explicit. |
| 21 | - Ownership/mode/capability assumptions must be validated. |
| 22 | - Resource limits and cgroup boundaries must be visible. |
| 23 | - Locale/timezone/encoding assumptions must be explicit. |
| 24 | |
| 25 | ## Runtime and Service Contracts |
| 26 | - systemd unit contracts (dependencies, restart, environment) must be deterministic. |
| 27 | - Startup arguments and runtime env must be explicit. |
| 28 | - Signal handling and graceful shutdown behavior must be validated. |
| 29 | - Worker/daemon ownership of subprocesses must be clear. |
| 30 | |
| 31 | ## Permissions and Filesystem Contracts |
| 32 | - Validate uid/gid ownership and mode bits on critical paths. |
| 33 | - Keep mount and symlink boundaries explicit. |
| 34 | - Avoid implicit root assumptions. |
| 35 | - Confirm write boundaries before mutating operations. |
| 36 | |
| 37 | ## Resource and Isolation Contracts |
| 38 | - Check `ulimit` and file descriptor ceilings for IO-heavy services. |
| 39 | - Validate memory/cpu/pid constraints in cgroups/containers. |
| 40 | - Distinguish host saturation from app-level faults. |
| 41 | - Keep capacity symptoms separated from logic bugs. |
| 42 | |
| 43 | ## Diagnostics Baseline |
| 44 | - journal logs + structured application logs. |
| 45 | - Process tree, open-file/socket state, and limit visibility. |
| 46 | - Resource pressure timeline (cpu/memory/io) before deep app debugging. |
| 47 | - One concise root-cause and prevention delta per incident. |
| 48 | |
| 49 | ## Metrics |
| 50 | - Mean time to identify Linux host root cause. |
| 51 | - Frequency of limit/ownership-related incidents. |
| 52 | - Recovery time for service restarts. |
| 53 | - Reopen rate for Linux environment incidents. |
| 54 | |
| 55 | ## Revisit Triggers |
| 56 | - Repeated “works only on one distro/host profile” failures. |
| 57 | - Frequent permission/ownership regressions. |
| 58 | - Recurring incidents from invisible resource constraints. |
| 59 | |
| 60 | |