Ship It Weekly – DevOps, SRE, and Platform Engineering News with Brian Teller

This week on Ship It Weekly, LiteLLM's release pipeline was hijacked again: stolen GitHub CLI tokens from three engineers changed the pipeline, kicked off the release and approved it, and a credential-stealing 1.107.0 went out to PyPI and Docker Hub. GitHub put a package firewall into public preview, with advisory and supply chain checks and a cooldown for new releases. PHP fixed PHP2Shell, two critical core runtime bugs (CVE-2026-109618 and CVE-2026-109619) that someone tried to exploit four days before disclosure. And eMudhra, a certificate authority in India, found out the private keys for eight intermediate certificates could be worked out from its own public certificates.

The thread: most of this came through things you depend on but don't run. A library's release pipeline, the registry you pull from, the runtime under your app, and the trust store in your images.

In the lightning round: another GitLab CVSS 10 (CVE-2026-110264), Google's report on the europe-west2 London outage, AWS taking its European Sovereign Cloud off the global backbone on October 24, and a GitHub audit log change that can quietly break SIEM rules.

The human closer, from Brent Chapman: page on what users actually do. Your users don't care how your machines feel. They care whether they can log in.

On Monday, Jinger Meilani joins me to talk about running retros like a tabletop RPG.

Highlights

• LiteLLM 1.107.0: two stolen identities passed a two-person approval

• GitHub package firewall: advisories, attestations and a cooldown

• PHP2Shell: an exploit attempt four days before disclosure

• Patching PHP is easy, finding all of it is hard

• eMudhra: CA private keys worked out from public certificates

• Page on what users feel, not how the machine feels

Links

LiteLLM 1.107.0 compromise (Wiz): https://tsn.io/yQtmu

GitHub package firewall in public preview: https://tsn.io/QjofJ

PHP2Shell, CVE-2026-109618 and CVE-2026-109619 (Help Net Security): https://tsn.io/GUyop

Eight eMudhra CA keys compromised (The Hacker News): https://tsn.io/K2jqM

GitLab 19.5.1 patch release: https://tsn.io/0wsPX

Google Cloud europe-west2 incident report: https://tsn.io/nVk2N

AWS European Sovereign Cloud independent operation: https://tsn.io/gKnQl

GitHub secret scanning updates: https://tsn.io/JDGmT

Stop paging on how your machines feel (Brent Chapman): https://tsn.io/MAItM

SRE Weekly #539: https://tsn.io/XAsKz

This week's On Call Brief: https://tsn.io/nBAVg

More episodes and show notes: https://tsn.io/NqkdP

On Call Brief: https://tsn.io/Gpz2d

LMGT Awards: https://tsn.io/lAERA

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps.

In this Ship It Conversations episode, I talk with Rob Koch about what accessible cloud native actually looks like, why sign language is still one of the hardest problems in AI, and why the fastest way to make your tooling accessible is to hire deaf engineers.

Rob is a senior principal at Slalom, an AWS Hero, and co-chair of the CNCF Deaf and Hard of Hearing Working Group. Rob is Deaf, and his ASL interpreter, Amelia Fruehsamer, voices his answers. Special thanks to Amelia for interpreting the conversation.

We start with an analogy Rob comes back to all episode: cloud native is the toolbox, and platform engineering is the workbench. Then we get into what's overhyped about AI, the guardrails it still needs, and what auditable AI observability looks like.

A big part of the conversation is accessibility in practice. Rob explains how captions pushed for deaf attendees ended up helping neurodivergent folks too, and what conferences get wrong: accessibility planned at the very end instead of the start, and the real cost of interpreters versus captions.

Then we get technical. Rob walks through using Kubernetes and Kubeflow to train models that recognize sign language, and why it's so hard: regional signs, lighting, skin tone and backgrounds, and a camera that can't tell which way a hand is facing. His joke: we don't have AGI until AI understands sign language.

The big takeaway: start with people, not a checklist. Hire deaf and hard of hearing engineers, ask them how they want to work, and your tools get more accessible because they're in the room.

Highlights

• Cloud native as the toolbox, platform engineering as the workbench

• What's overhyped about AI, and the guardrails it still needs

• What accessible cloud native means in practice

• Captions that help far more people than you planned for

• What conferences get wrong about accessibility, and what it costs

• Kubernetes and Kubeflow for sign language recognition

• Gesture-based computing and why speech-to-text had a 30-year head start

• Where to start: hire deaf engineers

Links

Rob Koch on LinkedIn: https://tsn.io/4qmFo

Rob Koch on GitHub: https://tsn.io/Mbxrs

Amelia Fruehsamer (Rob's ASL interpreter) on LinkedIn: https://tsn.io/UJB88

CNCF Deaf and Hard of Hearing Working Group: https://tsn.io/tB1lS

CNCF conference accessibility best practices: https://tsn.io/BZvpv

CNCF Merge Forward: https://tsn.io/QPVNQ

Kubeflow: https://tsn.io/4OVKf

More episodes and show notes: https://tsn.io/NqkdP

On Call Brief: https://tsn.io/Gpz2d

LMGT Awards: https://tsn.io/lAERA

This week on Ship It Weekly, GitHub Actions workflow lockfiles land in public preview, pinning actions and their dependencies to exact commits. GitLab shipped a critical patch release for CVE-2026-97229, a CVSS 10 CI/CD regex flaw that could let an unauthenticated attacker run code on the server. Google published its final report on two Cloud Storage incidents in us-central1, including a recurrence caused by an incomplete rollback. And Linus Torvalds pushed an experimental kernel branch with page-cache patches built using Claude Code, as maintainers debate dropping the mandatory AI attribution tag.

The thread this week: know what you're actually running, from the commit you reviewed to the state a rollback really left behind and code someone can still explain.

In the lightning round: a record Patch Tuesday with more than 1,400 CVEs and an actively exploited Windows Shell bypass, GitHub AI Scan for pull requests and agentic autofix going GA, AWS Strands Box, an open-source agent sandbox that keeps real credentials out of the agent's environment, and a Microsoft 365 outage through Global Secure Access.

The human closer comes from Paul LaPosta's “Your error budgets don't know AI exists.” AI has split shipping a change from understanding it, and SLOs don't measure comprehension. Your SLO can tell you the service is up. It cannot tell you whether anyone still understands it.

Programming note: Ship It Weekly turns one next month, so expect more conversations over the next couple of months, usually two a week, starting Monday with Rob Koch.

Links

GitHub Actions workflow lockfiles

GitLab 19.4.2 / 19.3.5 / 19.2.8 patch release

Google Cloud Storage incident report

Linus Torvalds and Claude Code (Phoronix)

Linux drops the Assisted-by requirement (Phoronix)

October 2026 Patch Tuesday

GitHub AI Scan for pull requests GA

Agentic autofix GA

AWS Strands Box

Microsoft 365 Global Secure Access outage

Your error budgets don't know AI exists (LeadDev)

This week's On Call Brief

This is a guest conversation episode of Ship It Weekly, separate from the weekly news recaps.

In this Ship It Conversations episode, I talk with Erik Osterman of Cloud Posse about Terraform modules, infrastructure as code frameworks, internal developer platforms, and why AI makes conventions more valuable, not less.

Erik is the founder and CEO of Cloud Posse, which has published more than 180 open source Terraform modules, and he hosts Cloud Posse's weekly DevOps Office Hours.

We start with what AI has done to open source infrastructure projects, then get into what makes a good Terraform module: consistent naming, a consistent interface, not so small that it wraps a single resource, and not so large that it has a hundred inputs nobody can test.

We also talk about CloudFormation, Pulumi, and why HCL's constraints helped build the module ecosystem. Erik explains where Crossplane fits, and why accounts, IAM, networking, DNS and certificates all come before the cluster.

A big part of the conversation is frameworks. Erik argues that nobody actually runs "vanilla Terraform," because Terraform alone doesn't handle state layout, promotion, authentication, tool installation or repository structure. Every team ends up building its own framework, usually undocumented. We talk about Terragrunt, Atmos, naming conventions, brownfield adoption, and breaking a monolithic root module into services.

Near the end, we get into AI and self-service. Erik's team now generates most of its infrastructure work with AI, and it works because the conventions, agent skills, MCPs and pipelines are already in place. Self-service doesn't have to mean another no-code platform.

The big takeaway: AI can generate infrastructure code faster than ever, but without conventions it is generating into chaos.

Highlights

• What makes a good Terraform module

• Root modules vs. child modules

• CloudFormation, Terraform, Pulumi, and why constraints helped HCL win

• Where Crossplane fits, and what has to exist before the cluster

• Why internal developer platforms like Backstage come after zero-to-one

• Why infrastructure as code needs frameworks, not just modules

• Agent skills, MCPs, and self-service without a no-code platform

Links

Cloud Posse: https://cloudposse.com

Atmos: https://atmos.tools

Cloud Posse on GitHub: https://github.com/cloudposse

DevOps Office Hours: https://tsn.io/2gq7v

Erik Osterman on LinkedIn: https://www.linkedin.com/in/osterman

More episodes and show notes: https://tsn.io/NqkdP

On Call Brief: https://tsn.io/Gpz2d

LMGT Awards: https://lmgt.org

This week on Ship It Weekly: Atlassian disclosed a critical arbitrary file access vulnerability affecting eight Data Center products, including Jira, Confluence, Bitbucket, and Bamboo. GitHub Copilot local sandboxing is now generally available, giving teams enforceable controls around filesystem, network, credentials, and local tooling. AWS published a deeper look at Lambda SnapStart for container images, including the operational risks of restoring initialized state. And Google Spanner Omni is now generally available, bringing Spanner outside Google Cloud and putting more of the availability responsibility back on the customer.

The bigger theme this week is control. Developer tools are production infrastructure. AI agents need real execution boundaries, not just instructions. Faster startup through snapshots means understanding exactly what state is being restored. And moving a managed database onto your own infrastructure gives you portability, but also gives you the pager.

In the lightning round: Citrix has a NetScaler SAML vulnerability under active exploitation. Cloudflare is working toward becoming a publicly trusted certificate authority. And attackers are actively scanning for Rejetto HFS servers vulnerable to a flaw that can lead to account takeover and remote code execution.

And the human closer comes from Lorin Hochstein and a recent Google Cloud networking incident. One part of the outage involved someone correctly following an incorrect procedure, while another involved someone incorrectly following a correct one. Training matters, but systems also need guardrails that keep normal human mistakes from turning into large outages. If your system only works when nobody makes a mistake, the system is the problem.

Links

Atlassian CVE-2026-21589

https://tsn.io/ZlqKv

GitHub Copilot Local Sandboxing

https://tsn.io/rQngM

AWS Lambda SnapStart for Container Images

https://tsn.io/uXqkK

Google Spanner Omni GA

https://tsn.io/GoZZ5

Citrix NetScaler CVE-2026-88779

https://tsn.io/yN5a9

Cloudflare Certificate Authority

https://tsn.io/egMN5

Rejetto HFS CVE-2026-61500

https://tsn.io/oQrkU

Google Cloud Networking Incident

https://www.tellerstech.com/go/s-dc989ad2/

There Was a Light That Never Went Out

https://www.tellerstech.com/go/s-cb0bc459/

This Week’s On Call Brief

https://tsn.io/8yaIY

Ship It Weekly

https://tsn.io/NqkdP

On Call Brief

https://tsn.io/Gpz2d