Keep Daily Deployments Going Even When Automation Stops—How to Return to a Manual Process After Failures, Expired Permissions, and Specification Changes
- AI
- GitHub
- Deploy Now
- Automation
- Project Management
Keep Daily Deployments Going Even When Automation Stops—How to Return to a Manual Process After Failures, Expired Permissions, and Specification Changes
In the previous article, we classified the work that remains each day into three options: “delegate it to AI or a system,” “have a person continue it,” and “stop doing it.”
Automating small aggregation and verification tasks lightens the burden of deploying every day. However, automation is not equipment that will run forever once it has been built.
- Credentials expire
- The input file format changes
- The screen or API specifications change
- The network is temporarily unavailable
- Items that require AI cannot be extracted
- The automated process succeeds, but the result is incorrect
- The person responsible changes, leaving no one who can fix it
What matters is not that automation must never be stopped.
It is to quickly detect anomalies, stop automated processing, and safely return to a procedure that people can understand.
This article organizes automation into five stages—“Detection,” “Stopping,” “Switchover,” “Recovery,” and “Resumption”—so that even non-engineers can work with it. It also establishes operating practices for daily deployments using AI, GitHub, and Deploy Now so that you do not lose your judgment when anomalies occur.
Automation needs both a “standard procedure” and a “fallback procedure”
When looking only at normal operation, automation appears as follows.
Actual operations require a different route.
Without this route, the moment automation stops, you will not even know what the work involves.
Before introducing automation, document the following two things:
- The procedure used before automation
- The minimal procedure for when automation cannot be used
The minimal procedure does not need to reproduce all the work that was done previously.Limit the scope to only the work needed to make safe publishing decisions that day.
Classify anomalies into four types
“It didn’t work” alone is not enough to determine the next action. Classify anomalies into four types.
| Type | Examples | Initial response |
|---|---|---|
| Execution failure | Connection failure, timeout, interruption midway | Check the status without rerunning |
| Expired permissions | Authentication expiration, access denied | Check the scope and validity period of the permissions |
| Specification change | Changes to field names, storage locations, or response format | Check differences between the input and output |
| Abnormal result | A success message is displayed, but the count or content is abnormal | Cross-check against the source data |
Execution failure
The processing itself does not complete.
If you rerun it immediately, it may result in duplicate registration or duplicate publishing. First check “how far the execution had progressed.”
Expired permissions
A state in which locations that were readable until yesterday can no longer be read or written to.
Do not immediately restore broad permissions just because you see a permissions error. Check the required targets and operations, and restore only the minimum scope.
Specification change
Even if the processing runs, the form of the input or output no longer matches.
Specification changes are anomalies that are difficult to detect unless previous assumptions are compared with the current state.
Abnormal result
The state that warrants the most attention is when “success” is displayed but the result is incorrect.
Verify the item count, update timestamp, target scope, and expected content, rather than just checking the exit code or success indicator.
Look at “whether it finished correctly,” not “whether it ran”
Three types of evidence are used to detect anomalies.
1. Evidence of execution
Check whether processing started and ended.
2. Evidence of results
Check whether the expected deliverables exist.
3. Semantic evidence
A person checks whether the results meet the objective.
It is important to distinguish between “successful execution” and “successful operation.”
Define stop conditions first
To operate automation safely, write the stop conditions before the success conditions.
Stopping is not a failure. It is normal behavior that prevents progress when there is insufficient information to make a judgment.
Operations eligible for automatic re-execution
- Temporary read failure
- Retrieval operations that do not modify the source data
- Verification operations that do not add results even when repeated
Operations that should avoid automatic re-execution
- Publishing
- Sending
- Deletion
- Overwriting
- Billing
- Registration with an external service
- Recovery operations
For the latter, resume only after a person has checked the current state.
Decisions to switch to a manual procedure
When an anomaly occurs, there is no need to stop daily work until the repair is complete. Switch to a safe, minimal procedure.
Return to the manual procedure if any of the following applies.
- The cause cannot be explained within 10 minutes
- It is unclear how far the automated process has progressed
- The number of results does not match expectations
- Processing cannot continue unless permissions are expanded
- It affects production, deletion, overwriting, and external publishing
- The same anomaly occurred twice in succession
- The source data and results cannot be cross-checked
To prevent the switching decision from relying on the “operator’s intuition,” it will be pinned to the control panel.
Create a manual fallback card
There is no time to search for a lengthy procedure manual when an anomaly occurs. Prepare a card that can be read on a single screen or page.
The card should state not only “what to do,” but also “what not to do.”
Freeze the State When Switching
When switching from automatic to manual mode, record the state first.
If possible, also retain logs and screen records. However, if they contain credentials, secrets, or personal information, do not share them as-is.
Beginning an investigation without freezing the state may allow further operations to alter the evidence, making it impossible to determine the cause.
Procedure for Recovering from Expired Permissions
For permission errors, check in the following order.
- Which operation was denied?
- Whether it involved reading, writing, or publishing
- Which repository, folder, or service was the target?
- What was the previous authorized scope?
- Was it due to expiration, a configuration change, or a change of owner?
- Can it be reconnected with least privilege?
- Does the read-only test succeed?
Make the first test read-only, such as retrieving a list.
Do not attempt to publish to production immediately after restoring permissions.
Criteria for identifying specification changes
Specification changes are easier to find by adding a “shape check” to the routine checklist.
| Items to check | Criteria for saving |
|---|---|
| Input | Required field names, item count, and format |
| Output | File name, save destination, and column names |
| Screen | Main operations and expected results |
| Permissions | Read and write targets |
| AI output | Required headings, sources, and unverified indicator |
You do not need to compare everything every time. Perform a quick check before processing.
If any prerequisite is no longer valid, stop without beginning processing.
Anomaly detection when using AI
AI output may vary in wording even for the same input. Therefore, check the required conditions rather than performing an exact full-text match.
When instructing the AI, also specify how it should behave in the event of an anomaly.
We prioritize making the state of uncertainty visible over producing plausible-looking results.
Do not immediately return to automated operations even after recovery
Even after fixing the cause, do not immediately return to normal operations. Pass through four gates.
Gate 1: Read test
Confirm whether the inputs can be obtained correctly without changing the targets.
Gate 2: Test with only one item
Check the output using one test item or one item with minimal impact.
Gate 3: Parallel operation with the manual procedure
Compare the automated results with the manual results.
Gate 4: Restart approval
A human verifies the cause, the changes made, the test results, the stop conditions, and the method for returning.
Return to standard operations after passing all four gates.
Separate what to monitor daily, weekly, and monthly
Daily
- Last execution time
- Success / Stopped / Failure
- Number of targets
- Difference from the expected count
- Human verification results
- Whether there was a manual switchover
Weekly
- Recurrence of the same anomaly
- Number of manual switchovers
- Time required for recovery
- False positives and false negatives
- Missing stop conditions
Monthly
- Permissions and connection destinations
- Person in charge and contact information
- Manual fallback card
- How to record the last known-good point
- Value of maintaining automation
Even when there are no anomalies, review the manual procedures monthly. This is because procedures that are not used tend to become outdated.
Do not use incident records to assign blame
The purpose of the record is not to identify who made the mistake, but to ensure that we can stop safely next time.
Even if the cause was human action, we examine “which checks or stop conditions could have prevented it.”
Common failures
Not monitoring until an error appears
Even if a success message is displayed, the result may be incorrect. Check the number of items, timestamps, and target scope.
Rerunning immediately
The data may have been written partway through. Check the current progress before deciding.
Restoring broad permissions
If permissions are granted in bulk without investigating the cause, even unnecessary targets may end up being modified.
Returning to production immediately after the fix
Resume in the following order: read-only, single-item processing, concurrent operation, and approval.
Discarding the manual procedure
While the automation is functioning normally, update the manual fallback card.
Treating the automation stopping as a failure
Stopping when in a dangerous state is the correct safety behavior.
The first 7 days
Day 1
Choose one automated task and write down its inputs, outputs, and targets affected by changes.
Day 2
Write one example each of an execution failure, permission expiration, specification change, and abnormal result.
Day 3
Decide on three stop conditions.
Day 4
Summarize the minimum manual procedure on a card.
Day 5
Practice switching to manual mode without publishing or writing anything.
Day 6
Check the four gates for recovery after an incident.
Day 7
Place the status, stop conditions, manual card, and restart approval on the operations dashboard.
Summary
When incorporating automation into standard operations, ease of recovery during abnormal conditions is just as important as speed under normal conditions.
The key points are as follows.
- Leave the pre-automation procedure and the minimum steps for handling anomalies
- Classify anomalies as execution failures, permission expirations, specification changes, and abnormal results
- Review three types of evidence: execution, results, and meaning
- Write the stopping conditions before the success conditions
- If the cause cannot be explained, switch to the manual procedure
- Secure the current state before switching
- Restore permissions beginning with the minimum scope
- After recovery, resume through read-only access, a single item, parallel operation, and approval
- Update the manual fallback card regularly
- Treat stopping safely as normal operation
The first step is to select one task being automated and write in three lines, “At what state should it stop?”
Next, we will organize a method for compiling a daily deployment system that includes automation and manual fallback into handover documentation that can be continued even when the person in charge changes.
