One thing I have learned over the years is that programming the automatic cycle is sometimes the easy part.
The machine starts in a known condition.
You tell it what to do.
It moves through the sequence.
Everything happens in the order you expected.
Then somebody asks:
“What happens if it stops right here?”
And suddenly the simple machine you programmed yesterday has developed about 47 possible personalities.
Recovery can be harder to program than the cycle itself.
I think most controls programmers have experienced some version of this during a machine startup.
The normal cycle is working, everyone is happy, and then the customer starts asking perfectly reasonable questions.
What happens if the operator presses Stop here?
What happens if the servo faults there?
What happens if the sensor fails after the clamp closes?
Can we resume the weld halfway through?
Can we restart without scrapping the part?
Can the machine automatically figure out where it was?
And eventually, if the discussion goes on long enough, you begin to feel like the next question is going to be:
“Okay, but what happens if the building burns down during a hurricane?”
At some point, recovery has to have limits.
But deciding where those limits are is an important part of machine design.
The Happy Path Is Easy to Understand
Most automatic sequences are built around what programmers sometimes call the happy path.
Everything works.
Part present.
Clamp closed.
Axis ready.
Laser ready.
Gas pressure good.
Cycle start.
Go.
That is relatively straightforward because every step has an expected beginning and an expected end.
Recovery is different.
If a cycle contains 30 steps, the machine can stop during any of those 30 steps—and the machine’s physical condition may be different at each one.
The part may be clamped.
An axis may be halfway through a move.
The laser process may have already started.
A shutter may have changed state.
Gas may still be flowing.
A robot may have completed a move, but the PLC may not have received the confirmation yet.
Now you have to answer a much harder question:
What does “start again” actually mean?
Reset Is Not the Same as Recovery
It is tempting to solve these problems with one big Reset button.
A fault occurs.
Operator presses Reset.
Everything goes back to normal.
Except machines don’t really work that way.
Resetting a fault bit does not change where an axis physically is.
It does not tell you whether a part has already been partially processed.
It does not guarantee a clamp is in the position the sequence expected.
And it definitely does not make a half-completed laser weld magically become an unprocessed part again.
One distinction that helps is to separate:
Reset — clear the fault condition when it is safe to do so.
from
Recovery — return the machine to a known condition from which production can safely continue.
Those are not always the same operation.
Sometimes the Correct Recovery Is “Start Over”
Customers understandably want machines that recover automatically.
And when practical, they should.
But in some situations, trying to recover exactly where the process stopped creates more complexity — and sometimes more risk — than simply returning the machine to a known state.
A positioning move might be easy to resume.
A laser cut interrupted halfway through a feature may not be.
A weld that stops halfway through may require operator evaluation before anything continues.
A partially completed additive process may have an entirely different set of consequences.
So one of the first questions during machine design should be:
Which failures can we recover from automatically, and which ones require an operator or technician to make a decision?
That conversation is much easier before the machine is on the customer’s floor.
Design Recovery While You Design the Sequence
One thing I think helps is asking a recovery question every time you create a machine state.
If the machine stops here:
- What physical condition will it probably be in?
- What conditions do we need to verify?
- Can we safely retry this state?
- Do we need to back out first?
- Does the part have to be removed?
- Is operator intervention required?
You don’t necessarily need a sophisticated automatic recovery routine for every possible fault.
You do need to decide what should happen.
That distinction matters.
A machine that deliberately says:
“Cycle interrupted during weld. Remove part and reset machine before restarting.”
may be much better designed than one that automatically tries to continue because somebody decided the machine should “recover from everything.”
The HMI Becomes Very Important Here
This is also where good diagnostics earn their keep.
After a fault, the operator is usually asking two questions:
What happened?
and
What do I do now?
The HMI should help answer both.
Compare:
Cycle Fault
with:
Cycle interrupted — X axis lost servo ready while part was clamped. Reset drive, verify axis position, then select Recovery.
The second message gives the operator somewhere to start.
Even better is when the machine can show its current state and the conditions preventing recovery.
For example:
Recovery unavailable:
- X axis not homed
- Clamp position not confirmed
- Laser not ready
That can save a lot of staring at the screen and pressing Reset repeatedly to see whether anything changes.
I have done my share of that too.
Test the Bad Cycle, Not Just the Good One
One of the best ways to evaluate recovery is surprisingly simple:
Break the machine on purpose.
During commissioning:
Press Stop during motion.
Interrupt a sensor.
Fault a drive.
Drop a process permissive.
Press the E-stop at several points in the sequence.
Interrupt the process itself.
Then give the machine back to someone who did not write the program.
Can they figure out what happened?
Can they get it running again?
Or does the controls programmer have to walk over with the laptop?
That is a very useful test.
Because a machine may run 500 perfect cycles and still be miserable to support the first time something goes wrong.
Recovery Requirements Need Boundaries
This also has an important project-management side.
“Recover from a fault” sounds like one requirement.
It isn’t.
Recover from which faults?
At what points in the sequence?
Automatically or manually?
Can the part be reused?
Can motion resume from the interrupted location?
What if power is lost?
What if position is no longer known?
The deeper you go, the larger the requirement becomes.
That doesn’t mean the customer is wrong for asking.
It means the machine builder and customer should agree on what reasonable recovery actually looks like.
Otherwise, near the end of startup, you may discover that the automatic cycle was only half the software project.
The other half is figuring out how to recover from every possible thing that can happen to it.
Including, apparently, hurricanes.
The Practical Test
A good machine should not only know how to run.
It should know enough about its own condition to help people understand why it stopped and what needs to happen next.
But recovery does not have to mean recovering from absolutely anything.
Sometimes the best recovery strategy is automatic.
Sometimes it is a guided manual sequence.
And sometimes the correct answer really is:
“Remove the part, return the machine to a known state, and start over.”
The important part is that somebody decided that intentionally.
Because getting a machine through the perfect cycle is automation.
Getting it back into production after the imperfect one is where a lot of the real programming begins.
Absolutely. I think the humor works best if it comes from the reality that recovery requirements can quietly expand from “recover from a sensor fault” to “apparently survive the apocalypse.”
MWL Automation Note
The Cycle Works. But Can You Recover When It Doesn’t Finish?
One thing I have learned over the years is that programming the automatic cycle is sometimes the easy part.
The machine starts in a known condition.
You tell it what to do.
It moves through the sequence.
Everything happens in the order you expected.
Then somebody asks:
“What happens if it stops right here?”
And suddenly the simple machine you programmed yesterday has developed about 47 possible personalities.
Recovery can be harder to program than the cycle itself.
I think most controls programmers have experienced some version of this during a machine startup.
The normal cycle is working, everyone is happy, and then the customer starts asking perfectly reasonable questions.
What happens if the operator presses Stop here?
What happens if the servo faults there?
What happens if the sensor fails after the clamp closes?
Can we resume the weld halfway through?
Can we restart without scrapping the part?
Can the machine automatically figure out where it was?
And eventually, if the discussion goes on long enough, you begin to feel like the next question is going to be:
“Okay, but what happens if the building burns down during a hurricane?”
At some point, recovery has to have limits.
But deciding where those limits are is an important part of machine design.
The Happy Path Is Easy to Understand
Most automatic sequences are built around what programmers sometimes call the happy path.
Everything works.
Part present.
Clamp closed.
Axis ready.
Laser ready.
Gas pressure good.
Cycle start.
Go.
That is relatively straightforward because every step has an expected beginning and an expected end.
Recovery is different.
If a cycle contains 30 steps, the machine can potentially stop during any of those 30 steps — and the physical condition of the machine may be different at every one of them.
The part may be clamped.
An axis may be halfway through a move.
The laser process may have already started.
A shutter may have changed state.
Gas may still be flowing.
A robot may have completed a move, but the PLC may not have received the confirmation yet.
Now you have to answer a much harder question:
What does “start again” actually mean?
Reset Is Not the Same as Recovery
It is tempting to solve these problems with one big Reset button.
A fault occurs.
Operator presses Reset.
Everything goes back to normal.
Except machines don’t really work that way.
Resetting a fault bit does not change where an axis physically is.
It does not tell you whether a part has already been partially processed.
It does not guarantee a clamp is in the position the sequence expected.
And it definitely does not make a half-completed laser weld magically become an unprocessed part again.
One distinction that helps is to separate:
Reset — clear the fault condition when it is safe to do so.
from
Recovery — return the machine to a known condition from which production can safely continue.
Those are not always the same operation.
Sometimes the Correct Recovery Is “Start Over”
Customers understandably want machines that recover automatically.
And when practical, they should.
But in some situations, trying to recover exactly where the process stopped creates more complexity — and sometimes more risk — than simply returning the machine to a known state.
A positioning move might be easy to resume.
A laser cut interrupted halfway through a feature may not be.
A weld that stops halfway through may require operator evaluation before anything continues.
A partially completed additive process may have an entirely different set of consequences.
So one of the first questions during machine design should be:
Which failures can we recover from automatically, and which ones require an operator or technician to make a decision?
That conversation is much easier before the machine is on the customer’s floor.
Design Recovery While You Design the Sequence
One thing I think helps is asking a recovery question every time you create a machine state.
If the machine stops here:
- What physical condition will it probably be in?
- What conditions do we need to verify?
- Can we safely retry this state?
- Do we need to back out first?
- Does the part have to be removed?
- Is operator intervention required?
You don’t necessarily need a sophisticated automatic recovery routine for every possible fault.
You do need to decide what should happen.
That distinction matters.
A machine that deliberately says:
“Cycle interrupted during weld. Remove part and reset machine before restarting.”
may be much better designed than one that automatically tries to continue because somebody decided the machine should “recover from everything.”
The HMI Becomes Very Important Here
This is also where good diagnostics earn their keep.
After a fault, the operator is usually asking two questions:
What happened?
and
What do I do now?
The HMI should help answer both.
Compare:
Cycle Fault
with:
Cycle interrupted — X axis lost servo ready while part was clamped. Reset drive, verify axis position, then select Recovery.
The second message gives the operator somewhere to start.
Even better is when the machine can show its current state and the conditions preventing recovery.
For example:
Recovery unavailable:
- X axis not homed
- Clamp position not confirmed
- Laser not ready
That can save a lot of staring at the screen and pressing Reset repeatedly to see whether anything changes.
I have done my share of that too.
Test the Bad Cycle, Not Just the Good One
One of the best ways to evaluate recovery is surprisingly simple:
Break the machine on purpose.
During commissioning:
Press Stop during motion.
Interrupt a sensor.
Fault a drive.
Drop a process permissive.
Press the E-stop at several points in the sequence.
Interrupt the process itself.
Then give the machine back to someone who did not write the program.
Can they figure out what happened?
Can they get it running again?
Or does the controls programmer have to walk over with the laptop?
That is a very useful test.
Because a machine may run 500 perfect cycles and still be miserable to support the first time something goes wrong.
Recovery Requirements Need Boundaries
There is also an important project-management side to this.
“Recover from a fault” sounds like one requirement.
It isn’t.
Recover from which faults?
At what points in the sequence?
Automatically or manually?
Can the part be reused?
Can motion resume from the interrupted location?
What if power is lost?
What if position is no longer known?
The deeper you go, the larger the requirement becomes.
That doesn’t mean the customer is wrong for asking.
It means the machine builder and customer should agree on what reasonable recovery actually looks like.
Otherwise, near the end of startup, you may discover that the automatic cycle was only half the software project.
The other half is figuring out how to recover from every possible thing that can happen to it.
Including, apparently, hurricanes.
The Practical Test
A good machine should not only know how to run.
It should know enough about its own condition to help people understand why it stopped and what needs to happen next.
But recovery does not have to mean recovering from absolutely anything.
Sometimes the best recovery strategy is automatic.
Sometimes it is a guided manual sequence.
And sometimes the correct answer really is:
“Remove the part, return the machine to a known state, and start over.”
The important part is that somebody decided that intentionally.
Because getting a machine through the perfect cycle is automation.
Getting it back into production after the imperfect one is where a lot of the real programming begins.