129
dbtng
iiiiiiittttttttttttI work for a company that works for a company that works for a company.
We have clients that are clients of clients, and may be direct clients in some other aspect.
We host services for these entities.
That is all fine until they don't pay us.
When they don't pay us, there are multiple rounds of contacting someone who will contact someone who will contact someone. Someday.
Sometimes, someday does not arrive.
At that point, we conduct the Scream Test.
The essence of the Scream Test is:
- This thing is theoretically not needed.
- Let's turn it off.
- And see who screams?
We shut down some stuff today. Waiting on the cacophony...
That's what it's called. The mystery server that know one knows who set it up and don't know what it dose won't know what's hit them when I'm back in on Monday.
Love a good ol scream test. Only problem is you have to make sure to keep the disconnected thing or the data within for a solid year in case it's only used once annually by "very important process that makes sure the doors stay open" and no one realizes that thing you disconnected is where it's pulled from. Learned that one the hard way
In this case, all we did was disable some ACLs. The machines are still running.
But ya. The point is to preserve whatever is actually needed. So don't nuke it before they can scream.
But that urge to nuke it can really make some good arguments
My boss designed a whole decomm system with wait periods and shit built into an app.
He still tells me, "You can just shut that down and delete it".
I don't. I make that jerk use his own system. "Put in a decomm, and I'll take care of it."
This post feels like poetry.
With scream tests I unusually unplug the nic or whatever the equivalent is in a VM because a server can always have weird startup requirements that no one knows anymore while an unplugged nic doesn't change the running state if you need to bring it back online.
What would you do if the power went out? Or you needed to upgrade the kernel?
That's an unplanned power outage. A scream test would be a planed test. You also should do the email to all departments to see who knows what they are for and what you are doing. Usually gets the managers in a panic and those emails runs up and down the chains faster than "can you tell me what srv-15935 does"
Everything was on a ups with enough run time to turn the genny on.
As for updates, that wasn't my problem.
We left that shit running and blocked it with ACLs on the switch.
The guests are kinda black-box stuff to us. Don't want to mess with em.
The difficulty in this structure is that someone, somewhere along the chain is laid off, fired made redundant and that person may have been in charge of the invoice.
That type of disconnect can be exploited: billing for services not rendered intermittently, etc.... We used to receive thermal printing paper at random times from a random company because they knew the invoice was sent to a different address and...who really commits fraud for printer paper? It could have just been a mistake because we received that paper....Right? Such a low key crime thats hard to spot and prove mal intentions.
Anxiously waiting for a follow up.
Is the official name for this "the scream test"?
I do this in lots of jobs; I tend to come in on cleanup jobs after gross incompetence, sometimes for many years. In addition to fixing problems, I have to help companies learn what should be happening and realign them to normal operating practices. Stopping pointless, worthless, outdated and redundant practices are something I do, and sometimes I apply the method above, though with many fewer degrees of separation than you have to those who may be affected. I find cutting pointless stuff to be one of the best ways to focus energy on things that do matter, get people to a better place and sometimes even save money doing it.
Ya man. Google it. I'm not that creative. Did not make it up.
Interesting, results seem to be focused in IT as a somewhat common approach but I'm not in IT, curious it has settled there in the vernacular.
