|
Home > Archive > CCNP > May 2002 > Network Stories
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
|
|
| paul_tribe 2002-05-22, 1:32 pm |
| I hope I am not breaking any rules or anything but I thought as light relief from all our hard studies we could have a thread for network administrator cock up stories. I have a couple but I'll wait for someone else to start. | |
|
| Ill bite
How about whilst working on a Wan problem for a well known UK International Telecoms company, shutting down and enabling a link port on a node with a view to clearing a fault.
Whats the problem with that you say? well......the port I shut down was at the remote node, which only happened to have one link...DOH!! Queue me ringing the office in that particular country and reporting a strange fault with the node....
The morale of the story....always be able to delete the alarm logs
Or how about....3 o'clock in the morning during a particularly busy nightshift ringing the on call engineer in Germany (I was in the UK) and dialling the wrong number...German House Fraus are not very pleasent at the that time of the morning.
What did I learn? That German house Fraus swear like troopers. | |
| Detour 2002-05-23, 11:28 am |
| How about when I learned about STP + Netware ...the hard way.
The switches were 3com i believe, replacements for hubs. They didn't seem to have a portfast function. Ended up disabling STP, after i figured out what the hell had happened.
I once found a bug in 11.x that could crash a router with a single packet. i think cisco's fixed it since(i've been meaning to try it out on some 12.x IOS's). The sniffer i was using could send raw packets, and i was using it to experiment with sending various packets to see how the router would react. One particular packet crashed the router.....unfortunately, it was a production router....ouch...ah well, it came back up...and then i did it again just to verify it was the packet..am i smart or what  | |
| paul_tribe 2002-05-23, 11:48 am |
| OK This is my 1st:
I was checking a downed serial link on a 3com Netbuilder II at the core of our production network and it was down. The NBII had connections to our FDDI backbone and various remote offices (about 25%) of our network. Sometimes on a serial link if you disable and re-enable the port the link would bounce back.
I'm sure you can tell which port I disabled on the router...yes the FDDI which immediatley disconnected it from the rest of the network. Furthermore this router was the default gateway for soooo many things! | |
| peterd 2002-05-24, 3:40 am |
| Hi guys,
erm, well...
we are a food productions site and we supply a couple of the major supermarket chains in the UK. One of these companies has 3 - 4 other suppliers for our line of produce (we kill about 2,000,000 chickens a week across four factories) and the way they used to place orders with their suppliers was to run an internet auction.
Basically each supply company offered a certain amount of produce at a particular price for the coming week and the auction took place at 10.30am on each Tuesday.
I knew nothing about this at the time!
So on a particular Tuesday morning I was poking around with the routers on site. The previous techie had set them up with all the PC's pointing at the first router as their default gateway.
This router pointed at a second router, and the second one pointed at the internet router. It all worked ok so I'd never changed the system.
I'd inserted a spare router into this chain for testing various things and on a Tuesday morning decided to remove it, forgetting to tweak the upstream router to send unknown address packets to it's 'new' default gateway (the internet router).
So the sales guys had their internet connection running, in the middle of the auction, and suddenly they lost the connection. In the 10 minutes it took us to figure out what had happened, the auction ended and we had no sales from our site that week!
Something in the order of 1,000,000 pounds (sterling) in lost sales (but I presume we punted the stock out to other, smaller, outlets during the week).
I blamed atmospherics, there was a storm in the next county...
Fortunately the supermarket chain dropped the auction idea and appointed us as their #1 supplier, so it all ended happily...
Regards
Peter | |
|
| I was doing a small upgrade on a mux. for a Large mining client and as everything was hot swap and I was just adding another analogue card I said it could be done during office hours.
As you can probably guess.. I plugged the card into the mux and the green lights instantly turned red. (This mux does compressed voice and data over atm for all offices). And yes their sysadmin guy was standing right beside me when it happened.
As luck would have it the bloody system wouln't come back even after cold boots!! As the mobile phones ringing all around me became deafening - I decided to remove the analogue card and try one last reset before I went mad. Luckily this did the trick! Something to do with an old software version not taking x number of analogue cards!
Bugz. |
|
|
|
|