Author: Steve Sinchak | 9/25/2003 10:47:27 AM
Starting mid-day September 24, we noticed a increasing spike in our traffic. By the end of the day our server was at its maximum capacity with over 2000 people connected at once. Our IBM eServer could not take our load and caused the website to slow to a crawl.
Working late into the night we discovered that a malicious program has been circulating the web modifying Windows users “hosts” files. For those of you that are not familiar with the Windows “hosts” file, the “hosts” file is place that Windows looks first when trying to find out where a domain name points. The malicious program added entries for many popular search sites that would redirect their search request to http://126.96.36.199. At that site, an unknown person has placed a link to the TweakXP.com Computer Support Forum as a place to get help to fix their system.
As the number of people infected by the malicious program increased, our webserver started having difficulties with the extremely high demand placed on it. To solve this problem, we replaced the file that http://188.8.131.52 was linking to on our site with a small low-res page that would not take up as much CPU usage and require as much bandwidth to upload. That has significantly helped the performance of our site and now it is accessible but still not at peak performance. However, this has left parts of our forum software crippled as we had to replace a key file in it operation. Currently, we are going through the thousands of lines of code to replace any references to old page with the new renamed page.
For those of you that are infected, please click here for instructions on how to fix your computer or for more information on how this happened.
We are sorry for the poor performance of our webserver. We are working hard trying to deal with the spike in traffic.
We answer calls for customers who were shut off for having trojans, worms and viruses on their computers. Often they are sending SPAM unwittingly. Norton, McAfee and others are supposed to pick these bugs up and remove them. But you have to actually run the program first....
Clueless Customer: OK, can you turn my service back on now?
Haywood: Have you found the worm on the computer?
Clueless Customer: Was I supposed to? She just said I have to get Norton. I did that.
Haywood: Did you install it.
Clueless Customer: Yes.
Haywood: Did you run it?
Clueless Customer: No. Am I supposed to?
Haywood: Yes, how else will it find the bug?
Clueless Customer: Oh, OK.
Haywood: Call me back when it has found and removed something called.....
Customer gets transfered to tier 2 with no warning apparently because the first thing out of this one's mouth:
Clueless Customer: OK I found "Run"
Haywood: Congratulations. Why were you looking for it?
Clueless Customer: That is what he wanted me to find?
Clueless Customer: The last guy I spoke with.
Haywood: OK, I am new to this conversation. What were you guys trying to do?
Clueless Customer: Well, I opened up all my ports in Norton, but I still can't get on the Internet.
Haywood: Well, maybe you should take that up with Norton, since this is not an ISP problem.
Clueless Customer: He said you could help me.
Haywood: Well, we can't help you with Norton.
Clueless Customer: But this isn't a problem with Norton.
Haywood: Yes it is.
Clueless Customer: It's worked fine up till now.
This is a conversation that I have on a dailytwice daily thrice daily basis.... It's getting really old... Perhaps I should figure out some way to automate....
I was just browsing thru the Tracker logs for this little site of mine, and looking thru the referers, it seems I showed up in the Error page of this Blog Indexing site. Guess the blog was down for a bit at that time... Hmmmm, I'll have to see about getting some credit for that downtime.... :P
I searched thru the list of rankings to find myself (I know, I shoulda just gone for a long vacation or taken the summer off, but that is not what I was looking for :P ), but there was no real way to find my site, other than by going next, next, next, next, next..... I knew I wouldn't be anywhere near the same page as Movabletype.org, Slashdot, Where is Raed, or any of the other Top 100 (or even top 1000), but I know I must be in there somewhere.... So I left the site after submitting a form to include my site (if it isn't already in there - or maybe is blacklisted...) and came here to write about them (and provide a link) so that they will look at my link quicker (nudge, nudge, wink, wink, knowhaddamean, knowhaddamean).
I have no idea how many thousands (or millions?) of blogs they index. Anyone wanna venture a guess? IMHO, they should add a "find your site's rank" so I can instantly find how I rank. I am always amazed when I look thru the logs and see where people are coming from and how they find me. If you wish to add a link to me, speak up and I will make some sort of code type thingee you can paste into your webpage....
Thanks for reading and feel free to ask me questions in the forums I just set up, or visit the chatroom which is always open..... Oh and the Bank, Shop and Lottery are all running now as well. I will be adding more features soon, and if anyone has any ideas for shop items, please speak up!
Clueless Customer: I cant get on the Internet because I have no address line
Me: Well, you can do File>Open and then type in the address there.
But some how I knew there was more to the problem than this...
Me: OK, I can help you with that
After verifying his information he goes more into what the problem really is
Clueless Customer: My IP address isn't showing up
Me (somewhat confused): HUH????
He describes all these steps he takes (mostly repairing the connection in the network connections control panel) and then says
Clueless Customer: It says I don't have an ISP number
Me: ISP number????
Customer was rude to me because we are having an outage on the east coast right now... Due to a HURRICANE!
Mr. Agro Customer: This is not acceptable! I am running a business here!
Me: And your point is? We run a RESIDENTIAL service. If you need that level of support, we have a service for you. And this ain't it! It is called < ISP .net > Business Services.
So we went back and forth over why I was not able to schedule a tech to come out immediately (maybe due to the hurricane?? and the fact that they don't due Sunday appointments in your system...) And eventually he demanded to speak with a supervisor. Of course, he initially thought that I was one... Someone at Tier 1 told him that I was.... sheesh.... Well at least he will believe the sup when he DENIES him a sooner appointment!
Sometimes I wish I could be the BOFH.... Simon woulda asked him for his username...
BOFH: You won't be needing that tech to come out as I have done as you requested and canceled your service with us.
Mr. Agro Customer: wha... wha... what!!!!!!!!!!!!!!???????
BOFH: You said you were going to cancel your service if I couldn't schedule a tech to come out sooner. So I canceled it. You want it reconnected?
Mr. Agro Customer: Please.
BOFH: OK, I will apply the $50 reconnect charge and then reactivate the service. Are you still not interested in my first available appointment?
Mr. Agro Customer: (sheepishly) Uhm, no, I will take your first available...
BOFH: Oh, well, it looks like someone else grabbed that appointment and I don't have anything until next week.
Clueless customer: Everything has been working fine for the last year. Now none of my computers can get online, so I don't think there is a problem with the router. We had 2 computers getting on wirelessly, and it worked fine until we hooked this computer up with the wire tonite.
How can they not see that there is a commonality there??? I just don't get it.
I am going on 28 32 hours straight with no sleep... This schedule (working nites) has gotten me all messed up. I came home from work last nite (this morning) and stayed up on the computer workingonsomewebsites.
So today, around hour 23 or so, I was reading a book to my 3 year old Bella and 1 year old and I feel myself drifting off to sleep, tho the words are still dripping out of my mouth. At the present time, I cannot even recall which book it was, but it was like a Doctor Suess book or something.
So anyway, I remember reading a sentence that was something like "and then the dog went down the street" or something equally inane... And I coulda swore I heard myself say, almost in a dream state, "and then you couldn't renew the IP address?" and then I snapped out of it like a flash. Bella seemed a little confused, but I read on. I did it again, a little later, and I don't recall exactly the phrase I used but I again slipped into a very surreal work/dream state for a brief second. It was just toooo weird...
So here I am now almost in delirium, pushing for 36 hours straight with NO SLEEP!!! I am having trouble remembering what it is that I was going to have this customer do...
Confused Customer: OK, now what?
Clueless Haywood: What was it that I asked you to look for?
Confused Customer: You said to click on the plus next to HKEY_Local_Machine.
OK, next we are looking for, uhm, System. Click on zzzzzzzzzzzzzzzzzzzzzzzz....
Haywood: Go ahead and type ping then Space
Clueless customer: Wait, where is the space?
Haywood: It's the big one across the bottom of the keyboard. The space bar.
Clueless customer: Wait, I put in a dash, is that what you wanted?
Haywood: No, we need a space, in between the words.
Clueless customer: oh, that. At the bottom of the keyboard. I am sorry, I am not thinking right today.
Haywood: That's ok, this is one of those high pressure sitauations.
And I did a winsock restore procedure with her. This is the kind of lady who has to read everything to you before you can go on. She is so busy reading the screen she cannot pay attention to what I am telling her to do... She is like one of those customers who have the philosophy of "if I don't hear anything from him for, oh, about 1 and 1/2 seconds, I will start yacking at him with what I am seeing." She then proceedes to tell me everything that she sees on the screen, completely unaware that I have seen that exact screen like a million times in the past....
Also not realizing that if she would just stfu and tell me "It's done" or "Now what?" when it's done instead, we would get done a lot quicker. She is also one of those "techno-parlance word estimaters". As in "Microsoft Family LOGAN" which she repeated several times...
Haywood: OK, try typing in E-Colon- Backslash - W-i-n-9-8
Clueless customer: i type it where the D colon backslash stuff is?
Haywood: Yes, type it there
Clueless customer: Do I need to erase this stuff in here.
Haywood: Yes! grrrrrrrr
After doing the winsock fix, upon reboot she gets a message that TCP/IP is not installed and of course she has to read it to me. This is a good thing... Tells me the call is almost over as she is gonna have to fix this on her own....
Clueless customer: "It says I should click here to go to Sim-en-tic. Should I?
Haywood: Symantec is the company that makes Norton. You will not be able to connect until we get this winsock problem fixed.
later after completing the winsock restore procedure and trying to get online:
Haywood: What is the address in the address line
Clueless customer: It says h-t-t-p-colon-slash-slash hot bar period com.
Haywood: I think your browser has been hijacked.
Clueless customer: What does that mean?
Haywood: That means that some 3rd party program was installed that is affecting your ability to connect to the internet. Possibly corrupting files along the way.
Haywood: See if you can go to google.
Clueless customer: How do I do that.
Haywood: Type it in.
Clueless customer: Where?
Haywood: In the address line that you are looking at right now.
Clueless customer: Ok, it says that the search page is unavailable, please try again.
Haywood: OK, you are going to need to call Dell, Gateway or HP, because this is a computer problem and not an ISP problem.
(btw, that is my newest favorite expression for unsupported issues. Not an ISP issue. I can't stress that enough... It really drives the point home...
So basically it took us 60 minutes for me to fail to repair what normally takes me 15 minutes tops (!!!) to fix (with someone fairly competent : knows the letter of their CD drive, doesn't have to read me everything in the screen, etc...) Live and learn i guess....
At one time I leaned towards yes, then I decided, no, with a clause, but am now leaning firmly towards YES! Especially with Blaster and Welchia and Nachi. People UPDATE!!!! Patch your Operating System. If you had a car, and they recalled the tires on the thing because they were faulty (never happens....) you would be damn sure to return your tires to the dealer and get the new ones. Right? I mean, come on....
Clueless Customer: I have been going at this for 3 hours now and it still isn't working.
Well, maybe if you could follow a simple instruction like unplug your modem, then your time on the phone might decrease... I swear I asked the guy to electrically unplug the cable modem. He says "OK" Are all the lights off on your modem? Yes. Hmm, that is interesting. I am able to send a signal to your modem and it looks like its online right now. Yes, it's off. Really? Then in the background I hear him whispering to someone about needing to bypass the router and reboot the modem. Some people will never get it...
Installing drivers by phone is always fun.... Especially with Windows 98. Consider yourself lucky if they have the CD, or even the one we provide them (our CD actually has the CAB files that Windows 95/98 often asks for when installing things... The prompt reads "Please insert your CD labeled Windows 98 Second Edition")
Haywood: And what does it say for the path below?
Clueless Customer: D:\Install\WinCabs\Win98
Haywood: Let's go ahead and change that to D:\Win98
Clueless Customer: but it says D:\Install\WinCabs\Win98.
Duh! You just read that to me. I know what it says Haywood: Then you need to change it.
While talking with this customer I had to blog about him...
Clueless Customer: You sure are typing alot.
Haywood: I have a lot of, uhm, er, uh, notes to write about this ticket...
General Motors doesn't have a "help line" for people who don't know how to drive, because people don't buy cars like they buy computers -- but imagine if they did . . .
Helpline: General Motors HelpLine, how can I help you?
Customer: I got in my car and closed the door, and nothing happened!
Helpline: Did you put the key in the ignition slot and turn it?
Customer: What's an ignition?
Helpline: It's a starter motor that draws current from your battery and turns over the engine.
Customer: Ignition? Motor? Battery? Engine? How come I have to know all of these technical terms just to use my car?
Turns out in my case it was a corrupt folder. Had to create a new store folder (or a new identity if you don't want to mess with that) and then import my messages into there... Am all good now... But this is the error I was getting:
When a customer is under pressure to report back to me all that he or she is showing and some of the words are "big words", they often guess what they are, instead of reading them. A classic example:
Me: What do you see listed there?
Clueless Customer: Client for Microsoft Networks, Dial-up Adapter, Microsoft Vital Private Networking Adapter, blah, blah
That shoulda been Microsoft Virtual Private Networking Adapter. She clearly was nervous, and I think the technology frightens them.... I see (er... hear) that all the time... Ah well, she got most of the letters in the word...
A time-out occurred while communicating with the server 0x800ccc19
This problem may occur when the following conditions are true:
You have Norton AntiVirus 2003 or Norton AntiVirus 2002 installed.
You have Scan incoming Email and Scan outgoing Email turned on in Norton AntiVirus.
This lady wanted to know something, and I think she is one of those people who knows just enough to be dangerous (to herself).
Clueless Customer: "What does Use Passive FTP for compatability with some firewalls and DSL modems? I have a firewall, do I need to have that checked or unchecked?"
Me: "Are you going to be doing FTP transfers?"
Clueless Customer: "What's that?"
Me: "It lets you download and upload files to and from a server."
Clueless Customer: "Oh, I won't be doing that. "
Me: "Then I would not worry about."
Customer is having trouble getting to some websites, and he happens to be a network engineer, or think he is one... This is an example of someone who thinks he can troubleshoot your network for you, and is the type of customer who gets on my nerves a little.
He can get to some websites, but not others, including our status page. He tells me, "There's a router out there eating packets." (chomp, chomp)
Then with an air of smugness, he says "I know something about the Internet." As if to say that I, a stupid phone jockey, am clueless about the Internet. Never mind my 2 and 1/2 years at the help desk, or my CCNA and MCP ceritifications. Those are meaningless to him. I didn't even bother mentioning this. No need to wave my certifications around like I am showing off my dick.
Then, in an attempt to troubleshoot the network for us, while examining his traceroutes, he says "I am not seeing anything in common, or I would have told you."