From guy at extragalactic.net Thu Jan 14 15:24:08 2010 From: guy at extragalactic.net (Guy B. Purcell) Date: Thu, 14 Jan 2010 15:24:08 -0800 Subject: How to report a bot-net Message-ID: Hi All, Been too long since we had real posts here & this is an interesting problem, so I'm hoping for some good discussion, if not resolution. I have an SSH portal at home that gets a remarkable amount of traffic--almost all of which is brute-force attacks either from individuals or from bot-net(s). Here's an example individual (I'm showing logged timestamp, source IP address, and username attempted): Jan 7 12:30:01 89.140.94.122 ke Jan 7 12:30:03 89.140.94.122 ke Jan 7 12:30:05 89.140.94.122 ke Jan 7 12:30:07 89.140.94.122 ke Jan 7 12:30:10 89.140.94.122 ke Jan 7 12:30:12 89.140.94.122 ke Jan 7 12:30:14 89.140.94.122 ke Jan 7 12:30:16 89.140.94.122 kg Jan 7 12:30:18 89.140.94.122 kg Jan 7 12:30:20 89.140.94.122 kg Jan 7 12:30:23 89.140.94.122 kg Jan 7 12:30:25 89.140.94.122 kg Jan 7 12:30:27 89.140.94.122 kg Jan 7 12:30:29 89.140.94.122 kg Jan 7 12:30:31 89.140.94.122 kg Jan 7 12:30:33 89.140.94.122 kg Jan 7 12:30:36 89.140.94.122 kg Jan 7 12:30:38 89.140.94.122 kg Jan 7 12:30:40 89.140.94.122 kg Jan 7 12:30:42 89.140.94.122 kh Jan 7 12:30:45 89.140.94.122 kh Jan 7 12:30:47 89.140.94.122 kh Jan 7 12:30:50 89.140.94.122 kh Jan 7 12:30:53 89.140.94.122 kh Jan 7 12:30:55 89.140.94.122 kh Here's an example bot-net: Jan 7 13:43:01 211.115.234.143 petang Jan 7 13:48:29 84.246.69.21 peter Jan 7 13:53:33 90.182.211.25 petkuo Jan 7 13:58:45 211.115.234.143 peyjiumc Jan 7 14:04:06 217.8.61.146 pgsql Jan 7 14:09:32 88.49.16.234 phantom Jan 7 14:14:42 124.31.204.53 phchang Jan 7 14:19:57 62.219.4.105 phire Jan 7 14:25:39 84.201.180.130 phoebe Jan 7 14:30:31 220.162.241.11 photo Jan 7 14:41:10 121.52.215.180 phws Jan 7 14:46:26 90.182.211.25 phy Jan 7 14:51:39 125.5.157.51 phyllis Jan 7 14:56:55 80.153.59.28 picture Jan 7 15:02:04 212.243.41.9 pig Jan 7 15:07:29 148.233.140.193 pigeon Jan 7 15:12:39 220.162.241.11 pigg This host is running ipfw, and I could write a script that couples that with the log to block the unwanted attention (easy for the individual attacks, non-trivial for the bot-nets--at least AFAICT), but that just kicks the can down the road to someone else's IP address. I could use the log info to attempt to notify the various ISPs' of the abuse, but they would just see it as a bunch of individual complaints. What I think would be better would be to use blocking & notification tactics (as automated as possible; I believe David kicked off a small discussion thread here about that) for the individual attackers, but to be able to treat the whole bot-net as a single-source attack & have it shut down. But how to go about that? The Internet is a global confederation with no real central authority over such a broad attack base (I have IP addresses from China, Korea, Australia, Isreal, Brazil, Italy, & the US--to name just the handful I happened to look up). Who would you turn to? If there's no authority with the ability or responsibility to shut bot-nets down, what do you think could be done to improve matters? -Guy From louisk at cryptomonkeys.org Thu Jan 14 16:13:18 2010 From: louisk at cryptomonkeys.org (Louis Kowolowski) Date: Thu, 14 Jan 2010 16:13:18 -0800 Subject: How to report a bot-net In-Reply-To: References: Message-ID: <7D068711-C3DA-4794-9955-46B942E78B78@cryptomonkeys.org> On Jan 14, 2010, at 3:24 PM, Guy B. Purcell wrote: > Hi All, > > Been too long since we had real posts here & this is an interesting problem, so I'm hoping for some good discussion, if not resolution. > > I have an SSH portal at home that gets a remarkable amount of traffic--almost all of which is brute-force attacks either from individuals or from bot-net(s). Here's an example individual (I'm showing logged timestamp, source IP address, and username attempted): > > This host is running ipfw, and I could write a script that couples that with the log to block the unwanted attention (easy for the individual attacks, non-trivial for the bot-nets--at least AFAICT), but that just kicks the can down the road to someone else's IP address. I could use the log info to attempt to notify the various ISPs' of the abuse, but they would just see it as a bunch of individual complaints. What I think would be better would be to use blocking & notification tactics (as automated as possible; I believe David kicked off a small discussion thread here about that) for the individual attackers, but to be able to treat the whole bot-net as a single-source attack & have it shut down. > > But how to go about that? The Internet is a global confederation with no real central authority over such a broad attack base (I have IP addresses from China, Korea, Australia, Isreal, Brazil, Italy, & the US--to name just the handful I happened to look up). Who would you turn to? If there's no authority with the ability or responsibility to shut bot-nets down, what do you think could be done to improve matters? > I get these all the time. I mostly don't pay attention to it. I have SSH configured so that passwords don't provide access, only ssh public keys work. If I had more time, I would probably write a script that would add hosts to a block table (ipfw), and once a large number of IPs in a netblock were there, I would add the netblock. Of course, the thing to weigh here is whether your blocking IPs that will later turn around and be legit traffic. For your home system, this probably isn't an issue, although if you travel and need (remote) access to this system, you may find yourself blocked by your own cleverness. -- Louis Kowolowski louisk at cryptomonkeys.org Cryptomonkeys: http://www.cryptomonkeys.org/~louisk Making life more interesting for people since 1977 From david at catwhisker.org Sat Jan 16 05:37:52 2010 From: david at catwhisker.org (David Wolfskill) Date: Sat, 16 Jan 2010 05:37:52 -0800 Subject: How to report a bot-net In-Reply-To: References: Message-ID: <20100116133752.GQ86359@bunrab.catwhisker.org> On Thu, Jan 14, 2010 at 03:24:08PM -0800, Guy B. Purcell wrote: > Hi All, > > Been too long since we had real posts here & this is an interesting problem, so I'm hoping for some good discussion, if not resolution. Well, in fairness, I brought up a closely-related topic not too long ago.... :-} > I have an SSH portal at home that gets a remarkable amount of traffic--almost all of which is brute-force attacks either from individuals or from bot-net(s). As do I, though usually I wouldn't call the amount of activity all that remarkable -- not until the number of probes/day gets measured in the thousands, I suppose. :-} > Here's an example individual (I'm showing logged timestamp, source IP address, and username attempted): > Jan 7 12:30:01 89.140.94.122 ke > Jan 7 12:30:03 89.140.94.122 ke > Jan 7 12:30:05 89.140.94.122 ke > .... > Here's an example bot-net: > > Jan 7 13:43:01 211.115.234.143 petang > Jan 7 13:48:29 84.246.69.21 peter > Jan 7 13:53:33 90.182.211.25 petkuo > ... > > This host is running ipfw, and I could write a script that couples > that with the log to block the unwanted attention (easy for the > individual attacks, non-trivial for the bot-nets--at least AFAICT), > but that just kicks the can down the road to someone else's IP > address. As Louis (and others, in other lists) have pointed out, this is generally a fairly effective way to commit a self-inflicted DoS attack on yourself. There may be circumstances where the risk is justified; you would likely know those circumstances (for yourself) better than most. > I could use the log info to attempt to notify the various > ISPs' of the abuse, but they would just see it as a bunch of > individual complaints. Perhaps. I actually do this (generally); the boiler-plate message I send makes it clear that my purpose is to provide a "heads up" that there is something going on on the network in question that bears investigation, and that the information I'm providing may be helpful in providing direction to that investigation. Fundamentally, I try to treat the folks running the other nets as I would like them to treat me were our roles reversed. That said, once those allegedly "responsible" for a given netblock have repeatedly demonstrated that they are unwilling or unable to address such issues, I give up. (A trivial case is when my attempts to notify them get rejected.) And in this case, that does not mean that I merely stop sending the notifications to them. Rather, it means that I add the netblock in question to a particular IPFW table for which I block all incoming and outgoing traffic of any kind. And If I find other netblocks that have equivalent contact information, I add those, as well. That is how I came to add all CHINANET*, CNCGROUP*, and UNICOM* netblocks to that table, for example. Note: I do this for my "firewall" (packet filter, actually -- though it does a bit more than that) box at home; I also do it for my laptop, as it is also exposed to the Outside World -- recall that when we met at Apple, Apple's DHCP server handed out routable 17/8 addresses. > What I think would be better would be to > use blocking & notification tactics (as automated as possible; I > believe David kicked off a small discussion thread here about that) :-} > for the individual attackers, but to be able to treat the whole > bot-net as a single-source attack & have it shut down. > But how to go about that? The Internet is a global confederation > with no real central authority over such a broad attack base (I > have IP addresses from China, Korea, Australia, Isreal, Brazil, > Italy, & the US--to name just the handful I happened to look up). > Who would you turn to? If there's no authority with the ability > or responsibility to shut bot-nets down, what do you think could > be done to improve matters? > ... I don't really attempt to address the issue in that way, as I don't know of a way to trace back where the "control point" is. Rather, I concentrate on what I can verify: I have pretty good evidence that the source IP addresses I'm seeing actually correspond to the traffic in question and try to help the folks responsible for the resources in question be aware that there may be a problem -- but ultimately, I will take whatever steps I deem necessary to protect my network, up to the point of effectively dyking the offending netblocks out of the part of the Internet with which I can directly interact. Mind, I probably wouldn't have quite that much flexibility were I performing this activity on behalf of someone else -- especially a large global corporation, for example. For amusement, Here are some log entries I see this morning. Note that in addition to the log entries generated by sshd(8), I also have IPFW configured to log all incoming session-establishment requests for SSH, so: Jan 15 07:42:51 bunrab sshd[86724]: Did not receive identification string from 59.76.81.109 Jan 15 07:48:41 bunrab sshd[86734]: Illegal user marine from 59.76.81.109 Jan 15 07:48:41 bunrab sshd[86736]: Illegal user cadi from 59.76.81.109 Jan 15 07:49:38 bunrab sshd[86732]: fatal: Timeout before authentication for 59.76.81.109 Jan 15 07:42:51 janus kernel: ipfw: 10000 Accept TCP 59.76.81.109:36589 172.16.8.11:22 out via dc0 Jan 15 07:47:38 janus kernel: ipfw: 10000 Accept TCP 59.76.81.109:51504 172.16.8.11:22 out via dc0 Jan 15 07:47:41 janus kernel: ipfw: 10000 Accept TCP 59.76.81.109:51719 172.16.8.11:22 out via dc0 Jan 15 07:47:45 janus kernel: ipfw: 10000 Accept TCP 59.76.81.109:51942 172.16.8.11:22 out via dc0 Jan 15 07:47:51 janus kernel: ipfw: 10000 Accept TCP 59.76.81.109:52318 172.16.8.11:22 out via dc0 Peace, david -- David H. Wolfskill david at catwhisker.org Depriving a girl or boy of an opportunity for education is evil. See http://www.catwhisker.org/~david/publickey.gpg for my public key. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available URL: From guy at extragalactic.net Tue Jan 19 00:19:49 2010 From: guy at extragalactic.net (Guy B. Purcell) Date: Tue, 19 Jan 2010 00:19:49 -0800 Subject: Meeting reminder Message-ID: Hi All, Please remember that this Thursday, Jan. 21, is our monthly BayLISA meeting. The talk this month will be a two-parter by Tom Limoncelli--his "Top 5 Tips: Time Management for System Administrators", and "Ganeti: Open Source Virtual Cluster Manager". Meeting starts at our usual 19:30 time, with Tom speaking after our announcement session; location is LinkedIn. For more details, please see the website . And please try & remember to send a quick "I'll be there, and bringing N others," or whatever, email to rsvp at baylisa.org so I can gauge attendance. Thanks! -Guy From guy at extragalactic.net Tue Jan 19 14:08:37 2010 From: guy at extragalactic.net (Guy B. Purcell) Date: Tue, 19 Jan 2010 14:08:37 -0800 Subject: How to report a bot-net In-Reply-To: <20100116133752.GQ86359@bunrab.catwhisker.org> References: <20100116133752.GQ86359@bunrab.catwhisker.org> Message-ID: <9CC95AA4-1433-4757-A6C4-3856F9E104A4@extragalactic.net> On Jan 16, 2010, at 5:37 AM, David Wolfskill wrote: > On Thu, Jan 14, 2010 at 03:24:08PM -0800, Guy B. Purcell wrote: >> I have an SSH portal at home that gets a remarkable amount of traffic--almost all of which is brute-force attacks either from individuals or from bot-net(s). > > As do I, though usually I wouldn't call the amount of activity all that > remarkable -- not until the number of probes/day gets measured in the > thousands, I suppose. :-} It varies from a "norm" of around 350/day to almost an order of magnitude more than that, which puts me squarely into the realm of thousands per day on bad days--bad enough that I stop sshd on the portal when it happens & I'm at home trying to work over the 'net :^( >> This host is running ipfw, and I could write a script that couples >> that with the log to block the unwanted attention (easy for the >> individual attacks, non-trivial for the bot-nets--at least AFAICT), >> but that just kicks the can down the road to someone else's IP >> address. > > As Louis (and others, in other lists) have pointed out, this is > generally a fairly effective way to commit a self-inflicted DoS attack > on yourself. There may be circumstances where the risk is justified; > you would likely know those circumstances (for yourself) better than > most. Yep--exactly what I don't want to happen--and I'm sure to forget about this exactly when I'm heading off on an excursion to someplace unusual, from which it'd likely be critical if I wanted to ssh back home (and I have in-laws in China, which has netblocks I'd be likely to blacklist). >> I could use the log info to attempt to notify the various >> ISPs' of the abuse, but they would just see it as a bunch of >> individual complaints. > > Perhaps. > > I actually do this (generally); the boiler-plate message I send makes it > clear that my purpose is to provide a "heads up" that there is something > going on on the network in question that bears investigation, and that > the information I'm providing may be helpful in providing direction to > that investigation. The problem with this is that each incident is seen individually by an ISP: no matter how politely you say it, you're still saying, "hey--I'm getting brute-force-attacked by *this* IP address that's under your control; please make it stop". They take some action (assuming they care to; I'll give 'em the benefit of the doubt that they do), eventually taking that circuit/IP offline--but the user eventually make's enough noise (or maybe just reboots their modem) & returns online--likely with a spiffy new IP address, that eventually starts knocking on my front door again. Round & round it goes. Others have suggested just use an alternate port for SSH. Definitely doable & fairly trivial, but would require me to modify my behavior (i.e. remember to use "-p xxx" with every connection to my home net), and is just hiding, not a solution (as Jesse pointed out). Eventually, bot-nets will become large enough and enough people will have hidden their networks in this way that it becomes worth the evildoers' whiles and within their power to locate the new port. >> What I think would be better would be to...treat the whole >> bot-net as a single-source attack & have it shut down. > >> But how to go about that? The Internet is a global confederation >> with no real central authority over such a broad attack base (I >> have IP addresses from China, Korea, Australia, Isreal, Brazil, >> Italy, & the US--to name just the handful I happened to look up). >> Who would you turn to? If there's no authority with the ability >> or responsibility to shut bot-nets down, what do you think could >> be done to improve matters? >> ... > > I don't really attempt to address the issue in that way, as I don't know > of a way to trace back where the "control point" is. Exactly--there's no consortium/group/whatever that has the ability to ask the various country ISPs to investigate what IP addresses a particular address is making connections to--at least not one I know of. That's the only way to locate the master(s)--find the set of IP addresses the collection as a whole is connecting to, and that requires global coordination of some sort. The coordinator doesn't need access to ISP systems--just the authority to ask 'em for a list of connections (probably don't need actual packet snoop data--just connections) with the expectation of getting a truthful answer, and the trust of local law enforcement to deal with the problem when asked by the group (which should have sufficient evidence by then). It'd be like a kind of global Internet grand jury :^) Wouldn't be easy to establish, but would be doable, IMHO. -Guy From rob.markovic at gmail.com Tue Jan 19 17:08:37 2010 From: rob.markovic at gmail.com (Rob Markovic) Date: Tue, 19 Jan 2010 17:08:37 -0800 Subject: How to report a bot-net In-Reply-To: <9CC95AA4-1433-4757-A6C4-3856F9E104A4@extragalactic.net> References: <20100116133752.GQ86359@bunrab.catwhisker.org> <9CC95AA4-1433-4757-A6C4-3856F9E104A4@extragalactic.net> Message-ID: <97a9d8c81001191708g61dbe2bi2087df84cb0545e1@mail.gmail.com> I believe we (Guy and I) discussed contacting the local FBI office and seeing what they can do. Perhaps you'll be put in a new beta program for a cyber crime task force and given ways to report such anomalous activity or to feed some new data to a super investigative program.. :) -- Rob On Tue, Jan 19, 2010 at 2:08 PM, Guy B. Purcell wrote: > > On Jan 16, 2010, at 5:37 AM, David Wolfskill wrote: > >> On Thu, Jan 14, 2010 at 03:24:08PM -0800, Guy B. Purcell wrote: >>> I have an SSH portal at home that gets a remarkable amount of traffic--almost all of which is brute-force attacks either from individuals or from bot-net(s). >> >> As do I, though usually I wouldn't call the amount of activity all that >> remarkable -- not until the number of probes/day gets measured in the >> thousands, I suppose. :-} > > It varies from a "norm" of around 350/day to almost an order of magnitude more than that, which puts me squarely into the realm of thousands per day on bad days--bad enough that I stop sshd on the portal when it happens & I'm at home trying to work over the 'net :^( > >>> This host is running ipfw, and I could write a script that couples >>> that with the log to block the unwanted attention (easy for the >>> individual attacks, non-trivial for the bot-nets--at least AFAICT), >>> but that just kicks the can down the road to someone else's IP >>> address. >> >> As Louis (and others, in other lists) have pointed out, this is >> generally a fairly effective way to commit a self-inflicted DoS attack >> on yourself. ?There may be circumstances where the risk is justified; >> you would likely know those circumstances (for yourself) better than >> most. > > Yep--exactly what I don't want to happen--and I'm sure to forget about this exactly when I'm heading off on an excursion to someplace unusual, from which it'd likely be critical if I wanted to ssh back home (and I have in-laws in China, which has netblocks I'd be likely to blacklist). > >>> I could use the log info to attempt to notify the various >>> ISPs' of the abuse, but they would just see it as a bunch of >>> individual complaints. >> >> Perhaps. >> >> I actually do this (generally); the boiler-plate message I send makes it >> clear that my purpose is to provide a "heads up" that there is something >> going on on the network in question that bears investigation, and that >> the information I'm providing may be helpful in providing direction to >> that investigation. > > The problem with this is that each incident is seen individually by an ISP: ?no matter how politely you say it, you're still saying, "hey--I'm getting brute-force-attacked by *this* IP address that's under your control; please make it stop". ?They take some action (assuming they care to; I'll give 'em the benefit of the doubt that they do), eventually taking that circuit/IP offline--but the user eventually make's enough noise (or maybe just reboots their modem) & returns online--likely with a spiffy new IP address, that eventually starts knocking on my front door again. ?Round & round it goes. > > Others have suggested just use an alternate port for SSH. ?Definitely doable & fairly trivial, but would require me to modify my behavior (i.e. remember to use "-p xxx" with every connection to my home net), and is just hiding, not a solution (as Jesse pointed out). ?Eventually, bot-nets will become large enough and enough people will have hidden their networks in this way that it becomes worth the evildoers' whiles and within their power to locate the new port. > >>> What I think would be better would be to...treat the whole >>> bot-net as a single-source attack & have it shut down. >> >>> But how to go about that? ?The Internet is a global confederation >>> with no real central authority over such a broad attack base (I >>> have IP addresses from China, Korea, Australia, Isreal, Brazil, >>> Italy, & the US--to name just the handful I happened to look up). >>> Who would you turn to? ?If there's no authority with the ability >>> or responsibility to shut bot-nets down, what do you think could >>> be done to improve matters? >>> ... >> >> I don't really attempt to address the issue in that way, as I don't know >> of a way to trace back where the "control point" is. > > Exactly--there's no consortium/group/whatever that has the ability to ask the various country ISPs to investigate what IP addresses a particular address is making connections to--at least not one I know of. ?That's the only way to locate the master(s)--find the set of IP addresses the collection as a whole is connecting to, and that requires global coordination of some sort. ?The coordinator doesn't need access to ISP systems--just the authority to ask 'em for a list of connections (probably don't need actual packet snoop data--just connections) with the expectation of getting a truthful answer, and the trust of local law enforcement to deal with the problem when asked by the group (which should have sufficient evidence by then). > > It'd be like a kind of global Internet grand jury :^) ?Wouldn't be easy to establish, but would be doable, IMHO. > > -Guy > > From lgj at usenix.org Fri Jan 29 15:04:48 2010 From: lgj at usenix.org (Lionel Garth Jones) Date: Fri, 29 Jan 2010 15:04:48 -0800 Subject: USENIX LISA '10 Call for Participation Now Available Message-ID: <14D9CEB7-5A46-45D4-9972-962E53189198@usenix.org> On behalf of the Program Committe, I would like to invite you to contribute refereed papers, experience reports, and proposals for invited talks and workshops to the 24th Large Installation System Administration Conference (LISA '10). We're also looking for any ideas you have for Guru Is In sessions, Work-in-Progress Reports, posters, and training sessions. The theme for LISA '10 is "Share your experiences, both real-world and in research." The Call for Participation with submission guidelines and sample topics can be found on the USENIX Web site at: http://www.usenix.org/lisa10/cfpa The annual LISA conference is the meeting place of choice for system and network administrators and engineers. The conference serves as a venue for a lively, diverse, and rich mix of technologists of all specialties and levels of expertise. LISA is the place to exchange ideas, sharpen old and new skills, learn new techniques, debate current and controversial issues, and meet industry gurus, colleagues, and friends. The conference's diverse group of participants are matched by an equally broad spectrum of activities: * NEW! LISA '10 will now include practice and experience reports. These reports will offer valuable insight into completed projects in system administration. * A training program offering state-of-the-art tutorials from top experts in their fields. Topics cover every level from introductory to highly advanced. * Refereed papers explore techniques, tools, theory, and case histories that extend our understanding of system and network administration. * Workshops, invited talks, and panels discuss important and timely topics in depth and typically include lively and/or controversial debates and audience interaction. * Work-in-Progress Reports (WiPs) and poster sessions provide brief looks ahead to next year's innovations. GET INVOLVED! * Share your results with a practice and experience report. * Submit a paper. * Propose a tutorial topic. * Suggest an invited talk or panel discussion. * Share your experience by leading a Guru Is In session. * Create and lead a workshop. * Present a Work-in-Progress Report (WiP) or submit a poster. * Organize a Birds-of-a-Feather (BoF) session. * Email an idea to the program chair: lisa10ideas at usenix.org LISA '10 takes place November 7-12, 2010, San Jose, CA. We look forward to hearing from you! On behalf of the LISA '10 Organizers, Rudi van Drunen, Competa IT and Xlexit Technology, The Netherlands LISA '10 Program Chair lisa10chair at usenix.org ------------------------------------------------------------ IMPORTANT DATES Extended abstracts, experience reports, and proposals for invited talks, workshops, and tutorials due: May 17, 2010 Notification to all submitters: July 1, 2010 Final papers and reports due: August 24, 2010 Poster proposals due: September 17, 2010 Notification to poster presenters: September 24, 2010 Submission guidelines and more information can be found at http://www.usenix.org/lisa10/cfpa