Heute schrieb Paul Frields auf der Fedora-announce-list:
https://www.redhat.com/archives/fedora-announce-list/2009-March/msg00010...
Am 12. August 2008 schlug ein Cron-Job auf einem Fedora-Server fehl, was die Administratoren alarmierte.
Beim durchschauen der Logdateien fanden die Admins dann Hinweise dass die Pakete auf dem Server veraendert wurden.
Sobald man sich eines Einbruchs sicher war wurde die Community informiert:
https://www.redhat.com/archives/fedora-announce-list/2008-August/msg0000...
Grund fuer den Einbruch war kein Software-Problem, der Einbrecher hatte den privaten SSH-Keys eines Admins.
Das Problem bei der Sache war, dass der SSH-Key nicht mit einem Passwort geschuetzt war.
Da der Signierschluessel fuer die Fedora-Pakete sich auf diesem Server befand, schlossen die Admins nicht aus, dass der Angreifer diesen Key nun hatte.
Der Angreiffer erstellte auch neue Pakete fuer openssh und rpm.
Mit diesen veraenderten Paketen konnte der Angreifer nachher allezu alle Fedora Rechner kontrollieren, da die Pakete ueber die yum Updates eingespielt wurden.
Es fehlte jedoch noch der Schritt des signierens, diesen konnte der Angreiffer wohl nicht ausfuehren, da er das Passwort fuer den Signaturschluessel der Pakete nicht hatte.
Da die Pakete aber doch auf einigen Rechnern des Projekts installiert wurden, entschloss man sich einen neuen Key zu erstellen und alle Pakete neu zu signieren, was natuerlich einige Zeit in Anspruch genommen hat.
Das Projekt hat aus dem Vorfall gelernt und eine neue Policy fuer die Admins eingefuehrt:
http://infrastructure.fedoraproject.org/csi/security-policy/
Hier die Historie (alle Zeiten in UTC):
= = = = =
2008-08-12 01:00:00 - Last packaging signing process from a Fedora admin. Key would have been on host temporarily up until this time.
2008-08-12 07:49:05 - Standard Fedora 'pkgconfig' package installed by the intruder. This package is required to build an 'openssh' package. Intruder proceeds to build a modified 'openssh' package.
2008-08-12 08:10:46 - modified 'openssh' package installed by intruder.
2008-08-12 17:46:50 - Standard Fedora 'gettext' package installed by intruder. This package is required to build an 'rpm' package.
2008-08-12 20:18:36 - Standard Fedora 'mc' package installed by intruder, possibly for convenience of stealth.
2008-08-12 21:33:59 - Bacula backup started (scheduled job)
2008-08-12 22:01:54 - Bacula backup Ended
2008-08-12 22:31:51 - modified 'rpm' package installed (along with standard Fedora package dependencies for 'rpm').
2008-08-12 22:51:00 - Cron job failed, notified admins.
2008-08-12 22:53:00 - Fedora Infrastructure admins first noticed and started poking around at why RPM had changed.
2008-08-12 23:11:00 - Infrastructure team lead is notified and more prodding begins.
2008-08-12 23:38:00 - Infrastructure team members gather for discussions on dedicated, private IRC channel and conference call.
2008-08-13 01:50:00 - It becomes more clear that a script is not at fault. LVM snapshot taken.
2008-08-13 04:00:14 - Bacula backup (during the intrusion) restored to secure location
2008-08-13 04:04:14 - Discovery of an RPM in /root/.ssh/ provides proof of malicious intent.
2008-08-13 04:05:00 - Red Hat security team notified.
2008-08-13 04:46:00 - Compromised host prohibited from routing out or in. All machines on its network are preventing access from it. Outbound connections logged.
2008-08-13 05:16:00 - Fedora Project Leader notified.
2008-08-13 06:13:00 - Host state saved (Xen guest). We have a running copy of the host as it was without a reboot.
2008-08-13 06:14:00 - Users who have accessed the machine during the intrusion advised to change their passwords and SSH keys.
2008-08-13 10:13:00 - Work continues in concert with Red Hat security team members. Preliminary announcement prepared
2008-08-14 17:36:00 - All passwords and SSH keys disabled.
2008-08-14 23:15:13 - Preliminary announcement to fedora-announce-list, 1+19:11 after initial determination of malicious event.
2008-08-15 02:47:00 - All administrator access forced to shell access only for partial re-enabling of account system.
2008-08-15 12:00:00 - (approximate) Fedora's package build system, koji, patched to revoke all access.
2008-08-15 13:11:00 - Last package build routine allowed to complete before shutdown. Comprehensive verification of the build system database contents begins, comparing against known source for malicious content.
2008-08-16 15:30:03 - Update announcement to fedora-announce-list, 3+11:26 after initial determination of malicious event.
2008-08-17 22:34:00 - Members of sysadmin-web group allowed back on app servers.
2008-08-18 04:06:31 - Primary content verification of build system and CVS completed.
2008-08-18 18:06:00 - CVS admins allowed back on servers, and handle additional verification for hosted projects.
2008-08-19 02:07:45 - Update announcement to fedora-announce-list, 5+22:03 after initial determination of malicious event.
2008-08-19 02:37:00 - Hosted project verification completed, and Fedora Hosted back online.
2008-08-19 20:19:00 - Anonymous access via cvspserver allowed.
2008-08-20 02:53:00 - Writable access to cvs1 reactivated.
2008-08-20 18:35:00 - Koji build system officially open and building again.
2008-08-22 12:00:02 - Update announcement to fedora-announce-list, 9+07:56 after initial determination of malicious event.
2008-09-19 02:41:29 - Update announcement to fedora-announce-list, 37+22:37 after initial determination of malicious event. Investigation and issue resolution continues.
2009-03-30 14:00:00 - Final report to fedora-announce-list, 229+9:56 after initial determination of malicious event.
= = = = =