Differences between revisions 190 and 654 (spanning 464 versions)
⇤ ← Revision 190 as of 2012-10-10 15:05:32 →
Size: 29195
Editor: jkoller
Comment:
← Revision 654 as of 2022-07-16 05:01:03 → ⇥
Size: 9992
Editor: maegger
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
#rev 2018-08-27 mreimers
#rev 2020-08-31 alders

= General Informations =
 * This page lists announcements and status messages for IT services managed by [[http://www.isg.ee.ethz.ch/|ISG.EE]].
 * For notifications and announcements of central IT services managed by ID, please visit https://www.ethz.ch/services/de/it-services/service-desk.html
 * For a detailed status overview of central IT services managed by ID, please visit https://ueberwachung.ethz.ch
Line 2: Line 10:
||<style="border: medium none;"> {{attachment:green.gif}} ||<style="border: medium none;">Resolved ||
||<style="border-width: medium medium 1px; border-top: medium none rgb(85, 136, 238); border-left: medium none rgb(85, 136, 238); border-right: medium none rgb(85, 136, 238); border-color: rgb(85, 136, 238);"> {{attachment:red.gif}} ||<style="border-width: medium medium 1px; border-top: medium none rgb(85, 136, 238); border-left: medium none rgb(85, 136, 238); border-right: medium none rgb(85, 136, 238); border-color: rgb(85, 136, 238);">Pending ||		 ||<style="border: medium none;"> {{attachment:Status/green.gif}} ||<style="border: medium none;">Resolved ||
||<style="border: medium none;"> {{attachment:Status/orange.gif}} ||<style="border: medium none;">Still working but with some errors ||
||<style="border-width: medium medium 1px; border-top: medium none rgb(85, 136, 238); border-left: medium none rgb(85, 136, 238); border-right: medium none rgb(85, 136, 238); border-color: rgb(85, 136, 238);"> {{attachment:Status/red.gif}} ||<style="border-width: medium medium 1px; border-top: medium none rgb(85, 136, 238); border-left: medium none rgb(85, 136, 238); border-right: medium none rgb(85, 136, 238); border-color: rgb(85, 136, 238);">Pending ||
Line 5: Line 14:
<<Anchor(2012-10-16-VPP Plotter at ETZSPEZ J66)>> = Current status reports =
Line 7: Line 16:
= Outage Poster printer at ETL J 66 / ETZSPEZ - 16. October 2012 =
'''STATUS:''' {{attachment:red.gif}}		 <<Anchor(2022-07-16-planned-server-downtime)>>
== Planned ISG.EE Mailsystem downtime because of IMAP server platform switch ==
'''Status:''' {{attachment:Status/red.gif}}
Line 10: Line 20:
 2012-10-16: 08:30 - 12:30 :: plotter HP6100 at ETZ J66 will be under maintenance and is not available during that time   2022-07-16 07:00:: Opdate of our IMAP server to a new platform has started. Therefore the mail system will not be available for our customers.
  2022-07-16 07:00:: On 2022-07-16 between 07:00 and 12:00 ISG is going to migrate the IMAP Server to another system. During the migration there will be no access to the E-Mails for the following Domains:
   * ee.ethz.ch
   * vision.ee.ethz.ch
   * isi.ee.ethz.ch
   * mwe.ee.ethz.ch
   * mins.ee.ethz.ch
   * nari.ee.ethz.ch
Line 12: Line 29:
---- <<Anchor(2022-07-09-netscratch-maintenance)>>
== NetScratch Server Filesystem Maintenance ==
'''Status:''' {{attachment:Status/green.gif}}
Line 14: Line 33:
<<Anchor(2012-10-10-lists.ee.ethz.ch_downtime)>>   2022-07-10 10:55:: Fileysstem for [[Services/NetScratch|NetScratch]] is online and in read-write mode.
  2022-07-10 09:30:: First run of checks are completed and we are proceeding with the next steps to put the filesystem again online.
  2022-07-09 13:00:: The [[Services/NetScratch|NetScratch]] filesystem will be put into read-only mode for maintenance.
Line 16: Line 37:
= Mailinglist Downtime (lists.ee.ethz.ch) =
'''STATUS:''' {{attachment:green.gif}}		 <<Anchor(2022-06-16-exchange-support)>>
== Unreachable ee.ethz.ch Email recipients over ID Exchange Mailserver ==
'''Status:''' {{attachment:Status/green.gif}}
Line 19: Line 41:
 2012-10-10: 09:00 - 17:00 :: Due to the migration of our mailing list software to sympa, we will take down the [[http://lists.ee.ethz.ch|lists.ee.ethz.ch]] website. Mails sent to lists.ee.ethz.ch will put into a HOLD queue and delivered to the mailinglist after migration. So '''no mails get lost'''!   2022-06-16 16:45:: Configuration issue has been resolved.
  2022-06-16 15:00:: Emails with ee.ethz.ch recipients sent over the ID Exchange Server do not reach destination. ID Exchange Admins are working on fixing the problem.
Line 21: Line 44:
 2012-10-10: 17:04 :: Mailinglists converted. Services up on running. <<Anchor(2021-11-01-home-maintenance)>>
== HOME Server maintenance to repair fileystem inconsistence ==
'''Status:''' {{attachment:Status/green.gif}}

  2021-11-01 06:30:: System back online and HOME directories are again accessible for all D-ITET user
  2021-11-01 05:50:: HOME Server will be put offline to start a repair of a filesystem inconsistence

<<Anchor(2021-07-05-linux-printing)>>
== Linux printing affected by PrintNightmare vulnerability patch ==
'''Status:''' {{attachment:Status/green.gif}}
  2021-07-05 13:00:: ID resolved the issue
  2021-07-05 09:41:: Workaround: Use [[Printing#Platform-independent_printing| platform-independent printing]]
  2021-07-05 09:41:: Authentification for printing fails. Ticket at ID servicedesk opened.

<<Anchor(2021-04-27-itetmaster01-update)>>
== Downtime various D-ITET services for server maintenance ==
'''Status:''' {{attachment:Status/green.gif}}

  2021-04-27 08:30:: Condor is back online, all services restored.
  2021-04-27 08:15:: Matrix/Element Chat services back online.
  2021-04-27 08:00:: Database upgrade done and online.
  2021-04-27 07:30:: Slurm services are back online.
  2021-04-27 07:00:: Base system has been upgraded, main database services in progress of upgrade.
  2021-04-27 06:00:: On 2021-04-27 between 06:00 and 08:30 ISG is going to update a server providing access to various D-ITET services. During the migration the following services will be affected and offline:
   * Matrix/Element Chat services (the instances will be unavailable)
   * IFA/Control Website: Access to the IFA database is blocked
   * Slurm (D-ITET Arton Cluster): It won't be possible to submit new jobs or view Slurm statistics. Already running jobs will not be affected.
   * Condor: Condor clients will be shut down the evening before to avoid running jobs during the migration.

<<Anchor(2021-03-31-network-disruption)>>
== Network disruption affecting several ISG.EE services ==
'''Status:''' {{attachment:Status/green.gif}}

  2021-03-31 09:30:: The configuration error was found. The configuration change will be deployed on '''2021-04-01 around 06:15''' and a short network of about 1min is expected.
  2021-03-31 08:00:: ID Networking team has rolled-back a deployed configuration, pending further investigation/analysis.
  2021-03-31 07:30:: There are currently disruption affecting a VPZ with servers managed by ISG.EE. Networking team of ID is investigating the issue. There are several ISG.EE services affected/malfunctioning due to this in particluar the FindYourData service.

<<Anchor(2021-03-11-mira-upgrade)>>
== login.ee.ethz.ch: downtime for server upgrade ==
'''Status:''' {{attachment:Status/green.gif}}

  2021-03-11 06:30:: Upgrade completed and service is up and running again.
  2021-03-11 06:00:: The server servicing login.ee.ethz.ch will be upgraded to a new OS version (Debian buster). During the time of the update logins might not be possible.

<<Anchor(2020-07-11-storage-downtime)>>
== Planned project/ archive storage downtime and client reboot ==
'''Status:''' {{attachment:Status/green.gif}}

  2020-07-11 12:00:: Migration has been completed, all services are back to operational state.

  2020-07-11 08:00:: Migration started, services are shutdown

  2020-07-11 8:00-12:00:: Start of planned maintenance work. Project/ archive storage services (known under the names "ocean", "bluebay", "lagoon" and "benderstor") will not be available. ISG-managed Linux clients will be rebooted.



<<Anchor(2020-06-04-svnsrv-upgrade)>>
== svn.ee.ethz.ch downtime for server upgrade ==
'''Status:''' {{attachment:Status/green.gif}}

  2020-06-04 07:05:: Webservices for managing SVN repositories are enabled.
  2020-06-04 06:15:: Systemupgrade is done and access to the SVN repositories via the `svn` and `https` transport protocols are back online.
  2020-06-04 06:00:: The server servicing the SVN repositories will be upgraded to a new operating system version. During this timeframe outages for access to the SVN repositories are expected.

<<Anchor(2020-05-17-cluster-abuse)>>
== European HPC cluster abuse ==
'''Status:''' {{attachment:Status/green.gif}}<<BR>>
Recently European HPC clusters have been attacked and abused for mining purposes. The D-ITET Slurm and SGE clusters have not been compromised. We are monitoring the situation closely.
  2020-05 17 08:30:: No successful login from known attacker IP addresses could be determined, none of the files indicating being compromised have been found on our file systems
  2020-05-16 14:30:: No unusal cluster job activity was observed

<<Anchor(2020-05-04-itetnas04-upgrade)>>
== D-ITET Netscratch downtime for server upgrade ==
'''Status:''' {{attachment:Status/green.gif}}

  2020-05-04 06:00:: Server upgrade has been completed.
  2020-05-04 06:00:: The server servicing the D-ITET Netscratch service will be upgraded to a new operating system version. During this timeframe outages for the NFS service will be expected.

<<Anchor(2020-04-07-network-interuption)>>
== Network outage ETx router ==
'''Status:''' {{attachment:Status/green.gif}}
  2020-04-07 05:30:: There was an issue on the Router `rou-etx`. ID networking team trackled and solved the issue. There was about a 10min interuption for the ETx networking zone affecting almost all ISG.EE maintained systems.

<<Anchor(2020-04-06-mira-maintenance)>>
== login.ee.ethz.ch: Reboot for maintenance ==
'''Status:''' {{attachment:Status/green.gif}}
  2020-04-06 05:35:: System behind `login.ee.ethz.ch` has been rebootet for maintenance and increase available resources.

See the [[RemoteAccess|information on access D-ITET resources remotely]]. To distribute better the load user are encouraged to use the VPN service whenever possible.

<<Anchor(2020-02-18-nostro-maintenance)>>
== itet-stor (FindYourData) Server maintenance: Reconfiguration of VM parameters ==
'''Status:''' {{attachment:Status/green.gif}}

  2020-02-18 19:03:: System again up and running.
  2020-02-18 19:00:: Scheduled downtime for the [[Workstations/FindYourData|itet-stor/FindYourData service]] due to maintenance work on the underlying server.

<<Anchor(2020-01-20-nostro-os-upgrade)>>
== itet-stor (FindYourData) Server migration: New operating system version ==
'''Status:''' {{attachment:Status/green.gif}}

  2020-01-20 07:15:: OS upgrade done, there were short interruptions to the [[Workstations/FindYourData|itet-stor/FindYourData service]].
  2020-01-20 06:00:: We will update the server servicing the [[Workstations/FindYourData|FindYourData service]] from Debian jessie 8 to Debian stretch 9. There will be short downtimes accessing this service during the update.


= Archived status reports =

[[Status/Archive/2010|2010]]
[[Status/Archive/2011|2011]]
[[Status/Archive/2012|2012]]
[[Status/Archive/2013|2013]]
[[Status/Archive/2014|2014]]
[[Status/Archive/2015|2015]]
[[Status/Archive/2016|2016]]
[[Status/Archive/2017|2017]]
[[Status/Archive/2018|2018]]
[[Status/Archive/2019|2019]]
Line 24: Line 163:

<<Anchor(2012-09-08-Ldap-Migration)>>

= LDAP Migration =
'''STATUS:''' {{attachment:green.gif}}

 2012-09-08: 10:00 - 17:00 :: All systems and services are not available during migration
 
 2012-09-08: 17:30 :: All systems are back online

----

<<Anchor(2012-08-17-Network-Problem-ETH)>>

= Network outage ETH =
'''STATUS:''' {{attachment:green.gif}}

 2012-08-17: 16:30:: Currently on ETH network there seem to be problems related on networking level. No information is available yet.

 2012-08-17: 17:15:: Systems are back online.

----

<<Anchor(2012-06-28-power-outage)>>

= Power outage at ETH =
'''STATUS:''' {{attachment:green.gif}}

 2012-06-28: 19:17:: ETH had power outage affecting many services. D-ITET infrastructure was partially affected too.

 2012-06-29: 08:00:: We are currently working on resolving the outstanding issues and bringing back online services which are still down.

 2012-06-29: 09:30:: On ID Status website https://www1.ethz.ch/id/servicedesk/sysstat/index you can find now further information.

 2012-06-29: 10:05:: The cause was a transformator on fire in the main building, causing a power downtime in the computer centres from 19:30 to 22:00.

----

<<Anchor(2012-06-20-oenone-servercrash)>>

= Server oenone crash =
'''STATUS:''' {{attachment:green.gif}}

 2012-06-20: 00:00:: oenone crashed. Affecting users of '''AUT''', '''BIWI''', '''COLLEGIUM''', '''IKT''' and '''VAW'''.

 2012-06-05: 00:30:: oenone rebooted and is running again.


(!) We are still investigating what caused the crash and will report further information here.

----

<<Anchor(2012-06-05-oenone-server-problem)>>

= oenone: hanging lockd affecting some User homes and webpages =
'''STATUS:''' {{attachment:green.gif}}

 2012-06-05: 21:37:: oenone is unresponsive. Affecting users of '''AUT''', '''BIWI''', '''COLLEGIUM''', '''IKT''' and '''VAW'''.

 2012-06-05: 00:00:: oenone was rebooted and is running again.

----

<<Anchor(2012-06-05-alumni-mailserver-problem)>>

= Outage on 2/3 of alumni.ethz.ch mailservers =
'''STATUS:''' {{attachment:green.gif}}

 2012-06-04: 16:00:: Our mailserver can't deliver emails to alumni.ethz.ch addresses. Reason: The receiving Servers have temporary errors: "451 unable to verify user". Looks like something is misconfigured there.
 2012-06-05: 10:00:: Looks like all alumni servers hosted on tophost.ch have a problem. We have created a temporary transport map which delivers the mails to the genotec.ch (the only one which works) alumni mailserver. As long as you use our mailserver to send emails to alumni addresses they will be delivered immediately.
----
<<Anchor(2012-05-13-oenone-server-problem)>>

= oenone: hanging lockd affecting some User homes and webpages =
'''STATUS:''' {{attachment:green.gif}}

 2012-05-13: 22:50:: oenone is unresponsive. Affecting users of '''AUT''', '''BIWI''', '''COLLEGIUM''', '''IKT''' and '''VAW'''.

 2012-05-14: 00:15:: oenone was rebooted and is running again.

----
<<Anchor(2012-05-08-drwho-server-problem)>>

= drwho: Main server outage affecting 64bit diskless Linux clients =
'''STATUS:''' {{attachment:green.gif}}

 2012-05-08: 11:50:: We currently experience some problems with one of our main server. All 64bit diskless clients are affected. We are working on the Problem. Furthermore some svn repositories might be affected.

 2012-05-08: 13:15:: The system is going back to normal but needs some time to fully recover.

 2012-05-08: 13:45:: The system should be back to normal. We still are working on some single hosts to recover them.

----


<<Anchor(2011-11-12-solaris-server-patching)>>

= Maintenance work on D-ITET's central IT infrastructure =
'''STATUS:''' {{attachment:green.gif}}

 2012-03-13: 21:30 :: All systems are back online.

 2012-03-13: 19:00 - 22:00 :: To keep our systems up to date with the newest software and security releases, we need to update our servers on a regular basis. For this reason we are going to install latest Oracle patches on our main servers. The servers will be rebooted during this maintenance

To prevent data corruption/loss please do the following:
 * save all open files
 * close all running applications
 * logout from all ITET systems (Linux, Solaris, Windows, Mac OS X)
 * shutdown your personal PC/Desktop
 * do not establish any connection from outside

More details:
 * servers concerned: DRWHO, TARDIS, OENONE, SPITFIRE, YOSEMITE, MALINA
 * webpages hosted on these systems are NOT available
 * NO mail access, NO outgoing mails (incoming mails WON'T get lost)

----


<<Anchor(2012-02-16-zaan-maintenance)>>

= Maintenance downtime of Server behind ipp2vpp.ee.ethz.ch (printing/licenses/dhcp for selfmanaged hosts) =
'''STATUS:''' {{attachment:green.gif}}

On '''2012-02-16''' around '''8:00AM''' we are scheduling a maintenance downtime of zaan (serving printing, license Server and DHCP Server for D-ITET). The downtime will be as short as possible. We plan to have it down for around 45 minutes at most.

 '''2012-02-16 08:45:''' Server is up and running again.

----

<<Anchor(2012-02-13-oenone-crash)>>

= oenone home server crash =
'''STATUS:''' {{attachment:green.gif}}

 2012-02-13:: During this night at around 22:40 '''oenone''' one of our home-servers crashed. Users with homes on oenone where affected, these are '''BIWI''', '''VAW''', '''Collegium Helveticum''', '''Control''', '''ISI''', '''IKT'''. The system was up again at 00:40.

We are sorry for the caused inconvenience and we are investigating the problem.


----
<<Anchor(2012-01-31-oenone-emergency-reboot)>>

= Emergency reboot of ITET's server OENONE =
'''STATUS:''' {{attachment:green.gif}}

 2012-01-31: 21:00:: Emergency reboot of server OENONE due to a not responding storage area. User concerned: '''BIWI, VAW, Collegium Helveticum, Control, ISI, IKT, EEH'''

 2012-01-31: 21:45:: Server oenone is up again and all services are running

----
<<Anchor(2012-01-23-linux-reboots)>>

= Emergency reboots of all Linux Clients and Servers =
'''STATUS:''' {{attachment:green.gif}}

 2012-01-23: 15:00 PM:: Due to a critical issue we were forced to reboot all affecting hosts We are sorry for the short notice and the inconvenience caused to you.

 2012-01-23: 20:45 PM:: All Clients rebooted.

----
<<Anchor(2011-11-29-oenone-crash)>>

= Maintenance work on D-ITET's home server TARDIS and OENONE =
'''STATUS:''' {{attachment:green.gif}}

 2011-12-17: 11:00 AM:: successful reboot of TARDIS and OENONE
 
 2011-12-17 10:00AM - 11:00AM:: During the last installation of Oracle patches a bug within the automount daemon was introduced causing high CPU load on systems with a high number of auto-mounted file systems. We have investigated this problem together with Oracle. Now, a bug fix is available, but requests a server reboot. Due to this requirement we are going to reboot TARDIS and OENONE at

To prevent data corruption/loss please do the following:
 * save all open files
 * close all running applications
 * logout from all ITET systems (Linux, Solaris, Windows, Mac OS X)
 * shutdown your personal PC/Desktop
 * do not establish any connection from outside


More details:
 * servers concerned: TARDIS, OENONE
 * webpages hosted on these systems are NOT available
 * NO mail access, NO outgoing mails (incoming mails WON'T get lost)

----
<<Anchor(2011-11-29-oenone-crash)>>

= oenone home server crash =
'''STATUS:''' {{attachment:green.gif}}

 2011-11-29:: During this night at around 21:30 '''oenone''' one of our home-servers crashed. Users with homes on oenone where affected, these are '''BIWI''', '''VAW''', '''Collegium Helveticum''', '''Control''', '''ISI''', '''IKT'''. The system was up again at 23:00.

We are sorry for the caused inconvenience and we are investigating the problem.


----

<<Anchor(2011-11-14-power-outage)>>

= Poweroutage affecting compute clusters =
'''STATUS:''' {{attachment:green.gif}}
 2011-11-14:: Due to a power outage some racks containing our compute clusters went down.

 2011-11-14 08:00:: All compute clusters should be up and running again.

----

<<Anchor(2011-11-12-solaris-server-patching)>>

= Maintenance work on D-ITET's central IT infrastructure =
'''STATUS:''' {{attachment:green.gif}}

 2011-11-12: 05:50 PM:: Final reboot of TARDIS successfully terminated.

 2011-11-12: 04:00 PM:: The final reboot of tardis is still outstanding due to a broken disk within TARDIS internal RAID (boot device). The broken disk has been successfully replaced but the RAID is still syncing. The reboot is postponed until the sync process is finished and the reboot can safely be carried out. So be prepared for a short interrupt today or tomorrow.

 2011-11-12: 04:00 PM:: All systems are back online.

 2011-11-12: 10:00 AM - 04:00 PM:: To keep our systems up to date with the newest software and security releases, we need to update our servers on a regular basis. For this reason we are going to upgrade specific storage software and install latest Oracle patches on our main servers. The servers will be rebooted multiple times during this maintenance

To prevent data corruption/loss please do the following:
 * save all open files
 * close all running applications
 * logout from all ITET systems (Linux, Solaris, Windows, Mac OS X)
 * shutdown your personal PC/Desktop
 * do not establish any connection from outside

More details:
 * servers concerned: DRWHO, TARDIS, OENONE, SPITFIRE, YOSEMITE, MALINA
 * webpages hosted on these systems are NOT available
 * NO mail access, NO outgoing mails (incoming mails WON'T get lost)

----

<<Anchor(2011-08-19-switch-routing-loop)>>

= Routing problems on switch.ch network =
'''STATUS:''' {{attachment:green.gif}}
 2011-08-19: 05:00 PM:: Routing Problems solved.
 2011-08-19: 02:22 PM:: Because of a routing problem on the switch network, all traffic to http://www.virginia.edu and their mailserver is disturbed.

----

<<Anchor(2011-10-14-cronbox-migration)>>
= Migration of cronbox.ee.ethz.ch to Debian Squeeze =
'''STATUS:''' {{attachment:green.gif}}
 2011-10-14: 07:30:: We plan to migrate the server behind cronbox.ee.ethz.ch to the new version of the Debian operating system. Expected downtime is from 07:30 up to around 10:00. The affected services are '''cron''' and '''ssh''' logins to the machine.
 2011-10-14: 11:00:: Migration completed.

----

<<Anchor(2011-09-30-matlab-license-server-down)>>
= Matlab License Server down =
'''STATUS:''' {{attachment:green.gif}}
 2011-09-30: 07:00:: Currently the Server from Informatik Dienste providing the license server service is down. You can track the curren status at their [[https://www.komcenter.ethz.ch/home/idServices/show|ID-Service Status]] page under Lizenzen -> 1965@vnava.

 2011-09-30: 08:45:: License server from Informatikdienste is now up again.

 2011-09-30: 15:30:: The license server lic-matlab.ethz.ch is unavailable again. Due it is outside our control we cannot estimate when it will work again.

 2011-09-30: 16:00:: lib-matlab.ethz.ch is available again.

----

<<Anchor(2011-08-30-nfs-problems-on-oenone)>>
= NFS outage on oenone =
'''STATUS:''' {{attachment:green.gif}}
 2011-08-30: 02:00 AM - 08:30 AM:: During this night at around 02:00 the NFS Services on '''oenone''' crashed. Users with homes on oenone where affected, these are '''BIWI''', '''VAW''', '''Collegium Helveticum''', '''Control''', '''IBT''', '''IKT'''. This crash also affected all webservices which depend on oenone and the mailserver (at least for those having the home directory on oenone). No mails are lost as we put them into a hold queue!

 Update: 08:30:: oenone is now up and running again. All Mails on the hold queue are now gradualy delivered. The webservices are also all available now.

We are sorry for the caused inconvenience and we are investigating the problem.

----

<<Anchor(2011-08-23-admin-ch-connection-problems)>>
= Connection problems to all admin.ch servers =
'''STATUS:''' {{attachment:green.gif}}
 2011-08-23: 03:00 PM:: All admin.ch websites an mailservers are reachable.
 2011-08-23: 01:25 PM:: Currently all traffic to the admin.ch servers is disturbed. This includes the websites and also email connections. Your sent mails are not lost as our server keeps them until it can connect to the destination.

----

<<Anchor(2011-08-23-solaris-server-patching)>>

= Maintenance work on D-ITET's central IT infrastructure =
'''STATUS:''' {{attachment:green.gif}}

 2011-08-23: CANCELLATION:: Due to unresolved error within Solaris operating system introduced by latest patch set

 2011-08-23: 7:00 PM - 10:00 PM:: To keep our systems up to date with the newest software and security releases, we need to update our servers on a regular basis. For this reason we are going to patch and reboot some of our Solaris servers.

To prevent data corruption/loss please do the following:
 * save all open files
 * close all running applications
 * logout from all ITET systems (Linux, Solaris, Windows, Mac OS X)
 * shutdown your personal PC/Desktop
 * do not establish any connection from outside

More details:
 * servers concerned: DRWHO, TARDIS, OENONE, SPITFIRE, YOSEMITE, MALINA
 * webpages hosted on these systems are NOT available
 * NO mail access, NO outgoing mails (incoming mails WON'T get lost)
----

<<Anchor(2011-08-19-switch-routing-loop)>>

= Routing problems on switch.ch network =
'''STATUS:''' {{attachment:green.gif}}
 2011-08-19: 05:00 PM:: Routing Problems solved.
 2011-08-19: 02:22 PM:: Because of a routing problem on the switch network, all traffic to http://www.virginia.edu and their mailserver is disturbed.

----

<<Anchor(2011-08-08-drwho-problems)>>

= Outage drwho.ee.ethz.ch =
'''STATUS:''' {{attachment:green.gif}}

 2011-08-13 09:00:: Since around 1:00 we experience server problems on one of our main servers affecting most of the services.

 2011-08-13 11:00:: All services back to normal.

----

<<Anchor(2011-08-08-dns_outage)>>

= ETH wide DNS outage =
'''STATUS:''' {{attachment:green.gif}}

 2011-08-08 10:00:: ETH wide DNS outage. All Services using Name Resolution do not work. Our Mailserver denys all incoming messages with an {{{450 4.3.2 Service currently unavailable}}}. Properly configured Mailservers should retry the message delivery later, so no mail is lost.

 2011-08-08 10:20:: DNS works again. All Services up and running.


----

<<Anchor(2011-06-22-colombo04)>>

= colombo04 not available =
'''STATUS:''' {{attachment:green.gif}}

 2011-06-23 11:00:: Powersupply of colombo04 replaced. colombo04 up and ready.

 2011-06-22 07:00:: Powersupply of colombo04 broke. Now waiting for replacement from Oracle.

----

<<Anchor(2011-06-06-biwinas01)>>

= biwinas01 not available =
'''STATUS:''' {{attachment:green.gif}}

 2011-06-08 08:00:: biwinas01 is now back online.

 2011-06-07 15:00:: The hardware supplier returned the server today. They had to replace the following hardware:
 * 1 CPU
 * 1 power supply
 * 1 fan

We will test the server now and bring it back online as soon as possible.

 2011-06-06 15:00:: biwinas01 is currently out of order due to a hardware failure. It might take several days until biwinas01 is back online.

----
<<Anchor(2011-06-06-ifhlux11)>>

= ifhlux11 not available =
'''STATUS:''' {{attachment:green.gif}}

 2011-06-08 11:20:: iflux11 is now back online.

 2011-06-07 15:00:: We are in contact with the supplier. Unfortunately, the reason for the crash of ifhlux11 is not known yet.

 2011-06-06 15:00:: ifhlux11 is currently out of order due to a hardware failure. It might take several days until ifhlux11 is back online.

----
<<Anchor(2011-06-03-Cooling-System-Replacement)>>

= Coming soon: Outage of several IT services due to cooling system replacement =

'''STATUS:''' {{attachment:green.gif}}

 2011-06-03 15:00:: All servers are again up and running with the exceptions of '''biwinas01''' and '''ifhlux11''' (see above).

'''UPCOMING: 2011-06-03 09:00 - 2011-06-06 09:00'''

The refrigeration supply in the ETZ building will be replaced. One server room
has to be shutdown completely (ETZ/D/96.2; all compute servers of the institutes).
The other server room will get an emergency cooling system (ETZ/F/66). This will,
however, not allow the cooling of all currently running servers within that room.
Consequently we will shut down as many servers as possible.

'''Basic services like email, user homes and other network shares, printing and login to
Windows or Linux workstations are not expected to be affected by this construction work.'''
In general, if you are not familiar with the server names or service terms below, you
should not be affected at all.

'''These services will not be available during the construction work times:'''
 * Shut down 2011-06-03 at '''09:00'''. Back online 2011-06-06 at 09:00:
  * Remote-Desktop-Access to the Windows Terminal Servers
   * quinn
   * sivi
   * vega7
  * All institute NAS servers (no access via Samba, link in home, ssh, etc.)
   * biwinas01-03
   * hamam01
   * ifenas01
   * ibtnas01
  * Most IFE compute servers
   * bernstein
   * coltrane
   * dylan
   * haydn
   * marley
   * mozart
  * Publication databases on sato (IFA)

 * Shut down 2011-06-03 at '''16:00'''. Back online 2011-06-06 at 09:00:
  * Computer rooms for students
   * ETZ D 61.1
   * ETZ D 61.2
   * ETZ D 96
  * All institute compute servers
   * autserv*
   * bender*
   * biwilux*
   * casseri*
   * colombo*
   * IFE compute servers cash and elvis
   * ifhlux*
   * nariwork*
   * tik*x
   * vierzack*

----
<<Anchor(2011-05-26-Firewall-Problem)>>

= Firewall Problem: some Services not reachable =
'''STATUS:''' {{attachment:green.gif}}

'''2011-05-26 07:15 - 2011-05-26 16:00'''

Due to problems with the firewall hardware, some services of the D-ITET-servers were not reachable.

----
<<Anchor(2011-05-19-polaris-and-zaan-reboot)>>

= Maintenance reboot of Servers behind login.ee.ethz.ch and ipp2vpp.ee.ethz.ch (printing/licenses) =
'''STATUS:''' {{attachment:green.gif}}

On '''2011-05-19''' around '''7:00AM''' we will perform a maintenance reboot of polaris (serving login.ee.ethz.ch) and zaan (serving printing at D-ITET). During this downtime it will not be possible to print via samba or cups.

----
<<Anchor(2011-05-09-galen-reboot)>>

= Maintenance reboot of Server behind people.ee.ethz.ch =
'''STATUS:''' {{attachment:green.gif}}

'''2011-05-10 07:45'''

On '''2011-05-10''' around '''7:00AM''' we will perform a maintenance reboot of galen. galen is the Server serving your personal homepage on people.ee.ethz.ch.

----

<<Anchor(2011-05-02-horde-outage)>>

= Horde Webmail outage =
'''STATUS:''' {{attachment:green.gif}}

'''2011-05-02 23:49 - 2011-05-03 02:27'''

The [[https://email.ee.ethz.ch|Horde Webmail Client]] had to be taken down for security reasons. It was not clear if someone used a zero day exploit or a phished account to send spam over our server. It turned out, the attackers used a phished account.

/!\ '''Please Remember: We at ISG.EE will NEVER ask you for your password.'''

----

<<Anchor(2011-03-01-short-smtp-outage)>>

= Short SMTP outage =
'''STATUS:''' {{attachment:green.gif}}

 2011-02-28 14:49 - 15:04:: As a result of an LDAP failure we had to stop the mail server for 15 minutes, to prevent the rejection of incoming emails. While the Mainserver was down, our two backup MX collected incoming emails.

----

<<Anchor(2011-02-28-zhadum-crash)>>

= Windows Terminal Server zhadum out-of-operation =
'''STATUS:''' {{attachment:green.gif}}

 2011-02-28 15:00:: Please use the server '''vega7''' from now on. If you had access to zhadum before, you should also be able to access vega7. '''Please use your NETHZ username and password to log in.'''

 2011-02-28 14:00:: The hardware of the departemental Windows terminal server '''zhadum''' is broken. A replacement server should be available soon...

----
<<Anchor(2011-02-22-jabba-upgrade)>>

= Upgrade of Backup Server JABBA =
'''STATUS:''' {{attachment:green.gif}}

 2011-02-22 18:15 PM:: The migration is finished and the new JABBA server is online.

 2011-02-22 7:00 AM - 7:00 PM (approx.):: We are going to upgrade the departments backup server JABBA. This upgrade includes changes in software as well as in hardware. During the upgrade no restore request of lost data can be fulfilled.

'''The complete backup infrastructure and all belonging services are NOT available during the upgrade'''

----
<<Anchor(2011-02-07-agilent-license-server-change)>>

= Agilent ADS/ICCAP License Server Change =
'''STATUS:''' {{attachment:green.gif}}

 2011-02-07 10:00:: As announced a week ago, the license server for Agilent ADS and ICCAP software has changed. If your client software still uses the old license server information, please make sure you change the license server to '''lic-agilent.ee.ethz.ch'''. The port stays the same.

----
<<Anchor(2011-01-31-svn-firewall)>>

= Subversion server not reachable from outside ETH Zurich =
'''STATUS:''' {{attachment:green.gif}}

 2011-01-31 16:30:: Our subversion server is now available again for users outside of ETH Zurich.

 2011-01-31: 16:15:: We have been told that the central firewall rules cannot be changed right now due to other (completely unrelated) problems. At the moment it is unknown when this will be fixed.

 2011-01-31: 15:30:: Our subversion server svn.ee.ethz.ch is at the moment not accessible through the svn:// protocol from outside ETH Zurich due to a firewall configuration error. We are working on it...

----
<<Anchor(2011-01-26-mailser-problems-after-patching)>>

= Mailserver problems while patching =
'''STATUS:''' {{attachment:green.gif}}

 2011-01-26: 11:00 AM:: During the server upgrade last night a patch has temporarily misconfigured the mail server. The server accepted incoming mails but could not place them into the users mailbox. EVERY such mail created a bounce. Because of this, our statement that no mails get lost while updating the servers, is not fully true anymore. Incoming Mails which bounced back have to be resent by the sender.

----
<<Anchor(2010-12-14-solaris-server-patching)>>

= Solaris Server Patching =
'''STATUS:''' {{attachment:green.gif}}

 2011-01-25: 10:30 PM:: All servers and services are back online

 2011-01-25: 7:00 PM - 10:00 PM:: To keep our systems up to date with the newest software and security releases, we need to update our servers on a regular base. For this reason we are going to patch and reboot some of our Solaris servers.

Servers concerned: '''drwho''', '''tardis''', '''oenone''', '''spitfire''', '''yosemite''', '''malina'''.

To prevent data corruption/loss please do the following:

 * All diskless clients (Linux): please logout and shutdown all DL clients
 * All Windows systems: please logout and shutdown all Windows clients
 * All Mac systems: please logout and shutdown all Mac clients
 * All user homes: please logout from these servers
 * No NFS or SAMBA access to user homes
 * No mail access, no outgoing mails (incoming mails WON'T get lost)
 * Webpages hosted on these systems are unavailable

----
<<Anchor(2010-12-19-delayed-email-delivery)>>

= Delayed Email delivery =
'''STATUS:''' {{attachment:green.gif}}

 2011-01-19: 11:00 AM - 4:30 PM:: As a result of a faulty [[http://lurker.clamav.net/thread/20110119.125839.2b4ce0e1.en.html|ClamAV signature File]] every Email that contained a PDF-file was marked as infected. Before we could resend the quarantined emails we had to fix the issue. No mail was lost and everything was resent.

 Update: 2011-01-20: 10:33 AM:: ClamAV Signatues have been updated and tested. Everything is working as it should.

----
<<Anchor(2010-12-07-reboot-yosemite)>>

= Maintenance Reboot of Solaris Server Yosemite =
'''STATUS:''' {{attachment:green.gif}}

 2010-12-07: 7:30 AM:: Server yosemite has been rebooted successfully. All services are available.

 2010-12-07: 7:00 AM:: Due to a shortage of available memory we are forced to reboot the solaris server yosemite. Downtime approx. 30 minutes.

----
<<Anchor(2010-11-26-servers-down)>>

= cooling water system outage on clusters =
'''STATUS:''' {{attachment:green.gif}}

 2010-11-26: 5:00 PM:: Host autserv02 is running as well. All hosts can be used.

 2010-11-26: 4:40 PM:: Server racks are cooled again, all hosts except of autserv02 are running and can be used.

 2010-11-26: 4:00 PM:: Server racks are still down. --> Update follows at 5 PM or earlier

 2010-11-26: 3:10 PM:: One of the cooling water pumps installed in ETZ/D/96.2 does not work correctly. This forces some of the racks in this server room to shutdown in order to protect the servers from thermal damage. '''clusters from IFH, IBT, BIWI, TIK, IKT and VAW are affected.''' the facility management is working on solving the problem. --> Update follows at 4 PM

----
<<Anchor(2010-11-23-email-phishing-attack)>>

= email phishing attack =
'''STATUS:''' {{attachment:green.gif}}

 2010-11-23:: Yesterday between 18:20 and 19:50 about 320 Phishing Mails have been sent to different Users at D-ITET. The Mails pretend to come from ''IT Support Group'' and contain the subject ''ISG.EE Webmail Alert''. The mail tells something about ''spammers'' that have compromised ''the'' ISG.EE Webmail Account and that you should provide your '''Username, Password''' and some '''Alternate Email'''. Please remember, that the ISG.EE Team will '''NEVER ask you for your Password!''' If you still have replied to this phishing mail please contact us '''immediately''' under support@ee.ethz.ch so that we can plan with you the next steps to keep your account safe.

----
<<Anchor(2010-11-17-oenone-crash)>>

= oenone home server crash =
'''STATUS:''' {{attachment:green.gif}}

 2010-11-17:: During this night at around 00:15 '''oenone''' one of our home-servers crashed. Users with homes on oenone where affected, these are '''BIWI''', '''VAW''', '''Collegium Helveticum''', '''Control''', '''IBT''', '''IKT'''. The server is now checking the filesystems and comming up again.

We are sorry for the caused inconvenience and we are investigating the problem.

 Update: 08:00:: oenone is now up and running again.
 Update: 2010-11-18 07:30:: We opened a support case at Sun/Oracle for this server.

----
<<Anchor(2010-11-16-servers-down)>>

= cooling water system outage for some clusters =
'''STATUS:''' {{attachment:green.gif}}

 2010-11-16:: On last friday evening one of the cooling water pumps installed in ETZ/D/96.2 stopped working correctly. This forced some of the racks in this server room to shutdown in order to protect the servers from thermal damage. '''All clusters from IFH, IBT, BIWI, TIK, IKT and VAW were affected.'''

The facility management is working on solving the problem.

The servers are currently (08:35) down again. Please, even if they come up again, do not use them for long-timed computations as we still do not know when exactly the technician has solved the issue.

 Update: 2010-11-17 08:25:: The rack systems are running now with only one cooling water pump. A new pump is ordered by the rack company.
 Update: 2010-11-18 16:00:: Planed substitution of broken pump will be on 25.11 or 26.11.

----		 [[CategoryEDUC]]

General Informations

Status-Key

Status/green.gif

Resolved

Status/orange.gif

Still working but with some errors

Status/red.gif

Pending

Current status reports

Planned ISG.EE Mailsystem downtime because of IMAP server platform switch

Status: Status/red.gif

2022-07-16 07:00
Opdate of our IMAP server to a new platform has started. Therefore the mail system will not be available for our customers.
2022-07-16 07:00
On 2022-07-16 between 07:00 and 12:00 ISG is going to migrate the IMAP Server to another system. During the migration there will be no access to the E-Mails for the following Domains:
  • ee.ethz.ch
  • vision.ee.ethz.ch
  • isi.ee.ethz.ch
  • mwe.ee.ethz.ch
  • mins.ee.ethz.ch
  • nari.ee.ethz.ch

NetScratch Server Filesystem Maintenance

Status: Status/green.gif

2022-07-10 10:55

Fileysstem for NetScratch is online and in read-write mode.

2022-07-10 09:30
First run of checks are completed and we are proceeding with the next steps to put the filesystem again online.
2022-07-09 13:00

The NetScratch filesystem will be put into read-only mode for maintenance.

Unreachable ee.ethz.ch Email recipients over ID Exchange Mailserver

Status: Status/green.gif

2022-06-16 16:45
Configuration issue has been resolved.
2022-06-16 15:00
Emails with ee.ethz.ch recipients sent over the ID Exchange Server do not reach destination. ID Exchange Admins are working on fixing the problem.

HOME Server maintenance to repair fileystem inconsistence

Status: Status/green.gif

2021-11-01 06:30
System back online and HOME directories are again accessible for all D-ITET user
2021-11-01 05:50
HOME Server will be put offline to start a repair of a filesystem inconsistence

Linux printing affected by PrintNightmare vulnerability patch

Status: Status/green.gif

2021-07-05 13:00
ID resolved the issue
2021-07-05 09:41

Workaround: Use platform-independent printing

2021-07-05 09:41
Authentification for printing fails. Ticket at ID servicedesk opened.

Downtime various D-ITET services for server maintenance

Status: Status/green.gif

2021-04-27 08:30
Condor is back online, all services restored.
2021-04-27 08:15
Matrix/Element Chat services back online.
2021-04-27 08:00
Database upgrade done and online.
2021-04-27 07:30
Slurm services are back online.
2021-04-27 07:00
Base system has been upgraded, main database services in progress of upgrade.
2021-04-27 06:00
On 2021-04-27 between 06:00 and 08:30 ISG is going to update a server providing access to various D-ITET services. During the migration the following services will be affected and offline:
  • Matrix/Element Chat services (the instances will be unavailable)
  • IFA/Control Website: Access to the IFA database is blocked
  • Slurm (D-ITET Arton Cluster): It won't be possible to submit new jobs or view Slurm statistics. Already running jobs will not be affected.
  • Condor: Condor clients will be shut down the evening before to avoid running jobs during the migration.

Network disruption affecting several ISG.EE services

Status: Status/green.gif

2021-03-31 09:30

The configuration error was found. The configuration change will be deployed on 2021-04-01 around 06:15 and a short network of about 1min is expected.

2021-03-31 08:00
ID Networking team has rolled-back a deployed configuration, pending further investigation/analysis.
2021-03-31 07:30

There are currently disruption affecting a VPZ with servers managed by ISG.EE. Networking team of ID is investigating the issue. There are several ISG.EE services affected/malfunctioning due to this in particluar the FindYourData service.

login.ee.ethz.ch: downtime for server upgrade

Status: Status/green.gif

2021-03-11 06:30
Upgrade completed and service is up and running again.
2021-03-11 06:00
The server servicing login.ee.ethz.ch will be upgraded to a new OS version (Debian buster). During the time of the update logins might not be possible.

Planned project/ archive storage downtime and client reboot

Status: Status/green.gif

2020-07-11 12:00
Migration has been completed, all services are back to operational state.
2020-07-11 08:00
Migration started, services are shutdown
2020-07-11 8:00-12:00
Start of planned maintenance work. Project/ archive storage services (known under the names "ocean", "bluebay", "lagoon" and "benderstor") will not be available. ISG-managed Linux clients will be rebooted.

svn.ee.ethz.ch downtime for server upgrade

Status: Status/green.gif

2020-06-04 07:05
Webservices for managing SVN repositories are enabled.
2020-06-04 06:15

Systemupgrade is done and access to the SVN repositories via the svn and https transport protocols are back online.

2020-06-04 06:00
The server servicing the SVN repositories will be upgraded to a new operating system version. During this timeframe outages for access to the SVN repositories are expected.

European HPC cluster abuse

Status: Status/green.gif
Recently European HPC clusters have been attacked and abused for mining purposes. The D-ITET Slurm and SGE clusters have not been compromised. We are monitoring the situation closely.

2020-05 17 08:30
No successful login from known attacker IP addresses could be determined, none of the files indicating being compromised have been found on our file systems
2020-05-16 14:30
No unusal cluster job activity was observed

D-ITET Netscratch downtime for server upgrade

Status: Status/green.gif

2020-05-04 06:00
Server upgrade has been completed.
2020-05-04 06:00
The server servicing the D-ITET Netscratch service will be upgraded to a new operating system version. During this timeframe outages for the NFS service will be expected.

Network outage ETx router

Status: Status/green.gif

2020-04-07 05:30

There was an issue on the Router rou-etx. ID networking team trackled and solved the issue. There was about a 10min interuption for the ETx networking zone affecting almost all ISG.EE maintained systems.

login.ee.ethz.ch: Reboot for maintenance

Status: Status/green.gif

2020-04-06 05:35

System behind login.ee.ethz.ch has been rebootet for maintenance and increase available resources.

See the information on access D-ITET resources remotely. To distribute better the load user are encouraged to use the VPN service whenever possible.

itet-stor (FindYourData) Server maintenance: Reconfiguration of VM parameters

Status: Status/green.gif

2020-02-18 19:03
System again up and running.
2020-02-18 19:00

Scheduled downtime for the itet-stor/FindYourData service due to maintenance work on the underlying server.

itet-stor (FindYourData) Server migration: New operating system version

Status: Status/green.gif

2020-01-20 07:15

OS upgrade done, there were short interruptions to the itet-stor/FindYourData service.

2020-01-20 06:00

We will update the server servicing the FindYourData service from Debian jessie 8 to Debian stretch 9. There will be short downtimes accessing this service during the update.

Archived status reports

2010 2011 2012 2013 2014 2015 2016 2017 2018 2019

CategoryEDUC

Status (last edited 2023-10-16 11:24:17 by alders)