|
Size: 4663
Comment:
|
Size: 10127
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| #rev 2018-08-27 mreimers #rev 2020-08-31 alders |
|
| Line 13: | Line 16: |
| <<Anchor(2017-10-18-outage-etz-d-96-2)>> == Outage of Servers in Serverroom ETZ/D/96.2 == '''Status:''' {{attachment:Status/red.gif}} 2017-10-18 10:30:: Outage of most racks in ETZ/D/96.2 (cooling problem) . Most compute servers are offline. 2017-10-18 12:50:: The problem has been localized and repaired. We need to wait that the circuit is cooling down. 2017-10-18 14:00:: Cooling system is still not working correctly, we only selectively powered on a couple of compute machines. <<Anchor(2017-05-13-outage-etz-d-96-2)>> == Outage Servers in Serverroom ETZ/D/96.2 == |
<<Anchor(2022-07-16-planned-server-downtime)>> == Planned ISG.EE Mailsystem downtime because of IMAP server platform switch == |
| Line 29: | Line 20: |
| 2017-05-13 20:00:: Outage of some racks in ETZ/D/96.2. Several compute servers offline. 2017-05-13 23:59:: Most of the servers are back online. 2017-05-15 08:45:: Status of remaining servers verified. All back online. |
2022-07-16 11:00:: The update of our IMAP server to a new platform has completed successfully. The Mailsystem is available again. 2022-07-16 07:00:: Update of our IMAP server to a new platform has started. Therefore the mail system will not be available for our customers. 2022-07-16 07:00:: On 2022-07-16 between 07:00 and 12:00 ISG is going to migrate the IMAP Server to another system. During the migration there will be no access to the E-Mails for the following Domains: * ee.ethz.ch * vision.ee.ethz.ch * isi.ee.ethz.ch * mwe.ee.ethz.ch * mins.ee.ethz.ch * nari.ee.ethz.ch |
| Line 33: | Line 30: |
| <<Anchor(2017-03-24-cronbox-login-ssh-keys)>> == Cronbox/Login Server migration: new SSH host key == |
<<Anchor(2022-07-09-netscratch-maintenance)>> == NetScratch Server Filesystem Maintenance == |
| Line 38: | Line 34: |
| 2017-03-24 17:00:: The cronbox and login server has moved to a new host. A new SSH host key has been generated: {{{ 4096 MD5:fc:a8:00:5b:64:90:86:a1:fb:49:75:ef:55:58:90:b3 (RSA) 4096 SHA256:v48HAAAjr+avnPAESdQzazSriKYZeTGGtIPKfoE8Dg0 (RSA) }}} The SSH host key is as well listed on: https://people.ee.ethz.ch/ |
2022-07-10 10:55:: Fileysstem for [[Services/NetScratch|NetScratch]] is online and in read-write mode. 2022-07-10 09:30:: First run of checks are completed and we are proceeding with the next steps to put the filesystem again online. 2022-07-09 13:00:: The [[Services/NetScratch|NetScratch]] filesystem will be put into read-only mode for maintenance. |
| Line 45: | Line 38: |
| Remember:: '''Always''' verify a fingerprint of a SSH host key before accepting it. | <<Anchor(2022-06-16-exchange-support)>> == Unreachable ee.ethz.ch Email recipients over ID Exchange Mailserver == '''Status:''' {{attachment:Status/green.gif}} |
| Line 47: | Line 42: |
| <<Anchor(2017-01-07-Mailsystem migration)>> | 2022-06-16 16:45:: Configuration issue has been resolved. 2022-06-16 15:00:: Emails with ee.ethz.ch recipients sent over the ID Exchange Server do not reach destination. ID Exchange Admins are working on fixing the problem. |
| Line 49: | Line 45: |
| == EE Mailsystem migration == '''STATUS:''' {{attachment:Status/green.gif}} '''Mailsystem up''' |
<<Anchor(2021-11-01-home-maintenance)>> == HOME Server maintenance to repair fileystem inconsistence == '''Status:''' {{attachment:Status/green.gif}} |
| Line 52: | Line 49: |
| 2017-01-08 15:00:: The new mailsystem is now started. In case of unattended problems we will stop it again to prevent data loss and to analyze the problem. | 2021-11-01 06:30:: System back online and HOME directories are again accessible for all D-ITET user 2021-11-01 05:50:: HOME Server will be put offline to start a repair of a filesystem inconsistence |
| Line 54: | Line 52: |
| 2017-01-07 24:00:: Not all testcases could be performed. We now plan to enable the new system about noon. | <<Anchor(2021-07-05-linux-printing)>> == Linux printing affected by PrintNightmare vulnerability patch == '''Status:''' {{attachment:Status/green.gif}} 2021-07-05 13:00:: ID resolved the issue 2021-07-05 09:41:: Workaround: Use [[Printing#Platform-independent_printing| platform-independent printing]] 2021-07-05 09:41:: Authentification for printing fails. Ticket at ID servicedesk opened. |
| Line 56: | Line 59: |
| 2017-01-07 20:45:: Old Mailserver Configuration migrated, starting the mailserver testing | <<Anchor(2021-04-27-itetmaster01-update)>> == Downtime various D-ITET services for server maintenance == '''Status:''' {{attachment:Status/green.gif}} |
| Line 58: | Line 63: |
| 2017-01-07 14:00:: User mailbox data migrated, starting mailserver configuration migration | 2021-04-27 08:30:: Condor is back online, all services restored. 2021-04-27 08:15:: Matrix/Element Chat services back online. 2021-04-27 08:00:: Database upgrade done and online. 2021-04-27 07:30:: Slurm services are back online. 2021-04-27 07:00:: Base system has been upgraded, main database services in progress of upgrade. 2021-04-27 06:00:: On 2021-04-27 between 06:00 and 08:30 ISG is going to update a server providing access to various D-ITET services. During the migration the following services will be affected and offline: * Matrix/Element Chat services (the instances will be unavailable) * IFA/Control Website: Access to the IFA database is blocked * Slurm (D-ITET Arton Cluster): It won't be possible to submit new jobs or view Slurm statistics. Already running jobs will not be affected. * Condor: Condor clients will be shut down the evening before to avoid running jobs during the migration. |
| Line 60: | Line 74: |
| 2017-01-07 07:00:: All mail services are stopped. Mailbox data copy started. | <<Anchor(2021-03-31-network-disruption)>> == Network disruption affecting several ISG.EE services == '''Status:''' {{attachment:Status/green.gif}} |
| Line 62: | Line 78: |
| <<Anchor(2016-09-12-network-outage)>> | 2021-03-31 09:30:: The configuration error was found. The configuration change will be deployed on '''2021-04-01 around 06:15''' and a short network of about 1min is expected. 2021-03-31 08:00:: ID Networking team has rolled-back a deployed configuration, pending further investigation/analysis. 2021-03-31 07:30:: There are currently disruption affecting a VPZ with servers managed by ISG.EE. Networking team of ID is investigating the issue. There are several ISG.EE services affected/malfunctioning due to this in particluar the FindYourData service. |
| Line 64: | Line 82: |
| == Networkoutage ETH == '''STATUS:''' {{attachment:Status/green.gif}} |
<<Anchor(2021-03-11-mira-upgrade)>> == login.ee.ethz.ch: downtime for server upgrade == '''Status:''' {{attachment:Status/green.gif}} |
| Line 67: | Line 86: |
| 2016-02-09 08:20:: ETH wide network outage due to hardware problems for the firewall infrastructure. In any case, please reboot your computer before continue. | 2021-03-11 06:30:: Upgrade completed and service is up and running again. 2021-03-11 06:00:: The server servicing login.ee.ethz.ch will be upgraded to a new OS version (Debian buster). During the time of the update logins might not be possible. |
| Line 69: | Line 89: |
| 2016-02-09 12:35:: Network is back online and services are being recovered. Due to the hardware failure 53 network zones were affected. The problem got localized and resolved. | <<Anchor(2020-07-11-storage-downtime)>> == Planned project/ archive storage downtime and client reboot == '''Status:''' {{attachment:Status/green.gif}} |
| Line 71: | Line 93: |
| 2016-02-09 14:25:: Our systems should be all back to normal. In case you experience any problem please contact support via mailto:support@ee.ethz.ch. | 2020-07-11 12:00:: Migration has been completed, all services are back to operational state. |
| Line 73: | Line 95: |
| <<Anchor(2016-02-10-maintenance-polaris)>> | 2020-07-11 08:00:: Migration started, services are shutdown |
| Line 75: | Line 97: |
| == Maintenance login.ee.ethz.ch and cronbox.ee.ethz.ch service == '''STATUS:''' {{attachment:Status/green.gif}} |
2020-07-11 8:00-12:00:: Start of planned maintenance work. Project/ archive storage services (known under the names "ocean", "bluebay", "lagoon" and "benderstor") will not be available. ISG-managed Linux clients will be rebooted. |
| Line 78: | Line 99: |
| 2016-02-10: 06:05:: The server for the [[Services/Cronjob|cronbox]] and login service is currently beeing updated from Debian Wheezy to Debian Jessie. The services will be temporarly unavailable. | |
| Line 80: | Line 100: |
| 2016-02-10: 12:00:: Server update is done. | <<Anchor(2020-06-04-svnsrv-upgrade)>> == svn.ee.ethz.ch downtime for server upgrade == '''Status:''' {{attachment:Status/green.gif}} 2020-06-04 07:05:: Webservices for managing SVN repositories are enabled. 2020-06-04 06:15:: Systemupgrade is done and access to the SVN repositories via the `svn` and `https` transport protocols are back online. 2020-06-04 06:00:: The server servicing the SVN repositories will be upgraded to a new operating system version. During this timeframe outages for access to the SVN repositories are expected. <<Anchor(2020-05-17-cluster-abuse)>> == European HPC cluster abuse == '''Status:''' {{attachment:Status/green.gif}}<<BR>> Recently European HPC clusters have been attacked and abused for mining purposes. The D-ITET Slurm and SGE clusters have not been compromised. We are monitoring the situation closely. 2020-05 17 08:30:: No successful login from known attacker IP addresses could be determined, none of the files indicating being compromised have been found on our file systems 2020-05-16 14:30:: No unusal cluster job activity was observed <<Anchor(2020-05-04-itetnas04-upgrade)>> == D-ITET Netscratch downtime for server upgrade == '''Status:''' {{attachment:Status/green.gif}} 2020-05-04 06:00:: Server upgrade has been completed. 2020-05-04 06:00:: The server servicing the D-ITET Netscratch service will be upgraded to a new operating system version. During this timeframe outages for the NFS service will be expected. <<Anchor(2020-04-07-network-interuption)>> == Network outage ETx router == '''Status:''' {{attachment:Status/green.gif}} 2020-04-07 05:30:: There was an issue on the Router `rou-etx`. ID networking team trackled and solved the issue. There was about a 10min interuption for the ETx networking zone affecting almost all ISG.EE maintained systems. <<Anchor(2020-04-06-mira-maintenance)>> == login.ee.ethz.ch: Reboot for maintenance == '''Status:''' {{attachment:Status/green.gif}} 2020-04-06 05:35:: System behind `login.ee.ethz.ch` has been rebootet for maintenance and increase available resources. See the [[RemoteAccess|information on access D-ITET resources remotely]]. To distribute better the load user are encouraged to use the VPN service whenever possible. <<Anchor(2020-02-18-nostro-maintenance)>> == itet-stor (FindYourData) Server maintenance: Reconfiguration of VM parameters == '''Status:''' {{attachment:Status/green.gif}} 2020-02-18 19:03:: System again up and running. 2020-02-18 19:00:: Scheduled downtime for the [[Workstations/FindYourData|itet-stor/FindYourData service]] due to maintenance work on the underlying server. <<Anchor(2020-01-20-nostro-os-upgrade)>> == itet-stor (FindYourData) Server migration: New operating system version == '''Status:''' {{attachment:Status/green.gif}} 2020-01-20 07:15:: OS upgrade done, there were short interruptions to the [[Workstations/FindYourData|itet-stor/FindYourData service]]. 2020-01-20 06:00:: We will update the server servicing the [[Workstations/FindYourData|FindYourData service]] from Debian jessie 8 to Debian stretch 9. There will be short downtimes accessing this service during the update. |
| Line 84: | Line 152: |
| [[Status/Archive/2010|2010]] [[Status/Archive/2011|2011]] [[Status/Archive/2012|2012]] [[Status/Archive/2013|2013]] [[Status/Archive/2014|2014]] |
|
| Line 85: | Line 158: |
| [[Status/Archive/2014|2014]] [[Status/Archive/2013|2013]] [[Status/Archive/2012|2012]] [[Status/Archive/2011|2011]] [[Status/Archive/2010|2010]] |
[[Status/Archive/2016|2016]] [[Status/Archive/2017|2017]] [[Status/Archive/2018|2018]] [[Status/Archive/2019|2019]] |
General Informations
This page lists announcements and status messages for IT services managed by ISG.EE.
For notifications and announcements of central IT services managed by ID, please visit https://www.ethz.ch/services/de/it-services/service-desk.html
For a detailed status overview of central IT services managed by ID, please visit https://ueberwachung.ethz.ch
Status-Key |
|
|
Resolved |
|
Still working but with some errors |
|
Pending |
Current status reports
Planned ISG.EE Mailsystem downtime because of IMAP server platform switch
Status: 
- 2022-07-16 11:00
- The update of our IMAP server to a new platform has completed successfully. The Mailsystem is available again.
- 2022-07-16 07:00
- Update of our IMAP server to a new platform has started. Therefore the mail system will not be available for our customers.
- 2022-07-16 07:00
- On 2022-07-16 between 07:00 and 12:00 ISG is going to migrate the IMAP Server to another system. During the migration there will be no access to the E-Mails for the following Domains:
- ee.ethz.ch
- vision.ee.ethz.ch
- isi.ee.ethz.ch
- mwe.ee.ethz.ch
- mins.ee.ethz.ch
- nari.ee.ethz.ch
NetScratch Server Filesystem Maintenance
Status:
- 2022-07-10 10:55
Fileysstem for NetScratch is online and in read-write mode.
- 2022-07-10 09:30
- First run of checks are completed and we are proceeding with the next steps to put the filesystem again online.
- 2022-07-09 13:00
The NetScratch filesystem will be put into read-only mode for maintenance.
Unreachable ee.ethz.ch Email recipients over ID Exchange Mailserver
Status:
- 2022-06-16 16:45
- Configuration issue has been resolved.
- 2022-06-16 15:00
- Emails with ee.ethz.ch recipients sent over the ID Exchange Server do not reach destination. ID Exchange Admins are working on fixing the problem.
HOME Server maintenance to repair fileystem inconsistence
Status:
- 2021-11-01 06:30
- System back online and HOME directories are again accessible for all D-ITET user
- 2021-11-01 05:50
- HOME Server will be put offline to start a repair of a filesystem inconsistence
Linux printing affected by PrintNightmare vulnerability patch
Status:
- 2021-07-05 13:00
- ID resolved the issue
- 2021-07-05 09:41
Workaround: Use platform-independent printing
- 2021-07-05 09:41
- Authentification for printing fails. Ticket at ID servicedesk opened.
Downtime various D-ITET services for server maintenance
Status:
- 2021-04-27 08:30
- Condor is back online, all services restored.
- 2021-04-27 08:15
- Matrix/Element Chat services back online.
- 2021-04-27 08:00
- Database upgrade done and online.
- 2021-04-27 07:30
- Slurm services are back online.
- 2021-04-27 07:00
- Base system has been upgraded, main database services in progress of upgrade.
- 2021-04-27 06:00
- On 2021-04-27 between 06:00 and 08:30 ISG is going to update a server providing access to various D-ITET services. During the migration the following services will be affected and offline:
- Matrix/Element Chat services (the instances will be unavailable)
- IFA/Control Website: Access to the IFA database is blocked
- Slurm (D-ITET Arton Cluster): It won't be possible to submit new jobs or view Slurm statistics. Already running jobs will not be affected.
- Condor: Condor clients will be shut down the evening before to avoid running jobs during the migration.
Network disruption affecting several ISG.EE services
Status:
- 2021-03-31 09:30
The configuration error was found. The configuration change will be deployed on 2021-04-01 around 06:15 and a short network of about 1min is expected.
- 2021-03-31 08:00
- ID Networking team has rolled-back a deployed configuration, pending further investigation/analysis.
- 2021-03-31 07:30
There are currently disruption affecting a VPZ with servers managed by ISG.EE. Networking team of ID is investigating the issue. There are several ISG.EE services affected/malfunctioning due to this in particluar the FindYourData service.
login.ee.ethz.ch: downtime for server upgrade
Status:
- 2021-03-11 06:30
- Upgrade completed and service is up and running again.
- 2021-03-11 06:00
- The server servicing login.ee.ethz.ch will be upgraded to a new OS version (Debian buster). During the time of the update logins might not be possible.
Planned project/ archive storage downtime and client reboot
Status:
- 2020-07-11 12:00
- Migration has been completed, all services are back to operational state.
- 2020-07-11 08:00
- Migration started, services are shutdown
- 2020-07-11 8:00-12:00
- Start of planned maintenance work. Project/ archive storage services (known under the names "ocean", "bluebay", "lagoon" and "benderstor") will not be available. ISG-managed Linux clients will be rebooted.
svn.ee.ethz.ch downtime for server upgrade
Status:
- 2020-06-04 07:05
- Webservices for managing SVN repositories are enabled.
- 2020-06-04 06:15
Systemupgrade is done and access to the SVN repositories via the svn and https transport protocols are back online.
- 2020-06-04 06:00
- The server servicing the SVN repositories will be upgraded to a new operating system version. During this timeframe outages for access to the SVN repositories are expected.
European HPC cluster abuse
Status:
Recently European HPC clusters have been attacked and abused for mining purposes. The D-ITET Slurm and SGE clusters have not been compromised. We are monitoring the situation closely.
- 2020-05 17 08:30
- No successful login from known attacker IP addresses could be determined, none of the files indicating being compromised have been found on our file systems
- 2020-05-16 14:30
- No unusal cluster job activity was observed
D-ITET Netscratch downtime for server upgrade
Status:
- 2020-05-04 06:00
- Server upgrade has been completed.
- 2020-05-04 06:00
- The server servicing the D-ITET Netscratch service will be upgraded to a new operating system version. During this timeframe outages for the NFS service will be expected.
Network outage ETx router
Status:
- 2020-04-07 05:30
There was an issue on the Router rou-etx. ID networking team trackled and solved the issue. There was about a 10min interuption for the ETx networking zone affecting almost all ISG.EE maintained systems.
login.ee.ethz.ch: Reboot for maintenance
Status:
- 2020-04-06 05:35
System behind login.ee.ethz.ch has been rebootet for maintenance and increase available resources.
See the information on access D-ITET resources remotely. To distribute better the load user are encouraged to use the VPN service whenever possible.
itet-stor (FindYourData) Server maintenance: Reconfiguration of VM parameters
Status:
- 2020-02-18 19:03
- System again up and running.
- 2020-02-18 19:00
Scheduled downtime for the itet-stor/FindYourData service due to maintenance work on the underlying server.
itet-stor (FindYourData) Server migration: New operating system version
Status:
- 2020-01-20 07:15
OS upgrade done, there were short interruptions to the itet-stor/FindYourData service.
- 2020-01-20 06:00
We will update the server servicing the FindYourData service from Debian jessie 8 to Debian stretch 9. There will be short downtimes accessing this service during the update.
Archived status reports
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019