All Systems Operational

About This Site

CityCloud status page for public regions

BUF1 ? Operational
Compute Operational
Storage Operational
Network Operational
API Operational
DX1 ? Operational
Compute Operational
Storage Operational
Network Operational
API Operational
FRA1 ? Operational
Compute Operational
Storage Operational
Network Operational
API Operational
KNA1 ? Operational
Compute Operational
Storage Operational
Object Storage ? Operational
Network Operational
API Operational
LON1 ? Operational
Compute Operational
Storage Operational
Network Operational
API Operational
STO2 ? Operational
Compute Operational
Storage Operational
Network Operational
API Operational
TKY1 ? Operational
Compute Operational
Storage Operational
Network Operational
API Operational
Online backup ? Operational
City Control Panel ? Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Scheduled Maintenance
Dear Valued Customer,
City Network would like to inform you that we have planned maintenance as scheduled below.


Customer Intervention NOT required.
Region/datacenter: OpenStack region STO2 (Stockholm, Sweden)

Maintenance window starts: 2021-09-27 08.00 CEST
Maintenance window ends: 2021-10-01 17.00 CEST

Maintenance Scope:
Throughout the maintenance window, a number of improvements will be implemented - from functionality to security alignment.

The major part of the scope is an Operating System and Ceph upgrade, which will offer certain new functions but also address a list of known bugs.

Please note that there may be a shorter time frame, still to be defined, involving a brief disruption of running services. Our ambition is to carry out the majority of the maintenance window with little to no impact on running services.

All infrastructure services will be restarted in a controlled and timely manner, in order for proper security patching.

Expected customer impact:
- This maintenance will minimize the impact on customers virtual machines and cloud resources reachability while upgrading the network nodes. Controlled router failover will be done where you may experience network outage for a few seconds.
- During maintenance you may experience API issues or unavailability during new resource creation or editing already created ones. It is recommended to postpone operations with OpenStack API when possible to the time after maintenance.

Follow status at https://cnstatus.com for updates.

If you have further questions, please contact our support at support@citycloud.com.

Respectfully,
City Network
Posted on Sep 17, 07:21 CEST
Past Incidents
Sep 21, 2021

No incidents reported today.

Sep 20, 2021

No incidents reported.

Sep 19, 2021

No incidents reported.

Sep 18, 2021

No incidents reported.

Sep 17, 2021
Completed - The scheduled maintenance has been completed.
Sep 17, 17:00 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Sep 13, 08:00 CEST
Scheduled - Dear Valued Customer,
City Network would like to inform you that we have planned maintenance as scheduled below.


Customer Intervention NOT required.
Region/datacenter: OpenStack region DX1 (Dubai, UAE)

Maintenance window starts: 2021-09-13 08.00 CEST
Maintenance window ends: 2021-09-17 17.00 CEST

Maintenance Scope:
Throughout the maintenance window, a number of improvements will be implemented - from functionality to security alignment.

The major part of the scope is an Operating System and Ceph upgrade, which will offer certain new functions but also address a list of known bugs.

Please note that there may be a shorter time frame, still to be defined, involving a brief disruption of running services. Our ambition is to carry out the majority of the maintenance window with little to no impact on running services.

All infrastructure services will be restarted in a controlled and timely manner, in order for proper security patching.

Expected customer impact:
- This maintenance will minimize the impact on customers virtual machines and cloud resources reachability while upgrading the network nodes. Controlled router failover will be done where you may experience network outage for a few seconds.
- During maintenance you may experience API issues or unavailability during new resource creation or editing already created ones. It is recommended to postpone operations with OpenStack API when possible to the time after maintenance.

Follow status at https://cnstatus.com for updates.

If you have further questions, please contact our support at support@citycloud.com.

Respectfully,
City Network
Sep 7, 02:36 CEST
Sep 16, 2021
Resolved - This incident is now resolved and postmortem will we published shortly.
Sep 16, 11:41 CEST
Update - We have now completed all synchronisation tasks to the new storage servers, and it will not be any more slow iops from the cluster.

Service is fully operational and we will update with the next steps in this change around 13:00.
Sep 16, 09:34 CEST
Update - UPDATE: We would like to inform you about the current status for the storage work in STO2.

We have now increased the speed of the synchronisation moving VM data from the old storage nodes into the new NVMe based storage nodes. The goal is to have this done before tomorrow morning to solve the initial problem with high latency and slow IOPS.

As long this synchronisation is ongoing it will be recurring slow IOPS that might affect VMs, but we are doing close monitoring to solve those as soon it occurs.
Sep 15, 14:05 CEST
Update - We would like to inform you about the current status for the storage work in STO2.

* Syncronization of VM images started yesterday evening (Tuesday, September 14th), and will be done continuously in the background.
Time. We will update with expected time frame for this sync soon.

Customer impact:

* Continuous improvements on the stability of our storage system as more and more data is offloaded onto the new storage.
* We sometimes experience recurring slow IOPS meanwhile the sync is ongoing.
Sep 15, 11:35 CEST
Monitoring - Slow ops are gone and we are continue with close monitoring of this issue
Sep 13, 13:13 CEST
Identified - Engineers investigating slow ops in the cluster at the moment.
Sep 13, 12:53 CEST
Monitoring - We are continuing to closely monitor this situation.
If you still experience issues with your VM, please contact support.
Sep 10, 12:15 CEST
Update - It's still work ongoing to solve this incident, but the situation is stable since the last update.

If you still experience any issues, hard reboot might help.
Please contact support if you have any issues.
Sep 10, 11:05 CEST
Update - Work is still ongoing.
At this time, we have managed to resolve most of the urgent problems and we notice that the storage system is returning to normal operation. We are still cautious and team is still working, but you may notice that your instances and volumes are functional again. We are closely monitoring and will continue to update.
Sep 10, 09:43 CEST
Update - The work with restoring the service is still ongoing all night.
Engineering team works together with external resources with specific actions at this moment.
We have no ETA at this time.

For affected VMs it will not help with hard reboot.

We will update this status 09:30
Sep 10, 08:41 CEST
Update - The work with restoring the service is still ongoing all night.
Engineering team works together with external resources with specific actions at this moment.
We have no ETA at this time.

For affected VMs it will not help with hard reboot.

We will update this status 09:30
Sep 10, 08:40 CEST
Update - We continue to work on a resolution for this on-going issue.
Next update will be at 08:00 CEST
Sep 10, 00:26 CEST
Update - We continue to work on a resolution for this on-going issue.
Next update will be at 00:30 CEST
Sep 9, 23:29 CEST
Update - We continue to work on a resolution for this on-going issue.
Next update will be at 23:30 CEST
Sep 9, 22:36 CEST
Update - We continue to work on a resolution for this on-going issue.
The next update will be at 22:30 CEST
Sep 9, 21:28 CEST
Identified - We have identified another issue related to the new nodes in the cluster. Our storage team is currently working on it.
Sep 9, 17:28 CEST
Monitoring - We have now restored the service and keep monitoring this situation closely.

Affected VMs shall be working again, and if not do a hard reboot.
If you still see issues, please contact support.
Sep 9, 15:39 CEST
Update - This issue is related to the ongoing sync to new storage nodes, which one of the new server failed.
Storage team is working on restoring the service.
Sep 9, 15:20 CEST
Update - We are still working on this storage issue.
Rebooting servers will not help at this point.
Sep 9, 14:57 CEST
Identified - Slow operation mitigated, storage team is currently working on this issue.

If you still have issues with VM, please try do a hard reboot.
Sep 9, 12:00 CEST
Investigating - Our Engineering team is investigating storage issue in Stockholm region (STO2).

We apologize for the inconvenience and will share an update once we have more information.
Sep 9, 10:58 CEST
Resolved - As we have 2 open incidents in parallell, we will close this one as the migration work is done.
Next update will be in the other incident post.
Sep 16, 09:36 CEST
Update - Dear customer,

We would like to inform you about the current status for the storage work in STO2.

* The process of populating the disks with data continues.
* Syncronization of volumes and VM images has started today (Wednesday, September 8th).
* Volume syncronization will take approximately a week (-> Monday, September 13th), and will be done continuously in the background.

Customer impact:

* Continuous improvements on the stability of our storage system as more and more data is offloaded onto the new storage.
* We do not expect the sync of data to cause any disruptions to your services.
Sep 8, 10:05 CEST
Update - Dear customer,

As you know, our STO2 region has had degraded performance during the last two months. We are resolving this issue by installing NVMe-based hardware. Here is an update on the current status.

* New disks are now installed and are production ready.
* The process of populating the disks with data has started 4/9.
* This process will take some time to complete, but we expect to get continuous improvements on the stability of our storage system as more and more data is offloaded onto the new storage.

We do not expect the sync of data to cause disruptions to your services.

If you have further questions, please reach out to the City Network support.
Kind regards,
City Network
Sep 7, 07:13 CEST
Update - We are still working with installation of the new storage nodes. This work will take a couple of days and including a test procedure before going into production.

We have also made some minor changes to and for the time being, these changes have improved the stability of the cluster.
Aug 18, 15:05 CEST
Update - Update:
Today Monday, we have continued the work to keep storage-related disruptions to a minimum.

In addition to monitoring and performing needed adjustments, we are also underway to rack-mount and prepare new storage nodes that will be added to the storage cluster.

The physical installation will be completed today, and the next step with software installation will continue tomorrow.

We will update the daily progress here on cnstatus.
Aug 16, 17:18 CEST
Update - We are continuing to monitor this incident closely and we expect to have more information to share regarding this incident on Monday.
Aug 15, 10:39 CEST
Monitoring - We are continuing with close monitoring of this incident of storage cluster. Meanwhile we are preparing next steps to permanent fix this issue.
Aug 13, 16:07 CEST
Update - Technicians are still working on this incident. We are currently working on decreasing the load on the storage cluster.
Aug 13, 10:30 CEST
Identified - The issue has been identified and a fix is being implemented.
Aug 13, 07:47 CEST
Investigating - We are currently investigating this issue.
Aug 13, 07:27 CEST
Sep 15, 2021
Sep 14, 2021

No incidents reported.

Sep 13, 2021
Sep 12, 2021

No incidents reported.

Sep 11, 2021

No incidents reported.

Sep 10, 2021
Resolved - This incident has been resolved.
Sep 10, 16:13 CEST
Investigating - We are experiencing limited disturbances in outgoing e-mail to some destinations
Sep 7, 21:55 CEST
Sep 9, 2021
Sep 8, 2021
Sep 7, 2021