City Network CityCloud Status

All Systems Operational

About This Site

CityCloud status page for public regions

BUF1 Operational

Compute Operational

Storage Operational

Network Operational

API Operational

DX1 Operational

Compute Operational

Storage Operational

Network Operational

API Operational

FRA1 Operational

Compute Operational

Storage Operational

Network Operational

API Operational

KNA1 Operational

Compute Operational

Storage Operational

Object Storage Operational

Network Operational

API Operational

LON1 Operational

Compute Operational

Storage Operational

Network Operational

API Operational

STO2 Operational

Compute Operational

Storage Operational

Network Operational

API Operational

TKY1 Operational

Compute Operational

Storage Operational

Network Operational

API Operational

Online backup Operational

City Control Panel Operational

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

Scheduled Maintenance

Planned Maintenance region STO2 (Stockholm) for w.39 2021-09-27 08.00 CEST -- 2021-10-01 17.00 CEST Sep 27, 08:00 - Oct 1, 17:00 CEST

Dear Valued Customer,
City Network would like to inform you that we have planned maintenance as scheduled below.

Customer Intervention NOT required.
Region/datacenter: OpenStack region STO2 (Stockholm, Sweden)

Maintenance window starts: 2021-09-27 08.00 CEST
Maintenance window ends: 2021-10-01 17.00 CEST

Maintenance Scope:
Throughout the maintenance window, a number of improvements will be implemented - from functionality to security alignment.

The major part of the scope is an Operating System and Ceph upgrade, which will offer certain new functions but also address a list of known bugs.

Please note that there may be a shorter time frame, still to be defined, involving a brief disruption of running services. Our ambition is to carry out the majority of the maintenance window with little to no impact on running services.

All infrastructure services will be restarted in a controlled and timely manner, in order for proper security patching.

Expected customer impact:
- This maintenance will minimize the impact on customers virtual machines and cloud resources reachability while upgrading the network nodes. Controlled router failover will be done where you may experience network outage for a few seconds.
- During maintenance you may experience API issues or unavailability during new resource creation or editing already created ones. It is recommended to postpone operations with OpenStack API when possible to the time after maintenance.

Follow status at https://cnstatus.com for updates.

If you have further questions, please contact our support at support@citycloud.com.

Respectfully,
City Network
Posted on Sep 17, 07:21 CEST

Past Incidents

Sep 21, 2021

No incidents reported today.

Sep 20, 2021

No incidents reported.

Sep 19, 2021

No incidents reported.

Sep 18, 2021

No incidents reported.

Sep 17, 2021

Planned Maintenance region DX1 (Dubai) for w.37 2021-09-13 08.00 CEST -- 2021-09-17 17.00 CEST

Completed - The scheduled maintenance has been completed.
Sep 17, 17:00 CEST

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Sep 13, 08:00 CEST

Scheduled - Dear Valued Customer,
City Network would like to inform you that we have planned maintenance as scheduled below.

Customer Intervention NOT required.
Region/datacenter: OpenStack region DX1 (Dubai, UAE)

Maintenance window starts: 2021-09-13 08.00 CEST
Maintenance window ends: 2021-09-17 17.00 CEST

Maintenance Scope:
Throughout the maintenance window, a number of improvements will be implemented - from functionality to security alignment.

The major part of the scope is an Operating System and Ceph upgrade, which will offer certain new functions but also address a list of known bugs.

Please note that there may be a shorter time frame, still to be defined, involving a brief disruption of running services. Our ambition is to carry out the majority of the maintenance window with little to no impact on running services.

All infrastructure services will be restarted in a controlled and timely manner, in order for proper security patching.

Expected customer impact:
- This maintenance will minimize the impact on customers virtual machines and cloud resources reachability while upgrading the network nodes. Controlled router failover will be done where you may experience network outage for a few seconds.
- During maintenance you may experience API issues or unavailability during new resource creation or editing already created ones. It is recommended to postpone operations with OpenStack API when possible to the time after maintenance.

Follow status at https://cnstatus.com for updates.

If you have further questions, please contact our support at support@citycloud.com.

Respectfully,
City Network
Sep 7, 02:36 CEST

Sep 16, 2021

[STO2] Storage issue in STO2 region

Resolved - This incident is now resolved and postmortem will we published shortly.
Sep 16, 11:41 CEST

Update - We have now completed all synchronisation tasks to the new storage servers, and it will not be any more slow iops from the cluster.

Service is fully operational and we will update with the next steps in this change around 13:00.
Sep 16, 09:34 CEST

Update - UPDATE: We would like to inform you about the current status for the storage work in STO2.

We have now increased the speed of the synchronisation moving VM data from the old storage nodes into the new NVMe based storage nodes. The goal is to have this done before tomorrow morning to solve the initial problem with high latency and slow IOPS.

As long this synchronisation is ongoing it will be recurring slow IOPS that might affect VMs, but we are doing close monitoring to solve those as soon it occurs.
Sep 15, 14:05 CEST

Update - We would like to inform you about the current status for the storage work in STO2.

* Syncronization of VM images started yesterday evening (Tuesday, September 14th), and will be done continuously in the background.
Time. We will update with expected time frame for this sync soon.

Customer impact:

* Continuous improvements on the stability of our storage system as more and more data is offloaded onto the new storage.
* We sometimes experience recurring slow IOPS meanwhile the sync is ongoing.
Sep 15, 11:35 CEST

Monitoring - Slow ops are gone and we are continue with close monitoring of this issue
Sep 13, 13:13 CEST

Identified - Engineers investigating slow ops in the cluster at the moment.
Sep 13, 12:53 CEST

Monitoring - We are continuing to closely monitor this situation.
If you still experience issues with your VM, please contact support.
Sep 10, 12:15 CEST

Update - It's still work ongoing to solve this incident, but the situation is stable since the last update.

If you still experience any issues, hard reboot might help.
Please contact support if you have any issues.
Sep 10, 11:05 CEST

Update - Work is still ongoing.
At this time, we have managed to resolve most of the urgent problems and we notice that the storage system is returning to normal operation. We are still cautious and team is still working, but you may notice that your instances and volumes are functional again. We are closely monitoring and will continue to update.
Sep 10, 09:43 CEST

Update - The work with restoring the service is still ongoing all night.
Engineering team works together with external resources with specific actions at this moment.
We have no ETA at this time.

For affected VMs it will not help with hard reboot.

We will update this status 09:30
Sep 10, 08:41 CEST

Update - We continue to work on a resolution for this on-going issue.
Next update will be at 08:00 CEST
Sep 10, 00:26 CEST

Update - We continue to work on a resolution for this on-going issue.
Next update will be at 00:30 CEST
Sep 9, 23:29 CEST

Update - We continue to work on a resolution for this on-going issue.
Next update will be at 23:30 CEST
Sep 9, 22:36 CEST

Update - We continue to work on a resolution for this on-going issue.
The next update will be at 22:30 CEST
Sep 9, 21:28 CEST

Identified - We have identified another issue related to the new nodes in the cluster. Our storage team is currently working on it.
Sep 9, 17:28 CEST

Monitoring - We have now restored the service and keep monitoring this situation closely.

Affected VMs shall be working again, and if not do a hard reboot.
If you still see issues, please contact support.
Sep 9, 15:39 CEST

Update - This issue is related to the ongoing sync to new storage nodes, which one of the new server failed.
Storage team is working on restoring the service.
Sep 9, 15:20 CEST

Update - We are still working on this storage issue.
Rebooting servers will not help at this point.
Sep 9, 14:57 CEST

Identified - Slow operation mitigated, storage team is currently working on this issue.

If you still have issues with VM, please try do a hard reboot.
Sep 9, 12:00 CEST

Investigating - Our Engineering team is investigating storage issue in Stockholm region (STO2).

We apologize for the inconvenience and will share an update once we have more information.
Sep 9, 10:58 CEST

[STO2] Storage issue in STO2 region

Resolved - As we have 2 open incidents in parallell, we will close this one as the migration work is done.
Next update will be in the other incident post.
Sep 16, 09:36 CEST

Update - Dear customer,

We would like to inform you about the current status for the storage work in STO2.

* The process of populating the disks with data continues.
* Syncronization of volumes and VM images has started today (Wednesday, September 8th).
* Volume syncronization will take approximately a week (-> Monday, September 13th), and will be done continuously in the background.

Customer impact:

* Continuous improvements on the stability of our storage system as more and more data is offloaded onto the new storage.
* We do not expect the sync of data to cause any disruptions to your services.
Sep 8, 10:05 CEST

Update - Dear customer,

As you know, our STO2 region has had degraded performance during the last two months. We are resolving this issue by installing NVMe-based hardware. Here is an update on the current status.

* New disks are now installed and are production ready.
* The process of populating the disks with data has started 4/9.
* This process will take some time to complete, but we expect to get continuous improvements on the stability of our storage system as more and more data is offloaded onto the new storage.

We do not expect the sync of data to cause disruptions to your services.

If you have further questions, please reach out to the City Network support.
Kind regards,
City Network
Sep 7, 07:13 CEST

Update - We are still working with installation of the new storage nodes. This work will take a couple of days and including a test procedure before going into production.

We have also made some minor changes to and for the time being, these changes have improved the stability of the cluster.
Aug 18, 15:05 CEST

Update - Update:
Today Monday, we have continued the work to keep storage-related disruptions to a minimum.

In addition to monitoring and performing needed adjustments, we are also underway to rack-mount and prepare new storage nodes that will be added to the storage cluster.

The physical installation will be completed today, and the next step with software installation will continue tomorrow.

We will update the daily progress here on cnstatus.
Aug 16, 17:18 CEST

Update - We are continuing to monitor this incident closely and we expect to have more information to share regarding this incident on Monday.
Aug 15, 10:39 CEST

Monitoring - We are continuing with close monitoring of this incident of storage cluster. Meanwhile we are preparing next steps to permanent fix this issue.
Aug 13, 16:07 CEST

Update - Technicians are still working on this incident. We are currently working on decreasing the load on the storage cluster.
Aug 13, 10:30 CEST

Identified - The issue has been identified and a fix is being implemented.
Aug 13, 07:47 CEST

Investigating - We are currently investigating this issue.
Aug 13, 07:27 CEST

Sep 15, 2021

Sep 14, 2021

No incidents reported.

Sep 13, 2021

Sep 12, 2021

No incidents reported.

Sep 11, 2021

No incidents reported.

Sep 10, 2021

[General] Disturbance in outgoing e-mail to some destinations

Resolved - This incident has been resolved.
Sep 10, 16:13 CEST

Investigating - We are experiencing limited disturbances in outgoing e-mail to some destinations
Sep 7, 21:55 CEST

Sep 9, 2021

Sep 8, 2021

Sep 7, 2021