Forum Discussion

Scott_Griffin's avatar
Scott_Griffin
Icon for Novice I rankNovice I
2 years ago
Solved

We have 3 arrays at this datacenter in which we run VM's leveraging vVols

 In all 3 upgrades we saw VM's that reported 200-85k ms worth of latency while the controllers were being upgraded/rebooted. From a reporting aspect within Pure1 we see the majority of latency at the VM level and not at the array level. At the array level we see Latency hitting 3-14 ms. As we have seen this across multiple systems it feels like it's either an issue with Purity or vCenter/VASA configuration. A few things that we have looked at/checked: • We have checked/confirmed that vCenter is running on a different cluster/storage platform • We have checked/confirmed that all VM's do not have any QOS enabled via SPBMs • We do see that the Storage Provider versions are reporting different between the arrays controllers. However, it's our understanding that the reported version within vCenter does not update and only reports at the time the provider is registered. • Reported Provider Versions: 1.1.1 to 2.0.0 • vCenter version: 7.0.3.01200 • Purity Upgrade: 6.3.8 to 6.3.13 Due to the lengthy pause the VM's are seeing it kind of feels like they are not failing/rolling over to the secondary controller/provider. Our next set of array upgrades are the week of Sept 11th so we have about 2 weeks to figure out what's happening. If anybody has any suggestions or things to try we are open to all suggestions.

vmware
  • Scott_Griffin's avatar
    Scott_Griffin
    2 years ago
    Hello Cody, Pure: CS0401273 VMware: SR 23461443009 Few more specifics about the last Purity upgrade. We would see latency ramp up on the host but then ramp back down via ESXTOP reporting. This would occur for about 3-5 ESXTOP refreshes. From external monitoring systems we would see latency being reported for 1-3 polling cycles (overall less than 1 minute). Latency levels reported varied but usually less 30ms. Again, some were higher. While these levels might be considered high they were much better than the previous Purity upgrades in which the same level of reporting usually lasted for 10+ minutes and we say ~500ms or higher levels.

8 Replies

  • Cody_Hosterman's avatar
    Cody_Hosterman
    Puritan
    2 years ago
    Engineering is in discussion on this right now--we are digging in. I will make sure this is root caused asap
  • Scott_Griffin's avatar
    Scott_Griffin
    Icon for Novice I rankNovice I
    2 years ago
    Little bit of follow-up... After digging into things with Support we have yet to find a "silver bullet". We have seen some similar latency increases (significant but not massive) when performing controller reboots. However, the significant increases haven't been enough to trigger events at the application layer. We will be performing another round of Purity upgrades (6.3.13 to 6.4.10) tomorrow with Support on the call just in case we experience the issue again and they can witness first hand what's happening.
  • mjacobson's avatar
    mjacobson
    Novice I
    2 years ago
    We are doing some upgrades soon, and will keep an eye out for this.
  • Scott_Griffin's avatar
    Scott_Griffin
    Icon for Novice I rankNovice I
    2 years ago
    Hello All, Wanted to provide an update to this adventure We recently did another round of Purity upgrades on our fleet to 6.5.3 and we had a much better experience. We still had some latency on our vVol VM's but nothing like before (previous round saw ~500ms for about 10-15 minutes; This round saw less than 100ms for maybe 1-2 minutes). Still working through support channels to confirm things.
  • Cody_Hosterman's avatar
    Cody_Hosterman
    Puritan
    2 years ago
    i want to get a readout on this from engineering--this is still very troublesome to me that there is any disruption like this. whats the latest ticket number on this?
  • Scott_Griffin's avatar
    Scott_Griffin
    Icon for Novice I rankNovice I
    2 years ago
    Hello Cody, Pure: CS0401273 VMware: SR 23461443009 Few more specifics about the last Purity upgrade. We would see latency ramp up on the host but then ramp back down via ESXTOP reporting. This would occur for about 3-5 ESXTOP refreshes. From external monitoring systems we would see latency being reported for 1-3 polling cycles (overall less than 1 minute). Latency levels reported varied but usually less 30ms. Again, some were higher. While these levels might be considered high they were much better than the previous Purity upgrades in which the same level of reporting usually lasted for 10+ minutes and we say ~500ms or higher levels.