PDA

View Full Version : Troubleshooting Active Directory issues



chiefnz
22-08-2017, 01:51 PM
I have a client who is having some issues with Active Directory.

There are 3 specific issues which I'm looking into though I suspect if I fix one it is likely to resolve the others or at the very least shed light on how to fix the other issues.

So a bit of background (will try to keep it as short as possible)

Two years back they paid Fujitsu to demote the then Primary DC and left the secondary DC running.
Now 2 years later and several administrators later the current admin is having issues and she's asked me to give her a hand.

Issue 1

So after a week of tinkering the first anomaly I found was that after running netdom query fsmo the result showed that the decomissioned PDC was still holding all the FSMO roles. From this I assumed Fujitsu did not demote the DC in the correct way and further evidence of this was that there were still references to the old PDC in Active Directory Sites and Services as well as AD users and computers and DNS. So the first thing I did was perform a FSMO seizure of all the roles to the current domain controller (it is also the only domain controller). I cleaned up DNS and also removed any references to the "dead" domain controller.

The only thing I'm not sure about now in regards to this is that there is an entry in Active Directory Sites and Services for the Exchange server? Is this normal? I have confirmed that the server does not have the AD role installed so from that I'm assuming it shouldn't be in AD Sites and Services? I'm not an Exchange Expert so if anyone can shed light on this that would be great. It's Exchange 2013 by the way.

Now onto the other 2 issues we've been seeing on the domain;

Issue 2

The Domain Admins group keeps getting changed (currentlty every 3 hours) and it has some weird permissions on it which is as follows;

Members = Domain Admins
Member of = Domain Admins, Administrators

This is obviously incorrect;

They should be;

Members = list of all accounts which are granted domain admin privilege - in this instance the client has specific accounts admins have to use for domain admin access only.

Member of = Administrators and RODC replication password denied - this is the standard out of the box config.

Now here's the strange thing... when the domain admins group "reverts" to the defunct state above I can log into the DC with my Domain admin account but cannot login to ANY other domain computer whether it be a server or desktop. I fix this by changing the domain admins group members and members of tab to what they should be as per above. So this one has me stumped and I have to do this every 3 hours.

Issue 3

User accounts are constantly getting locked, we don't know why, it can happen to any user multiple times (and has happened to multiple users) and there appears to be no pattern in terms of timing and frequency. We have ruled out outdated passwords on users' other devices such as tablets and mobile phones and it doesn't appear that the the Default Domain Password Policy has anything to do with this either. So again i'm stumped here....

My gut is telling me it has something to so with the way the old DC was forced off the network. Clearly it was not done correctly if at all and AD is kind of just revertingg to its "tombstone" state which is set at intervals of 180 minutes.

Part of me wants to suggest commissioning a new DC setting everything up the way it should be and then somehow add it to the domain WITHOUT it receiving replication from the current DC. I'm not sure if this is even possible or advisable but it's safe to say the current state if AD is not good and it is also creating issues in the share permissions area. I'm trying to fix this problem with a new standalone File server... currently the DC is also a file server for all the client's user data.... they have everything in one basket.

Today I fnished tidying up AD sites and services as well as AD users and computers by removing all the references to the old DC (including DNS). Hopefully, this will have helped but I will only know in the morning when I try to login with my domain admin account.

Apologies for the novel but I couldn't explain it any shorter.

I'd appreciate any advice or tips and tricks on how to resolve this one... ideally, without having to commission a new server.

Cheers,

CYaBro
22-08-2017, 02:27 PM
Had a quick read through but haven't had a chance to look at it fully yet.

Couple of things, what server OS?
For issue 2 could it be a group policy doing that?

chiefnz
22-08-2017, 04:53 PM
The DC is running Server 2016.

I haven't actually looked at group policy as being the culprit here. I will check and post findings.

Why would you even use Group Policy that way? Surely, that wouldl create more of an administrative overhead?

chiefnz
26-08-2017, 02:42 AM
So no changes after I made the modifications. The Domain Admins group is still reverting to its defunct state.

Some other things I found, the current DC was pointing to itself for DNS via its assigned static IP, I don't think that should be an issue but usually when a DC is the DNS server the DNS setting for that DC is usually 127.0.0.1 so I made that change. There were issues with some new machines not being able to locate an LDAP server but since making the above change this seems to have now gone away.

Another thing I'm thinking about is raising the domain's functional level from Server 2003 to at least Server 2008 R2, not sure what effect this will have but the benefits should largely outway the current state.

I need to find a way of setting up a new DC which will take over all the FSMO roles WITHOUT replicating the current state from the existing DC or at least force the current DC to replicate from the new one rather than the other way around.... Is that even possible? Turning the current DC nto an RODC and then set the new DC as the primary holding the GC?

chiefnz
26-08-2017, 10:06 AM
Not much progress has been made but I did raise the domain functional level to Server 2008 R2 and there were no issues. I also found an AD Health Check Script here (http://carlwebster.com/downloads/download-info/active-directory-health-check/) which I ran on the DC and apart from a few obvious no-nos (mainly around account password expiry etc) the most significant item was according to the script no domain controller was found and there had been no contact with a domain controller for the last 3 months?

I'm not sure what this all means or what my options will be to remediate.

What I'm thinking about doing is the following;


Setup a new DC running Server 2012 R2 or Server 2016
Join the new DC to the domain
Transfer all the FSMO roles to the new DC
Make the new DC the GC
Remove GC from the existing server
Disconnect the current DC from the network and see if any issue arise - authenticaton issues etc.


If all goes well, demote the old DC properly and rebuild it from scratch and then rejoin to the domain.

wainuitech
26-08-2017, 10:06 AM
Dont know if the following article is of any use or not https://support.microsoft.com/en-us/help/324801/how-to-view-and-transfer-fsmo-roles-in-windows-server-2003 or https://www.petri.com/transferring_fsmo_roles

If changing a server, why use an outdated, (Server2008) better to go with Server 2012 or 2016. Found that the other day trying to get a backup software for a few PC's to auto backup. Server 2008 had so many things no longer working it became a pain, so used server 2012.

chiefnz
26-08-2017, 03:08 PM
Wainui are you refering to the Domain Functional Level or Server OS?

The functional level is set to 2008 R2 and the current server is running 2012 R2.

The new server will have 2012 R2 or 2016 installed on it.

I hav transferred all the FSMO roles to the current server and have verified they have been changed to the current server.

There are are few things in the first article which I didn't exeplicitly do (according to the process detailed) but I'm sure that the transfer has been completed.
I will check on these items as per the article and post back

Thanks

wainuitech
26-08-2017, 03:41 PM
I was only referring to the comment :)
I'm thinking about is raising the domain's functional level from Server 2003 to at least Server 2008 R2 If a server upgrade is on the cards, better to go with at least 2012, as Server 2008 R2 is coming up upon end of supported life, where as 2012 is more up to date and stable. The only downside is 2012 has the W8 style interface, where as 2016 is closer to W10.

chiefnz
29-08-2017, 11:13 AM
Update:

Added a secondary DC running Server 2008 R2.

Transfer of all the FSMO roles was successful apart from the schema master.
I tried to register the Schmmgmt.dll library by running regsvr32 schmmgmt.dll but got an error message saying the module was registered successfully but the schema snap-in could not be loaded.

When I tried to add the Schema snap-in in wasn't there.

Next I forced the schema to the new server use FSMO seizure which was successful.

I then let all the AD elements replicate to the new server including DNS.
Everything appeared to be ok but then I started getting errors from domain computers not being able to locate a domain controller. I also had trouble logging in with domain accounts as well as authenticating domain accounts... I kept getting errors saying a domain controller could not be located.

At this point I didn't want to risk any more potential damage so then started moving things back to the original DC.
FSMO roles transfer completed successfully and then I had to delete the second server from DNS as well as AD sites and Services etc.

Once this was done I was able to log back into domain computers and there were no longer issues with domain level authentication.

Really at a loss here, part of me thinks a reboot may be a good start but could just be a waste of time. The next reboot is scheduled for the night of 31 August so just going to wait until then and then start troubleshooting again.

The client is considering implementing a new domain design etc. which is probably not a bad idea but it's a lot of work and obviously a lot of things could go wrong in terms of migrating existing users and computers to the new domain. There is also the issue of the current DC being the file server as well so all those shares and permissions would have to be migrated to a new server... which is on order still. So alot of decisions to make and a few design documents to draw up. It's going to be a long couple of weeks ahead.

linw
29-08-2017, 03:10 PM
We feel your pain. A helluva responsibility on a live system.

Hopefully, something will go your way.

CYaBro
29-08-2017, 04:39 PM
We feel your pain. A helluva responsibility on a live system.

Hopefully, something will go your way.

Yea agree here, can you get a full backup image to take away and mount in a VM or something to play with and see if you can get it fixed?

chiefnz
30-08-2017, 04:13 AM
They have no spare infrastructure, it's an overseas government department with very little funding. Took me 5 weeks to convince them to buy 3 more servers.

I may just blow the second DC away I just built and install Hyper-V on there and try to get a backup copy of the current DC restored to a VM.

Whilst doing everything from scratch will be a lot of effort at least I know it will be done properly, to be honest anything would be better than the bowl of noodle soup they have now.

I will continue to troubleshoot on the VM once I have the backup restored.

Cheers,

chiefnz
02-09-2017, 02:12 PM
I have found that the Exchange server is a member of the "Domain Controllers" group?

Not sure if this is meant to be here as the Exchange server doesn't have the AD role installed. Is this normal?

I tried to remove it from the "Domain Controllers" group but this returned a message saying that this is the server's "Primary Group" and that I have to select a new Primary Group before I can remove it.

We are running Exchange 2013.

Can any AD or Exchange gurus provide some advice on this?

Thanks,

Alex B
06-09-2017, 10:55 AM
Before you go any further make sure you have good backups, use Windows Backup to backup AD system state at least.

Consider opening a support case with MS. I think the cost is about $500, but that can work out cheap with AD issues like you're looking at, they are rarely straightforward to resolve.

I would raise the domain/forest functional level to at least 2008 R2. There hasn't been any improvements since that level, so no real point going past it that I'm aware of.

You may also want to upgrade from FRS to DFSR Replication: https://blogs.technet.microsoft.com/filecab/2014/06/25/streamlined-migration-of-frs-to-dfsr-sysvol/#quick

As for the Exchange Box being in Domain Controllers. IIRC if a DC is properly demoted, it is automatically removed from the Domain Controllers OU.

How are the logs looking in event viewer? Especially around AD and DNS. AD is very dependent on DNS being healthy.

When you changed to a new DC, did you update DHCP to point the endpoints to the new DC for DNS?

chiefnz
20-09-2017, 03:10 PM
Sorry for the delay.

I've been through the logs and here is what I've found... which isn't much really as you will see from the log entry types:

For the DNS, Directory Services and DHCP Server logs:

They all have entries similar to this; they only vary in number, most of them occur in the Directory services log and i'm pretty sure that's because there is only one domain controller, from what I can tell there are many errors relating to AD replication etc. which is to be expected.


The description for Event ID xxxx from source xxxx cannot be found. Either the component that raises this event is not installed
on your local computer or the installation.

The System log is flooded with the following error:


No suitable default server credential exists on this system. This will prevent server applications that expect to make use of the
system default credentials from accepting SSL.

I've had to stop working on the client's system for now as they have run out of funding.

I do have copies of the log files so could potentially answer a few queries if any arise from this post.

Cheers,