Web lists-archives.com

Re: [Samba] [Patches] AD Database corruption after upgrade from <= 4.6 to 4.7 (bug #13228)






Am 23.01.2018 um 00:05 schrieb Achim Gottinger via samba:


Am 22.01.2018 um 22:12 schrieb Ralph Böhme:
On Mon, Jan 22, 2018 at 05:24:44PM +0100, Achim Gottinger via samba wrote:
Am 22.01.2018 um 10:49 schrieb Stefan Metzmacher via samba:
Also DO NOT repair the following errors with samba-tool dbcheck!
"Remove duplicate links in attribute"
and
"ERROR: orphaned backlink"
as this removes the ability to repair the database
in the next round of patches!

I had this error after upgrading from 4.7.3 to 4.7.4 and used samba-tool
dbcheck --clean to get rid of them.
Replication is still working. What kind of unrepairable corruption can i
expect now?
see the bug report for details, this can eg cause loss of group memberships or
generally speaking loss of linked-attributes.

The only remede is comparing all objects for differences in linked-attributes
and restore overwritten forward-links from now dangling backlinks.

We're currently also working on an improvement to dbcheck so it can detect such corruption and fix it, but this will only work if you did *not* run dbcheck
--fix on the affected database.

-slow

Thank you for the infos!

I took a look at my notes.

I updates from 4.6.8 to 4.7.3 on 25.11.2017.

Back then i found error like this all related to siteList before the update.

ERROR: no target object found for GUID component for siteList in object CN=DEFAULTIPSITELINK,CN=IP,CN=Inter-Site Transports,CN=Sites,CN=Configuration,DC=samba-list,DC=loc - <GUID=d4f41749a1595a43871ab1d72f24fe6b>;<RMD_ADDTIME=130015150890000000>;<RMD_CHANGETIME=130015150890000000>;<RMD_FLAGS=0>;<RMD_INVOCID=af301252bb781543b57dbd7cb773d46f>;<RMD_LOCAL_USN=4762>;<RMD_ORIGINATING_USN=4762>;<RMD_VERSION=0>;CN=Test,CN=Sites,CN=Configuration,DC=samba-list,DC=loc
Not removing dangling forward link
ERROR: no target object found for GUID component for siteList in object CN=DEFAULTIPSITELINK,CN=IP,CN=Inter-Site Transports,CN=Sites,CN=Configuration,DC=samba-list,DC=loc - <GUID=596bd8ae9e8bc94eab99ad3c12e22132>;<RMD_ADDTIME=130739077850000000>;<RMD_CHANGETIME=130739077850000000>;<RMD_FLAGS=0>;<RMD_INVOCID=af301252bb781543b57dbd7cb773d46f>;<RMD_LOCAL_USN=453494>;<RMD_ORIGINATING_USN=453494>;<RMD_VERSION=0>;CN=Grafing,CN=Sites,CN=Configuration,DC=samba-list,DC=loc
Not removing dangling forward link
Please use --fix to fix these errors

I updated to 4.7.3 and back then edited the ldb file and deleted the links to old expunged sites whom did no longer exist with the given GUID.

#~ldbedit -e nano -H /varLib/samba/private/sam.ldb.d/CN=CONFIGURATION,DC=SAMBA-LIST,DC=LOC.ldb
#~samba-tool dbcheck --reindexdb

An month later on 26.12.2017 at about 5 am a few groups suddenly had an messed up member list, some users showed up twice some where missing. I fixed it by deleting and recreating the affected groups, erros where deceted but could not be fixed with samba-tool dbcheck for the affected users/groups. Also deleting those twice listed users did not work. Thought it was caused by an forced kill -9 to the samba service from an cron job at that time.

I maintain two separate networks with samba addc's and this only happend at one of these networks, both run samba adds's on 5 and 7 sites. My thombstoneLifetime is set to 30 days ab both networks.

On 12.01.2018 i updated from 4.7.3 to 4.7.4. dbcheck ran clean before the update but showed a few dangling forward errors whom i then fixed with dbcheck --fix. Till now no group corruption had happened. I can think of restoring an backup from 11.01.2018 to an vm with 4.7.4 here to inspect the errors from dbcheck again and maybe recreate these deleted links again. As far as i remember the errors where different on the ad's of whom i run a dozend, so this may become complicated.

I assume the errors caused by the 4.6.8->4.7.3 update happened 30 days later and I fixed these by recreating the affected groups. But i'm unsure if the fixes i ran after the 4.7.3->4.7.4 update may cause another corruption on 11.02.2018. dbcheck --cross-ncs did not find any errors before the update only afterwards. So the question is will the fixing of the newly detected  errors (by dbcheck version 4.7.4) cause issues or are these unrelated.

Achim
Did a few tests to answer my own questions.

Restored an backup from 23.12.2017 to an VM. At this point only one Computer Group had been comprimised. I used the -kcc workaround to prevent an immediate tomstone expunge.

With samba 4.7.3 i get these results:

#~samba-tool dbcheck
Checking 556 objects
ERROR: orphaned backlink attribute 'memberOf' in CN=WIN7-G-ADMIN,CN=Computers,DC=domain,DC=loc for link member in CN=CG Grafing Laden,CN=Computers,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=WIN7-G-BUERO1,CN=Computers,DC=domain,DC=loc for link member in CN=CG Grafing Laden,CN=Computers,DC=domain,DC=loc
Not removing orphaned backlink memberOf
Please use --fix to fix these errors
Checked 556 objects (2 errors)

The errors can not be fixed with --fix.

With 4.7.4 the errors look different

#~samba-tool dbcheck
Checking 556 objects
ERROR: orphaned backlink attribute 'memberOf' in CN=WIN7-G-ADMIN,CN=Computers,DC=domain,DC=loc for link member in CN=CG Grafing Laden,CN=Computers,DC=domain,DC=loc
Not removing orphaned backlink memberOf
WARNING: Link (back) mismatch for 'memberOf' (1) on 'CN=WIN7-G-BUERO1,CN=Computers,DC=domain,DC=loc' to 'member' (2) on 'CN=CG Grafing Laden,CN=Computers,DC=domain,DC=loc' ERROR: Duplicate link values for attribute 'member' in 'CN=CG Grafing Laden,CN=Computers,DC=domain,DC=loc' Duplicate link '<GUID=2eb2053a-19b3-4f0e-beaf-7c64fe577855>;<RMD_ADDTIME=130755196240000000>;<RMD_CHANGETIME=130755196240000000>;<RMD_FLAGS=0>;<RMD_INVOCID=521230af-78bb-4315-b57d-bd7cb773d46f>;<RMD_LOCAL_USN=457188>;<RMD_ORIGINATING_USN=457188>;<RMD_VERSION=0>;<SID=S-1-5-21-1446910239-1605792192-310601177-9714>;CN=WIN7-G-BUERO1,CN=Computers,DC=domain,DC=loc' Correct   link '<GUID=2eb2053a-19b3-4f0e-beaf-7c64fe577855>;<RMD_ADDTIME=130755196240000000>;<RMD_CHANGETIME=130755196240000000>;<RMD_FLAGS=0>;<RMD_INVOCID=521230af-78bb-4315-b57d-bd7cb773d46f>;<RMD_LOCAL_USN=457188>;<RMD_ORIGINATING_USN=457188>;<RMD_VERSION=0>;<SID=S-1-5-21-1446910239-1605792192-310601177-9714>;CN=WIN7-G-BUERO1,CN=Computers,DC=domain,DC=loc'
Not removing duplicate links in attribute 'member'
Please use --fix to fix these errors
Checked 556 objects (2 errors)

The i forced the tombstone expunge

#~samba-tool domain tombstones expunge

Afterwards a few more groups where compromised.

samba-tool dbcheck
Checking 556 objects
ERROR: orphaned backlink attribute 'memberOf' in CN=haar,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=haar,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=fhe,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=fhe,CN=Users,DC=domain,DC=loc for link member in CN=Email Einlagen Intern,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=an,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=an,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=an,CN=Users,DC=domain,DC=loc for link member in CN=Email Einlagen Intern,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=lr,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=lho,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=lho,CN=Users,DC=domain,DC=loc for link member in CN=Email Einlagen Intern,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=poing,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=poing,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=marktschwaben,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=marktschwaben,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=rs,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=rs,CN=Users,DC=domain,DC=loc for link member in CN=Email Einlagen Intern,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=rr,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=rr,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=mb,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=mb,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=mb,CN=Users,DC=domain,DC=loc for link member in CN=Email Einlagen Intern,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=fs,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=fs,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=sw,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=sw,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=gd,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=tib,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=tib,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=bf,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=ke,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=ke,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=tb,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=tb,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=mg,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=mg,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=fg,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=fg,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=hg,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=hg,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=ag,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=ag,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=jg,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=jg,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=sf,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=sf,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=schwabing,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=ug,CN=Users,DC=domain,DC=loc for link member in CN=Email Einlagen Intern,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=ug,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=alg,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=alg,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=WIN7-G-BUERO1,CN=Computers,DC=domain,DC=loc for link member in CN=CG Grafing Laden,CN=Computers,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=rg,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=rg,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=reitz,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=fh,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=fh,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=sk,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=sk,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=lk,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=lk,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=grafing,CN=Users,DC=domain,DC=loc for link member in CN=DG Email,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
ERROR: orphaned backlink attribute 'memberOf' in CN=grafing,CN=Users,DC=domain,DC=loc for link member in CN=Email Mitarbeiter,CN=Users,DC=domain,DC=loc
Not removing orphaned backlink memberOf
Please use --fix to fix these errors
Checked 556 objects (62 errors)

Back then i had to delete and recreate this groups to fix the issues.

With 4.7.4 and the patch "fix linked attribute corruption on databases with" running "samba-tool domain tombstones expunge" does not cause the corruption of the above groups.

Afterwards i tested an backup from 11.01.2018 (before i upgraded from 4.7.3 to 4.7.4). (Un)fortunately i can not reproduce the dbcheck errors i had seen on the production system. As far as i remeber these where small site related issues and not caused by bug #13228. Also did another tombstone expunge which did not remove any object and

So i assume with the groups issues already fixed and the patch applied to 4.7.4 I'm save from future issues by this bug.

Thanks for the info's and the patch

Sincere,
Achim~












--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba