[Date Prev][Date Next] [Chronological] [Thread] [Top]

Small tip to speed up large imports with ldapadd



Hi,

as I am in the situation where I have to do large imports with ldapadd, I wanted to share a little tip that is saving me a lot of time and may be helpful to others too.

As I don't care for data consistency *while* importing (if the import should abort, I can alway restart the process), I'm stopping the regular slapd and run it manually under the control of 'eatmydata', a little tool that LD_PRELOAD's all filesystem sync operations to noops. This has increased the speed of my import tests by several orders of magnitude.

Here's my script (slightly obfuscated):

systemctl stop slapd
rm -f /srv/ldap/data/*
systemd-run --unit=slapd-fastimport eatmydata /usr/sbin/slapd -d 0 -s 0 -h ldapi:/// -g openldap -u openldap -F /etc/ldap/slapd.d
sleep 2
ldapadd -Y EXTERNAL -H ldapi:/// -Q -c -S errorlog -f data.ldif
systemctl stop slapd-fastimport
sleep 2
systemctl start slapd
grep Error: errorlog | sort | uniq -c | sort -n

Best regards
Karsten