pam_ldap passwd entry when using kerberos

I have both libpam-ldap and libpam-krb5 installed because I am using Kerberos for authentication here. The implication is that I am not using passwords in ldap.

When I try to change my password I get this in the auth.log:

Apr 1 23:21:30 foo passwd[4927]: pam_unix(passwd:chauthtok): user "brian" does not exist in /etc/passwd
Apr 1 23:21:38 foo passwd[4927]: pam_krb5(passwd:chauthtok): user brian changed Kerberos password
Apr 1 23:21:38 foo passwd[4927]: pam_unix(passwd:chauthtok): user "brian" does not exist in /etc/passwd
Apr 1 23:21:38 foo passwd[4927]: pam_ldap: ldap_modify_s Insufficient access

The tty where I changed my password shows:

$ passwd
Current Kerberos password:
Enter new Kerberos password:
Retype new Kerberos password:
LDAP password information update failed: Insufficient access
passwd: Permission denied
passwd: password unchanged

Presumably this is all because PAM is trying to manipulate passwords in LDAP but they just don't/shouldn't exist there.

My /etc/pam.d/common-passwd looks like this:

# here are the per-package modules (the "Primary" block)
password requisite pam_krb5.so minimum_uid=1000
password [success=2 default=ignore] pam_unix.so obscure use_authtok try_first_pass sha512
password [success=1 user_unknown=ignore default=die] pam_ldap.so use_authtok try_first_pass
# here's the fallback if no module succeeds
password requisite pam_deny.so
# prime the stack with a positive return value if there isn't one already;
# this avoids us returning an error just because nothing sets a success code
# since the modules above will each just jump around
password required pam_permit.so
# and here are more per-package modules (the "Additional" block)
password optional pam_gnome_keyring.so
password optional pam_ecryptfs.so
# end of pam-auth-update config

Does the configuration need to allow for whatever failure is causing the "ldap_modify_s Insufficient access" in the case where LDAP is not being used for authentication?

ProblemType: Bug
DistroRelease: Ubuntu 11.04
Package: libpam-ldap 184-8.4ubuntu1
ProcVersionSignature: Ubuntu 2.6.38-13.56-generic 2.6.38.8
Uname: Linux 2.6.38-13-generic i686
Architecture: i386
Date: Sun Apr 1 23:37:37 2012
ProcEnviron:
 LANGUAGE=en_CA:en
 PATH=(custom, no user)
 LANG=en_CA
 LC_MESSAGES=en_CA.UTF-8
 SHELL=/bin/bash
SourcePackage: libpam-ldap
UpgradeStatus: No upgrade log present (probably fresh install)