How to enable windows and NTLM authentication on WCF

Objective

This document will explain various combinations of IIS and WCF Ntlm/Windows authentication settings

What is difference between NTLM and WINDOWS authentication in WCF?

Windows authentication = authentication in NTLM + authentication in Active Directory
NTLM authentication = authentication in only NTLM

IIS configuration

For all scenario IIS is configured for Windows authentication. What I mean is Windows Authentication is enabled and all other authentication is disabled.
Navigate to below path to open ApplicationHost.Config file of IIS.

C:\Windows\System32\inetsrv\config\applicationHost.config

Binding used in WCF service

For all scenario basicHttpBinding being used for WCF service.

Scenario #1
Default setting for IIS Applicationhost.Config is

<windowsAuthentication enabled="false">
                    <providers>
                        <add value="Negotiate" />
                        <add value="NTLM" />
                    </providers>
                </windowsAuthentication>

If IIS APP.Config file is having default setting, then we can have any authentication for WCF service corresponding IIS configured; WCF service will run as expected without any error.

Note: SharePoint is running as expected

Browsers Behavior with default settings

1. IE 7.0 is not asking for authentication
2. Fire Fox 3.5.6 is asking user to authenticate

3. Safari 4.0.4 is asking user to authenticate 

Scenario #2
If IIS Applicationhost.Config File setting has been modified as below, where forcefully Windows authentication is enabled for Kerberos then we have to modify service with Windows authentication.  

<windowsAuthentication enabled="true">
                    <providers>
                        <add value="Negotiate" />
                        <!--<add value="NTLM" />-->
                    </providers>
                </windowsAuthentication>

WCF configuration setting for Windows authentication should be 

<basicHttpBinding>
        <binding name="BasicHttpBinding">
          <security mode ="TransportCredentialOnly">
            <transport clientCredentialType ="Windows"/>
          </security>
        </binding>
      </basicHttpBinding>

Browsers Behavior with default settings

1. IE 7.0 is not asking for authentication
2. Fire Fox 3.5.6 is asking user to authenticate

3. Safari 4.0.4 is asking user to authenticate

Scenario #3
If IIS Applicationhost.Config File setting has been modified as below, where forcefully Windows authentication is enabled for NTLM

<windowsAuthentication enabled="true">
                    <providers>
                        <!--<add value="Negotiate" />-->
                    </providers>
 </windowsAuthentication>

 And we go with Windows authentication for the service, we will get below error 

<basicHttpBinding>
        <binding name="BasicHttpBinding">
          <security mode ="TransportCredentialOnly">
            <transport clientCredentialType ="Windows"/>
          </security>
        </binding>
      </basicHttpBinding>

So to remove above error, WCF configuration setting for should be modified for the NTLM authentication. 

<basicHttpBinding>
        <binding name="BasicHttpBinding">
          <security mode ="TransportCredentialOnly">
            <transport clientCredentialType ="Ntlm"/>
          </security>
        </binding>
      </basicHttpBinding>

 Note: SharePoint is running as expected

Browsers Behavior with default settings
4. IE 7.0 is not asking for authentication
5. Fire Fox 3.5.6 is asking user to authenticate

6. Safari 4.0.4 is asking user to authenticate

So,
1. If we have ApplicationHost.Config of IIS configured as default, we can have either of Ntlm or Windows authentications for WCF service.
2. If we have ApplicationHost.Config of IIS configured as Ntlm, we can have only Ntlm authentication for WCF service.
3. If we have ApplicationHost.Config of IIS configured as Windows, we can have only Windows authentication for WCF service.