Yes I am.

@bathory
I believe that zone file is for the localhost zones, I added it anyway to be safe.


Still, if I remove the forwarder option, i get:
[root@ns1 data]# service named restart
Stopping named: . [ OK ]
Starting named: [ OK ]
[root@ns1 data]# dig @127.0.0.1 google.com

; <<>> DiG 9.7.3-P3-RedHat-9.7.3-8.P3.el6_2.2 <<>> @127.0.0.1 google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached


BUT if I add the +trace...
[root@ns1 data]# dig @127.0.0.1 google.com +trace

; <<>> DiG 9.7.3-P3-RedHat-9.7.3-8.P3.el6_2.2 <<>> @127.0.0.1 google.com +trace
; (1 server found)
;; global options: +cmd
. 518351 IN NS e.root-servers.net.
. 518351 IN NS b.root-servers.net.
. 518351 IN NS c.root-servers.net.
. 518351 IN NS k.root-servers.net.
. 518351 IN NS l.root-servers.net.
. 518351 IN NS j.root-servers.net.
. 518351 IN NS f.root-servers.net.
. 518351 IN NS g.root-servers.net.
. 518351 IN NS h.root-servers.net.
. 518351 IN NS d.root-servers.net.
. 518351 IN NS i.root-servers.net.
. 518351 IN NS m.root-servers.net.
. 518351 IN NS a.root-servers.net.
;; Received 320 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms

com. 172800 IN NS a.gtld-servers.net.
com. 172800 IN NS b.gtld-servers.net.
com. 172800 IN NS c.gtld-servers.net.
com. 172800 IN NS d.gtld-servers.net.
com. 172800 IN NS e.gtld-servers.net.
com. 172800 IN NS f.gtld-servers.net.
com. 172800 IN NS g.gtld-servers.net.
com. 172800 IN NS h.gtld-servers.net.
com. 172800 IN NS i.gtld-servers.net.
com. 172800 IN NS j.gtld-servers.net.
com. 172800 IN NS k.gtld-servers.net.
com. 172800 IN NS l.gtld-servers.net.
com. 172800 IN NS m.gtld-servers.net.
;; Received 488 bytes from 193.0.14.129#53(k.root-servers.net) in 109 ms

google.com. 172800 IN NS ns2.google.com.
google.com. 172800 IN NS ns1.google.com.
google.com. 172800 IN NS ns3.google.com.
google.com. 172800 IN NS ns4.google.com.
;; Received 164 bytes from 192.31.80.30#53(d.gtld-servers.net) in 259 ms

google.com. 300 IN A 74.125.115.147
google.com. 300 IN A 74.125.115.99
google.com. 300 IN A 74.125.115.103
google.com. 300 IN A 74.125.115.105
google.com. 300 IN A 74.125.115.106
google.com. 300 IN A 74.125.115.104
;; Received 124 bytes from 216.239.38.10#53(ns4.google.com) in 8 ms

Huh, does not make sense at all
Run the following to see what happens:

It works because it's cached, when I try something thats not in cache it works but takes REALLY long.


[root@ns1 data]# dig @127.0.0.1 hammer.com +tcp

; <<>> DiG 9.7.3-P3-RedHat-9.7.3-8.P3.el6_2.2 <<>> @127.0.0.1 hammer.com +tcp
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44857
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 0

;; QUESTION SECTION:
;hammer.com. IN A

;; ANSWER SECTION:
hammer.com. 10800 IN A 12.130.6.150

;; AUTHORITY SECTION:
hammer.com. 10800 IN NS ns2.empirix.com.
hammer.com. 10800 IN NS ns1.empirix.com.

;; Query time: 4358 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Jan 23 14:06:40 2012
;; MSG SIZE rcvd: 88

It's odd, just tried sledge.com because I know there is no way of it being in cache.


The first time failed, the second time with +tcp worked fine, then it answered fine without the +tcp because it was in cache.

So here I tried again with something I know will not be in cache (hair.com). The first time it times out, but the second time it works fine, I presume its in cache now.

[root@ns1 data]# dig +short @127.0.0.1 hair.com
;; connection timed out; no servers could be reached
[root@ns1 data]# dig +short @127.0.0.1 hair.com
67.207.67.108

what in named.run file?

Just as a test, I did it again, works fine the second time. It's like it doesn't want to answer the first time but once the record is in cache it all a go.

[root@ns1 data]# dig +short @127.0.0.1 paper.com
;; connection timed out; no servers could be reached
[root@ns1 data]# dig +short @127.0.0.1 paper.com
205.157.102.45

At least it caching, lol
The fact that is taking a long time could be an issue. Maybe dig timesout before getting the answer. You may try this:
Of course you need to flush the cache, or lookup different domains every time

Nothing is written to the file when the query fails. even at debug level 2

That seems to work, but why would it be taking so long?

[root@ns1 data]# dig @127.0.0.1 tools.com +time=10

; <<>> DiG 9.7.3-P3-RedHat-9.7.3-8.P3.el6_2.2 <<>> @127.0.0.1 tools.com +time=10
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7141
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 0

;; QUESTION SECTION:
;tools.com. IN A

;; ANSWER SECTION:
tools.com. 900 IN A 208.76.1.70

;; AUTHORITY SECTION:
tools.com. 900 IN NS ns2.idealab.com.
tools.com. 900 IN NS ns1.idealab.com.

;; Query time: 9858 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Jan 23 14:48:52 2012
;; MSG SIZE rcvd: 87

I can only guess that these query times are due to some short of networking errors. A router dropping fragmented udp packets or flaky nic/cables could be a reason:
Try to comment out:
and see if it helps.

Also what is the query time if you use a third party dns like google:

No difference with or without edns-udp-size option.

Query times seem normal using thrid party dns:

[root@ns1 ~]# dig @8.8.8.8 google.com

; <<>> DiG 9.7.3-P3-RedHat-9.7.3-8.P3.el6_2.2 <<>> @8.8.8.8 google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42198
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;google.com. IN A

;; ANSWER SECTION:
google.com. 300 IN A 74.125.113.105
google.com. 300 IN A 74.125.113.147
google.com. 300 IN A 74.125.113.104
google.com. 300 IN A 74.125.113.103
google.com. 300 IN A 74.125.113.99
google.com. 300 IN A 74.125.113.106

;; Query time: 38 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Jan 23 15:12:35 2012
;; MSG SIZE rcvd: 124

It was expected...

Unless you're using the root servers.
Anyway as I'm running out of ideas, try to blacklist the ipv6 module if it's loaded into the kernel (I saw you're running named with the -4 option for ipv4) and do some queries to see what you get (try plain dig and dig -4)

Great minds think alike. I just did that and still the same issue. I added the following to /etc/sysconfig/network:

NETWORKING_IPV6=off

Also added the following to modprobe:
alias net-pf-10 off
alias ipv6 off


Still bad, actually its worse, it doesnt seem to be answering even after caching now.

So I figured out why

dig @127.0.0.1 google.com +trace

works and

dig @127.0.0.1 google.com

doesnt.

I had another DNS server in my /etc/resolv.conf file and it was using that somehow.

Doesn't make sense. You query your dns at localhost in both cases (i.e. you connect to 127.0.0.1 on port 53). A nameserver from /etc/resolv.conf is used when you are not using @<server> in dig.