Hmm. The plot thickens.

I thought that while waiting to hear back on this one I'd try testing with the rest of the many test pcaps at https://archive.wrccdc.org/pcaps/2018/ but with that detect-MHR line commented out, since if those tests all worked it would help confirm detect-MHR was a root cause and that other egregious runtime problems aren't lurking in there.

It didn't take long to find https://archive.wrccdc.org/pcaps/2018/wrccdc.2018-03-23.010356000000000.pcap.gz also triggers a segfault. So maybe that has a different root cause or maybe the root cause is the same and commenting out detect-MHR just somehow gave more headroom before causing a segfault.

Anyway, just figured I'd throw that on the pile in case it helps with isolating the problem and verifying a fix.

Well detect-mhr specifically is potentially very relevant.

That script does one thing: look for downloads of a certain mime type: https://github.com/zeek/zeek/blob/master/scripts/policy/frameworks/files/detect-MHR.zeek#L18-L24

Then for any matches it looks up the file hash using a TXT DNS query via lookup_hostname_txt to fmt("%s.malware.hash.cymru.com", hash); inside of a when statement.

when with DNS queries has been a bit buggy even on linux.

There's only one other script that does this that might be triggering on a random pcap run: ssh/interesting-hostnames.zeek.

Does that other pcap crash if you disable both the MHR and the ssh script?

Thanks @JustinAzoff! Indeed, your theory panned out. I just commented out the SSH interesting-hostnames line in my local.zeek on my test Windows VM and was able to successfully run Zeek against that pcap dozen of times without triggering the segfault.

(I also just realized that I pasted the wrong pcap into my prior comment, but I just went back and fixed it. Should have been https://archive.wrccdc.org/pcaps/2018/wrccdc.2018-03-23.010356000000000.pcap.gz.)

So does knowing the problem reproduces reliably on Windows actually make it any easier to fix? 😬 😄

Does this script make things crash?

event zeek_init()
{
	when ( local result = lookup_hostname_txt("example.com"))
	{
		print result;
	}
}

We have some tests for dns, but I think they all run using the fake 'test' resolver, so if it's the real resolver that has the issue this could have been missed.

Assuming I did it right, it doesn't seem to cause a crash. I put your script into a file justin.zeek and ran it repeatedly. Output seems to toggle. No crash.

Administrator@EC2AMAZ-NVIH7LJ MINGW64 ~/zeek/build (master)
$ cat justin.zeek
event zeek_init()
{
        when ( local result = lookup_hostname_txt("example.com"))
        {
                print result;
        }
}

Administrator@EC2AMAZ-NVIH7LJ MINGW64 ~/zeek/build (master)
$ src/zeek justin.zeek
wgyf8z8cgvm2qmxpnbnldrcltvk4xqfn

Administrator@EC2AMAZ-NVIH7LJ MINGW64 ~/zeek/build (master)
$ src/zeek justin.zeek
v=spf1 -all

Administrator@EC2AMAZ-NVIH7LJ MINGW64 ~/zeek/build (master)
$ src/zeek justin.zeek
wgyf8z8cgvm2qmxpnbnldrcltvk4xqfn

Administrator@EC2AMAZ-NVIH7LJ MINGW64 ~/zeek/build (master)
$ src/zeek justin.zeek
v=spf1 -all

Administrator@EC2AMAZ-NVIH7LJ MINGW64 ~/zeek/build (master)
$ src/zeek justin.zeek
wgyf8z8cgvm2qmxpnbnldrcltvk4xqfn

Administrator@EC2AMAZ-NVIH7LJ MINGW64 ~/zeek/build (master)
$ src/zeek justin.zeek
v=spf1 -all

Administrator@EC2AMAZ-NVIH7LJ MINGW64 ~/zeek/build (master)
$ src/zeek justin.zeek
wgyf8z8cgvm2qmxpnbnldrcltvk4xqfn

It's definitely possible there's some bug in c-ares on Windows. I'll try to get a backtrace out of a Windows build tomorrow for this.

Got a more useful backtrace finally (after I actually read your repro steps above 🤦🏼‍♂️ ):

Exception thrown: read access violation.
filt was 0xFFFFFFFFFFFFFFD7.

>	zeek.exe!windows_kevent_copyout(kqueue * kq, int nready, kevent * eventlist, int nevents) Line 142	C
 	zeek.exe!kevent(int kqfd, const kevent * changelist, int nchanges, kevent * eventlist, int nevents, const timespec * timeout) Line 451	C
 	zeek.exe!zeek::iosource::Manager::Poll(std::vector<zeek::iosource::Manager::ReadySource,std::allocator<zeek::iosource::Manager::ReadySource>> * ready, double timeout, zeek::iosource::IOSource * timeout_src) Line 180	C++
 	zeek.exe!zeek::iosource::Manager::FindReadySources(std::vector<zeek::iosource::Manager::ReadySource,std::allocator<zeek::iosource::Manager::ReadySource>> * ready) Line 174	C++
 	zeek.exe!zeek::run_state::detail::run_loop() Line 270	C++
 	zeek.exe!main(int argc, char * * argv) Line 95	C++
 	zeek.exe!invoke_main() Line 79	C++
 	zeek.exe!__scrt_common_main_seh() Line 288	C++
 	zeek.exe!__scrt_common_main() Line 331	C++
 	zeek.exe!mainCRTStartup(void * __formal) Line 17	C++
 	kernel32.dll!BaseThreadInitThunk()	Unknown
 	ntdll.dll!RtlUserThreadStart()	Unknown

Ah, @philrz another thing you can try is use the stock scripts, but set ZEEK_DNS_FAKE=1 in the environment. if it crashes without that, but runs ok with that set, then yeah, it's definitely something with c-ares or as the backtrace above shows, the event loop.

@JustinAzoff: Ok, I just tried that, and it seems to validate your "it's definitely something" theory. I was able to use the stock scripts (i.e., both detect-MHR and interesting-hostnames enabled in my invoked local.zeek) and both those test wrccdc pcaps could be processed to completion several dozen times in a row without triggering the segfault.

It's odd because DNS_Mgr and c-ares aren't doing anything untoward that I can tell. It drops the nameserver connection a couple of times but re-establishes at the same time both times. I've tried running against another large pcap and it's not failing there either, though it's doing the same thing. I'll open an issue on the kqueue repo with the crash and see if they have any pointers about what could cause that memory to be invalid.

Update: I spotted mheily/libkqueue#155 as the issue @timwoj mentioned in the last comment above.