When in the recon phase of a security assessment or penetration test, quite often you want to collect the dns names for all hosts in a scope of IP addresses. I covered how to do that with a few different APIs in this story: (https://isc.sans.edu/diary/Using+Passive+DNS+sources+for+Reconnaissance+and+Enumeration/28596)
On the flip-side, quite often you want to collect DNS records of all hosts in a domain. With more folks using wildcard certificates these days, certificate transparency isn't always the goldmine that it used to be for that (https://isc.sans.edu/diary/Using+Certificate+Transparency+as+an+Attack+Defense+Tool/24114)
What to do? The various OSINT repositories (and commercial intelligence repos) have an answer for you.
For DNS Dumpster, this will list what you seek:
curl -k "https://api.hackertarget.com/hostsearch/?q=sans.edu"
isc.sans.edu,45.60.31.34
www.sans.edu,45.60.31.34
Note that DNS Dumpster limits you on requests-per-day, you'll need an API key if you want more queries. They also have a nice web front-end if you're not feeling particulary code-y that day .. https://hackertarget.com/ip-tools/
That seems like a short list to me though, let's look at Cisco Umbrella, which uses OpenDNS as it's back-end database - note that this function has been there a while in the web UI, but is fairly new to the API:
curl -s -k "https://investigate.umbrella.com/subdomains/sans.edu" -H "Authorization: Bearer <APIKEY>" -H "Content-Type: application/json" | jq
[
{
"securityCategories": [],
"firstSeen": "1627675727",
"name": "_dmarc.sans.edu"
},
{
"securityCategories": [],
"firstSeen": "1627675727",
"name": "_domainkey.sans.edu"
},
..... and so on
Umbrella also has a web front-end (login required) - https://dashboard.umbrella.com/o/7966391/#/investigate
Back to the code - getting a count:
curl -s -k "https://investigate.umbrella.com/subdomains/sans.edu" -H "Authorization: Bearer <APIKEY>" -H "Content-Type: application/json" | jq | grep name | wc -l
20
This is because this API returns values 20 at a time, you use the last value returned as an offset to get the next batch.
What's that, you say? Sounds like a script? Great idea you!
Looking one step ahead, after this list is created, we want to collect all of them that are still valid A records, so that we have a list of target hosts to dig deeper in to.
So along the way, let's take all the records that are found and divvy them up into 3 lists:
$validARecords - this is mostly what we're looking for - valid DNS A records for hosts in the domain, which we can use for targets
$validotherrecords - these are other DNS records (MX, NS, SOA, TXT etc). Useful, but not usually direct (in scope) targets
$notvalidrecords - these are dns records that no longer resolve, these records did exist at one time but have since been removed
This API call also returns a "first seen" date in Unix Epoch time (seconds since 1 Jan, 1970) - since we're coding, let's convert that to readable text along the way, it might be useful in subsequent phases of your gig.
Putting this all into code:
$apikey = "YOUR API KEY GOES HERE"
$headers = @{
'Authorization'="Bearer "+$apikey
}
$domain = "sans.edu"
$validArecords = @()
$validOtherRecords =@()
$notvalidrecords = @()
$dnsrecords =@()
# set the countval to 20 so that the loop will start
$loopcount = 0
$countval = 20
while($countval -ge 20) {
if ($loopcount -gt 0) {
# if this is not the first loop through, get the offset and apply it
$offsetname = ($dnsrecords | select-object -last 1).name
$callstring = "https://investigate.umbrella.com/subdomains/" + $domain + "?offsetName="+$offsetname
} else {
$callstring = "https://investigate.umbrella.com/subdomains/" + $domain
}
$retvals = invoke-webrequest -Method 'Get' -uri $callstring -headers $headers -ContentType "application/json"
$records = ($retvals.content | convertfrom-json) | select firstseen, name
$countval = $records.count
$dnsrecords += $records
write-host "Count is " $dnsrecords.count
$loopcount += 1
}
# Convert all the "first seen" dates from Unix Epoch to Strings
# also test each records and assign each to the correct list
foreach ($val in $dnsrecords) {
# First, fix the "first seen" date
$date2 = (Get-Date 01.01.1970).AddSeconds($val.firstseen)
$val.firstseen = ($date2).ToString("yyyy-MM-dd")
#next, separate out the current A records and expired A records
if($record = resolve-dnsname -name $val.name -type a -ErrorAction SilentlyContinue) {
# record is valid - add the ip and update thev valid list
# check for other record types (SOA, NS, MX etc)
if($record.type -ne "A") {
$validotherrecords += $record
} else {
# these are the target valid A records
$tempval = $val
$tempval | add-member -membertype noteproperty -name ipaddress -value $record.ipaddress
$validArecords += $tempval
}
} else {
# record is not valid, update the list of invalid records
$notvalidrecords += $val
}
}
So, what's in the final lists:
These list of course are the primary targets - straight up A records (in your customer's domain of course):
$validarecords
firstSeen name ipaddress
--------- ---- ---------
2021-01-25 email.sans.edu 136.147.129.27
2022-09-28 localhost.email.sans.edu 127.0.0.1
2017-03-13 isc.sans.edu {45.60.31.34, 45.60.103.34}
2014-08-03 www.sans.edu 45.60.31.34
These are also valid records and are of potential use, the CNAME records in particular will be of interest (for instance for multiple websites on the same host):
$validotherrecords | select name,querytype
Name QueryType
---- ---------
sans.edu SOA
sans.edu SOA
autodiscover.alumni.sans.edu CNAME
autodiscover.outlook.com CNAME
autod.ha-autod.office.com CNAME
.. and so on
Finally, the invalid list - you'll find that these are records that are of historic interest, and have been removed. Don't discount them though, that doesn't mean that the hosts are gone, just the DNS records - - these are all still worth checking! (see what I meant about a DNS Time Machine?)
$notvalidrecords
firstSeen name
--------- ----
2021-07-30 _dmarc.sans.edu
2021-07-30 _domainkey.sans.edu
2020-09-15 eiqu3eingae1ha9ja4phepaivahqu9xo._domainkey.sans.edu
2020-11-01 isc._domainkey.sans.edu
.. and so on
As always, as this script evolves I'll maintain it on my github: https://github.com/robvandenbrink/DNS-PDNS-Recon
Cisco Umbrella is my current front-running for this type of work, mainly because of the back-end that it uses (OpenDNS), the nice API and the amount of data returned. Do you have a DNS recon site that has more or better information? Please, share using our comment form!
===============
Rob VandenBrink
rob@coherentsecurity.com
https://github.com/robvandenbrink
When in the recon phase of a security assessment or penetration test, quite often you want to collect the dns names for all hosts in a scope of IP addresses. I covered how to do that with a few different APIs in this story: (https://isc.sans.edu/diary/Using+Passive+DNS+sources+for+Reconnaissance+and+Enumeration/28596)
On the flip-side, quite often you want to collect DNS records of all hosts in a domain. With more folks using wildcard certificates these days, certificate transparency isn't always the goldmine that it used to be for that (https://isc.sans.edu/diary/Using+Certificate+Transparency+as+an+Attack+Defense+Tool/24114)
What to do? The various OSINT repositories (and commercial intelligence repos) have an answer for you.
For DNS Dumpster, this will list what you seek:
curl -k "https://api.hackertarget.com/hostsearch/?q=sans.edu"
isc.sans.edu,45.60.31.34
www.sans.edu,45.60.31.34
Note that DNS Dumpster limits you on requests-per-day, you'll need an API key if you want more queries. They also have a nice web front-end if you're not feeling particulary code-y that day .. https://hackertarget.com/ip-tools/
That seems like a short list to me though, let's look at Cisco Umbrella, which uses OpenDNS as it's back-end database - note that this function has been there a while in the web UI, but is fairly new to the API:
curl -s -k "https://investigate.umbrella.com/subdomains/sans.edu" -H "Authorization: Bearer <APIKEY>" -H "Content-Type: application/json" | jq
[
{
"securityCategories": [],
"firstSeen": "1627675727",
"name": "_dmarc.sans.edu"
},
{
"securityCategories": [],
"firstSeen": "1627675727",
"name": "_domainkey.sans.edu"
},
..... and so on
Umbrella also has a web front-end (login required) - https://dashboard.umbrella.com/o/7966391/#/investigate
Back to the code - getting a count:
curl -s -k "https://investigate.umbrella.com/subdomains/sans.edu" -H "Authorization: Bearer <APIKEY>" -H "Content-Type: application/json" | jq | grep name | wc -l
20
This is because this API returns values 20 at a time, you use the last value returned as an offset to get the next batch.
What's that, you say? Sounds like a script? Great idea you!
Looking one step ahead, after this list is created, we want to collect all of them that are still valid A records, so that we have a list of target hosts to dig deeper in to.
So along the way, let's take all the records that are found and divvy them up into 3 lists:
$validARecords - this is mostly what we're looking for - valid DNS A records for hosts in the domain, which we can use for targets
$validotherrecords - these are other DNS records (MX, NS, SOA, TXT etc). Useful, but not usually direct (in scope) targets
$notvalidrecords - these are dns records that no longer resolve, these records did exist at one time but have since been removed
This API call also returns a "first seen" date in Unix Epoch time (seconds since 1 Jan, 1970) - since we're coding, let's convert that to readable text along the way, it might be useful in subsequent phases of your gig.
Putting this all into code:
$apikey = "YOUR API KEY GOES HERE"
$headers = @{
'Authorization'="Bearer "+$apikey
}
$domain = "sans.edu"
$validArecords = @()
$validOtherRecords =@()
$notvalidrecords = @()
$dnsrecords =@()
# set the countval to 20 so that the loop will start
$loopcount = 0
$countval = 20
while($countval -ge 20) {
if ($loopcount -gt 0) {
# if this is not the first loop through, get the offset and apply it
$offsetname = ($dnsrecords | select-object -last 1).name
$callstring = "https://investigate.umbrella.com/subdomains/" + $domain + "?offsetName="+$offsetname
} else {
$callstring = "https://investigate.umbrella.com/subdomains/" + $domain
}
$retvals = invoke-webrequest -Method 'Get' -uri $callstring -headers $headers -ContentType "application/json"
$records = ($retvals.content | convertfrom-json) | select firstseen, name
$countval = $records.count
$dnsrecords += $records
write-host "Count is " $dnsrecords.count
$loopcount += 1
}
# Convert all the "first seen" dates from Unix Epoch to Strings
# also test each records and assign each to the correct list
foreach ($val in $dnsrecords) {
# First, fix the "first seen" date
$date2 = (Get-Date 01.01.1970).AddSeconds($val.firstseen)
$val.firstseen = ($date2).ToString("yyyy-MM-dd")
#next, separate out the current A records and expired A records
if($record = resolve-dnsname -name $val.name -type a -ErrorAction SilentlyContinue) {
# record is valid - add the ip and update thev valid list
# check for other record types (SOA, NS, MX etc)
if($record.type -ne "A") {
$validotherrecords += $record
} else {
# these are the target valid A records
$tempval = $val
$tempval | add-member -membertype noteproperty -name ipaddress -value $record.ipaddress
$validArecords += $tempval
}
} else {
# record is not valid, update the list of invalid records
$notvalidrecords += $val
}
}
So, what's in the final lists:
These list of course are the primary targets - straight up A records (in your customer's domain of course):
$validarecords
firstSeen name ipaddress
--------- ---- ---------
2021-01-25 email.sans.edu 136.147.129.27
2022-09-28 localhost.email.sans.edu 127.0.0.1
2017-03-13 isc.sans.edu {45.60.31.34, 45.60.103.34}
2014-08-03 www.sans.edu 45.60.31.34
These are also valid records and are of potential use, the CNAME records in particular will be of interest (for instance for multiple websites on the same host):
$validotherrecords | select name,querytype
Name QueryType
---- ---------
sans.edu SOA
sans.edu SOA
autodiscover.alumni.sans.edu CNAME
autodiscover.outlook.com CNAME
autod.ha-autod.office.com CNAME
.. and so on
Finally, the invalid list - you'll find that these are records that are of historic interest, and have been removed. Don't discount them though, that doesn't mean that the hosts are gone, just the DNS records - - these are all still worth checking! (see what I meant about a DNS Time Machine?)
$notvalidrecords
firstSeen name
--------- ----
2021-07-30 _dmarc.sans.edu
2021-07-30 _domainkey.sans.edu
2020-09-15 eiqu3eingae1ha9ja4phepaivahqu9xo._domainkey.sans.edu
2020-11-01 isc._domainkey.sans.edu
.. and so on
As always, as this script evolves I'll maintain it on my github: https://github.com/robvandenbrink/DNS-PDNS-Recon
Cisco Umbrella is my current front-running for this type of work, mainly because of the back-end that it uses (OpenDNS), the nice API and the amount of data returned. Do you have a DNS recon site that has more or better information? Please, share using our comment form!
===============
Rob VandenBrink
rob@coherentsecurity.com
https://github.com/robvandenbrink
