• 🖨️ Stats 24Hr⏲️ ➼ 2024_06_03

--> 🌐 Total
[+] New/ReNewed SSL Certs (ALL): +2552622

• 🖨️ Stats 7Days⏲️ ➼ 2024_06_03 <--> 2024_05_27

--> 🌐 Total
[+] New/ReNewed SSL Certs (ALL): +48715899

• Info
• What? | Why?
• Enriching/Investigating the Data
• Sources
• In The Wild
• Future Ideas
• Thanks & Appreciation

• Info

• [Automated | UpToDate] Daily (@24 Hrs) Dumps of CertStream Certificate Logs Data

1. All the Scripts & Tools used are OpenSource & Public, as such all this comes with no Guarantees | Liabilities.
2. Due to Github's File Size Limit, all data is Compressed using 7z.
3. View Latest Data from the last 24 Hr at: Raw/Latest

• Download : certstream_domains_latest.txt (Warning: May Crash Browser)

!# Download with wget
 wget "https://pub.ajam.dev/datasets/certstream/all_latest.txt"
!# View without Downloading (Spikes Memory Usage)
 curl -qfsSL "https://pub.ajam.dev/datasets/certstream/all_latest.txt" | less

• Parse (If for some reason, you want to do it manually)

!# Create a Directory
mkdir "./certstream-latest" && cd "./certstream-latest"

!# Download all .7z file
for url in $(curl -qfsSL "https://api.github.com/repos/Azathothas/CertStream-Domains/contents/Raw/Latest" -H "Accept: application/vnd.github.v3+json" | jq -r '.[].download_url'); do echo -e "\n[+] $url\n" && curl -qfLJO "$url"; done

!# Extract all .7z files
!# Install 7z: sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/7z" -o "/usr/local/bin/7z" && sudo chmod +xwr "/usr/local/bin/7z"
find . -iname "*.7z" -exec sh -c '7z x "{}" -o"$(dirname "{}")/$(basename "{}" .7z)"' \;

!# Cat all to a single text file
find . -type f -iname "certstream_domains.txt" -exec cat {} \; 2>/dev/null | sort -u -o "./certstream_domains_latest.txt" ; wc -l < "./certstream_domains_latest.txt"

!# Del .7z files
find . -maxdepth 1 -type f -iname "certstream*.7z" -exec rm {} \; 2>/dev/null

4. View Archival Data upto 7 Days at: Raw/Archive

• Download : certstream_domains_weekly.txt (Warning: May Crash Browser)

!# Download with wget
 wget "https://pub.ajam.dev/datasets/certstream/all_weekly.txt"
!# View without Downloading (DANGEROUS for your CPU/RAM)
 curl -qfsSL "https://pub.ajam.dev/datasets/certstream/all_weekly.txt" | less

• Parse (If for some reason, you want to do it manually)

!# Create a Directory
mkdir "./certstream-7days" && cd "./certstream-7days"

!# Download all .7z file
for url in $(curl -qfsSL "https://api.github.com/repos/Azathothas/CertStream-Domains/contents/Raw/Archive" -H "Accept: application/vnd.github.v3+json" | jq -r '.[].download_url'); do echo -e "\n[+] $url\n" && curl -qfLJO "$url"; done

!# Extract all .7z files
!# Install 7z: sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/7z" -o "/usr/local/bin/7z" && sudo chmod +xwr "/usr/local/bin/7z"
find . -iname "*.7z" -exec sh -c '7z x "{}" -o"$(dirname "{}")/$(basename "{}" .7z)"' \;

!# Cat all to a single text file
find . -type f -iname "certstream_domains.txt" -exec cat {} \; 2>/dev/null | sort -u -o "./certstream_domains_7days.txt" ; wc -l < "./certstream_domains_7days.txt"

!# Del .7z files
find . -maxdepth 1 -type f -iname "certstream*.7z" -exec rm {} \; 2>/dev/null

5. Do Whatever/However you want !

• Blue Teamers: Monitor for Phising Domains
• Red Teamers || Bug Bounty Hunters : Monitor for new assets for your target
• Statisticians || Chad Data Analysts : Have Fun

• Data

• Info: Certificate Transparency Logs only list issuance of website certificates. This data may not necessarily indicate newly registered domains, as Certificates expire and are renewed frequently.
• Instead, use cemulus/crt to check their history:

!# Install:
sudo curl -qfsSL "https://bin.ajam.dev/x86_64_Linux/crt" -o "/usr/local/bin/crt" && sudo chmod +xwr "/usr/local/bin/crt"

• Check

crt "$domain_name"
!# Example:
crt "rmb.info.np"

• Details

crt -json "$domain_name"

!# Example:
crt -json "rmb.info.np"

---

• Rationale

What?

• Azathothas/CertStream-Domains is an Append Only (RFC 6962)[https://datatracker.ietf.org/doc/html/rfc6962] dumps of logs.
• Azathothas/CertStream-Domains only extracts SAN & CN from ct-logs.

What not?

• It is not a database for pre-existing ones.
• There exist a million projects that do the Collection/Database thing a million times better than this repo could ever do. So look elsewhere if you want a DB of certificates & all the data.

Why?

1. There used to be internetwache/CT_subdomains which was very similar to this repo. But it didn't list everything, and also hasn't been updated since Oct 13, 2021. Read their Blog
2. crt.sh also monitors the same logs, but there's a delay (usually 24 Hrs) until it shows up on results. Furthermore, you will have to use additional filters to only list newly issued/renewed certs.
3. Services like SSLMate, Report-Uri & SecurityTrails either monitor only your domains || do not provide all the data || Put it behind Paywalls.

• SSLMate has opensourced their own monitor: SSLMate/certspotter but the data is behind a paywall.

• Sources

1. certstream.calidog.io uses it's own Server to fetch all logs exposing wss://certstream.calidog.io for libraries. Azathothas/certstream is a simple cli that uses the go library.

certstream.calidog.io has been dropped in favour of a completely self-hosted solution. See: #6

2. List of logs monitored: https://www.gstatic.com/ct/log_list/v3/all_logs_list.json

• Elsewhere

• Seeing how fast people will probe you after you get a new TLS certificate
• Azathothas/CertStream-Nepal : [Automated | UpToDate] Daily Dumps of CertStream Subdomains Data For Nepal 🇳🇵
• RebootEx/CertStream-Bangladesh : [Automated | UpToDate] Daily Dumps of CertStream Subdomains Data For Bangladesh 🇧🇩
• Azathothas/CertStream-World : [Automated | UpToDate] Daily Dumps of CertStream Subdomains Data For the World 🌐

• Ideas

1. Use something like mouday/domain-admin if looking to monitor only specific domains.
2. Use something like letsencrypt/ct-woodpecker/ct-woodpecker if looking for detailed output with stats & monitors (Prometheus) for Production.
3. Use something like drfabiocastro/certwatcher if looking for Automation. This is essentially nuclei for cert-logs.

• Thanks

1. The Hacker's Choice for proividing servers on segfault & being so generous.

• Telegram : @thcorg | Github : https://github.com/hackerschoice