October 21, 2020

Long list of Apple domains

This post represents an effort to find various filetypes that might exit against this 597 strong Apple subdomains list that was recently found on a random website. Unfortunately the original URL hosting the list was not documented.

Irregardless, a dry run was executed against all 597 domains using the wayback_machine_downloader with the -l option switched on. This prevents any downloads from occurring while outputting the process to stdout (the terminal window). The stdout was logged to a text file. A 1.05GB log of the process resulted. Althought the script was only setup with an interest in pkg, as, hqx, cpt, bin, sea, sit, sitx, dd, and pit files, it’s clear that many other filetypes are documented in the log file, including PDF files.

zsh script used for the dry run

while read -r line
do
    wayback_machine_downloader -a -d "$line"_IA -c6 --only "/\.(pkg|as|hqx|cpt|bin|sea|sit|sitx|dd|pit)$/i" "$line"
done < /path/to/domains.txt | tee -a log.txt

Descriptions of the options applied

Option Short Form	Option Unabbreviated	Description
-l	–list	Only list file urls in a JSON format with the archived timestamps, won’t download anything
-s	–all-timestamps	download all snapshots/timestamps for a given website
-d	–directory PATH	Directory to save the downloaded files into
-c	–concurrency NUMBER	Number of multiple files to dowload at a time

Number of files curated

The script considered or querried only certain filetypes so only these filetypes are considered to be curations. As mentioned other filetypes could be curated by the scrape, of notable interest are PDF files as would be any text files but that could result in many robots.txt files travelling inbound.

At first glance robots.txt files in themselves may appear to be of little interest but sometimes they reveal interesting subdomains that should be ignored by spiders, misc. notes and other tidbits of data. The internet archive’s wayback machine has dutifully ignored the contents of robots.txt files. Any subdomains contained therein have likely expired and only wget could potentially bring anything out of the dark as it were. apelle

Domain	No. of Files Curated
speech.apple.com	30
macos.apple.com	3
lists.apple.com	139
retail.apple.com	2
developer.apple.com	314
bananajr6000.apple.com	2
latinamerica.apple.com	3
mirror.apple.com	33,831
ppclinux.apple.com	7
qtj.apple.com	3
ws2.quicktime.apple.com	4

Block of domains

apple-wwnet.apple.com, nwk-www.apple.com, nwk-search.apple.com, nwk-qtannounce.apple.com, nwk-serialno.apple.com, nwk-qtinstall.apple.com, awpicts.apple.com, nwk-stream.qtv.apple.com, nwk-qttest.apple.com, nwk-qtsoftware.apple.com, helpqt.apple.com, helposx.apple.com, help.apple.com, ssl.apple.com, nwk-redirect.apple.com, livepage.apple.com, instore.apple.com, applebrasil.com, .br asia-red.apple.com, www.uk.apple.com, speech.apple.com, nwk-qtinstall.apple.com, nwk-qtpix.apple.com, nwk-www.support.apple.com, nwk-groups.mac.com, apple-order1.apple.com, ink.apple.com, ipod.com, itunes.com, rss.itunes.com, nwk-neo.qtv.apple.com, apple-darwin.com, nwk-hompage.mac.com, nwk-prohelp.apple.com, nwk-documentation.apple.com, bz-si-a.apple.com, bz-si-b.apple.com, fdns1.apple.com, fdns2.apple.com, time0.apple.com, time1.apple.com, time2.apple.com, time.apple.com, fdns5.apple.com, fdns6.apple.com, smtpin01.universityarts.com, mediaport01.apple.com, mediaport02.apple.com, qtannounce.apple.com, debit.apple.com, nsagate.apple.com, nserver.apple.com, mail-out2.apple.com, mail-out1.apple.com, forum.apple.com, mail-in1.apple.com, mail-in2.apple.com, nserver2.apple.com, littlebuddy.apple.com, connect1.apple.com, connect2.apple.com, scv-wsidecar.apple.com .0.254.17.in-addr.arpa, console1.apple.com, console2.apple.com, ray.apple.com, jay.apple.com, littlebuddyx.apple.com, wwb1.apple.com, wwb2.apple.com, mycroft.apple.com, tmpforum.apple.com, www.apple.com, bz1.apple.com, bz2.apple.com, tondero.apple.com, piglet.apple.com, netcache11-e0.apple.com, www.apple.com, netcache11-e5.apple.com, webmail.apple.com, qtbetastreamer1.apple.com, nosleep.apple.com, macos.apple.com, cab.apple.com, hwseeding.apple.com, developer1.apple.com, skippy.apple.com, developer2.apple.com, acrux-o.apple.com, spica.apple.com, pariah.apple.com, broadcaster.apple.com, cay.apple.com, asedb.apple.com, cay01.apple.com, cay02.apple.com, velocity.apple.com, internet1.apple.com, unihan.unicode.org lists.apple.com, applenews.lists.apple.com, craft.training.apple.com, search.apple.com, qtconfig.apple.com, qtconfig2.apple.com, qtd.apple.com, qtconfig.apple.com, netcache12-e0.apple.com, ext-stage.apple.com, qtstage.apple.com, qtstage2.apple.com, upload.qtv.apple.com, apple.com, ftpdev01.apple.com, ftpdev02.apple.com, campreg.apple.com, rns1.apple.com, rns1.apple.com, enfuego.apple.com, developer3.apple.com, developer4.apple.com, help1.apple.com, help2.apple.com, help3.apple.com, charple.apple.com, unsure.apple.com, sure.apple.com, sure01.apple.com, sure02.apple.com, rdns1.apple.com, ftp.qtv.apple.com, neo.qtv.apple.com, sfxf1.apple.com, sfxf2.apple.com, sfxm.apple.com, sfxm1.apple.com, sfxm2.apple.com, sfxm3.apple.com, sfx.apple.com, sfxt.apple.com, imercury2.apple.com, imercury3.apple.com, spica1.apple.com, spica2.apple.com, instanton.apple.com, gidget160.apple.com, gidget161.apple.com, gsearch.apple.com, xlate.apple.com, xlate1.apple.com, learnandearn.apple.com, signin.apple.com, devcon.apple.com, changeinfo.apple.com, register.apple.com, retail.apple.com, asw.apple.com, asw2.apple.com, aswf.apple.com, aswf2.apple.com, daw.apple.com, dawws01.apple.com, dawws02.apple.com, truecomp.apple.com, b2bstore.appple.com, education.apple.com, crt.apple.com, train.apple.com, bugreport.apple.com, itsa.apple.com, itse.apple.com, arthurwmml.apple.com, krazy.apple.com, kool.apple.com, topcat.apple.com, fritz.apple.com, mxwm.apple.com, seacoast.apple.com, etsdev1wm.apple.com, micronwm.apple.com, wmdev.apple.com, gra1.apple.com, gra2.apple.com, gcrm.apple.com, casis.apple.com, epp.apple.com, swupdate.apple.com, update10.apple.com, update11.apple.com, gra.apple.com, testswupdate.apple.com, webnfaws.apple.com, connect.apple.com, register-tmp.apple.com, stream.apple.akadns.net, stream.apple.akadns.net, stream.apple.akadns.net, stream.apple.akadns.net, stream.apple.akadns.net, stream.apple.akadns.net, apply-euro.apple.com, developer.apple.com, webgdv.apple.com, radarsubmissions.apple.com, scv-ipodrocks.apple.com, myinfo.apple.com, naswws.apple.com, chatbox-web1.apple.com, chatbox-web2.apple.com, milhousewm.apple.com, margewmt.apple.com, margewmm.apple.com, arthurwm.apple.com, arthurwmfw.apple.com, store7.apple.com, vs6-1.apple.com, vs6-2.apple.com, vs7-1.apple.com, vs7-2.apple.com, jobs.apple.com, search.lists.apple.com, iforgot.apple.com, aristo.apple.com, maosh.apple.com, salestraining.apple.com, gsx.apple.com, scv-searchcgi.apple.com, caesar.apple.com, wdg1.apple.com, wdg2.apple.com, jobsws1.apple.com, jobsws2.apple.com, adcsearch.apple.com, adcweb.apple.com, gcrmfg.apple.com, wdb.apple.com, monterey.apple.com, portland.apple.com, search-fr.apple.com, search-de.apple.com, search-nl.apple.com, search-no.apple.com, search-it.apple.com, search-es.apple.com, search-dk.apple.com, search-se.apple.com, search-fi.apple.com, chatbox-web.apple.com, t2castro.apple.com, t2cool.apple.com, t2valenn.apple.com, t2zathras.apple.com, t2submit.apple.com, t2guidejp.apple.com, nugget.apple.com, t2nugget.apple.com, eureka.apple.com, t2eureka.apple.com, guidejp2.apple.com, t2guidejp2.apple.com, xsurvey.apple.com, t2higgs.apple.com, higgs.apple.com, t2perkins.apple.com, perkins.apple.com, adcweb1.apple.com, adcweb2.apple.com, adcweb3.apple.com, adcweb4.apple.com, mp4reg.apple.com, designawardst.apple.com, trainingsun1.apple.com, trainingsun2.apple.com, trainingsun3.apple.com, trainingsun4.apple.com, trainingsun5.apple.com, trainingsun6.apple.com, trainingsun7.apple.com, trainingsun8.apple.com, trainingsun9.apple.com, dvdsp.apple.com, qtsoftware.apple.com, helpqt.apple.com, helposx.apple.com, help.apple.com, guide-n.apple.com, guide1-n.apple.com, guide2-n.apple.com, guidejp-n.apple.com, guidejp1-n.apple.com, guidejp2-n.apple.com, applemusic.com, vs5-2.apple.com, www.access.apple.com, store.apple.com, connect6.apple.com, connect7.apple.com, store6.apple.com, itunes.com, www.anat.apple.com, bananajr6000.apple.com, devseed.apple.com, sherlockdevelopert.apple.com, rodwm.apple.com, gdv.apple.com, connect3.apple.com, extlor.apple.com, appleorder.apple.com, www.applereg.com, sonarproxy.apple.com, rfaproxy.apple.com, topfritz.apple.com, customer.apple.com, radarproxy.apple.com, store8.apple.com, red2.apple.com, learnandearn1.apple.com, learnandearn2.apple.com, ipod.com, icards.mac.com, supplier.apple.com, www.latinamerica.apple.com, kangaroo.apple.com, websabc.apple.com, spider.apple.com, certifications.apple.com, war.apple.com, jobsat.apple.com, backstage.apple.com, webobjects.apple.com, survey.apple.com, spssprod.apple.com, survey.edu-research.com, redirect.apple.com, red1.apple.com .3.254.17.in-addr.arpa, adcstudentst.apple.com, firewire.apple.com, mirror.apple.com, hound.apple.com, veloxs.apple.com, ws7.apple.com, ws8.apple.com, iss.apple.com, ppclinux.apple.com, powerbook.apple.com, servers.apple.com, rss.itunes.com, xerxes.apple.com, devcd.apple.com, qtpartners.apple.com, appletalkback.apple.com, markets.apple.com, campusreps2.apple.com, salesready.apple.com, campusreps3.apple.com, ws1.quicktime.apple.com, ws2.quicktime.apple.com, campusreps4.apple.com, reg.devworld.apple.com, survey.devworld.apple.com, b2bpbzml.apple.com, speech.apple.com, cgi2.training.apple.com, qtj.apple.com, highered.apple.com, campusreps5.apple.com, campusreps6.apple.com, www.applenet.apple.com, seeding.apple.com, qtinstall.apple.com, www.apple.fr, arthurlwm.apple.com, galileo.apple.com, www.apple.nl, www.apple.de, galileo2.apple.com, www.appledesigns.com, www.apple.at, adc-kbaset.apple.com, www.apple.ch, www.apple.be, www.apple.co.jp prismo.com, www.apple.se, www.apple.no, www.apple.fi, applejava.apple.com, www.applestore.ch, www.apple.dk, www.apple.it, www.apple.za, www.apple.es, www.uk.apple.com, cgis.training.apple.com, cgi5.training.apple.com, applescript.apple.com, hypercard.apple.com, wwc1.apple.com, wwc2.apple.com, wwc3.apple.com, awpicts.apple.com, apple.com, www.euro.apple.com, www.apple.ca, stories.apple.com, colorsync.apple.com, www.dueval.apple.com, stories2.apple.com, clarisftp.apple.com, slg.apple.com, qtpix.apple.com, resellerapplication.apple.com, livepage.apple.com, ra.apple.com, stories3.apple.com, azis.apple.com, qtvr20test.apple.com, var.apple.com, clarisworks.apple.com, applemasters01.apple.com, adc10.apple.com, software.apple.com, www.support.apple.com, survey1.apple.com, unicode2.apple.com, stream.qtv.apple.com, qttest.apple.com, qtpartners.apple.com, www.homepage.mac.com, cgi4.training.apple.com, apple-order1.apple.com, apple-order1.apple.com, apple-order1.apple.com, apple-order1.apple.com, federal.apple.com, fonts.apple.com, red-train.apple.com, quicktimelive.com, product.info.apple.com, product.info.apple.com, product.info.apple.com, product.info.apple.com, www0.info.apple.com, www0.info.apple.com, kidsafe.apple.com, mp4ra.org, spruceuserforums.apple.com, sprucesupport.apple.com, spruceregistration.com, www.spruce-tech.com, www.webdvd.org, shake.apple.com, chatbox-smtp-in.apple.com, chatbox-smtp-out1.apple.com, chatbox-smtp-out2.apple.com, chatbox-smtp-out3.apple.com, chatbox-smtp-out4.apple.com, chatbox-smtp-out5.apple.com, chatbox-smtp-out6.apple.com, gateway221.apple.com, ex0-stephenbz-221.apple.com, ex1-stephenbz-221.apple.com, engbz-bi-3a-221.apple.com, engbz-bi-3b-221.apple.com, spsi1.apple.com, spsi2.apple.com, sp-bi-4ka.apple.com, sp-bi-4kb.apple.com, adcevents.apple.com, adcinvoice.apple.com, adc-kbase.apple.com, adckitchen.apple.com, adcrequest.apple.com, adcstudents.apple.com, adctv.apple.com, designawards.apple.com, sherlockdeveloper.apple.com, techtalk.apple.com, wwdcexhibitors.apple.com, wwdcvolunteers.apple.com, guide.apple.com, hotdeals.apple.com, mpgsubmit.apple.com, mpgmonitor.apple.com, mpgjp1.apple.com, mpgjp2.apple.com, adcpdub01.apple.com, adcpdub02.apple.com, adcpdub03.apple.com, adcpdub04.apple.com, adcpdub05.apple.com, adcpdub06.apple.com, adcftp1.apple.com, adcftp2.apple.com, adcftp3.apple.com, plugfest.apple.com, gateway223.apple.com, ex0-mail-bz-223.apple.com, ex1-mail-bz-223.apple.com, mail-in5.apple.com, mail-in4.apple.com, mail-in3.apple.com, mail-out3.apple.com, mail-out4.apple.com, bz3.apple.com, bz4.apple.com, gateway226.apple.com, ex0-efax-226.apple.com, ex1-efax-226.apple.com, embrun.apple.com, downey.apple.com, compton.apple.com, cassis.apple.com, hobart.apple.com, rostock.apple.com, tre.apple.com, ton.apple.com, ftp01.apple.com, ftp02.apple.com, ftp03.apple.com, privftp.apple.com, wc1.apple.com, wcstream.apple.com, wchttp.apple.com, lfcomponents.apple.com, canadaapp.apple.com, jobtasks.apple.com, mirror.apple.com, odenya.apple.com, fcptrain.apple.com, www.opensource.apple.com, anoncvs.opensource.apple.com, clearmater.apple.com, wwmarkets.apple.com, foup.apple.com, appleseed.apple.com, discuss.appleseed.apple.com, appleseedwo.apple.com, seedsurvey.apple.com, dm.apple.com, seoext.apple.com, tcext.apple.com, gdext.apple.com, unihan.unicode.org spazmozart.apple.com, campusreps.apple.com, educationstreaming.apple.com, loc.apple.com, index-backup.apple.com, locsubmit.apple.com, talkies.apple.com, communityloc.apple.com, communitydev.apple.com, dts.ftp.apple.com, electron.apple.com, www.seminars.apple.com, www2.seminars.apple.com, www3.seminars.apple.com, db1.seminars.apple.com, db3.seminars.apple.com, list.seminars.apple.com, consultants.apple.com, consultants2.apple.com, streaming.seminars.apple.com, agents.apple.com, programs.apple.com, edsol.apple.com, alibak1.apple.com, newali.apple.com, alimed1.apple.com, oldali.apple.com, alistr1.apple.com, alistg1.apple.com, homeroom.apple.com, uldemo.apple.com, qtamigos.apple.com, qtdevseed.apple.com, qtpartners.apple.com, qtvr20test.apple.com, qtsdemo.apple.com, atis.training.apple.com, cgidb.training.apple.com, cgi6.training.apple.com, cgi3.training.apple.com, benedict.apple.com, alpha.ns.apple.com, beta.ns.apple.com, gamma.ns.apple.com, as.apple.com, alink-gw.apple.com, www.apple.com, www2.apple.com, www3.apple.com, webmaster.apple.com, bananajr6000.apple.com, skunkworks.apple.com, macip.apple.com