Just Solve the File Format Problem - User contributions [en]

Sources

2018-10-11T03:22:54Z

More barbeque: /* List of Places Keeping Track of Formats */ add PC98 disk image formats page

__NOTOC__
If you're looking for places to pull either file format information, or to verify stuff you've found, or to go that extra mile to track down variations, here's some similar endeavors and items.

==List of Places Keeping Track of Formats==

* [http://www.forensicswiki.org/wiki/Category:File_Formats Forensics Wiki]
* http://archivematica.org/preservation
* http://www.fileformat.info/
* http://www.ace.net.nz/tech/TechFileFormat.html
* http://wotsit.org/ - their download page is broken. need to ping webmaster about that.
* Wikipedia: http://en.wikipedia.org/wiki/List_of_file_formats and http://en.wikipedia.org/wiki/List_of_file_formats_%28alphabetical%29
** Also [[wikipedia:Template:Infobox file format]] ([http://en.wikipedia.org/w/index.php?title=Special%3AWhatLinksHere&limit=500&target=Template%3AInfobox+file+format&namespace=0 pages using this template])
* http://unarchiver.c3.cx/formats
* [[PRONOM|The PRONOM format registry]]
* http://wiki.multimedia.cx/
* http://archive.org/download/file-format-encyclopedia/fileformat.zip/
* http://www.digitalpreservation.gov/formats/fdd/descriptions.shtml
* http://www.udfr.org/, see also http://gdfr.info/.
* http://gitorious.org/re-lab Tools & specs from re-lab- graphics formats, and office formats
* http://www-mmsp.ece.mcgill.ca/documents/AudioFormats/index.html - Documentation, details, specs, and samples for a handful of audio formats
* http://wiki.xentax.com/index.php/Game_File_Format_Central This wiki is the home of the most game (archive) file format knowledge in the world.
* http://rewiki.regengedanken.de has some more game related reverse engineering
* http://mark0.net/soft-trid-e.html Home to TrID, which can identify a vast number of file formats using an updateable database
* http://mark0.net/soft-trid-deflist.html Online searchable version of the TrID database file
* http://file-extension.net/seeker/ Metasearch engine for file extensions - Find info about unknown file extensions or filetypes
* http://www.vgmpf.com/Wiki/index.php?title=Main_Page - the Videogame Music Preservation Foundation Wiki has information on a number of formats used to play back videogame music
* http://hackipedia.org/File%20formats/ - loads of info on various formats here
* http://wiki.mobileread.com/wiki/E-book_formats - detailed information on eBook formats.
* http://mediapedia.nla.gov.au/ - "The Mediapedia resource is intended to enable the identification of various physical media carrier types for assisting with collection planning, assessment, documentation, infrastructure and preservation planning for the content they hold. These could include media across various genres such as cine, video, photo, audio, data, paper carriers, microfilm, etc."
* http://wiki.opf-labs.org/display/TR/Formats - Small list of formats with detailed descriptions of some known access/preservation problems.
* [http://registry.dcc.ac.uk:8080/RegistryWeb/Registry/ CASPAR/DCC Representation Information Repository]
* [http://theswo.sourceforge.net/ The Software Ontology]
* [http://keep-totem.co.uk/ KEEP Totem] also covers software, hardware etc.
* [http://cultureandcommunication.org/deadmedia/index.php/Main_Page Dead Media Archives]
* [http://www.fid3.com/file-formats The >4000 formats known to FI Tools], [http://www.fid3.com/formats-ext1 by extension], [http://www.fid3.com/formats-mime by MIME type]
* [http://hg.python.org/cpython/file/2.7/Lib/mimetypes.py#l402 Python sourcecode for mimetypes module]
* /etc/mimetypes on *nix systems defines a number of file formats and [[MIME types]]
* http://www.iasa-web.org/tc04/key-digital-principles - Recommended Audio formats by Audiovisual Archives and Institutions
* http://www.gpsbabel.org/capabilities.html - List of formats used by GPS devices
* http://www.microsoft.com/openspecifications/en/us/programs/osp/office-file-formats/default.aspx Microsoft's Open Document Specifications
* http://mamedev.org/source/src/lib/formats/index.html http://mamedev.org/source/src/mess/formats/index.html - format handlers for disk images, cassette images, and memory snapshots/quickloads in the MESS source code
* http://minidisc.org alot of minidisc related info
* http://cool.conservation-us.org/ Conservation Online
* http://filext.com/ File Extensions with MIME types
* http://en.wikipedia.org/wiki/List_of_file_formats_(alphabetical) Wikipedia list of file formats (alphabetical)
* http://ist.uwaterloo.ca/~schepers/formats.html The C64 file formats list
* http://3dbrew.org/wiki/Category:File_formats Nintendo 3DS file formats
* http://www.iana.org/assignments/media-types/index.html IANA MIME Media Types - lots of useful info here and often quite detailed as well
* http://gitorious.org/msoscheme msoscheme is a project that creates a machine readable description of the binary file formats of Microsoft Office
* http://www.schnarff.com/file-formats/ File Formats documentation collection
* http://the-fr.org/ The Format Registry
* http://www.freebase.com/computer/file_format?instances= Freebase instances in category "File Format"
* https://bitbucket.org/haypo/hachoir/src/default/hachoir-parser/hachoir_parser/ Hachoir file format parsers
* https://bitbucket.org/haypo/hachoir/wiki/FileFormatResources Hachoir list of file format resources
* https://theiphonewiki.com/wiki/Category:File_Formats iPhone related file formats
* https://www.synalysis.net/formats.xml Synalize It file format grammars
* [http://www.binwalk.org Binwalk] — can recognise and extract data in many formats embeddded inside firmware images
* http://www.math.utah.edu/~beebe/fonts/X-Window-System-fonts.html - Good info about the Fonts on Windows X system
* http://www.online-convert.com/file-type
* https://melog.info/archives/2004/09/30223144.html - PC98 disk image formats (Japanese)

==Emulation==

* http://nonmess.retrogames.com/ - Emulator listing for various computer formats, including MESS and single-machine emulators.
* [http://emuframework.sourceforge.net/ The Emulation Framework] is software developed by the international KEEP project, co-funded by the European Union's 7th Framework Programme.

==Books==

* [http://www.amazon.com/Compressed-Image-File-Formats-JPEG/dp/0201604434/ Compressed Image File Formats: JPEG, PNG, GIF, XBM, BMP] by John Miano (1999)
* [http://www.amazon.com/Encyclopedia-Graphics-File-Formats-Reference/dp/1565921615/ Encyclopedia of Graphics File Formats] by James D. Murray and William VanRyper (1994, 1996) '''Copy Acquired!''' Refer to the [[Encyclopedia of Graphics File Formats|main article]].
* [http://www.amazon.com/Graphics-File-Formats-David-Kay/dp/0070340250/ Graphics File Formats] by David C. Kay and John R. Levine (1994)
* [http://www.amazon.com/Graphics-File-Formats-Reference-Guide/dp/0133034054/ Graphics File Formats: Reference and Guide] by C. Wayne Brown, Barry J. Shepherd (1994)
* [http://www.amazon.com/Formats-Popular-Personal-Computer-Software/dp/0471836710/ File Formats for Popular Personal Computer Software: A Programmer's Reference (Self-teaching Guides)] by Jeff Walden (1986) '''Copy Acquired!'''
* [http://www.amazon.com/File-Formats-Internet-Guide-Users/dp/0871114410 File Formats on the Internet: A Guide for PC Users] by Allison B. Zhang (1996) '''Copy Acquired!'''
* [http://www.amazon.com/Graphics-File-Formats-Programmers-Reference/dp/0201488353/ 3D Graphics File Formats: A Programmer's Reference] by Keith Rule (1996)
* [http://www.amazon.com/More-File-Formats-Popular-Software/dp/0471850772/ More File Formats for Popular PC Software: A Programmer's Reference] by Jeff Walden (1987) '''Copy Acquired!'''
* [http://www.amazon.com/Lotus-File-Formats-1-2-3-Symphony/dp/0201168243/ Lotus File Formats for 1-2-3 Symphony & Jazz: File Structure Descriptions for Developers] from Lotus, Inc. (1986)
* [http://www.amazon.com/Inside-Windows-File-Formats-Swan/dp/0672303388/ Inside Windows File Formats] by Tom Swan (Nov 1993)
* [http://www.amazon.com/Internet-File-Formats-Complete-Receiving/dp/188357756X/ Internet File Formats: Your Complete Resource for Sending, Receiving, and Using Internet Files] by Tim Kientzle (Oct 15, 1995)
* [http://www.amazon.com/File-Formats-Popular-PC-Software/dp/0471528064/ File Formats for Popular PC Software] by Jeff Walden (Nov 1992)
* [http://www.amazon.com/File-Formats-Conversions-Book-Developers/dp/155755059X/ PC File Formats and Conversions/Book and Disk (Developers Series)] by Ralf Kussmann (Mar 1990)
* [http://www.amazon.com/Windows-Undocumented-Formats-Working-Inside/dp/0879304375/ Windows Undocumented File Formats; Working Inside 16- and 32- bit Windows] by Pete Davis (Jan 23, 1997)
* [http://www.amazon.com/Graphics-File-Formats-Complete-Reference/dp/0070375321/ Graphics File Formats: The Complete Reference] by John R. Levine and David C. Kay (May 1992)
* [http://www.amazon.com/Using-Pcx-Graphics-Files-Programmers/dp/0879304324/ Using Pcx Graphics Files: The Programmer's Definitive Guide to Pcx File Formats] by Roger T. Stevens (Apr 1, 1996)
* [http://www.amazon.com/File-Formats-Handbook-Gunter-Born/dp/1850321175/ The File Formats Handbook] by Gunter Born (Sep 1995)
* [http://www.amazon.com/Window-System-Formats-Application-Conventions/dp/1872630154/ X Window System File Formats and Application Conventions (X11R4)] by The Open Group (Jul 17, 1991)
* [http://www.iasa-web.org/audio-preservation-tc04 Guidelines on the Production and Preservation of Digital Audio Objects] by IASA (2009)

==Other Useful Materials and Services==

* http://www.nsrl.nist.gov/ National Software Reference Library (see also http://blogs.loc.gov/digitalpreservation/2012/05/life-saving-the-national-software-reference-library/)
* http://www.openplanetsfoundation.org/software/fido Format Identification for Digital Objects
* https://github.com/usnationalarchives/File-Analyzer File Analyzer
* https://sk1project.net/modules.php?name=Products&product=uniconvertor UniConvertor is a universal vector graphics translator. It is a command line tool which uses sK1 object model to convert one format to another
* http://www.textfiles.com/programming/FORMATS/
* http://wiki.opf-labs.org/display/SPR/Digital+Preservation+Tools Wiki list of digital preservation tool lists
* http://wiki.opf-labs.org/display/REQ/Digital+Preservation+and+Data+Curation+Requirements+and+Solutions Digital Preservation and Data Curation Requirements and Solutions. Not classified by formats, but by datasets, issues and solutions.
* http://fileformats.wordpress.com/ - The File Formats Blog
* http://www.archive.org/details/cdbbsarchive - The Internet Archive has over 1,500 CD-ROMs loaded with shareware, and with that comes converters, documentation and other vital works related to file formats.
* [http://wiki.opf-labs.org/display/SPR/Crowd+sourced+Representation+Information+for+Supporting+Preservation+(cRIsp) Crowd sourced Representation Information for Supporting Preservation (cRIsp)]
* [http://www.nsrl.nist.gov/ National Software Reference Library]
* [http://mediatypes.appspot.com/ Linked Data URIs for MIME Types]
* http://darwinsys.com/file/ UNIX file(1) format guessing utility
* http://www.filesignatures.net/ Online database of file signatures (magic numbers)
* http://www.garykessler.net/library/file_sigs.html List of file signatures (magic numbers)
* [http://paulbourke.net/dataformats/ Data Formats from Paul Bourke] Useful information on older formats. Categories include Geometric (e.g., 3-D), Audio, and Image.
* [http://www.martinreddy.net/gfx/ Graphics File Formats page from Martin Reddy] Useful information on older formats. Categories are: 2-D and 3-D, with supplementary resources.
* [http://samples.multimedia.cx/ LibAV Sample Files collection, over 50GB]
* http://www.corel.com/content/pdf/wpx4/corel-wordperfect-office-X4-reviewers-guide.pdf Corel WordPerfect Office Reviewers Guide, Appendix A gives useful overview of old Office formats + their typical file extensions.
* http://documentation.statsoft.com/STATISTICAHelp.aspx?path=common/CommonMenus/CompatibleDocumentTypes Some info on formats of / compatible with Statsoft Statistica
* http://web-beta.archive.org/web/20100210224635/http://medlem.spray.se/converters Loads of DOS / Windows conversion programs - this appears to be the most recent grab held by the Internet Archive
* https://github.com/kreativekorp/ksfl Project to read/write various container formats
* http://www.kreativekorp.com/swdownload/wildfire/HC%20FILE%20FORMAT%202010.TXT Description of the [[HyperCard stack]] file format
* [http://mediaarchaeologylab.com/ Media Archeology Lab]
* [https://archive.org/details/bitsavers_dec Bitsavers document library of DEC manuals]
* [http://www.documentliberation.org/ Document Liberation Project]
* [http://qanda.digipres.org/80/unusual-sources-of-file-format-signatures Unusual sources of file format signatures]
* [http://www.deadmedia.org/ The Dead Media Project]
* [http://archives.govt.nz/advice/guidance-and-standards/guidance-subject/digital-recordkeeping/advice-file-formats-creation-di Advice on File Formats for the Creation of Digital Records] (Archives New Zealand)
* [http://www.avpreserve.com/papers-and-presentations/an-introduction-to-using-the-command-line-interface-to-work-with-files-and-directories/ An introduction to using the command line interface to work with files and directories]
* [http://www.digipres.org/ DigiPres Commons Community-owned digital preservation resources]
* [http://www.digipres.org/tools/ POWRR Tool Grid]
* [http://ec2-52-37-126-112.us-west-2.compute.amazonaws.com/falstaff/ Falstaff: online file identification utility]
* [http://access.bitcurator.net/index.php?title=Main_Page BitCurator access tools]
* [http://www.documentliberation.org/projects/ Document Liberation Project projects]

[[Category:Sources| ]]

Anex86 PC98 floppy image

2018-10-11T00:25:53Z

More barbeque:

{{FormatInfo
| formattype = electronic
| subcat = Disk Image Formats
| extensions = {{ext|fdi}}
}}

The FDI disk image format for NEC PC-98 disk images seems to originate from the Anex86 PC98 emulator.

== FDI File Header ==
The FDI header is 4096 bytes long, Intel little-endian, and starts with this 32-byte header.

typedef struct {
BYTE dummy[4];
BYTE fddtype[4];
BYTE headersize[4];
BYTE fddsize[4];
BYTE sectorsize[4];
BYTE sectors[4];
BYTE surfaces[4];
BYTE cylinders[4];
} FDIHDR;

The next 4,064 bytes of the header seem to always be pad bytes.

== FDI file image contents ==
The remaining portion of the file is a raw dd-style image of the disk. In the case of an HDM image, you can prepend a 4096-byte header onto the image to turn it into an FDI (see example below).

== Examples ==
For a 1261568-byte "2HD" disk image (like *.HDM format), the header is as follows:

{|class="wikitable"
!Field
!Value
!Comment
|-
|| dummy || 0 ||
|-
|| fddtype || 144 || Unsure what this value actually means
|-
|| header size || 4096 || Includes the 32-bytes in the header
|-
|| fdd size || 1261568 ||
|-
|| sector size || 1024 ||
|-
|| sector count || 8 ||
|-
|| surfaces || 2 || double-sided disk
|-
|| cylinders || 77 ||
|}

Anex86 PC98 floppy image

2018-10-11T00:24:33Z

More barbeque: Created page with "{{FormatInfo | formattype = electronic | subcat = Disk Image Formats | extensions = {{fdi}} }} The FDI disk image format for NEC PC-98 disk images seems to or..."

{{FormatInfo
| formattype = electronic
| subcat = Disk Image Formats
| extensions = {{fdi}}
}}

The FDI disk image format for NEC PC-98 disk images seems to originate from the Anex86 PC98 emulator.

== FDI File Header ==
The FDI header is 4096 bytes long, Intel little-endian, and starts with this 32-byte header.

typedef struct {
BYTE dummy[4];
BYTE fddtype[4];
BYTE headersize[4];
BYTE fddsize[4];
BYTE sectorsize[4];
BYTE sectors[4];
BYTE surfaces[4];
BYTE cylinders[4];
} FDIHDR;

The next 4,064 bytes of the header seem to always be pad bytes.

== FDI file image contents ==
The remaining portion of the file is a raw dd-style image of the disk. In the case of an HDM image, you can prepend a 4096-byte header onto the image to turn it into an FDI (see example below).

== Examples ==
For a 1261568-byte "2HD" disk image (like *.HDM format), the header is as follows:

{|class="wikitable"
!Field
!Value
!Comment
|-
|| dummy || 0 ||
|-
|| fddtype || 144 || Unsure what this value actually means
|-
|| header size || 4096 || Includes the 32-bytes in the header
|-
|| fdd size || 1261568 ||
|-
|| sector size || 1024 ||
|-
|| sector count || 8 ||
|-
|| surfaces || 2 || double-sided disk
|-
|| cylinders || 77 ||
|}

Disk Image Formats

2018-10-11T00:10:10Z

More barbeque: /* Misc. */ Add Anex86 PC98 floppy image

{{FormatInfo
|formattype=electronic
|thiscat=Disk Image Formats
|image=Selanit floppy disk download icon.png
}}

[[Filesystem]]s and other file formats described on this page's parent page describe entities that exist on disks as they would normally be used.

However, there are also file formats that allow a physical disk (or tape or other medium) to be "imaged" into a file, either to be re-copied onto a physical disk later, or else used by an *emulator* of the computer system that would have originally used that sort of media.

These file formats represent a data image of one of the physical floppy disk (etc.) formats. In addition to the raw data from the disk/tape/etc, these formats may contain metadata that the corresponding emulator environment uses to correctly access the data.

Typically, when archiving a physical disk for later use, a physical disk will be imaged into one of these file formats. Sub-pages of this page should contain information on the format itself as well as instructions on how to extract a physical disk into that disk image format, including information about what physical drives are required, what software to use and how and what OS that is run under.

See also [[ROM and memory images]] for images of memory, and [[Floppy disk]]s for the physical floppies. See [[Archiving]] for archive and backup formats that are filesystem-based rather than raw-sector-based. See [[Filesystem]] for the filesystems. [[Forensics and Law Enforcement]] has formats for preserving drive contents with metadata for investigations.

==Floppy Disk Image Formats==
Images created directly from floppy disks.
===Generic Sector Images===

====Acorn====

* [[Acorn Archimedes Disk Image]]

====Amiga====

* [[ADF (Amiga)]]
* [[Disk Imploder]] (Amiga) (.dmp, .dex)
* [[Disk Masher System]] (Amiga) (.dms)

====Amstrad====

* [[DSK (CPCEMU)]], used for Amstrad and related disks (typically 3" disks)

====APF Imagination Machine====

* [[APD]] (APF Imagination Machine emulated disk) (.apd)

====Apple II series====

* [[Apple Disk Image]] (Apple II disk image with header) (.2mg)
* [[Disk Copy 4.2]] (Apple)
* [[Disk Copy 6]] (Apple)
* [[DSK (Apple II)]], used for Apple II disks (typically 5.25" disks)
* [[EDD disk image]]
* [[Nibblized disk image]] (.nib), an Apple II disk image format that works for copy-protected disks
* [[WOZ disk image]]

====Atari====

* [[ATR]] (SIO2PC)
* [[DCM]] (Disk Communicator)
* [[DI]] (800XL DJ emulator)
* [[DIM]] (F-Copy)
* [[MSA (Magic Shadow Archiver)]]
* [[PRO]] (APE ProSystem)
* [[SCP]] (Spartados)
* [[ST disk image]]
* [[STT]] (STEEM)
* [[STX]] (Pasti)
* [[XFD]] (Xformer)

====Commodore 64/128====

* [[C64 Nibbler Format]] (.nib) (created by mnib)
* [[D64]] and its variants (D71, D81, all with optional error information) from the C64
* [[G64]] C64 raw (GCR coded) disk image

====Macintosh====

* [[Apple Disk Image]] (MacOS or OS X disk image with header) (.dmg, .smi, .img)
* [[DART]] (Apple)

====PC (DOS/Windows)====

* [[CopyQM]]
* [[DDI]] (DiskDupe image file)
* [[FDI]] (Formatted Disk Image)
* [[IMA]]
* [[IMD]] (ImageDisk)
* [[IMG (DOSIMG)|IMG]] (HD-Copy disk image)
* [[Quick Release Sector Transfer]] (Compaq)
* [[TD0]] (TeleDisk)

====Tandy====

* [[DMK]]
* [[JV1/JV3]]

====Misc.====
* [[Anex86 PC98 floppy image]] (.fdi)
* [[DiscFerret floppy image]] (.dfi)
* [[Famicom Disk System]] (.fds)
* [[MESS floppy image]] (.mfi)
* [[Pasopia disk image]] (.d88)
* [[Raw disk image]] (.img) (ie, created with dd)

===Physical (Hardware) Images===

* [[DRAFT]] (Software Preservation Society intermediate format)
* [[IPF]] (Software Preservation Society image format)
* [[KryoFlux]]

==Optical Disc Image Formats==
CD-ROM, DVD-ROM, BluRay, GD-ROM etc.

* [[BWI]] (BlindWrite Image File)
* [[CDI]] (Disc Juggler)
* [[CloneCD Control File]] (.ccd, .img, .sub)
* [[CUE and BIN]]
* [[DDP]] (Disc Description Protocol)
* [[DMG]] (Apple)
* [[ISO image]] (.iso)
* [[ISZ]]
* [[MDF and MDS]]
* [[NRG]] (Nero Burning ROM)

==Hard Disk Image Formats==

* [[HDV (disk image)|HDV]]
* [[Raw disk image]] (.img)

==Virtual Machine Disks and Virtualization Image Formats==
* [[OVF (Open Virtualization Format)]]
* [[qcow]] (QEMU)
* [[VDI]] (Virtual Disk Image) (VirtualBox) (.vdi)
* [[VHD (Virtual Hard Disk)]] (Microsoft) (.vhd)
* [[VMDK]] (Virtual Machine Disk) (VMWare) (.vmdk)

==Tape Image Formats==

* [[APT]] (APF Imagination Machine emulated tape, logical bits) (.apt)
* [[APW]] (APF Imagination Machine emulated tape, raw audio) (.apw)
* [[CAS (Atari)|CAS]] (Atari; Digital Cassette Image system)
* [[PZX]] (ZX Spectrum)
* [[T64]]
* [[TAP (ZX Spectrum)]]
* [[Tap file]] (contains data from Commodore data cassette)
* [[TZX]] (ZX Spectrum)
* [[Unified Emulator Format]] (UEF; also does Acorn disks and ROMs)

==Unknown==
* [[lbl]] See here for details: [http://classiccmp.org/pipermail/cctalk/2012-December/326747.html http://classiccmp.org/pipermail/cctalk/2012-December/326747.html]

==Tools==
* [[The Sleuth Kit and Autopsy]] can read raw, Expert Witness, and AFF formats
* [http://www.winimage.com/ WinImage] to read, write and create images, also lots of info
* [http://foremost.sourceforge.net/ Foremost: console tool to recover files from disk images]
* [http://disktype.sourceforge.net/ disktype: detect format of disk or disk image]
* [http://mark0.net/soft-trid-e.html TrID File Identifier]: recognize many disk image formats
* [https://github.com/bitcurator/bca-webtools bca-webtools]
* [https://github.com/eaas-framework/xmount xmount: emulates various disk image formats for use in emulators requiring them]

See also [[Disk Imaging Software & Systems]], and [[Forensics and Law Enforcement]].

== Links ==
* [http://qanda.digipres.org/20/what-are-the-reasons-for-saving-disk-images?show=183 What are the reasons for saving disk images?]
* [http://ils.unc.edu/digccurr/curategear2016-talks/woods-curategear2016.pdf BCA-Webtools: Accessing and Visualizing Disk Images in a Web Browser]