Just Solve the File Format Problem - User contributions [en]

MikMod UNI format module

2025-09-23T04:02:10Z

Anonymoususer852: Added release date for this format.

{{FormatInfo
|formattype=electronic
|subcat=Audio and Music
|extensions={{ext|uni}}
|magic=[https://github.com/file/file/blob/master/magic/Magdir/audio#L173 55 4e 30 35]
|released=2005<ref>[https://sourceforge.net/p/mikmod/mikmod/ci/048f5f9552a1a08694703fedd08a86a1d97f2284/ Mikmod Sound System / Code / Commit <nowiki>[048f5f]</nowiki> - SourceForge]</ref>
}}

== Description ==
'''.uni''' (MikMod UNI format module) "is an internal format used by [https://mikmod.sourceforge.net/ MikMod]" module player/library.<ref>[https://tldp.org/HOWTO/Sound-Playing-HOWTO-2.html Sound Playing HOWTO 2 - TLDP.org]</ref><ref>[https://sourceforge.net/p/mikmod/mikmod/ci/master/tree/libmikmod/loaders/load_uni.c libmikmod/loaders/load_uni.c - MikMod VCS repository on SourceForge]</ref>

== Identification ==
'''.uni''' files begins with the HEX {{magic|55 4e 30 35}} which translates to "<code>UN05</code>" in ASCII.

== Software ==
* [https://mikmod.sourceforge.net/ MikMod]

== Sample files ==
* [https://discmaster.textfiles.com/search?mode=deep&extension=.uni&family=music&format=mikMod Search results of files with .uni extension - Discmaster.textfiles.com]
** Possibly the best known examples of this is the music that comes with a hacking game named "Uplink", inside the [https://discmaster.textfiles.com/browse/17646/Ambrosia%20Software%20CD-ROM%20(Ambrosia%20Software,%20Inc.)(2002).bin/pc/Games/Adventure/Uplink/Uplink%201.3.3/Data/music.dat/music music.dat] which the dat file itself is effectively a [[ZIP]] file.

== References ==
<references/>

[[Category:Game data files]]
[[Category:Introversion Software]]
[[Category:Uplink (Game)]]

Redshirt

2025-09-23T03:37:25Z

Anonymoususer852: Added note about redshirt2.

{{FormatInfo
|formattype=electronic
|subcat=Encryption
|magic=52 45 44 53 48 49 52 54 (REDSHIRT) 4F 4E 45 54 49 4D 45 52 45 44 53 48 49 52 54 (ONETIMEREDSHIRT)
|released=2001
}}
'''Redshirt''' was used in the [[wikipedia:Uplink (video game)|Uplink (video game)]]<ref>[https://forums.introversion.co.uk/viewtopic.php?f=44&t=32354 just want to asking about .dat in uplink - Introversion.co.uk forum]</ref> to bring challenges towards players who are curious in nature. The featured game itself is about computer hacking, so the pseudo encryption was used to encrypt save games for instance to prevent players cheating the game<ref>[https://forums.introversion.co.uk/viewtopic.php?p=416942#p416942 fatal crash on start up - Introversion.co.uk forum]</ref>, or by encrypting portions of game assets to cater for easter eggs, like the game bible series that were distributed on separate discs<ref>[https://uplinkfan.blogspot.com/2011/10/getting-book-2-of-uplink-game-bible.html Getting Book 2 of the Uplink Game Bible - UplinkFan on blogspot.com]</ref> or shared on peer-to-peer file sharing software like Kazaa.<ref>[https://uplinkfan.blogspot.com/2011/10/getting-book-4-of-uplink-game-bible.html Getting Book 4 of the Uplink Game Bible - UplinkFan on blogspot.com]</ref>

Earlier versions of Redshirt uses simple XOR encryption scheme<ref>[https://crypto.stackexchange.com/a/1557 How can I break REDSHIRT / REDSHIRT2 encryption? - Stack Exchange]</ref> Latter revisions incorporated features like one-time-pad (OTP) encryption<ref>[https://uplinkfan.blogspot.com/2011/10/getting-book-2-of-uplink-game-bible.html Getting Book 2 of the Uplink Game Bible - UplinkFan on blogspot.com]</ref>. Certain revisions of redshirt2 used a different technique.<ref>[https://forums.introversion.co.uk/viewtopic.php?p=424012&sid=acd88a8fd17623597924e191fe40d933#p424012 Redshirt2 and Hex Editor - Introversion.co.uk forum]</ref>

== Identification ==
'''Redshirt''' typically begins with HEX {{magic|52 45 44 53 48 49 52 54}} which corresponds to ASCII string "<code>REDSHIRT</code>". Redshirt with one-time-pad (OTP) encryption begins with HEX {{magic|4F 4E 45 54 49 4D 45 52 45 44 53 48 49 52 54}} which corresponds to ASCII string "<code> ONETIMEREDSHIRT</code>".

== Software ==
=== Windows ===
* https://web.archive.org/web/20040219012231/http://www.stewsburntmonkey.com/redshirt.exe
* Uplink Developer CD has these under tools folder.

=== Linux/Unix ===
* http://www.mario-konrad.ch/blog/software/redshirt/index.html

=== Macintosh ===
* http://www.cytheraguides.com/archives/ambrosia_addons/uplink/ - the file in question is called "RedshirtX.sit"

=== Non-OS specific ===
==== Python ====
* [https://crypto.stackexchange.com/a/1557 How can I break REDSHIRT / REDSHIRT2 encryption? - Stack Exchange]

==== Rust ====
* https://lib.rs/crates/redshirt - upstream repository: https://github.com/FaultyRAM/redshirt-rs

== Sample files ==
* [https://discmaster.textfiles.com/view/17646/Ambrosia%20Software%20CD-ROM%20(Ambrosia%20Software,%20Inc.)(2002).bin/pc/Games/Adventure/Uplink/Uplink%201.3.3/Data/data.dat data.dat - Uplink 1.3.3 - Ambrosia Software CD-ROM (Ambrosia Software, Inc.)(2002).bin - Discmaster.textfiles.com]
* [https://discmaster.textfiles.com/view/17646/Ambrosia%20Software%20CD-ROM%20(Ambrosia%20Software,%20Inc.)(2002).bin/pc/Games/Adventure/Uplink/Uplink%201.3.3/Data/world.dat world.dat - Uplink 1.3.3 - Ambrosia Software CD-ROM (Ambrosia Software, Inc.)(2002).bin - Discmaster.textfiles.com]

== References ==
<references/>

[[Category:Game data files]]
[[Category:Introversion Software]]
[[Category:Uplink (Game)]]

Category:Introversion Software

2025-09-22T17:27:48Z

Anonymoususer852: Created page with "Category:Companies and organizations"

[[Category:Companies and organizations]]

MikMod UNI format module

2025-09-22T17:27:07Z

Anonymoususer852: Added extra categories in which best exemplifies the use of such formats.

{{FormatInfo
|formattype=electronic
|subcat=Audio and Music
|extensions={{ext|uni}}
|magic=[https://github.com/file/file/blob/master/magic/Magdir/audio#L173 55 4e 30 35]
}}

== Description ==
'''.uni''' (MikMod UNI format module) "is an internal format used by [https://mikmod.sourceforge.net/ MikMod]" module player/library.<ref>[https://tldp.org/HOWTO/Sound-Playing-HOWTO-2.html Sound Playing HOWTO 2 - TLDP.org]</ref><ref>[https://sourceforge.net/p/mikmod/mikmod/ci/master/tree/libmikmod/loaders/load_uni.c libmikmod/loaders/load_uni.c - MikMod VCS repository on SourceForge]</ref>

== Identification ==
'''.uni''' files begins with the HEX {{magic|55 4e 30 35}} which translates to "<code>UN05</code>" in ASCII.

== Software ==
* [https://mikmod.sourceforge.net/ MikMod]

== Sample files ==
* [https://discmaster.textfiles.com/search?mode=deep&extension=.uni&family=music&format=mikMod Search results of files with .uni extension - Discmaster.textfiles.com]
** Possibly the best known examples of this is the music that comes with a hacking game named "Uplink", inside the [https://discmaster.textfiles.com/browse/17646/Ambrosia%20Software%20CD-ROM%20(Ambrosia%20Software,%20Inc.)(2002).bin/pc/Games/Adventure/Uplink/Uplink%201.3.3/Data/music.dat/music music.dat] which the dat file itself is effectively a [[ZIP]] file.

== References ==
<references/>

[[Category:Game data files]]
[[Category:Introversion Software]]
[[Category:Uplink (Game)]]

Category:Uplink (Game)

2025-09-22T17:25:34Z

Anonymoususer852: Created page with "This category includes formats and specifications of the 2001 hacking game, ''Uplink''. Category:Introversion Software Category:Game data files"

This category includes formats and specifications of the 2001 hacking game, ''Uplink''.

[[Category:Introversion Software]]
[[Category:Game data files]]

Redshirt

2025-09-22T17:22:21Z

Anonymoususer852: Created page with "{{FormatInfo |formattype=electronic |subcat=Encryption |magic=52 45 44 53 48 49 52 54 (REDSHIRT) 4F 4E 45 54 49 4D 45 52 45 44 53 48 49 52 54 (ONETIMEREDSHIRT) |released=..."

{{FormatInfo
|formattype=electronic
|subcat=Encryption
|magic=52 45 44 53 48 49 52 54 (REDSHIRT) 4F 4E 45 54 49 4D 45 52 45 44 53 48 49 52 54 (ONETIMEREDSHIRT)
|released=2001
}}
'''Redshirt''' was used in the [[wikipedia:Uplink (video game)|Uplink (video game)]]<ref>[https://forums.introversion.co.uk/viewtopic.php?f=44&t=32354 just want to asking about .dat in uplink - Introversion.co.uk forum]</ref> to bring challenges towards players who are curious in nature. The featured game itself is about computer hacking, so the pseudo encryption was used to encrypt save games for instance to prevent players cheating the game<ref>[https://forums.introversion.co.uk/viewtopic.php?p=416942#p416942 fatal crash on start up - Introversion.co.uk forum]</ref>, or by encrypting portions of game assets to cater for easter eggs, like the game bible series that were distributed on separate discs<ref>[https://uplinkfan.blogspot.com/2011/10/getting-book-2-of-uplink-game-bible.html Getting Book 2 of the Uplink Game Bible - UplinkFan on blogspot.com]</ref> or shared on peer-to-peer file sharing software like Kazaa.<ref>[https://uplinkfan.blogspot.com/2011/10/getting-book-4-of-uplink-game-bible.html Getting Book 4 of the Uplink Game Bible - UplinkFan on blogspot.com]</ref>

Earlier versions of Redshirt uses simple XOR encryption scheme<ref>[https://crypto.stackexchange.com/a/1557 How can I break REDSHIRT / REDSHIRT2 encryption? - Stack Exchange]</ref> Latter revisions incorporated features like one-time-pad (OTP) encryption<ref>[https://uplinkfan.blogspot.com/2011/10/getting-book-2-of-uplink-game-bible.html Getting Book 2 of the Uplink Game Bible - UplinkFan on blogspot.com]</ref>.

== Identification ==
'''Redshirt''' typically begins with HEX {{magic|52 45 44 53 48 49 52 54}} which corresponds to ASCII string "<code>REDSHIRT</code>". Redshirt with one-time-pad (OTP) encryption begins with HEX {{magic|4F 4E 45 54 49 4D 45 52 45 44 53 48 49 52 54}} which corresponds to ASCII string "<code> ONETIMEREDSHIRT</code>".

== Software ==
=== Windows ===
* https://web.archive.org/web/20040219012231/http://www.stewsburntmonkey.com/redshirt.exe
* Uplink Developer CD has these under tools folder.

=== Linux/Unix ===
* http://www.mario-konrad.ch/blog/software/redshirt/index.html

=== Macintosh ===
* http://www.cytheraguides.com/archives/ambrosia_addons/uplink/ - the file in question is called "RedshirtX.sit"

=== Non-OS specific ===
==== Python ====
* [https://crypto.stackexchange.com/a/1557 How can I break REDSHIRT / REDSHIRT2 encryption? - Stack Exchange]

==== Rust ====
* https://lib.rs/crates/redshirt - upstream repository: https://github.com/FaultyRAM/redshirt-rs

== Sample files ==
* [https://discmaster.textfiles.com/view/17646/Ambrosia%20Software%20CD-ROM%20(Ambrosia%20Software,%20Inc.)(2002).bin/pc/Games/Adventure/Uplink/Uplink%201.3.3/Data/data.dat data.dat - Uplink 1.3.3 - Ambrosia Software CD-ROM (Ambrosia Software, Inc.)(2002).bin - Discmaster.textfiles.com]
* [https://discmaster.textfiles.com/view/17646/Ambrosia%20Software%20CD-ROM%20(Ambrosia%20Software,%20Inc.)(2002).bin/pc/Games/Adventure/Uplink/Uplink%201.3.3/Data/world.dat world.dat - Uplink 1.3.3 - Ambrosia Software CD-ROM (Ambrosia Software, Inc.)(2002).bin - Discmaster.textfiles.com]

== References ==
<references/>

[[Category:Game data files]]
[[Category:Introversion Software]]
[[Category:Uplink (Game)]]

Category:File formats with extension .uni

2025-09-22T15:45:01Z

Anonymoususer852: Created page with "U"

[[Category:File formats by extension|U]]

MikMod UNI format module

2025-09-22T15:43:11Z

Anonymoususer852: Created page with "{{FormatInfo |formattype=electronic |subcat=Audio and Music |extensions={{ext|uni}} |magic=[https://github.com/file/file/blob/master/magic/Magdir/audio#L173 55 4e 30 35] }} =..."

{{FormatInfo
|formattype=electronic
|subcat=Audio and Music
|extensions={{ext|uni}}
|magic=[https://github.com/file/file/blob/master/magic/Magdir/audio#L173 55 4e 30 35]
}}

== Description ==
'''.uni''' (MikMod UNI format module) "is an internal format used by [https://mikmod.sourceforge.net/ MikMod]" module player/library.<ref>[https://tldp.org/HOWTO/Sound-Playing-HOWTO-2.html Sound Playing HOWTO 2 - TLDP.org]</ref><ref>[https://sourceforge.net/p/mikmod/mikmod/ci/master/tree/libmikmod/loaders/load_uni.c libmikmod/loaders/load_uni.c - MikMod VCS repository on SourceForge]</ref>

== Identification ==
'''.uni''' files begins with the HEX {{magic|55 4e 30 35}} which translates to "<code>UN05</code>" in ASCII.

== Software ==
* [https://mikmod.sourceforge.net/ MikMod]

== Sample files ==
* [https://discmaster.textfiles.com/search?mode=deep&extension=.uni&family=music&format=mikMod Search results of files with .uni extension - Discmaster.textfiles.com]
** Possibly the best known examples of this is the music that comes with a hacking game named "Uplink", inside the [https://discmaster.textfiles.com/browse/17646/Ambrosia%20Software%20CD-ROM%20(Ambrosia%20Software,%20Inc.)(2002).bin/pc/Games/Adventure/Uplink/Uplink%201.3.3/Data/music.dat/music music.dat] which the dat file itself is effectively a [[ZIP]] file.

== References ==
<references/>

Talk:Unified diff

2025-09-21T00:00:15Z

Anonymoususer852: Added topics for discussion.

== Identification mess ==
Not always does unified diff or patch files consistently begins with <code>@@</code>. Some, notably older unified diff files also contains <code>***</code> and <code>---</code> instead of <code>---</code> and <code>+++</code> to provide file changes, e.g.
*** lib/cap/atalkdbm.c.orig Wed Mar 20 17:02:57 1991
--- lib/cap/atalkdbm.c Wed Mar 20 17:03:30 1991
***************
*** 1,7 ****
/*
! * $Date: 91/03/15 14:56:00 $
! * $Header: atalkdbm.c,v 2.3 91/03/15 14:56:00 djh Exp $
! * $Revision: 2.3 $
Source: https://tsubaki.st.ryukoku.ac.jp/pub/network/cap/cap.patches/cap60.patch001

In general, a modern <code>patch</code> utility will often detect whether or not if the supplied file is in the specified unified diff format and prompt if necessary.

[[User:Anonymoususer852|Anonymoususer852]] ([[User talk:Anonymoususer852|talk]]) 00:00, 21 September 2025 (UTC)

== Interesting quirks when applying changes via the utilities ==
=== <code>.orig</code>/<code>.rej</code> files ===
<code>patch</code> will produce a file with <code>.orig</code> extension if:
* <code>--backup</code> or <code>--backup-if-mismatch</code> is set, or
* the targeted file is successfully applied with fuzzing
** fuzzing "only applies to diffs that have context, and causes patch to ignore up to that many lines of context in looking for places to install a hunk." -- https://www.gnu.org/software/diffutils/manual/html_node/Inexact.html

In addition, <code>patch</code> will produce a file with <code>.rej</code> extension if:
* the specified diff/patch file does not correctly match with the contents of the file to be patched itself, or
* the fuzz factor was set too low

<code>git apply --3way ...</code> in particular will not generate either of these files in applicable cases, if the changes did not apply cleanly.

=== Preserving timestamp ===
When applying changes via GNU <code>patch</code>, the updated file or files often has the last-modified time adjusted to the date when the changes were applied. This is a rather unfortunate side-effect in which those wanting to preserve timestamp that is as indicated (which is an optional field) in the supplied unified diff or patch be consequently lost in the process. While it is possible to re-adjust the last-modified timestamp to more accurately represent the time when it was initially modified, via <code>touch</code>, there is no trivial way of doing that as either,
* Indicated timestamp format is tied to the creator of unified diff's locale,
** i.e. it may or may not be in [[wikipedia:ISO_8601|ISO 8601]] format,
** timezone is specific to the creator of unified diff, which can cause time skew/variance
* The optional field is not in a time format, but RCS/VCS indicating changes from the version that it was previously from, or,
* The optional field is empty

[[User:Anonymoususer852|Anonymoususer852]] ([[User talk:Anonymoususer852|talk]]) 00:00, 21 September 2025 (UTC)

Unified diff

2025-09-20T22:56:21Z

Anonymoususer852: Indicate and provide example of git utility can be used to also generate and apply unified diff.

{{FormatInfo
|subcat=Archiving
|extensions={{ext|patch}}, {{ext|diff}}
|released=1991
|image=Unified diff.png
|caption=Unified diff example
}}
'''Unified diff''' (or '''diff unified format''', '''unified context diff''', '''unidiff''') is a [[diff]] format commonly used for patch files.

== Identification ==
Unlike other diff formats, Unified diff has lines that begin with "@@".

== Specifications ==
* [http://www.artima.com/weblogs/viewpost.jsp?thread=164293 Unified Diff Format] - Ad-hoc specification by Guido van Rossum
* [http://www.gnu.org/software/diffutils/manual/html_node/Detailed-Unified.html GNU Diffutilis manual: Detailed Description of Unified Format]

== Software ==
* [http://www.gnu.org/software/diffutils/ GNU Diffutils]: <code>diff</code> (with <code>-u</code> option), <code>patch</code>
* [https://git-scm.com/ git]: <code>git</code> with either <code>git diff...</code>, <code>git format-patch...</code>, or <code>git log -p</code> can be used to provide unified diff
** Conversely <code>git apply ...</code> can be used to apply diff or patches that were created via <code>git</code> and that the user is inside a functional git filesystem, where top directory has a folder named <code>.git</code>

== Sample files ==
* [http://www.gnu.org/software/diffutils/manual/html_node/Example-Unified.html Example from the GNU Diffutils manual]
* http://www.apache.org/dist/httpd/patches/ ...
* [https://patch-diff.githubusercontent.com/raw/git/git/pull/6.diff Updated git-remote documentation #6 in patch/diff format - GitHub] - via inputting <code>https://github.com/git/git/pull/6.diff</code> into the address bar will make GitHub provide the pull request (PR) in a suitable patch/diff format

== Links ==
* [[Wikipedia:Diff utility#Unified_format|Wikipedia:Diff utility]] has a section on Unified diff

User:Anonymoususer852

2025-09-05T16:59:35Z

Anonymoususer852:

Hi, I [[Special:Contributions/Anonymoususer852|waffle a lot]] lol.

== My ramblings ==
* [[EPWING]]
* [[EBZip]]
** [[Talk:EBZip]] - This especially :'D.
* [[DWARFS]]
* [[SquashFS]]
** [[Talk:Squashfs]]
* [[XP3]]
* [[Linux]]
* [[Initramfs]]
* [[Vmlinuz]]
* [[Firmware]]
* [[Solid-State Drive]]
* [[Advanced Forensics Format]]
…

User:Anonymoususer852

2025-09-05T16:41:38Z

Anonymoususer852: /* My ramblings */

Hi, I waffle a lot lol.

== My ramblings ==
* [[EPWING]]
* [[EBZip]]
** [[Talk:EBZip]] - This especially :'D.
* [[DWARFS]]
* [[SquashFS]]
** [[Talk:Squashfs]]
* [[XP3]]
* [[Linux]]
* [[Initramfs]]
* [[Vmlinuz]]
* [[Firmware]]
* [[Solid-State Drive]]
…

User:Anonymoususer852

2025-09-05T16:38:40Z

Anonymoususer852: /* My ramblings */

User:Anonymoususer852

2025-09-05T16:38:15Z

Anonymoususer852: /* My ramblings */

Hi, I waffle a lot lol.

== My ramblings ==
* [[EPWING]]
* [[EBZip]]
** [[Talk:EBZip]] - This especially :'D.
* [[DWARFS]]
* [[SquashFS]]
** [[Talk:SquashFS]]
* [[XP3]]
* [[Linux]]
* [[Initramfs]]
* [[Vmlinuz]]
* [[Firmware]]
…

User:Anonymoususer852

2025-09-05T16:35:11Z

Anonymoususer852: Created page with "Hi, I waffle a lot lol. == My ramblings == * EPWING * EBZip ** Talk:EBZip - This especially :'D. * DWARFS * SquashFS * XP3 * Linux * Initramfs..."

Hi, I waffle a lot lol.

== My ramblings ==
* [[EPWING]]
* [[EBZip]]
** [[Talk:EBZip]] - This especially :'D.
* [[DWARFS]]
* [[SquashFS]]
* [[XP3]]
* [[Linux]]
* [[Initramfs]]
* [[Vmlinuz]]
* [[Firmware]]
…

EBZip

2025-09-05T16:07:28Z

Anonymoususer852: Updated magic, after comparing it with other .ebz files like HONMON.ebz and GAI*.ebz files.

{{FormatInfo
|formattype=electronic
|subcat=Compression
|extensions={{ext|ebz}}
|compression=lossless
|magic=45 42 5A 69 70 15 00 00 00 00
|released=1999<ref>[https://web.archive.org/web/20010619200710/http://www.sra.co.jp/people/m-kasahr/eb/ChangeLog-ja.html ChangeLog - SRA website mirrored on Internet Archive]</ref>
}}
{{notice|1=Majority of the links/citations are in Japanese.}}

'''Electronic Book Zip or EBZip''' is a compression format used in electronic dictionaries that were distributed primarily on compact discs (CD). It is also one of the many frontend utilities that uses the EB library for manipulating [[EPWING]] format.

The electronic dictionaries stored on computers eliminates the need to have physical media, and allows remote referencing if it was connected to a network, using the likes of <code>dserver</code> or <code>ebnetd</code> for instance. It is likely that compression was needed due to the fact that hard disks were expensive compared to compact discs, when having a number of electronic dictionaries stored on a computer, it made little sense to retain physical media just to reference a number of words every so often, and to juggle around ejecting/inserting discs into the computer. For the expense of sacrificing few CPU cycles and potentially insignificant amount of memory, these electronic dictionaries could be made to take less storage space<ref>[https://web.archive.org/web/20170705202139/http://sci.lang.japan.narkive.com/X84h0clT/kenkyusha-dictionary-cd-and-epwing#post2 Kenkyusha dictionary CD and EPWING - sci.lang.japan NARKIVE - Internet Archive copy]</ref>, and allows faster dictionary lookups due to hard disks allowing random access.

The compression is lossless, initially using a modified [https://www.zlib.net/ zlib]<ref>[https://github.com/mistydemeo/eb/blob/742248c23cf7769d0fed412cbce458222a437795/zlib/README.altered Last version released with internal zlib - README.altered - version 4.1.3 - Misty De Méo]</ref>, then likely unmodified zlib by the original <code>ebzip</code> author<ref>[https://github.com/mistydemeo/eb/commit/742ddab4f255b7c34d7d1f1b4e3fd245bfc3ea18 Internal zlib removed - Commit 742ddab (4.2) - Misty De Méo]</ref>. Do keep in mind that not everything is compressed, EBZ format is not a container like [[ZIP]] for instance, selected files (typically <code>CATALOGS</code> and <code>HONMON</code>) are compressed and have their filenames bearing these extensions, e.g. <code>HONMON.ebz</code>.

== History ==
EB-like software for Unix-like began circulating on Japanese Usenet as early as September 1991 in the form of [[shar]] by Shingo Nishioka of Osaka University.<ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=NISHIOKA.91Sep3191513@sp4.sanken.osaka-u.ac.jp dserver/Electronic Book Dictionary server & clients (1/2) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=NISHIOKA.91Sep3191550@sp4.sanken.osaka-u.ac.jp dserver/Electronic Book Dictionary server & clients (2/2) - AncientFJ - katsu.watanabe.name]</ref> Versions before 2.1beta of <code>dserver</code> lacked compression, which was introduced as early as 1996-03-25 by Yura Shunsuke.<ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=4j5h0e$2am@isnews.is.s.u-tokyo.ac.jp dserver 2.1-beta patch for compressed dictionary - AncientFJ - katsu.watanabe.name]</ref> Gradually over time, Motoyuki Kasahara, a member of Software Research Associates (SRA) and the author of EB Library, began developing <code>ndtpd</code> which handles electronic dictionary or dictionaries over the network, replacing <code>dserver</code>.<ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5rp4qp$3sk$1@sranhh.sra.co.jp BookView 1.0 - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sctc3$irv$2@sranhh.sra.co.jp BookView 1.0.2 (Re: BookView 1.0) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=64emm8$mcf$1@sranhh.sra.co.jp BookView 1.1 (1/3) - AncientFJ - katsu.watanabe.name]</ref> The first release of such software was around 1997-06-04 as <code>ndtpd-1.0beta0</code> (via FTP) with which it included <code>eb</code> which was a library. Earliest release on fj.sources was 1997-08-07 as <code>ndtpd-1.0</code>, which also describes the relationship between <code>dserver</code> and <code>ndtpd</code>, as [[shar]].<ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc4ns$31a$2@sranhh.sra.co.jp ndtpd 1.0 (0/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc4pr$3o1$1@sranhh.sra.co.jp ndtpd 1.0 (1/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc4rn$3o1$2@sranhh.sra.co.jp ndtpd 1.0 (2/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc4te$3o1$3@sranhh.sra.co.jp ndtpd 1.0 (3/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc4vf$3o1$4@sranhh.sra.co.jp ndtpd 1.0 (4/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc514$3o1$5@sranhh.sra.co.jp ndtpd 1.0 (5/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc52p$3o1$6@sranhh.sra.co.jp ndtpd 1.0 (6/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc54s$3o1$7@sranhh.sra.co.jp ndtpd 1.0 (7/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc55j$3o1$8@sranhh.sra.co.jp ndtpd 1.0 (8/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc56d$3o1$9@sranhh.sra.co.jp ndtpd 1.0 (9/9) - AncientFJ - katsu.watanabe.name]</ref> Until the release of <code>ndtpd-3.0alpha0</code> (2000-06-04), <code>eb</code> has always been included. EB Library began as a separate project with the first release in 1999-06-16 as <code>eb-2.2</code>. Both of these projects continued until the discontinuation of <code>ndtpd</code> (as a release) on 2003-05-24 with the version 3.1.5. <code>eb</code> remained to be maintained until 2010-03-08 with the version 4.4.3.

== EBUZip ==
EBUZip is a fork of EBZip that is maintained by Ito Kazuhiro. The 'U' in EBUZip denotes the ability to support JIS X 4081 UTF-8 extensions provided that it is compiled and configured correctly. It also brings in new features that differs from EBZip such as,
* The ability to use libdeflate algorithm at compile time, which, "is a library for fast, whole-buffer DEFLATE-based compression and decompression."<ref>[https://github.com/ebiggers/libdeflate?tab=readme-ov-file#overview README.md Overview section - libdeflate - GitHub]</ref>, which was added on 2019-06-30.<ref>[http://green.ribbon.to/~ikazuhiro/dic/ebu.html EB Library with UTF-8 extension]</ref>. Additionally, the new maintainer provided benchmark results against uncompressed, zlib and libdeflate.<ref>[http://ikazuhiro.s206.xrea.com/article.php/20190701203038572 Compressing EPWING (using libdeflate)]</ref>
** libdeflate can also decompress EPWING e-books compressed with zlib, as well as recompressing them with potentially higher compression ratio. This could subsequently make it a viable candidate instead of needing to alternate between EBZip and EBUZip, see the [[Talk:EBZip#The_viability_ebuzip_as_a_possible_drop-in_replacement_candidate|Talk:EBZip section on this]] which is covered in detail.
* Fixes to work on a newer codebase, as opposed to maintaining a series of patches on top of libeb.<ref>[https://sources.debian.org/data/main/e/eb/4.4.3-14/debian/patches/010_debian.patch 010_debian.patch - eb-4.4.3-14 - sources.debian.org]</ref>

Mr. Kazuhiro have also previously contributed to Mr. Kasahara's EBZip project, on 2000-06-24.<ref>[https://github.com/mistydemeo/eb/blob/master/ChangeLog.1#L1371 ChangeLog.1 (line 1371) - EB Library - Misty De Méo]</ref>.

== Identification ==
A <code>EBZ</code> file begins with bytes {{magic|45 42 5a 69 70 15 00 00 00 00}}, in ASCII this would be <code>"EBZip...."</code>.

== Examples ==
'''These must be executed in the top level of (book) directory that contains <code>CATALOGS</code> file, otherwise it will default to your current directory <code>$(pwd)</code>.''' See [[EPWING#Content structure]] for the directory structure of book layout.
If you are using EBUZip, the name of the compiled binary is <code>ebuzip</code>, as opposed to <code>ebzip</code>.

Show information about the file:
ebzip --information
Decompress files:
ebzip --uncompress wikipedia-fpw
Compress files:
ebzip wikipedia-fpw
Test files:
ebzip --test wikipedia-fpw
Show help:
ebzip --help

== Sample file ==
{| border="1" align="center" style="text-align:center;" cellpadding="6"
| Filename
| Main server download link
| Mirror server download link
| Notes
|-
| kanjidic_en.fpwebz.tar
| [http://ftp.edrdg.org/pub/Nihongo/edict_en.fpwebz.tar EDRDG.org]
| [http://ftp.usf.edu/pub/ftp.monash.edu.au/pub/nihongo/kanjidic_en.fpwebz.tar USF.edu]
| Use [[7-Zip]] to open on Windows machines for instance, otherwise [[Tar]] should suffice.
|}

== Software (mostly Japanese only) ==
See also: [[Wikipedia:ja:EPWING#検索ソフトウェア]]
* Windows
** [ftp://ftp.sra.co.jp/pub/misc/eb/WinEBZip-0.0.exe WinEBZip.exe on SRA's FTP server - no longer maintained]
** [ftp://ftp.sra.co.jp/pub/misc/eb/WinEBZip-0.0.zip WinEBZip.zip on SRA's FTP server - no longer maintained]
** [https://ebstudio.info/manual/EBWin4/EBWin4.html EBWin4]
* Linux
** [ftp://ftp.sra.co.jp/pub/misc/eb/ EB Library directory on SRA's FTP server - no longer maintained]
** [http://green.ribbon.to/~ikazuhiro/dic/ebu.html EBU - EB Library with JIS X 4081 UTF-8 extension]
** [https://github.com/mvf/qolibri qolibri EPWING dictionary viewer]

== Links (mostly Japanese only) ==
* [[Wikipedia:ja:電子ブック (規格)]]
* [https://web.archive.org/web/20010606001016/http://www.sra.co.jp/people/m-kasahr/eb/index-ja.html Internet Archive's mirror of EB Library that was hosted on SRA's website]
* [http://green.ribbon.to/~ikazuhiro/dic/ebu.html EBU]

== References (mostly Japanese only) ==
<references/>

EBZip

2025-09-05T15:59:08Z

Anonymoususer852: /* EBUZip */ Added note as a possibility to just use EBUZip regardless, instead of needing to keep two compiled binaries or packages that uses slightly different compression libraries.

{{FormatInfo
|formattype=electronic
|subcat=Compression
|extensions={{ext|ebz}}
|compression=lossless
|magic=45 42 5a 69 70 15 00 00 00 00 00 00 10 00 8b e8
|released=1999<ref>[https://web.archive.org/web/20010619200710/http://www.sra.co.jp/people/m-kasahr/eb/ChangeLog-ja.html ChangeLog - SRA website mirrored on Internet Archive]</ref>
}}
{{notice|1=Majority of the links/citations are in Japanese.}}

'''Electronic Book Zip or EBZip''' is a compression format used in electronic dictionaries that were distributed primarily on compact discs (CD). It is also one of the many frontend utilities that uses the EB library for manipulating [[EPWING]] format.

The electronic dictionaries stored on computers eliminates the need to have physical media, and allows remote referencing if it was connected to a network, using the likes of <code>dserver</code> or <code>ebnetd</code> for instance. It is likely that compression was needed due to the fact that hard disks were expensive compared to compact discs, when having a number of electronic dictionaries stored on a computer, it made little sense to retain physical media just to reference a number of words every so often, and to juggle around ejecting/inserting discs into the computer. For the expense of sacrificing few CPU cycles and potentially insignificant amount of memory, these electronic dictionaries could be made to take less storage space<ref>[https://web.archive.org/web/20170705202139/http://sci.lang.japan.narkive.com/X84h0clT/kenkyusha-dictionary-cd-and-epwing#post2 Kenkyusha dictionary CD and EPWING - sci.lang.japan NARKIVE - Internet Archive copy]</ref>, and allows faster dictionary lookups due to hard disks allowing random access.

The compression is lossless, initially using a modified [https://www.zlib.net/ zlib]<ref>[https://github.com/mistydemeo/eb/blob/742248c23cf7769d0fed412cbce458222a437795/zlib/README.altered Last version released with internal zlib - README.altered - version 4.1.3 - Misty De Méo]</ref>, then likely unmodified zlib by the original <code>ebzip</code> author<ref>[https://github.com/mistydemeo/eb/commit/742ddab4f255b7c34d7d1f1b4e3fd245bfc3ea18 Internal zlib removed - Commit 742ddab (4.2) - Misty De Méo]</ref>. Do keep in mind that not everything is compressed, EBZ format is not a container like [[ZIP]] for instance, selected files (typically <code>CATALOGS</code> and <code>HONMON</code>) are compressed and have their filenames bearing these extensions, e.g. <code>HONMON.ebz</code>.

== History ==
EB-like software for Unix-like began circulating on Japanese Usenet as early as September 1991 in the form of [[shar]] by Shingo Nishioka of Osaka University.<ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=NISHIOKA.91Sep3191513@sp4.sanken.osaka-u.ac.jp dserver/Electronic Book Dictionary server & clients (1/2) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=NISHIOKA.91Sep3191550@sp4.sanken.osaka-u.ac.jp dserver/Electronic Book Dictionary server & clients (2/2) - AncientFJ - katsu.watanabe.name]</ref> Versions before 2.1beta of <code>dserver</code> lacked compression, which was introduced as early as 1996-03-25 by Yura Shunsuke.<ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=4j5h0e$2am@isnews.is.s.u-tokyo.ac.jp dserver 2.1-beta patch for compressed dictionary - AncientFJ - katsu.watanabe.name]</ref> Gradually over time, Motoyuki Kasahara, a member of Software Research Associates (SRA) and the author of EB Library, began developing <code>ndtpd</code> which handles electronic dictionary or dictionaries over the network, replacing <code>dserver</code>.<ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5rp4qp$3sk$1@sranhh.sra.co.jp BookView 1.0 - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sctc3$irv$2@sranhh.sra.co.jp BookView 1.0.2 (Re: BookView 1.0) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=64emm8$mcf$1@sranhh.sra.co.jp BookView 1.1 (1/3) - AncientFJ - katsu.watanabe.name]</ref> The first release of such software was around 1997-06-04 as <code>ndtpd-1.0beta0</code> (via FTP) with which it included <code>eb</code> which was a library. Earliest release on fj.sources was 1997-08-07 as <code>ndtpd-1.0</code>, which also describes the relationship between <code>dserver</code> and <code>ndtpd</code>, as [[shar]].<ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc4ns$31a$2@sranhh.sra.co.jp ndtpd 1.0 (0/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc4pr$3o1$1@sranhh.sra.co.jp ndtpd 1.0 (1/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc4rn$3o1$2@sranhh.sra.co.jp ndtpd 1.0 (2/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc4te$3o1$3@sranhh.sra.co.jp ndtpd 1.0 (3/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc4vf$3o1$4@sranhh.sra.co.jp ndtpd 1.0 (4/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc514$3o1$5@sranhh.sra.co.jp ndtpd 1.0 (5/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc52p$3o1$6@sranhh.sra.co.jp ndtpd 1.0 (6/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc54s$3o1$7@sranhh.sra.co.jp ndtpd 1.0 (7/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc55j$3o1$8@sranhh.sra.co.jp ndtpd 1.0 (8/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc56d$3o1$9@sranhh.sra.co.jp ndtpd 1.0 (9/9) - AncientFJ - katsu.watanabe.name]</ref> Until the release of <code>ndtpd-3.0alpha0</code> (2000-06-04), <code>eb</code> has always been included. EB Library began as a separate project with the first release in 1999-06-16 as <code>eb-2.2</code>. Both of these projects continued until the discontinuation of <code>ndtpd</code> (as a release) on 2003-05-24 with the version 3.1.5. <code>eb</code> remained to be maintained until 2010-03-08 with the version 4.4.3.

== EBUZip ==
EBUZip is a fork of EBZip that is maintained by Ito Kazuhiro. The 'U' in EBUZip denotes the ability to support JIS X 4081 UTF-8 extensions provided that it is compiled and configured correctly. It also brings in new features that differs from EBZip such as,
* The ability to use libdeflate algorithm at compile time, which, "is a library for fast, whole-buffer DEFLATE-based compression and decompression."<ref>[https://github.com/ebiggers/libdeflate?tab=readme-ov-file#overview README.md Overview section - libdeflate - GitHub]</ref>, which was added on 2019-06-30.<ref>[http://green.ribbon.to/~ikazuhiro/dic/ebu.html EB Library with UTF-8 extension]</ref>. Additionally, the new maintainer provided benchmark results against uncompressed, zlib and libdeflate.<ref>[http://ikazuhiro.s206.xrea.com/article.php/20190701203038572 Compressing EPWING (using libdeflate)]</ref>
** libdeflate can also decompress EPWING e-books compressed with zlib, as well as recompressing them with potentially higher compression ratio. This could subsequently make it a viable candidate instead of needing to alternate between EBZip and EBUZip, see the [[Talk:EBZip#The_viability_ebuzip_as_a_possible_drop-in_replacement_candidate|Talk:EBZip section on this]] which is covered in detail.
* Fixes to work on a newer codebase, as opposed to maintaining a series of patches on top of libeb.<ref>[https://sources.debian.org/data/main/e/eb/4.4.3-14/debian/patches/010_debian.patch 010_debian.patch - eb-4.4.3-14 - sources.debian.org]</ref>

Mr. Kazuhiro have also previously contributed to Mr. Kasahara's EBZip project, on 2000-06-24.<ref>[https://github.com/mistydemeo/eb/blob/master/ChangeLog.1#L1371 ChangeLog.1 (line 1371) - EB Library - Misty De Méo]</ref>.

== Identification ==
A <code>EBZ</code> file begins with bytes {{magic|45 42 5a 69 70 15 00 00 00 00 00 00 10 00 8b e8}}.

== Examples ==
'''These must be executed in the top level of (book) directory that contains <code>CATALOGS</code> file, otherwise it will default to your current directory <code>$(pwd)</code>.''' See [[EPWING#Content structure]] for the directory structure of book layout.
If you are using EBUZip, the name of the compiled binary is <code>ebuzip</code>, as opposed to <code>ebzip</code>.

Show information about the file:
ebzip --information
Decompress files:
ebzip --uncompress wikipedia-fpw
Compress files:
ebzip wikipedia-fpw
Test files:
ebzip --test wikipedia-fpw
Show help:
ebzip --help

== Sample file ==
{| border="1" align="center" style="text-align:center;" cellpadding="6"
| Filename
| Main server download link
| Mirror server download link
| Notes
|-
| kanjidic_en.fpwebz.tar
| [http://ftp.edrdg.org/pub/Nihongo/edict_en.fpwebz.tar EDRDG.org]
| [http://ftp.usf.edu/pub/ftp.monash.edu.au/pub/nihongo/kanjidic_en.fpwebz.tar USF.edu]
| Use [[7-Zip]] to open on Windows machines for instance, otherwise [[Tar]] should suffice.
|}

== Software (mostly Japanese only) ==
See also: [[Wikipedia:ja:EPWING#検索ソフトウェア]]
* Windows
** [ftp://ftp.sra.co.jp/pub/misc/eb/WinEBZip-0.0.exe WinEBZip.exe on SRA's FTP server - no longer maintained]
** [ftp://ftp.sra.co.jp/pub/misc/eb/WinEBZip-0.0.zip WinEBZip.zip on SRA's FTP server - no longer maintained]
** [https://ebstudio.info/manual/EBWin4/EBWin4.html EBWin4]
* Linux
** [ftp://ftp.sra.co.jp/pub/misc/eb/ EB Library directory on SRA's FTP server - no longer maintained]
** [http://green.ribbon.to/~ikazuhiro/dic/ebu.html EBU - EB Library with JIS X 4081 UTF-8 extension]
** [https://github.com/mvf/qolibri qolibri EPWING dictionary viewer]

== Links (mostly Japanese only) ==
* [[Wikipedia:ja:電子ブック (規格)]]
* [https://web.archive.org/web/20010606001016/http://www.sra.co.jp/people/m-kasahr/eb/index-ja.html Internet Archive's mirror of EB Library that was hosted on SRA's website]
* [http://green.ribbon.to/~ikazuhiro/dic/ebu.html EBU]

== References (mostly Japanese only) ==
<references/>

Talk:EBZip

2025-09-05T15:55:58Z

Anonymoususer852: /* The viability ebuzip as a possible drop-in replacement candidate */ new section

== <code>ebzip</code>/<code>ebuzip</code> caveats ==
Intended to be a place to note down issues, as well as tips and tricks with either <code>eb-4.4.3</code> and/or <code>ebu-4.5-20220808</code>.

=== Hardcoded to look specifically for a file that begins with <code>"CATALOGS"</code> (case-insensitive), or <code>"CATALOGS.ebz"</code> ===
Custom dictionaries doesn't always follow standard convention to store their content(s), like having "<code>START.ebz</code>" in the top level. Even if they do, they may also compress "<code>CATALOGS</code>" as "<code>CATALOGS.ebz</code>." <code>eb-4.4.3</code> specifically looks for either "<code>CATALOGS</code>" or "<code>CATALOGS.ebz</code>", and either decompress or compress the files specified within, per user's request, e.g. <code>ebzip -u</code> will tell it to decompress and rename any of the listed files that are noted in either "<code>CATALOGS</code>" or "<code>CATALOGS.ebz</code>", and has "<code>.ebz</code>" extension, ignoring any other files in the same path, and doesn't decompress "<code>CATALOGS.ebz</code>" even if instructed. There are ways to workaround that:

==== Decompress "<code>CATALOGS.ebz</code>" ====
In my case,
* Simply rename the decompressed "<code>HONMON</code>" to something like "<code>HONMON.old</code>",
* Copy the "<code>CATALOGS.ebz</code>" there, but have it named as "<code>HONMON.ebz</code>",
* Run decompression via the usual <code>ebzip -u</code> from the top level where it doesn't contain "<code>HONMON.ebz</code>" but only "<code>CATALOGS.ebz</code>",
* Once decompressed, move decompressed "<code>HONMON</code>" (not "<code>HONMON.old</code>") back to where "<code>CATALOGS.ebz</code>" was, have it renamed as "<code>CATALOGS</code>",
* Rename "<code>HONMON.old</code>" back to "<code>HONMON</code>".
Optionally, delete "<code>CATALOGS.ebz</code>" considering that is now redundant.

==== Decompress other <code>.ebz</code> that were not listed ====
Same as above, but instead of copying, you can just move it, as,
* <code>ebzip -u</code> will only look for "<code>CATALOGS</code>" or "<code>CATALOGS.ebz</code>", and
* It saves having to manually delete the redundant file(s) afterwards.

==== Maliciously/Erroneously renamed files ====
<code>ebuzip</code> specifically looks for the extension <code>.ebz</code> to determine the level of compression. If,
* <code>CATALOGS.ebz</code> which was compressed, but renamed as <code>CATALOGS</code>, running <code>ebuzip -i</code> would show the following error message, "ebuzip: failed to read a catalog file".
* <code>HONMON.ebz</code> which was compressed, but again renamed as <code>HONMON</code>, running <code>ebuzip -i</code> would indicate that file as not compressed.
This bug is likely to also affect original <code>ebzip</code>.

=== Zero-length (un)compressed files not overwritten upon manual intervention ===
When a path that has zero-length files, but are listed in either "<code>CATALOGS</code>" or "<code>CATALOGS.ebz</code>", <code>ebzip -u</code> for instance may not overwrite it. This may also apply if <code>ebzip -uf</code> is used. To fix this, just delete the zero-length files, ensuring that they have a duplicate file that does not have <code>.ebz</code> extension, then re-run the command. When compressing, the opposite is true, to look for duplicate zero-length files with <code>.ebz</code> extension, delete them and run the utility to compress instead.

On modern [[Linux]] environments, these could be achieved via the following:

==== Enumerating ====
find . -size 0 -type f

==== Deleting ====
find . -size 0 -type f -exec sudo rm -iv "{}" \;
Care must be taken to avoid any potential losses when running this command.

[[User:Anonymoususer852|Anonymoususer852]] ([[User talk:Anonymoususer852|talk]]) 15:02, 5 September 2025 (UTC)

== LZMA benchmarks against other compression/utilities ==
In response to maintainer of <code>ebu</code>'s blog entry,<ref>[http://ikazuhiro.s206.xrea.com/article.php/20190701203038572 EPWINGを圧縮する (libdeflateで) / Compressing EPWING via libdeflate - Kazuhiro's blog]</ref> I decided to run a couple of benchmarks with different dictionaries, books or other referencing materials on my end. The following packages and their versions are as follows; kernel-6.15.8, zlib-1.3.1, libdeflate-1.24, xz-5.4.2, dwarfs-0.12.4, squashfs-tools-4.7.1.

=== zlib versus libdeflate versus LZMA benchmark (single electronic book) ===
For the following, file sizes are reported from <code>ls -l</code>, therefore size would be reported in blocks<ref>[https://unix.stackexchange.com/questions/28780/file-block-size-difference-between-stat-and-ls file block size - difference between stat and ls - Unix & Linux - Stack Exchange]</ref>. The tested dictionaries using no compression, zlib, libdeflate, or [[XZ]] are [[tar|tarballed]] for consistency. For compressed file systems, XZ's LZMA2 is used. Both zlib and libdeflate has a maximum compression level of 5, anything beyond 5 would result in error, however XZ goes up to level 9, and has further options like <code>--extreme</code> as well as various other tweaks which I will keep it simple and use <code>xz -9evv</code> instead.

[[DWARFS]] example parameters:
mkdwarfs --input . --output=../JMdict_eng_2021-02-26_UTC.dwarfs --block-size-bits=26 --compression=lzma:level=9:extreme --schema-compression=lzma:level=9:extreme --metadata-compression=lzma:level=9:extreme --no-history \
--pack-metadata=all,force --file-hash=sha3-512 --no-history-timestamp --no-create-timestamp --no-history-command-line

[[SquashFS]] example parameters:
mksquashfs . ../JMdict_eng_2021-02-26_UTC.squashfs -b 1048576 -comp xz -Xdict-size 100% -progress -info -no-xattrs

Percentages are rounded to two decimal points, otherwise rounded up. Compression ratio against uncompressed in percentage is offered instead.

{| border="1" align="center" style="text-align:center;" cellpadding="6"
|
| [[EBZip#Sample_file|kanjidic]]
| NHKAccent
| Eijiro
| Kenkyusha eiwa daijiten v6
| Kenkyusha waei daijiten v5
|-
| ''No compression''
| 130,385,920
| 1,578,055,680
| 1,374,771,200
| 484,945,920
| 328,192,000
|-
| <code>ebuzip -z -l 5</code> (zlib)
| 42,035,200
| 1,393,920,000
| 379,975,680
| 117,596,160
| 109,721,600
|-
| ''Compression ratio against uncompressed in percentage''
| 32.24%
| 88.33%
| 27.63%
| 24.25%
| 33.43%
|-
| <code>ebuzip -z -l 5</code> (libdeflate)
| 40,171,520
| 1,383,505,920
| 353,269,760
| 107,765,760
| 102,113,280
|-
| ''Compression ratio against uncompressed in percentage''
| 30.81%
| 87.67%
| 25.70%
| 22.22%
| 31.11%
|-
| <code>xz -9evv -T1</code> (liblzma)
| 21,607,608
| 1,350,900,536
| 197,134,768
| 55,220,788
| 50,109,852
|-
| ''Compression ratio against uncompressed in percentage''
| 16.57%
| 85.61%
| 14.34%
| 11.39%
| 15.27%
|-
| <code>xz -9evv -T0</code> (liblzma)
| 21,607,616
| 1,351,560,168
| 202,934,760
| 56,670,664
| 50,747,640
|-
| ''Compression ratio against uncompressed in percentage''
| 16.57%
| 85.65%
| 14.76%
| 11.69%
| 15.46%
|-
| <code>mk[[DWARFS|dwarfs]]</code> (liblzma)
| 22,145,746
| 1,353,699,364
| 212,307,176
| 60,043,728
| 54,541,980
|-
| ''Compression ratio against uncompressed in percentage''
| 16.99%
| 85.78%
| 15.44%
| 12.38%
| 16.62%
|-
| <code>mk[[SquashFS|squashfs]]</code> (liblzma)
| 31,793,152
| 1,367,302,144
| 272,068,608
| 81,805,312
| 160,956,416
|-
| ''Compression ratio against uncompressed in percentage''
| 24.38%
| 86.64%
| 19.79%
| 16.87%
| 49.04%
|}

XZ compression results are provided for the sake of discussion only, more on this in conclusion. While it does show extremely good results against all other utilities, the seek times are abysmally poor, worse than DWARFS with LZMA2 on machines with weaker hardware. Tests confirms liblzma having better compression regardless, against both zlib and libdeflate.

NHKAccent is the exception here, with poor compression results. This is due to electronic book containing multimedia files, which won't compress well if it was already compressed.

It is worthy to note here that compression/archiving utilities embracing liblzma, could by default take advantage of multi-threading, at the expense of producing potentially larger file. This however, is a small trade off, for effectively making full use of available CPU in the given environment.

=== Modern compressed file system with multiple electronic books shootout ===
Author of <code>ebu</code> mentioned the case with smartphones, which I do personally find that these are also limited in performance, not just with storage capacities. [[SquashFS]] was or is used in Android devices,<ref>[https://android.stackexchange.com/questions/258236/squashfs-explanation-for-android-system SquashFS explanation for Android system - Android Enthusiasts - Stack Exchange]</ref> so it makes sense to try using that on non system files in addition.

For these tests, the sample data consists of ~189 dictionaries in a single directory. Both [[DWARFS]] and [[SquashFS]] are configured to use XZ's LZMA2 where possible, as it is known to be one of the best, freely available (especially under Unix-like platforms) compression. Both DWARFS and SquashFS parameters are the same as in prior section. Due to the inherent way of how <code>ebuzip</code> works, requiring "<code>CATALOGS</code>" to be in the top directory, both variations of <code>ebuzip</code> using either zlib or libdeflate are omitted from this test. Besides, single compression job per dictionary, not including other files in the that path, e.g. "<code>START</code>" would make the overall process time consuming, if the process isn't already tedious that it requires extensive scripting.

{| border="1" align="center" style="text-align:center;" cellpadding="6"
| Method
| Size (blocks)
|-
| Uncompressed [[tar]]
| 29,014,026,240
|-
| [[DWARFS]]
| 10,613,886,184
|-
| [[SquashFS]]
| 12,269,518,848
|}

Where DWARFS shines, is the ability to also save on segmenting data, in addition to file de-duplication. For a small penalty I've given to DWARFS, in that it is to use sha3-512 as hash, it still compresses the collection better and faster, than SquashFS. Then again, the focus here is on overall file size, not performance in file creation.

=== Conclusion ===
Now of course, there are plenty of other compression utilities available, such as, but are not limited to [[Zstandard]], [[BZip3]], [[LZO]], etc. From the various results I have read, indicates that these other compression benchmarks, while trading better performance, these fall behind in final size constraints. The benchmark showcase here outlines the need to have as small of a compressed file as possible, with modern compression utilities and modern PC hardware, therefore making (speed-wise) performance testing out of the scope, as is the idea of creating the likes of [[ZIP]] bombs, or nested compressed files out of the scope.

I have also tried to run <code>mkfs.[[EROFS|erofs]]</code> in order to include it against the modern compressed file system with multiple dictionaries shootout. With version 1.8.10, no matter what parameters I set with LZMA <code>level=9</code> or <code>level=109</code> (109 is the extreme variant), even with lowering <code>dictsize=</code>, the results would cause system to run out of memory (16GB of RAM) before even completing the file system creation.

These benchmarks are in no way discrediting or disavowing the use of libdeflate over zlib. If file size is a concern, more modern compression algorithms and/or compression techniques could be considered. Modern EPWING viewer programs generally don't care if the electronic book is compressed using zlib or not compressed at all, and there are many ways of handling decompression transparently in modern Unix-like environments. Even more so, are the ever growing hardware performance as well as affordability with storage capacities in modern computing, questions the use of the what is widely considered as legacy compression algorithms for data archival. Last but not least, while single-threaded compression (as witnessed in the likes of <code>xz -9evv -T1</code> above) or tasks might be extremely beneficial in reducing overheads, especially with LZMA2, it is also a slow process which may subsequently not translate well in modern, demanding contexts, where time honored traditions are casted away for practical, convenient, modernized, and easy to use solutions.

<references />
[[User:Anonymoususer852|Anonymoususer852]] ([[User talk:Anonymoususer852|talk]]) 18:46, 2 September 2025 (UTC)

== The viability <code>ebuzip</code> as a possible drop-in replacement candidate ==

As covered in the main page, that [[EBZip#EBUZip|EBUZip]] uses the more modern libdeflate, I decided to also test this theory with once again using the same [[EBZip#Sample_file|JMDict]] as the subject for these experimentation. On my modern Linux machine, running <code>ebuzip -i</code> on the path that contains JMdict as the name of subdirectory,
=== Using the sample JMDict tarball as is ===
==> /home/nobody/EPWING-research/JMDict-temp/a/JMdict_eng_2021-02-26_UTC/JMdict/DATA/HONMON.ebz <==
130332672 -> 42407287 bytes (32.5%, ebzip level 5 compression)

==> /home/nobody/EPWING-research/JMDict-temp/a/JMdict_eng_2021-02-26_UTC/JMdict/GAIJI/GAI16H.ebz <==
4096 -> 529 bytes (12.9%, ebzip level 5 compression)

==> /home/nobody/EPWING-research/JMDict-temp/a/JMdict_eng_2021-02-26_UTC/CATALOGS <==
2048 bytes (not compressed)

This is likely compressed using <code>ebzip</code> with zlib. Running <code>ebuzip -u</code> indicates the following,
==> uncompress /home/nobody/EPWING-research/JMDict-temp/a/JMdict_eng_2021-02-26_UTC/JMdict/DATA/HONMON.ebz <==
output to ./JMdict/DATA/HONMON
51.5% done (67108864 / 130332672 bytes)
completed (130332672 / 130332672 bytes)
==> uncompress /home/nobody/EPWING-research/JMDict-temp/a/JMdict_eng_2021-02-26_UTC/JMdict/GAIJI/GAI16H.ebz <==
output to ./JMdict/GAIJI/GAI16H
completed (4096 / 4096 bytes)
==> copy /home/nobody/EPWING-research/JMDict-temp/a/JMdict_eng_2021-02-26_UTC/CATALOGS <==
output to ./CATALOGS
the input and output files are the same, skipped.

Then I ran <code>ebuzip -q -z -l 5</code> which compresses it using the same level (5) compression, via libdeflate and silenced output. After a few minutes, I was greeted with my prompt, so I ran <code>ebuzip -i</code>,
==> /home/nobody/EPWING-research/JMDict-temp/a/JMdict_eng_2021-02-26_UTC/JMdict/DATA/HONMON.ebz <==
130332672 -> 40120400 bytes (30.8%, ebzip level 5 compression)

==> /home/nobody/EPWING-research/JMDict-temp/a/JMdict_eng_2021-02-26_UTC/JMdict/GAIJI/GAI16H.ebz <==
4096 -> 513 bytes (12.5%, ebzip level 5 compression)

==> /home/nobody/EPWING-research/JMDict-temp/a/JMdict_eng_2021-02-26_UTC/CATALOGS <==
2048 bytes (not compressed)

=== EBUZip with zlib decompressing the same sample compressed JMDict, then recompressing it using zlib at first, then recompressing it finally using libdeflate ===
I then decided to experiment JMDict but with <code>ebuzip</code> compiled to use zlib, in a separate path, which I used that to first decompress the original zlib level 5 JMDict, then recompress it again using zlib level 5, before it is finally recompressed (after decompressing) using libdeflate. Running <code>ebuzip -i</code> shows,
==> /home/nobody/EPWING-research/JMDict-temp/b/JMdict_eng_2021-02-26_UTC/JMdict/DATA/HONMON.ebz <==
130332672 -> 41985166 bytes (32.2%, ebzip level 5 compression)

==> /home/nobody/EPWING-research/JMDict-temp/b/JMdict_eng_2021-02-26_UTC/JMdict/GAIJI/GAI16H.ebz <==
4096 -> 529 bytes (12.9%, ebzip level 5 compression)

==> /home/nobody/EPWING-research/JMDict-temp/b/JMdict_eng_2021-02-26_UTC/CATALOGS <==
2048 bytes (not compressed)

As you can see, <code>HONMON.ebz</code> has different overall size/compression ratio compared to the sample compressed JMDict. zlib project is still being maintained, so it's not terribly surprising to see ever so slight increase in compression ratio of 0.3%.

Again, decompressing it using the same <code>ebuzip -u</code>,
==> uncompress /home/nobody/EPWING-research/JMDict-temp/b/JMdict_eng_2021-02-26_UTC/JMdict/DATA/HONMON.ebz <==
output to ./JMdict/DATA/HONMON
51.5% done (67108864 / 130332672 bytes)
completed (130332672 / 130332672 bytes)
==> uncompress /home/nobody/EPWING-research/JMDict-temp/b/JMdict_eng_2021-02-26_UTC/JMdict/GAIJI/GAI16H.ebz <==
output to ./JMdict/GAIJI/GAI16H
completed (4096 / 4096 bytes)
==> copy /home/nobody/EPWING-research/JMDict-temp/b/JMdict_eng_2021-02-26_UTC/CATALOGS <==
output to ./CATALOGS
the input and output files are the same, skipped.

Finally, with both <code>ebuzip -q -z -l 5</code> and <code>ebuzip -i</code>,
==> /home/nobody/EPWING-research/JMDict-temp/b/JMdict_eng_2021-02-26_UTC/JMdict/DATA/HONMON.ebz <==
130332672 -> 40120400 bytes (30.8%, ebzip level 5 compression)

==> /home/nobody/EPWING-research/JMDict-temp/b/JMdict_eng_2021-02-26_UTC/JMdict/GAIJI/GAI16H.ebz <==
4096 -> 513 bytes (12.5%, ebzip level 5 compression)

==> /home/nobody/EPWING-research/JMDict-temp/b/JMdict_eng_2021-02-26_UTC/CATALOGS <==
2048 bytes (not compressed)

=== Observation notes ===
<code>ebuzip</code> does not indicate whether or not,
* If compression was done by <code>ebzip</code>, and,
* The library used, if it was zlib compressed or that it was subsequently using libdeflate.
Therefore, no matter which one of the two different <code>ebuzip</code> was used overall, the results are the same, with compressed sizes being different.

As my aim with either <code>ebzip</code> or <code>ebuzip</code> is to simply decompress the files, before recompressing them all using LZMA via the likes of [[DWARFS]] or [[SquashFS]], I have not tested whether or not if JMDict with EBUZip via libdeflate compression could successfully load using a general e-book viewer. But considering that the decompressing and recompressing indicates that the file could still be read, even if I used <code>ebuzip</code> here instead of the original <code>ebzip</code>, I am going to assume it will successfully load.

[[User:Anonymoususer852|Anonymoususer852]] ([[User talk:Anonymoususer852|talk]]) 15:55, 5 September 2025 (UTC)

Talk:EBZip

2025-09-05T15:02:49Z

Anonymoususer852: /* ebzip caveats */ Updated section with topics about maliciously/errneously named files which can cause issues. Changed topic to also include EBUZip. Updated signed date.

EPWING

2025-09-05T14:35:05Z

Anonymoususer852: /* Identification */ Correction on GAI* files and provided more clarification in general.

{{FormatInfo
|formattype=electronic
|subcat=Electronic Publishing formats
|extensions={{noext}}
|wikidata={{wikidata|Q11197749}}
|compression=none
|released=1987<ref>[https://web.archive.org/web/19980214200328/http://www.epwing.or.jp/epwing/admission/index.html#history EP_入会について - 略歴 - EP_About admission - brief history - Internet Archive]</ref>
}}
{{notice|1=Majority of the links/citations are in Japanese.}}

'''E'''lectronic '''P'''ublishing-WING or EPWING, is a standardized format for electronic reference materials, typically dictionaries that were distributed on CD-ROM for PCs. [[wikipedia:electronic dictionaries|Electronic dictionaries]] in general were also popular in East Asia.<ref>[https://www.igaku-shoin.co.jp/digital/detail/89680 看護医学電子辞書10 | 電子商品詳細 | 電子商品（個人向け） | 医学書院]</ref>

The format itself is largely uncompressed, taking advantage of larger storage capacities CD-ROMs offered over hard disks, and what was the largely unjustifiable cost of storing duplicate data, as well as older PC hardware that have other resource limitations such as slower CPU or less RAM/memory capacities.

== Content structure ==
EPWING files generally do not have any extensions, with the exception of when they are compressed using the likes of [[EBZip]]. Below is a rough outline describing what it would look like:
├── (top directory)
│ ├── Catalogs
│ └── (e-book name)
│ ├── DATA
│ │ └── HONMON
│ ├── GAIJI
│ │ ├── GAI16F
│ │ ├── GAI16H
│ │ ├── GAI24F
│ │ └── GAI24H
│ └── MOVIE
│ ├── 0000115
There are many exceptions to this layout, for instance, the,
* File named "Catalogs" may be in all caps or all lower-case,
* "HONMON" may include one or more of similar names, like "HONMON2" or "HONMONG", etc,
* "MOVIE" path may or may not exist, depending on the digitized referencing material, subsequently the filenames may also vary.

== Identification ==
These identification are provided as a rough indication for the uncompressed EPWING format. Compressed EPWING format using the likes of [[EBZip|EBZip or EBUZip]] may differ. By no means should you otherwise use these as reference.
* <code>CATALOGS</code> (case-insensitive) begins with {{magic|00 00}} at offset 0x60 followed by the name of e-book, which usually consists of up to 8 or 9 alphanumeric characters in hexadecimal format.
** [[EBZip#Sample_file|JMDict]] for example, contains {{magic|00 00 4a 4d 64 69 63 74}} at offset 0x60, in ASCII, it would be "<code>..JMdict</code>"
** If the name of the e-book is not up to the required length, it is usually padded with {{magic|20 20}}. In the same example with the above JMDict, at offset 0x60 it would then contain {{magic|00 00 4a 4d 64 69 63 74 20 20}}, and again in ASCII "<code>..JMdict </code>"
* <code>HONMON</code> contains a series of {{magic|00}} at offset 0x05 up to 0x0f.
* <code>GAI[1,2,3,4][0,4,6,8][F,H]*</code> has a series of {{magic|00}} at offset 0x01 till 0x07 and contains {{magic|21}} (ASCII: "<code>!</code>") at offset 0x0b.

== History ==
EPWING was a consortium of five members; Iwanami Shoten, Dai Nippon Printing, Toppan Printing, Fujitsu and Sony, produced the Electronic Book (EB) extension based on WING standard in circa 1991, with the aim to promote electronic publications in a common format, and to standardize WING onto ISO 9660 (the logical format for CD-ROMs), establishing the first edition (V1) of the EPWING convention.<ref>[[Wikipedia:ja:EPWING#成立の経緯]]</ref>

== Modern use and alternatives ==
The format is still in use by users who have been accustomed to its software, however it is considered to be a niche crowd when compared to [[StarDict]]. It might be better known through the use of EB library's [[EBZip]] utility, which offers lossless, "transparent-like" compression that seamlessly works with modern EPWING reader software, without the need to decompress first before using the data.

As the format was intended for digitized hardcopies of published books<ref>[https://ascii.jp/elem/000/000/313/313351/ ASCII.jp：岩波書店、『広辞苑第五版 CD-ROM』を発売 - ASCII.jp: Iwanami shouten, Kojien 5th edition CD-ROM release]</ref><ref>[https://ccfrancais.net/royal-cd-app/ ロワイヤル仏和中辞典のCD-ROMを辞書アプリ化する方法 | 三十歳からのフランス語 - Using Royal French-Japanese-Chinese CD-ROM dictionary app | French (language) from 30 years old]</ref>, it is possible to create your own electronic referencing materials in EPWING format. A common example is Wikipedia where the user runs either pre-made FreePWING<ref>[http://ikazuhiro.s206.xrea.com/staticpages/index.php/wikipedia-fpw wikipedia-fpw Kazuhiro's blog]</ref><ref>[https://msakamoto-sf.github.io/oldwebarchive/glamenv-septzen/view-609.html 日記/2010/03/08/FreePWING, wikipedia-fpwでWikipedia-jaのデータを変換 - Glamenv-Septzen(ぐらめぬ・ぜぷつぇん)(archive)]</ref><ref>[https://github.com/botesan/wik-asterisk-fpw botesan/wik-asterisk-fpw: Wik* (Wiktionary etc) FreePWING converters. Base wikipedia-fpw.]</ref> or FreeUWING<ref>[http://green.ribbon.to/~ikazuhiro/dic/files/wikipedia-fuw-20091202-src.tar.gz wikipedia-fuw-20091202-src.tar.gz - Index of /~ikazuhir/dic/files - green.ribbon.to]</ref> generators, the latter FreeUWING, being a variant that uses UTF-8 extensions which the FreePWING lacked, as Unicode format wasn't standardized nor widespread.

The format has also seen its use as a conversion to [https://github.com/yomidevs/yomitan yomitan]/[https://github.com/FooSoft/yomichan yomichan] web browser extension/add-on, notably via FooSoft's [https://github.com/FooSoft/zero-epwing zero-epwing] that aims to convert specific electronic EPWING formatted dictionaries into a JSON format suitable for on-screen pop-up translation. Aedict also supports [https://mvysny.github.io/aedict-blog/EPWING-Support/ one very specific EPWING dictionary, Daijirin] with the developer citing the same niche format issues.

Japanese dictionaries in general, has also witnessed its role in assisting Input Method Editor (IME) software, whereby converting the electronic dictionary format into a format that IME tool can use, enables fast and accurate word lookups.<ref>[https://www.igaku-shoin.co.jp/em/cdrom/henkanji/IME2K.htm 医学書院／書籍・電子メディア／医学書院　医学看護変換辞書／「MS-IME2000用変換辞書」作成・登録マニュアル]</ref>

Despite its flaws, EPWING format is still regarded as a trusted source owing to its traditional use, over the more modern crowd-sourced citations. While it was succeeded by the ONESWING format, which adds compression functionality among other features that were missing in EPWING, ONESWING format is more popular for smartphones.<ref>[[wikipedia:ja:EPWING#%E5%BE%8C%E7%B6%99%E3%83%BB%E6%B4%BE%E7%94%9F%E8%A6%8F%E6%A0%BC|後継・派生規格 - Wikipedia (JP)]]</ref>

== Software ==
See also: [[wikipedia:EPWING#Software|Software section of EPWING on (English) Wikipedia]] and [[wikipedia:ja:EPWING#%E6%A4%9C%E7%B4%A2%E3%82%BD%E3%83%95%E3%83%88%E3%82%A6%E3%82%A7%E3%82%A2|Software section of EPWING on (Japanese) Wikipedia]] for a more comprehensive list of software.
* [https://ebstudio.info/manual/EBStudio2/EBStudio2.html EBStudio] - (Windows) Used to create EPWING electronic reference materials.
* [ftp://ftp.sra.co.jp/pub/misc/freepwing freepwing on SRA's FTP server] - (Unix-like) Used to create EPWING electronic reference materials, mainly through scripting.
* [http://green.ribbon.to/~ikazuhiro/dic/freeuwing.html FreeUWING - FreePWING with JIS X 4081 UTF-8 extension] - (Unix-like) same as FreePWING but with UTF-8 extensions.

== References ==
<references/>

EPWING

2025-09-05T14:13:23Z

Anonymoususer852: /* Identification */ Added section. You should not use this as a source of reference, as it could be disputed!

{{FormatInfo
|formattype=electronic
|subcat=Electronic Publishing formats
|extensions={{noext}}
|wikidata={{wikidata|Q11197749}}
|compression=none
|released=1987<ref>[https://web.archive.org/web/19980214200328/http://www.epwing.or.jp/epwing/admission/index.html#history EP_入会について - 略歴 - EP_About admission - brief history - Internet Archive]</ref>
}}
{{notice|1=Majority of the links/citations are in Japanese.}}

'''E'''lectronic '''P'''ublishing-WING or EPWING, is a standardized format for electronic reference materials, typically dictionaries that were distributed on CD-ROM for PCs. [[wikipedia:electronic dictionaries|Electronic dictionaries]] in general were also popular in East Asia.<ref>[https://www.igaku-shoin.co.jp/digital/detail/89680 看護医学電子辞書10 | 電子商品詳細 | 電子商品（個人向け） | 医学書院]</ref>

The format itself is largely uncompressed, taking advantage of larger storage capacities CD-ROMs offered over hard disks, and what was the largely unjustifiable cost of storing duplicate data, as well as older PC hardware that have other resource limitations such as slower CPU or less RAM/memory capacities.

== Content structure ==
EPWING files generally do not have any extensions, with the exception of when they are compressed using the likes of [[EBZip]]. Below is a rough outline describing what it would look like:
├── (top directory)
│ ├── Catalogs
│ └── (e-book name)
│ ├── DATA
│ │ └── HONMON
│ ├── GAIJI
│ │ ├── GAI16F
│ │ ├── GAI16H
│ │ ├── GAI24F
│ │ └── GAI24H
│ └── MOVIE
│ ├── 0000115
There are many exceptions to this layout, for instance, the,
* File named "Catalogs" may be in all caps or all lower-case,
* "HONMON" may include one or more of similar names, like "HONMON2" or "HONMONG", etc,
* "MOVIE" path may or may not exist, depending on the digitized referencing material, subsequently the filenames may also vary.

== Identification ==
These identification are provided as a rough indication. By no means should you use these as reference.
* <code>CATALOGS</code> (case-insensitive) begins with {{magic|00 00}} at offset 0x60 followed by the name of e-book, which usually consists of up to 8 alphanumeric characters in hexadecimal format.
* <code>HONMON</code> contains a series of {{magic|00}} at offset 0x05 up to 0x0f.
* <code>GAI[1,2,3,4][0,4,6,8][F,H]00</code> has a series of {{magic|00}} at offset 0x01 till 0x07 and contains {{magic|21}} at offset 0x0b.

== History ==
EPWING was a consortium of five members; Iwanami Shoten, Dai Nippon Printing, Toppan Printing, Fujitsu and Sony, produced the Electronic Book (EB) extension based on WING standard in circa 1991, with the aim to promote electronic publications in a common format, and to standardize WING onto ISO 9660 (the logical format for CD-ROMs), establishing the first edition (V1) of the EPWING convention.<ref>[[Wikipedia:ja:EPWING#成立の経緯]]</ref>

== Modern use and alternatives ==
The format is still in use by users who have been accustomed to its software, however it is considered to be a niche crowd when compared to [[StarDict]]. It might be better known through the use of EB library's [[EBZip]] utility, which offers lossless, "transparent-like" compression that seamlessly works with modern EPWING reader software, without the need to decompress first before using the data.

As the format was intended for digitized hardcopies of published books<ref>[https://ascii.jp/elem/000/000/313/313351/ ASCII.jp：岩波書店、『広辞苑第五版 CD-ROM』を発売 - ASCII.jp: Iwanami shouten, Kojien 5th edition CD-ROM release]</ref><ref>[https://ccfrancais.net/royal-cd-app/ ロワイヤル仏和中辞典のCD-ROMを辞書アプリ化する方法 | 三十歳からのフランス語 - Using Royal French-Japanese-Chinese CD-ROM dictionary app | French (language) from 30 years old]</ref>, it is possible to create your own electronic referencing materials in EPWING format. A common example is Wikipedia where the user runs either pre-made FreePWING<ref>[http://ikazuhiro.s206.xrea.com/staticpages/index.php/wikipedia-fpw wikipedia-fpw Kazuhiro's blog]</ref><ref>[https://msakamoto-sf.github.io/oldwebarchive/glamenv-septzen/view-609.html 日記/2010/03/08/FreePWING, wikipedia-fpwでWikipedia-jaのデータを変換 - Glamenv-Septzen(ぐらめぬ・ぜぷつぇん)(archive)]</ref><ref>[https://github.com/botesan/wik-asterisk-fpw botesan/wik-asterisk-fpw: Wik* (Wiktionary etc) FreePWING converters. Base wikipedia-fpw.]</ref> or FreeUWING<ref>[http://green.ribbon.to/~ikazuhiro/dic/files/wikipedia-fuw-20091202-src.tar.gz wikipedia-fuw-20091202-src.tar.gz - Index of /~ikazuhir/dic/files - green.ribbon.to]</ref> generators, the latter FreeUWING, being a variant that uses UTF-8 extensions which the FreePWING lacked, as Unicode format wasn't standardized nor widespread.

The format has also seen its use as a conversion to [https://github.com/yomidevs/yomitan yomitan]/[https://github.com/FooSoft/yomichan yomichan] web browser extension/add-on, notably via FooSoft's [https://github.com/FooSoft/zero-epwing zero-epwing] that aims to convert specific electronic EPWING formatted dictionaries into a JSON format suitable for on-screen pop-up translation. Aedict also supports [https://mvysny.github.io/aedict-blog/EPWING-Support/ one very specific EPWING dictionary, Daijirin] with the developer citing the same niche format issues.

Japanese dictionaries in general, has also witnessed its role in assisting Input Method Editor (IME) software, whereby converting the electronic dictionary format into a format that IME tool can use, enables fast and accurate word lookups.<ref>[https://www.igaku-shoin.co.jp/em/cdrom/henkanji/IME2K.htm 医学書院／書籍・電子メディア／医学書院　医学看護変換辞書／「MS-IME2000用変換辞書」作成・登録マニュアル]</ref>

Despite its flaws, EPWING format is still regarded as a trusted source owing to its traditional use, over the more modern crowd-sourced citations. While it was succeeded by the ONESWING format, which adds compression functionality among other features that were missing in EPWING, ONESWING format is more popular for smartphones.<ref>[[wikipedia:ja:EPWING#%E5%BE%8C%E7%B6%99%E3%83%BB%E6%B4%BE%E7%94%9F%E8%A6%8F%E6%A0%BC|後継・派生規格 - Wikipedia (JP)]]</ref>

== Software ==
See also: [[wikipedia:EPWING#Software|Software section of EPWING on (English) Wikipedia]] and [[wikipedia:ja:EPWING#%E6%A4%9C%E7%B4%A2%E3%82%BD%E3%83%95%E3%83%88%E3%82%A6%E3%82%A7%E3%82%A2|Software section of EPWING on (Japanese) Wikipedia]] for a more comprehensive list of software.
* [https://ebstudio.info/manual/EBStudio2/EBStudio2.html EBStudio] - (Windows) Used to create EPWING electronic reference materials.
* [ftp://ftp.sra.co.jp/pub/misc/freepwing freepwing on SRA's FTP server] - (Unix-like) Used to create EPWING electronic reference materials, mainly through scripting.
* [http://green.ribbon.to/~ikazuhiro/dic/freeuwing.html FreeUWING - FreePWING with JIS X 4081 UTF-8 extension] - (Unix-like) same as FreePWING but with UTF-8 extensions.

== References ==
<references/>

EPWING

2025-09-05T13:29:57Z

Anonymoususer852: /* Modern use and alternatives */ Typo fixes and more documentation on the successor of EPWING.

{{FormatInfo
|formattype=electronic
|subcat=Electronic Publishing formats
|extensions={{noext}}
|wikidata={{wikidata|Q11197749}}
|compression=none
|released=1987<ref>[https://web.archive.org/web/19980214200328/http://www.epwing.or.jp/epwing/admission/index.html#history EP_入会について - 略歴 - EP_About admission - brief history - Internet Archive]</ref>
}}
{{notice|1=Majority of the links/citations are in Japanese.}}

'''E'''lectronic '''P'''ublishing-WING or EPWING, is a standardized format for electronic reference materials, typically dictionaries that were distributed on CD-ROM for PCs. [[wikipedia:electronic dictionaries|Electronic dictionaries]] in general were also popular in East Asia.<ref>[https://www.igaku-shoin.co.jp/digital/detail/89680 看護医学電子辞書10 | 電子商品詳細 | 電子商品（個人向け） | 医学書院]</ref>

The format itself is largely uncompressed, taking advantage of larger storage capacities CD-ROMs offered over hard disks, and what was the largely unjustifiable cost of storing duplicate data, as well as older PC hardware that have other resource limitations such as slower CPU or less RAM/memory capacities.

== Content structure ==
EPWING files generally do not have any extensions, with the exception of when they are compressed using the likes of [[EBZip]]. Below is a rough outline describing what it would look like:
├── (top directory)
│ ├── Catalogs
│ └── (e-book name)
│ ├── DATA
│ │ └── HONMON
│ ├── GAIJI
│ │ ├── GAI16F
│ │ ├── GAI16H
│ │ ├── GAI24F
│ │ └── GAI24H
│ └── MOVIE
│ ├── 0000115
There are many exceptions to this layout, for instance, the,
* File named "Catalogs" may be in all caps or all lower-case,
* "HONMON" may include one or more of similar names, like "HONMON2" or "HONMONG", etc,
* "MOVIE" path may or may not exist, depending on the digitized referencing material, subsequently the filenames may also vary.

== History ==
EPWING was a consortium of five members; Iwanami Shoten, Dai Nippon Printing, Toppan Printing, Fujitsu and Sony, produced the Electronic Book (EB) extension based on WING standard in circa 1991, with the aim to promote electronic publications in a common format, and to standardize WING onto ISO 9660 (the logical format for CD-ROMs), establishing the first edition (V1) of the EPWING convention.<ref>[[Wikipedia:ja:EPWING#成立の経緯]]</ref>

== Modern use and alternatives ==
The format is still in use by users who have been accustomed to its software, however it is considered to be a niche crowd when compared to [[StarDict]]. It might be better known through the use of EB library's [[EBZip]] utility, which offers lossless, "transparent-like" compression that seamlessly works with modern EPWING reader software, without the need to decompress first before using the data.

As the format was intended for digitized hardcopies of published books<ref>[https://ascii.jp/elem/000/000/313/313351/ ASCII.jp：岩波書店、『広辞苑第五版 CD-ROM』を発売 - ASCII.jp: Iwanami shouten, Kojien 5th edition CD-ROM release]</ref><ref>[https://ccfrancais.net/royal-cd-app/ ロワイヤル仏和中辞典のCD-ROMを辞書アプリ化する方法 | 三十歳からのフランス語 - Using Royal French-Japanese-Chinese CD-ROM dictionary app | French (language) from 30 years old]</ref>, it is possible to create your own electronic referencing materials in EPWING format. A common example is Wikipedia where the user runs either pre-made FreePWING<ref>[http://ikazuhiro.s206.xrea.com/staticpages/index.php/wikipedia-fpw wikipedia-fpw Kazuhiro's blog]</ref><ref>[https://msakamoto-sf.github.io/oldwebarchive/glamenv-septzen/view-609.html 日記/2010/03/08/FreePWING, wikipedia-fpwでWikipedia-jaのデータを変換 - Glamenv-Septzen(ぐらめぬ・ぜぷつぇん)(archive)]</ref><ref>[https://github.com/botesan/wik-asterisk-fpw botesan/wik-asterisk-fpw: Wik* (Wiktionary etc) FreePWING converters. Base wikipedia-fpw.]</ref> or FreeUWING<ref>[http://green.ribbon.to/~ikazuhiro/dic/files/wikipedia-fuw-20091202-src.tar.gz wikipedia-fuw-20091202-src.tar.gz - Index of /~ikazuhir/dic/files - green.ribbon.to]</ref> generators, the latter FreeUWING, being a variant that uses UTF-8 extensions which the FreePWING lacked, as Unicode format wasn't standardized nor widespread.

The format has also seen its use as a conversion to [https://github.com/yomidevs/yomitan yomitan]/[https://github.com/FooSoft/yomichan yomichan] web browser extension/add-on, notably via FooSoft's [https://github.com/FooSoft/zero-epwing zero-epwing] that aims to convert specific electronic EPWING formatted dictionaries into a JSON format suitable for on-screen pop-up translation. Aedict also supports [https://mvysny.github.io/aedict-blog/EPWING-Support/ one very specific EPWING dictionary, Daijirin] with the developer citing the same niche format issues.

Japanese dictionaries in general, has also witnessed its role in assisting Input Method Editor (IME) software, whereby converting the electronic dictionary format into a format that IME tool can use, enables fast and accurate word lookups.<ref>[https://www.igaku-shoin.co.jp/em/cdrom/henkanji/IME2K.htm 医学書院／書籍・電子メディア／医学書院　医学看護変換辞書／「MS-IME2000用変換辞書」作成・登録マニュアル]</ref>

Despite its flaws, EPWING format is still regarded as a trusted source owing to its traditional use, over the more modern crowd-sourced citations. While it was succeeded by the ONESWING format, which adds compression functionality among other features that were missing in EPWING, ONESWING format is more popular for smartphones.<ref>[[wikipedia:ja:EPWING#%E5%BE%8C%E7%B6%99%E3%83%BB%E6%B4%BE%E7%94%9F%E8%A6%8F%E6%A0%BC|後継・派生規格 - Wikipedia (JP)]]</ref>

== Software ==
See also: [[wikipedia:EPWING#Software|Software section of EPWING on (English) Wikipedia]] and [[wikipedia:ja:EPWING#%E6%A4%9C%E7%B4%A2%E3%82%BD%E3%83%95%E3%83%88%E3%82%A6%E3%82%A7%E3%82%A2|Software section of EPWING on (Japanese) Wikipedia]] for a more comprehensive list of software.
* [https://ebstudio.info/manual/EBStudio2/EBStudio2.html EBStudio] - (Windows) Used to create EPWING electronic reference materials.
* [ftp://ftp.sra.co.jp/pub/misc/freepwing freepwing on SRA's FTP server] - (Unix-like) Used to create EPWING electronic reference materials, mainly through scripting.
* [http://green.ribbon.to/~ikazuhiro/dic/freeuwing.html FreeUWING - FreePWING with JIS X 4081 UTF-8 extension] - (Unix-like) same as FreePWING but with UTF-8 extensions.

== References ==
<references/>

EPWING

2025-09-05T13:06:43Z

Anonymoususer852: /* Software */ Added links to Wikipedia.

{{FormatInfo
|formattype=electronic
|subcat=Electronic Publishing formats
|extensions={{noext}}
|wikidata={{wikidata|Q11197749}}
|compression=none
|released=1987<ref>[https://web.archive.org/web/19980214200328/http://www.epwing.or.jp/epwing/admission/index.html#history EP_入会について - 略歴 - EP_About admission - brief history - Internet Archive]</ref>
}}
{{notice|1=Majority of the links/citations are in Japanese.}}

'''E'''lectronic '''P'''ublishing-WING or EPWING, is a standardized format for electronic reference materials, typically dictionaries that were distributed on CD-ROM for PCs. [[wikipedia:electronic dictionaries|Electronic dictionaries]] in general were also popular in East Asia.<ref>[https://www.igaku-shoin.co.jp/digital/detail/89680 看護医学電子辞書10 | 電子商品詳細 | 電子商品（個人向け） | 医学書院]</ref>

The format itself is largely uncompressed, taking advantage of larger storage capacities CD-ROMs offered over hard disks, and what was the largely unjustifiable cost of storing duplicate data, as well as older PC hardware that have other resource limitations such as slower CPU or less RAM/memory capacities.

== Content structure ==
EPWING files generally do not have any extensions, with the exception of when they are compressed using the likes of [[EBZip]]. Below is a rough outline describing what it would look like:
├── (top directory)
│ ├── Catalogs
│ └── (e-book name)
│ ├── DATA
│ │ └── HONMON
│ ├── GAIJI
│ │ ├── GAI16F
│ │ ├── GAI16H
│ │ ├── GAI24F
│ │ └── GAI24H
│ └── MOVIE
│ ├── 0000115
There are many exceptions to this layout, for instance, the,
* File named "Catalogs" may be in all caps or all lower-case,
* "HONMON" may include one or more of similar names, like "HONMON2" or "HONMONG", etc,
* "MOVIE" path may or may not exist, depending on the digitized referencing material, subsequently the filenames may also vary.

== History ==
EPWING was a consortium of five members; Iwanami Shoten, Dai Nippon Printing, Toppan Printing, Fujitsu and Sony, produced the Electronic Book (EB) extension based on WING standard in circa 1991, with the aim to promote electronic publications in a common format, and to standardize WING onto ISO 9660 (the logical format for CD-ROMs), establishing the first edition (V1) of the EPWING convention.<ref>[[Wikipedia:ja:EPWING#成立の経緯]]</ref>

== Modern use and alternatives ==
The format is still in use by users who have been accustomed to its software, however it is considered to be a niche crowd when compared to [[StarDict]]. It might be better known through the use of EB library's [[EBZip]] utility, which offers lossless, "transparent-like" compression that seamlessly works with modern EPWING reader software, without the need to decompress first before using the data.

As the format was intended for digitized hardcopies of published books<ref>[https://ascii.jp/elem/000/000/313/313351/ ASCII.jp：岩波書店、『広辞苑第五版 CD-ROM』を発売 - ASCII.jp: Iwanami shouten, Kojien 5th edition CD-ROM release]</ref><ref>[https://ccfrancais.net/royal-cd-app/ ロワイヤル仏和中辞典のCD-ROMを辞書アプリ化する方法 | 三十歳からのフランス語 - Using Royal French-Japanese-Chinese CD-ROM dictionary app | French (language) from 30 years old]</ref>, it is possible to create your own electronic referencing materials in EPWING format. A common example is Wikipedia where the user runs either a pre-made FreePWING<ref>[http://ikazuhiro.s206.xrea.com/staticpages/index.php/wikipedia-fpw wikipedia-fpw Kazuhiro's blog]</ref><ref>[https://msakamoto-sf.github.io/oldwebarchive/glamenv-septzen/view-609.html 日記/2010/03/08/FreePWING, wikipedia-fpwでWikipedia-jaのデータを変換 - Glamenv-Septzen(ぐらめぬ・ぜぷつぇん)(archive)]</ref><ref>[https://github.com/botesan/wik-asterisk-fpw botesan/wik-asterisk-fpw: Wik* (Wiktionary etc) FreePWING converters. Base wikipedia-fpw.]</ref> or FreeUWING<ref>[http://green.ribbon.to/~ikazuhiro/dic/files/wikipedia-fuw-20091202-src.tar.gz wikipedia-fuw-20091202-src.tar.gz - Index of /~ikazuhir/dic/files - green.ribbon.to]</ref> generators, the latter FreeUWING, being a variant that uses UTF-8 extensions which the FreePWING lacked, as Unicode format wasn't standardized nor widespread.

The format has also seen its use as a conversion to [https://github.com/yomidevs/yomitan yomitan]/[https://github.com/FooSoft/yomichan yomichan] web browser extension/add-on, notably via FooSoft's [https://github.com/FooSoft/zero-epwing zero-epwing] that aims to convert specific electronic EPWING formatted dictionaries into a JSON format suitable for on-screen pop-up translation. Aedict also supports [https://mvysny.github.io/aedict-blog/EPWING-Support/ one very specific EPWING dictionary, Daijirin] with the developer citing the same niche format issues.

Japanese dictionaries in general, has also witnessed its role in assisting Input Method Editor (IME) software, whereby converting the electronic dictionary format into a format that IME tool can use, enables fast and accurate word lookups.<ref>[https://www.igaku-shoin.co.jp/em/cdrom/henkanji/IME2K.htm 医学書院／書籍・電子メディア／医学書院　医学看護変換辞書／「MS-IME2000用変換辞書」作成・登録マニュアル]</ref>

Despite its flaws, EPWING format is still regarded as a trusted source owing to its traditional use, over the more modern crowd-sourced citations.

== Software ==
See also: [[wikipedia:EPWING#Software|Software section of EPWING on (English) Wikipedia]] and [[wikipedia:ja:EPWING#%E6%A4%9C%E7%B4%A2%E3%82%BD%E3%83%95%E3%83%88%E3%82%A6%E3%82%A7%E3%82%A2|Software section of EPWING on (Japanese) Wikipedia]] for a more comprehensive list of software.
* [https://ebstudio.info/manual/EBStudio2/EBStudio2.html EBStudio] - (Windows) Used to create EPWING electronic reference materials.
* [ftp://ftp.sra.co.jp/pub/misc/freepwing freepwing on SRA's FTP server] - (Unix-like) Used to create EPWING electronic reference materials, mainly through scripting.
* [http://green.ribbon.to/~ikazuhiro/dic/freeuwing.html FreeUWING - FreePWING with JIS X 4081 UTF-8 extension] - (Unix-like) same as FreePWING but with UTF-8 extensions.

== References ==
<references/>

EPWING

2025-09-05T12:35:36Z

Anonymoususer852: Wording corrections and topic expansion in Modern use and alternatives section. Added more references.

{{FormatInfo
|formattype=electronic
|subcat=Electronic Publishing formats
|extensions={{noext}}
|wikidata={{wikidata|Q11197749}}
|compression=none
|released=1987<ref>[https://web.archive.org/web/19980214200328/http://www.epwing.or.jp/epwing/admission/index.html#history EP_入会について - 略歴 - EP_About admission - brief history - Internet Archive]</ref>
}}
{{notice|1=Majority of the links/citations are in Japanese.}}

'''E'''lectronic '''P'''ublishing-WING or EPWING, is a standardized format for electronic reference materials, typically dictionaries that were distributed on CD-ROM for PCs. [[wikipedia:electronic dictionaries|Electronic dictionaries]] in general were also popular in East Asia.<ref>[https://www.igaku-shoin.co.jp/digital/detail/89680 看護医学電子辞書10 | 電子商品詳細 | 電子商品（個人向け） | 医学書院]</ref>

The format itself is largely uncompressed, taking advantage of larger storage capacities CD-ROMs offered over hard disks, and what was the largely unjustifiable cost of storing duplicate data, as well as older PC hardware that have other resource limitations such as slower CPU or less RAM/memory capacities.

== Content structure ==
EPWING files generally do not have any extensions, with the exception of when they are compressed using the likes of [[EBZip]]. Below is a rough outline describing what it would look like:
├── (top directory)
│ ├── Catalogs
│ └── (e-book name)
│ ├── DATA
│ │ └── HONMON
│ ├── GAIJI
│ │ ├── GAI16F
│ │ ├── GAI16H
│ │ ├── GAI24F
│ │ └── GAI24H
│ └── MOVIE
│ ├── 0000115
There are many exceptions to this layout, for instance, the,
* File named "Catalogs" may be in all caps or all lower-case,
* "HONMON" may include one or more of similar names, like "HONMON2" or "HONMONG", etc,
* "MOVIE" path may or may not exist, depending on the digitized referencing material, subsequently the filenames may also vary.

== History ==
EPWING was a consortium of five members; Iwanami Shoten, Dai Nippon Printing, Toppan Printing, Fujitsu and Sony, produced the Electronic Book (EB) extension based on WING standard in circa 1991, with the aim to promote electronic publications in a common format, and to standardize WING onto ISO 9660 (the logical format for CD-ROMs), establishing the first edition (V1) of the EPWING convention.<ref>[[Wikipedia:ja:EPWING#成立の経緯]]</ref>

== Modern use and alternatives ==
The format is still in use by users who have been accustomed to its software, however it is considered to be a niche crowd when compared to [[StarDict]]. It might be better known through the use of EB library's [[EBZip]] utility, which offers lossless, "transparent-like" compression that seamlessly works with modern EPWING reader software, without the need to decompress first before using the data.

As the format was intended for digitized hardcopies of published books<ref>[https://ascii.jp/elem/000/000/313/313351/ ASCII.jp：岩波書店、『広辞苑第五版 CD-ROM』を発売 - ASCII.jp: Iwanami shouten, Kojien 5th edition CD-ROM release]</ref><ref>[https://ccfrancais.net/royal-cd-app/ ロワイヤル仏和中辞典のCD-ROMを辞書アプリ化する方法 | 三十歳からのフランス語 - Using Royal French-Japanese-Chinese CD-ROM dictionary app | French (language) from 30 years old]</ref>, it is possible to create your own electronic referencing materials in EPWING format. A common example is Wikipedia where the user runs either a pre-made FreePWING<ref>[http://ikazuhiro.s206.xrea.com/staticpages/index.php/wikipedia-fpw wikipedia-fpw Kazuhiro's blog]</ref><ref>[https://msakamoto-sf.github.io/oldwebarchive/glamenv-septzen/view-609.html 日記/2010/03/08/FreePWING, wikipedia-fpwでWikipedia-jaのデータを変換 - Glamenv-Septzen(ぐらめぬ・ぜぷつぇん)(archive)]</ref><ref>[https://github.com/botesan/wik-asterisk-fpw botesan/wik-asterisk-fpw: Wik* (Wiktionary etc) FreePWING converters. Base wikipedia-fpw.]</ref> or FreeUWING<ref>[http://green.ribbon.to/~ikazuhiro/dic/files/wikipedia-fuw-20091202-src.tar.gz wikipedia-fuw-20091202-src.tar.gz - Index of /~ikazuhir/dic/files - green.ribbon.to]</ref> generators, the latter FreeUWING, being a variant that uses UTF-8 extensions which the FreePWING lacked, as Unicode format wasn't standardized nor widespread.

The format has also seen its use as a conversion to [https://github.com/yomidevs/yomitan yomitan]/[https://github.com/FooSoft/yomichan yomichan] web browser extension/add-on, notably via FooSoft's [https://github.com/FooSoft/zero-epwing zero-epwing] that aims to convert specific electronic EPWING formatted dictionaries into a JSON format suitable for on-screen pop-up translation. Aedict also supports [https://mvysny.github.io/aedict-blog/EPWING-Support/ one very specific EPWING dictionary, Daijirin] with the developer citing the same niche format issues.

Japanese dictionaries in general, has also witnessed its role in assisting Input Method Editor (IME) software, whereby converting the electronic dictionary format into a format that IME tool can use, enables fast and accurate word lookups.<ref>[https://www.igaku-shoin.co.jp/em/cdrom/henkanji/IME2K.htm 医学書院／書籍・電子メディア／医学書院　医学看護変換辞書／「MS-IME2000用変換辞書」作成・登録マニュアル]</ref>

Despite its flaws, EPWING format is still regarded as a trusted source owing to its traditional use, over the more modern crowd-sourced citations.

== Software ==
* [https://ebstudio.info/manual/EBStudio2/EBStudio2.html EBStudio] - (Windows) Used to create EPWING electronic reference materials.
* [ftp://ftp.sra.co.jp/pub/misc/freepwing freepwing on SRA's FTP server] - (Unix-like) Used to create EPWING electronic reference materials, mainly through scripting.
* [http://green.ribbon.to/~ikazuhiro/dic/freeuwing.html FreeUWING - FreePWING with JIS X 4081 UTF-8 extension] - (Unix-like) same as FreePWING but with UTF-8 extensions.

== References ==
<references/>

LHA

2025-09-04T14:28:37Z

Anonymoususer852: Added compression information to FormatInfo template. Added note on jca02266's GitHub repo on intentions to make the code free and easy to use. Addressed a number of old http:// only links which became redirects to https://

{{FormatInfo
|subcat=Archiving
|extensions={{ext|lzh}}, {{ext|lha}}
|mimetypes={{mimetype|application/x-lzh-compressed}}
|pronom={{PRONOM|fmt/626}}
|wikidata={{wikidata|Q368782}}
|compression=lossless, optional
|kaitai struct=lzh
|released=≤1989
}}
'''LHA''' is a family of archiving programs, and their associated file format, created by Haruyasu Yoshizaki (a.k.a. Yoshi). The software was originally called '''LHarc''', then was briefly '''LH''' (v2.02–2.04), then '''LHa''' (v2.05–2.06), before settling on '''LHA''' (v2.10+). The file format is also known as '''LZH'''. See the [[LZH|LZH disambiguation page]] for other "LZH" formats.

== Discussion ==
In the 1990s, LHA was the most popular archiving format on the Amiga platform. It also got some use on the PC platform, including in the installers for id Software games such as Doom and Quake, because [[ZIP]] compression was inferior until the release of PKZIP 2.0, which brought the formats to parity.

It was particularly popular in Japan. Most of the best information about it is in Japanese.

It supports a number of different compression schemes, most of which use [[LZ77 with Huffman coding|LZ77 combined with Huffman coding]].

This article covers the format used by LHarc/LHA, as well as "generalized" LHA format: the same file format, but with other compression schemes. The generalized format was possibly designed by Kazuhiko Miki for [[LArc]], but confirmation of this is needed. If so, it was soon borrowed by LHarc, with new compression schemes.

The format may have been released in 1988, but conclusive evidence is lacking. The earliest confirmed release date is 1989-02 for the [[LArc]] compression schemes (LArc v3.33), and 1989-03 for LHarc (v1.00).

== Format details ==
=== File structure ===
An LHA file consists mainly of a sequence of elements, each representing a member file or directory. The sequence is usually terminated by an end-of-archive marker consisting of a single 0x00 byte (but take care, as level 2 headers could start with 0x00). There is no global archive-level header.

=== Member format ===
There are at least four different formats that an element can have. (Note that this is independent of compression schemes.) In LHA jargon, the formats are known as "header levels", and are usually called "header level 0", "... 1", "... 2", and "... 3". The header level is determined by the byte at offset 20 from the beginning of that element.

The header levels are similar, but irritatingly different. They don't even follow the same principles with respect to how they must be parsed.

=== LZH compression overview ===
From a decompression perspective, the LZ77+Huffman schemes work roughly as follows. (This is oversimplified.) There is a ''codes'' Huffman tree, and a separate ''offsets'' tree. A symbol is read using the codes tree which, depending on its value, represents either a literal byte value, or a ''length''. If it is a length, then an additional symbol is read using the offsets tree. Based on the offset and length, a run of recently-decompressed bytes is repeated.

=== Compression schemes ===
Each member file has a 5-byte ''compression method'' field, composed of ASCII characters. The first and last characters are virtually always dashes ("<code>-</code>"), and might be left off when discussing LHA compression schemes. Known schemes:

{| class="wikitable"
! ID !! Category !! Description and remarks
|-
|<code>-lh0-</code> || || Uncompressed
|-
|<code>-lh1-</code> || || [[LZ77 with Huffman coding|LZ77+Huffman]], 4k window, [[Adaptive Huffman coding|adaptive Huffman]] for codes, offsets use a pre-defined Huffman tree. See also [[LZHUF]].
|-
|<code>-lh2-</code> || || LZ77+Huffman, 8k window, adaptive Huffman. Considered experimental/obsolete.
|-
|<code>-lh3-</code> || || LZ77+Huffman, 8k window, segmented, static Huffman for codes, offsets can use static Huffman or a pre-defined Huffman tree. Considered experimental/obsolete.
|-
|<code>-lh4-</code> || || Like lh5, but 4k window. Rare.
|-
|<code>-lh5-</code> || || [[LZ77 with Huffman coding|LZ77+Huffman]], 8k window, segmented, static Huffman. See also [[ar (Haruhiko Okumura)]].
|-
|<code>-lh6-</code> || || Like lh5, but 32k window
|-
|<code>-lh7-</code> || || Like lh5, but 64k window
|-
|<code>-lh7-</code> || LHARK extension || Refer to [[LHARK]].
|-
|<code>-lh8-</code> ||rowspan="5"| Joe Jared extensions || Like lh5, but 64k window. (Same as lh7.)
|-
|<code>-lh9-</code> || Like lh5, but 128k window. Probably never used.
|-
|<code>-lha-</code> || Like lh5, but 256k window. Probably never used.
|-
|<code>-lhb-</code> || Like lh5, but 512k window. Probably never used.
|-
|<code>-lhc-</code> || Like lh5, but 1M window. Probably never used.
|-
|<code>-lhd-</code> || Special || Not a compression scheme. Indicates that the element represents a subdirectory.
|-
|<code>-lhe-</code> || Joe Jared extensions || Like lh5, but 2M window. Probably never used.
|-
|<code>-lhx-</code> ||rowspan="2"| UNLHA32 extensions ||
|-
|<code>-lx1-</code> ||
|-
|<code>-lz2-</code> ||rowspan="7"| LArc methods ||rowspan="7"| Refer to [[LArc]].
|-
|<code>-lz3-</code>
|-
|<code>-lz4-</code>
|-
|<code>-lz5-</code>
|-
|<code>-lz7-</code>
|-
|<code>-lz8-</code>
|-
|<code>-lzs-</code>
|-
|<code>-pm0-</code> ||rowspan="3"| PMarc extensions ||rowspan="3"| Refer to [[PMA]].
|-
|<code>-pm1-</code>
|-
|<code>-pm2-</code>
|-
|<code>-ah0-</code> ||rowspan="3"| MAR extensions ||rowspan="3"| Refer to [[Micrognosis Compression Archiver]].
|-
|<code>-ari-</code>
|-
|<code>-hf0-</code>
|-
|<code>-lZ0-</code> ||rowspan="3"| PUT/GET variants ||rowspan="3"| Refer to [[PUT]].
|-
|<code>-lZ1-</code>
|-
|<code>-lZ5-</code>
|-
|<code>␠LH0␠</code> ||rowspan="2"| SAR variants ||rowspan="2"| Refer to [[SAR (Streamline Design)]]. The compression IDs begin and end with a space (0x20).
|-
|<code>␠LH5␠</code>
|}

The Wikipedia article has [[Wikipedia:LHA (file format)#Compression methods|more information]] about some of the schemes.

For reference, here are some other LHA-like identifiers:

{| class="wikitable"
! ID !! References and remarks
|-
|<code>-afx-</code> || Refer to [[AFX (Atari ST)]].
|-
|<code>-arn-</code> ||rowspan="2"| Possibly used by [[Micrognosis Compression Archiver]].
|-
|<code>-lzw-</code>
|-
|<code>-LD6-</code> ||rowspan="2"| Refer to [[LDArc and LDIFF]].
|-
|<code>-lz6-</code>
|-
|<code>-ll0-</code> ||rowspan="2"| Refer to [[PAKLEO]].
|-
|<code>-ll1-</code>
|-
|<code>-pc1-</code> || Used by [[PopCom!]].
|-
|<code>-pms-</code> || Used by [[PMsfx]] and [[PMexe]].
|-
|<code>-sqx-</code> || Refer to [[SQX]].
|-
|<code>-sw0-</code> ||rowspan="2"| Refer to [[SWG]].
|-
|<code>-sw1-</code>
|-
|<code>-TK0-</code> ||rowspan="2"| Refer to [[TPK (compressed archive)]].
|-
|<code>-TK1-</code>
|}

=== Extended headers ===
For header levels 1 and higher, each member file has an associated list of "extended headers", similar to [[ZIP#Extensible data fields|ZIP's extensible data fields]]. Each extended header is tagged with a single byte indicating its type. Extended headers are used to store platform-specific metadata, and to extend the format in other ways.

* [https://web.archive.org/web/20110912035449/http://homepage1.nifty.com:80/dangan/en/Content/Program/Java/jLHA/Notes/ExtHeaderList.html List of extended headers] (from archive.org)
* [https://github.com/libarchive/libarchive/blob/master/libarchive/archive_read_support_format_lha.c libarchive: archive_read_support_format_lha.c] (look for "EXT_HEADER_CRC")

Header level 0 supports extended data in a more limited way. It allows for just one set of extended header fields (called the "extended area"), the content of which is determined by the initial one-byte "OS type" field.

* [https://web.archive.org/web/20110909114523/http://homepage1.nifty.com:80/dangan/en/Content/Program/Java/jLHA/Notes/ExtendArea.html Extended area] (from archive.org)

== Identification ==
LHA can be identified with high accuracy, but doing so can be laborious, due to the lack of a signature, and other complicating factors.

Identification logic could be based on the header of the first member file. Check that the compression method (offset 2–6) and header level (offset 20) fields have valid values. When suitable and possible, validate the header checksum field -- this depends on the header level.

See also the "[[#See also]]" section, for some formats that could masquerade as LHA.

== See also ==
* [[LHA/LHarc self-extracting archive]]
* [[LArc]]
* [[PMA]]
* [[LHARK]]
* [[LHice]]
* [[PUT]]
* [[Micrognosis Compression Archiver]]
* [[SAR (Streamline Design)]]
* [[LZHUF]]
* [[ar (Haruhiko Okumura)]]

Other LHA-like formats to be aware of:
* [[AFX (Atari ST)]]
* [[ARX]]
* [[CAR (MylesHi!)]]
* [[LDArc and LDIFF]]
* [[SWG]]

== Format documentation ==
* [http://dangan.g.dgdg.jp/ jLHA software]: LHA Notes
** [http://dangan.g.dgdg.jp/Content/Program/Java/jLHA/Notes/Notes.html Japanese]
** [https://web.archive.org/web/20120211104049/http://homepage1.nifty.com/dangan/en/Content/Program/Java/jLHA/Notes/Notes.html English (translation?)] (from archive.org)
* [https://gswv.apple2.org.za/a2zine/GS.WorldView/Resources/The.MacShrinkIt.Project/ARCHIVES.TXT Archive format info]
* [http://www.textfiles.com/programming/FORMATS/arc_fmts.txt LZH file header format (among other archive types)]
* [http://www33146ue.sakura.ne.jp/staff/iz/formats/lzh.html LZH format]
* [https://hwiegman.home.xs4all.nl/fileformats/lzh/lzhformat.html LZH format] (Aeco Systems)
* [https://github.com/libarchive/libarchive/blob/master/libarchive/archive_read_support_format_lha.c libarchive: archive_read_support_format_lha.c] - Has comments with information about the header formats
* [https://web.archive.org/web/20021005080911/http://www.osirusoft.com/joejared/lzhformat.html Joe Jared's LHA specification] (from archive.org)

== Software ==
* [https://github.com/fragglet/lhasa lhasa] - Lhasa is a library for parsing LHA (.lzh) archives and a free replacement for the Unix LHA tool.
* [[7-Zip]]
* [https://www.ponsoftware.com/en/ Explzh for Windows]
* [https://web.archive.org/web/20130906133859/http://homepage1.nifty.com/dangan/en/Content/Program/Java/jLHA/jLHA.html Java library] (from archive.org)
* [https://www.libarchive.org libarchive]
* [https://web.archive.org/web/20230129210625/http://lha.osdn.jp/ LHa for Unix] (from archive.org) · [https://github.com/jca02266/lha GitHub project] - GitHub repository author states, "I am trying to make the code more free and easy to use as a component of LHa for UNIX."
** [https://web.archive.org/web/20200301124852/http://www2m.biglobe.ne.jp/~dolphin/lha/lha.htm LHa for Unix (Tsukao Okamoto)] (from archive.org)
* [https://micco.mars.jp/mysoft/unlha32.htm UNLHA32.DLL] and [https://micco.mars.jp/mysoft/lhmelt.htm LHMelt]
* LHarc/LHA
** For DOS
*** LHarc v1.00 - English (1989-03-04): [https://archive.org/details/RbbsInABoxVol1No2_640 RBBS in a Box, vol 1 no 2] → 014r/lharc10e.com
*** {{CdTextfiles|carousel344/013/LHARC12.ZIP|LHarc v1.12 Test version - English}} (1989-04-23)
*** LHarc v1.12b - English (1989-04-29): [https://archive.org/details/RbbsInABoxVol1No2_640 RBBS in a Box, vol 1 no 2] → add2/lharc12b.exe
*** {{CdTextfiles|bbox4/archiver/lharc113.exe|LHarc v1.13 Test version - English}} (1989-05-04)
*** {{CdTextfiles|hof91/ARC/LH113C.EXE|LHarc v1.13c - English}} (1989-05-31)
*** LHarc v1.13c - Japanese (1989-05-21): [https://archive.org/details/FMTownsFreeSoftwareCollection1 FM Towns Free Software Collection 1] → FREEWARE.{BIN,CUE} → ms_dos/lharc/*
*** {{CdTextfiles|garbo/PC/GOLDIES/LH113DE.COM|LHarc v1.13d - English}} (1989-12-22)
*** LHarc v1.13d - Japanese: [https://archive.org/download/FMTownsFreeSoftwareCollection3 FM Towns Free Software Collection 3] → FREEWARE.{BIN,CUE} → ms_dos/lharc/*
*** {{CdTextfiles|californiacollect/his008/lha205e.exe|LHa v2.05 test version - English}} (1991-01-27)
*** {{CdTextfiles|hof91/ARC/LH205.EXE|LHa v2.05 test version - Japanese}}
*** LHa v2.05b - Japanese (1991-02-07): [https://archive.org/download/FMTownsFreeSoftwareCollection3 FM Towns Free Software Collection 3] → FREEWARE.{BIN,CUE} → ms_dos/lha/*
*** {{CdTextfiles|hof91/ARC/LHA206E.EXE|LHa v2.06 - English}} (1991-02-14)
*** {{CdTextfiles|hof91/COMP/LHA210.EXE|LHA v2.10 - English}} (1991-02-24)
*** {{CdTextfiles|californiacollect/his008/lha211.exe|LHA v2.11 - English}} (1991-03-03)
*** {{CdTextfiles|californiacollect/his008/lha212.exe|LHA v2.12 - English}} (1991-03-21)
*** LHA v2.12 - Japanese: [https://archive.org/details/VectorSoftwarePackforWinGold20000Titles Vector Software Pack for Win Gold] → (Disc 8) → WIN95/EDU/SCIENCE/CHEMICAL/CHEMICAL.LZH → LHA.EXE
*** {{CdTextfiles|simtel/simtel20/MSDOS/ARCHIVRS/LHA213.EXE|LHA v2.13 - English}} (1991-07-20)
*** LHA v2.13 - Japanese: [https://archive.org/details/Nova_Win50GameVol7_Japan Win 50 Game+ Vol. 7 (Japan)] → Win 50 Game+ Vol. 7 (Japan).7z → Win 50 Game+ Vol. 7 (Japan).{bin,cue} → lha_file/lha/lha213.exe
*** LHA v2.52 - English (1992-09-07): [{{DiscMasterURL|view/30113/xtra0390-0399/xtra0395.tar/xtra0395/S2_3KIN1.ZIP/LHA.EXE}}]
*** {{CdTextfiles|pier02/002a/lha252.exe|LHA v2.52 - Japanese}} (1992-09-07)
*** LHA v2.54 - Japanese (1992-10-04): [https://archive.org/details/cg-network-4 CG Network 4] → pc/program/lha/lha.exe
*** LHA v2.55 - Japanese (1992-11-15): [https://archive.org/details/2014.03.ftp.eri.u-tokyo.ac.jp] → ftp.eri.u-tokyo.ac.jp/pub/DOS/tools/lha255.exe
*** {{CdTextfiles|pdos9606/ARCHIVER/TOOLS/LHA255B.EXE|LHA v2.55b}} (1992-11-24) - Japanese (LHA.EXE) and English (LHA_E.EXE)
*** [http://info.elf.stuba.sk/packages/pub/pc/pack/lha266.exe LHA v2.66 test version - Japanese] (1994-12-30)
**** [{{SACFTPURL|pack|lha266e.exe}} lha266e.exe] - Patch to convert error messages to English
*** Various versions at old-dos.ru: [http://old-dos.ru/index.php?page=files&mode=files&do=show&id=3432 LHarc], [http://old-dos.ru/index.php?page=files&mode=files&do=show&id=713 LHA]
** For Windows console
*** [http://info.elf.stuba.sk/packages/pub/pc/pack/lha267.exe LHA32 v2.67.00 test version - Japanese] (1995-10-07)
**** [{{SACFTPURL|pack|lha267e.exe}} lha267e.exe] - Patch to convert error messages to English
** Source code
*** {{CdTextfiles|simtel/simtel20/MSDOS/ARCHIVRS/LHARCSRC.ZIP|v1.13b}}
*** {{CdTextfiles|simtel/simtel20/MSDOS/ARCHIVRS/LHA211SR.ZIP|v2.11}}
* [http://old-dos.ru/index.php?page=files&mode=files&do=show&id=2836 Lha32] - by "Take"
* [[LZHUF]] - Source code related to "lh1" compression
* [[ar (Haruhiko Okumura)]] - Implementation of "lh5" compression
* [https://github.com/PascalVault/Lazarus_Unpacker Open-source library in Free Pascal]
* [https://github.com/temisu/ancient Ancient] - Has modern C++ code for decompressing most LHA schemes, but as of this writing there's no easy way to use it.
* {{Deark}} (e.g. with <code>-zip</code> option)
* {{XAD}}
* Huffman Compression Engine II, a.k.a. LH7, by Joe Jared
** v0.21q for DOS: [https://encode.su/threads/1563-DOS-Archiver-Benchmark?p=31913&viewfull=1#post31913] → [https://encode.su/attachment.php?attachmentid=2155&d=1357879541 DLH7021Q.ZIP]
** v0.21q for Windows: [https://encode.su/threads/1658-LH7-archiver?p=31932&viewfull=1#post31932] → [https://encode.su/attachment.php?attachmentid=2158&d=1358062452 WLH7021Q.7z]
** v0.21q for Linux: [https://web.archive.org/web/20010901000000*/http://www.osirusoft.com/llh7021q.zip llh7021q.zip] (from archive.org)

=== Software oddities ===
There are many customized versions of LHarc/LHA floating around. Some of them are listed here, either because they are notable, or because they are potentially misleading. (For DOS, unless otherwise indicated.)

Worth noting is that LHA 2.x has a tamper-detection feature, invoked by running "LHA t LHA.EXE" (or "LHA_E t LHA_E.EXE"). Most (but not all) modified files fail the test, and print "No file found" or "Broken archive".

* "LHarc v1.13" (1989-05-14): {{CdTextfiles|simtel0595/DISC1/CITADEL/K2NE608A.ZIP|K2NE608A.ZIP}} → LHARC.EXE - Suspect this is the v1.13 test version, edited to make it look like a full release.
* "LHarc v1.131c" by Steve Hoglund: [https://archive.org/details/bbs-1 BBS# 1] → DOCUMENT/TURBOBAS.LZH → LHARC.COM
* [[LHice]] - A hack of v1.13c.
* {{CdTextfiles|hof91/COMP/LHA114A.COM|"LHarc v1.14a"}} - A hack of v1.13c and/or LHice.
* {{CdTextfiles|hof91/ARC/LH114B.EXE|"LHARC v1.14β"}} - A hack of v1.13c and/or LHice.
* {{CdTextfiles|animfestival/SBPRO/LHARC.EXE|"LHarc v2.01a"}} - Apparently a hack of v1.13c.
* {{CdTextfiles|simtel/simtel9703/disk2/DISC2/ARCERS/LHA255E.EXE|"LHA v2.55E"}} (1992-11-15/1996-01-10) - English translation of v2.55, by Hitoshi Ozawa

== Sample files ==
* [https://github.com/fragglet/lhasa/tree/master/test/archives lhasa test files]
* [https://github.com/libarchive/libarchive/tree/master/libarchive/test libarchive test files] → test_read_format_lha_*.lzh.uu
* [https://aminet.net/ aminet]
* {{CdTextfiles|hof91/}} ...
* {{DexvertSamples|archive/lha}}

== Other links ==
* [[Wikipedia:LHA (file format)|Wikipedia article]]

Zoo

2025-09-04T14:11:57Z

Anonymoususer852: /* FormatInfo template */ Correction on mandatory compression, as it offers no packing/uncompressed.

{{FormatInfo
|subcat=Archiving
|extensions={{ext|zoo}}
|mimetypes={{mimetype|application/x-zoo}}
|pronom={{PRONOM|x-fmt/269}}
|wikidata={{wikidata|Q219983}}
|compression=lossless, optional
|magic=DC A7 C4 FD (at offset 20)
|released=1986
}}
'''Zoo''' is a compressed archive format developed by Rahul Dhesi. It was one of the many competing archive formats in the 1980s, and was based (originally) on [[LZW]] compression. It had some popularity with Open/VMS and Amiga systems, but didn't ultimately win out over [[ZIP]].

== Discussion ==
The Zoo software encompassed several different utilities, which were sometimes distributed individually, and which had their own version numbers.
* '''Zoo''' - The main program
* '''Ooz''' - Extractor
* '''Booz''' (Barebones Ooz) - Extractor; some versions can also list files
* '''Looz''' (Ooz with List) - Extract/list/execute
* '''Atoz''' - Helper utility for converting other formats ([[ARC (compression format)|ARC]], [[LBR]]) to Zoo
* '''Fiz''' - Analyze damaged Zoo files
* '''Stuff''' - File finder
* '''Sez''' - Self-Extracting Zoo utility

== Compression methods ==

{| class="wikitable"
! ID !! Name !! Description
|-
|0 || No packing || Uncompressed
|-
|1 || LZD || [[LZW]]. A fairly generic LZW implementation, with a dynamic code size from 9 to 13 bits, and two special codes: 256=Clear, 257=Stop.
|-
|2 || LZH || [[LZ77 with Huffman coding|LZ77+Huffman]]. Nearly identical to [[LHA]]'s "lh5" method. A small difference is that Zoo defines a way to mark the end of data. The format is pretty well documented in the comments in unzoo.c by Martin Schoenert (see below).
|}

== Identification ==
Files have bytes {{magic|DC A7 C4 FD}} at offset 20.

Zoo files typically start with "{{magic|ZOO ?.?? Archive.}}", followed by the byte <code>0x1a</code>, where each "<code>?</code>" is a digit. Regarding this field, the source code says "The contents of the text message are however not used by Zoo and they may be anything."

== Related formats ==
* [[Zoo filter format]]
* [[Zoo Z format‎]]
* [[Brik]] - same author

== Specifications ==
''[Ed. note: The Zoo format information in these documents is not of the highest quality. We haven't found a specification that is reasonably complete and correct.]''

* [https://www.fileformat.info/format/zoo/corion.htm The ZOO Archive File Format] from Corion.net, via FileFormat.info
* [http://apple2.org.za/gswv/a2zine/GS.WorldView/Resources/The.MacShrinkIt.Project/ARCHIVES.TXT Archive format info, including ZOO]
* [http://www.textfiles.com/programming/FORMATS/arc_fmts.txt ZOO file header format (among other archive types)]

== Software ==
* [[Konvertor]]
* Zoo - DOS binaries
** {{CdTextfiles|megarom/megarom1/ARC_LBR/ZOO120.ZIP|v1.20}} (1986-09-06)
** {{CdTextfiles|carousel344/013/ZOO121.ZIP|v1.21}} (1986-09-21)
** v1.22 (1986-10-01): [https://archive.org/details/ibm-wgam-wbiz-collection] → ibm-wgam-wbiz-collection/xtra0320-0329.zip → xtra0321.tar → FNW315-1.ZIP → ZOO122.EXE (bare executable)
** v1.40 (1987-01-24): [https://archive.org/details/shareware-grab-bag] → Shareware_Grab_Bag_1988.zip → 014/zoo140.arc
** v1.41: [https://archive.org/details/os2sharewarebbs OS/2 Shareware BBS Full Collection] → 08-Other.zip → S12215.ZIP → ZOO.EXE (bare executable)
** {{CdTextfiles|rbbsv3n1/zzap/zoo200.exe|v2.00}} (~1988-02-06)
** {{CdTextfiles|hof91/ARC/ZOO201.EXE|v2.01}} (1988-08-25) - Also includes Fiz 2.0 and Stuff 1.0
** {{CdTextfiles|simtel/simtel20/MSDOS/ARCHIVRS/ZOO210.EXE|v2.10}} (1991-07-09)
* Other official Zoo utilities:
** {{CdTextfiles|simtelctc/disk1/ZOO/BOOZ20.ZIP;1|Booz 2.0}} - Extractor; includes source code and DOS binary
** {{CdTextfiles|carousel344/013/LOOZ210.ZIP|Looz 2.1}}
** {{CdTextfiles|carousel/013B/OOZ130.ZIP|Ooz 1.30}}
** Sez {{CdTextfiles|carousel344/013/SEZ200.ZIP|2.0}}, {{CdTextfiles|microhaus/mhblackbox3/ARCHIVER/SEZ230.ZIP|2.3}} - Makes self-extracting archives
** {{CdTextfiles|simtelctc/disk1/ZOO/ATOZ112.ZIP;1|Atoz 1.12}} - Conversion helper
** {{CdTextfiles|simtel/simtel9211/msdos/zoo/stuff2.zip|Stuff 2.0}}
* [https://www.amiga-stuff.com/archivers-download.html amiga-stuff.com: Archivers] - Lists several versions of Zoo for Amiga
* [http://archives.math.utk.edu/software/multi-platform/gap/util/unzoo.c unzoo.c] by Martin Schoenert (Does not support LZD.)
* [https://github.com/PascalVault/Lazarus_Unpacker Open-source library in Free Pascal]
* {{Deark}}

Original(?) source code:
* v1.41 (1987-02-07): {{CdTextfiles|hof91/PROGLC/ZOO141_C.LZH|ZOO141_C.LZH}}
* v1.51 (1987-07-12)
** [https://archive.org/details/ASMEsMechanicalEngineeringToolkit1997December ASME's Mechanical Engineering Toolkit Dec. 1997] → C_LANG/Z150_SRC.LZH
** As posted to Usenet comp.sources.unix 1987-08: [{{CdTextfilesURL|sourcecode/usenet/compsrcs/unix/volume11/zoo/}}]
* v2.01 (1988-08-25)
** [http://www.nic.funet.fi/index/unix/tools/zoo/zoo-2.01.tar.gz zoo-2.01.tar.gz]
** [https://archive.org/details/ASMEsMechanicalEngineeringToolkit1997December ASME's Mechanical Engineering Toolkit Dec. 1997] → C_LANG/{Z201SRC1.Z00 + Z201SRC2.Z00}
** [https://archive.org/details/pegasus20 Pegasus 2.0] → pegasus/c_2/zoo_src.zip
* v2.10, a.k.a. v2.1 (1991-07-07)
** [http://www.nic.funet.fi/index/unix/tools/zoo/zoo-2.1.tar.gz zoo-2.1.tar.gz]
** {{CdTextfiles|aminet0294/AMINET/MISC/UNIX/ZOO_2_10.TAR|ZOO_2_10.TAR}}

Other source code and packages:
* [http://distcache.freebsd.org/ports-distfiles/zoo-2.10pl1.tar.gz zoo-2.10pl1.tar.gz]
* [http://www.ibiblio.org/pub/Linux/utils/compress/zoo-2.10-3.src.rpm zoo-2.10-3.src.rpm]
* [https://build.opensuse.org/package/show/openSUSE:Factory/zoo openSUSE maintained variant of zoo-2.10]
* [https://github.com/troglobit/zoo zoo - GitHub] - "A service to all computer archaeologists and acts as a focal point for patches and maintenance for the long haul." Provides updated licensing terms that it is now Unlicense/Public Domain.

== Sample files ==
* {{CdTextfilesURL|aminet0294/AMINET/UTIL/ARC/}} → *.ZOO
* {{CdTextfilesURL|garbo/PC/}} → .../*.ZOO
* {{CdTextfilesURL|geminiatari/ZIP/PROGRAM/}} → *.ZOO
* {{DexvertSamples|archive/zoo}}

== Links ==
* [[Wikipedia: Zoo (file format)]]
* {{CdTextfiles|rbbsv3n1/zzap/zooplan1.zip|zooplan1.zip}} → ZOOPLAN1.TXT - A Zoo Manifesto, by Rahul Dhesi (1986-11-01)
* {{CdTextfiles|simtelctc/disk1/ZOO/UGUIDE.ZIP;1|The zoo Archiver User's Guide}}, by Bill Davidsen (for Zoo 2.01)
* [https://manpages.ubuntu.com/manpages/bionic/man1/zoo.1.html Zoo man page] - Includes a change log going back to v1.71
* {{CdTextfiles|carousel344/013/ZOO121.ZIP|Zoo 1.21}} → ZOOHSTRY.TXT - Has some historical information about older versions

Zoo

2025-09-04T14:10:09Z

Anonymoususer852: /* FormatInfo template */ Added compression information.

{{FormatInfo
|subcat=Archiving
|extensions={{ext|zoo}}
|mimetypes={{mimetype|application/x-zoo}}
|pronom={{PRONOM|x-fmt/269}}
|wikidata={{wikidata|Q219983}}
|compression=lossless, mandatory
|magic=DC A7 C4 FD (at offset 20)
|released=1986
}}
'''Zoo''' is a compressed archive format developed by Rahul Dhesi. It was one of the many competing archive formats in the 1980s, and was based (originally) on [[LZW]] compression. It had some popularity with Open/VMS and Amiga systems, but didn't ultimately win out over [[ZIP]].

== Discussion ==
The Zoo software encompassed several different utilities, which were sometimes distributed individually, and which had their own version numbers.
* '''Zoo''' - The main program
* '''Ooz''' - Extractor
* '''Booz''' (Barebones Ooz) - Extractor; some versions can also list files
* '''Looz''' (Ooz with List) - Extract/list/execute
* '''Atoz''' - Helper utility for converting other formats ([[ARC (compression format)|ARC]], [[LBR]]) to Zoo
* '''Fiz''' - Analyze damaged Zoo files
* '''Stuff''' - File finder
* '''Sez''' - Self-Extracting Zoo utility

== Compression methods ==

{| class="wikitable"
! ID !! Name !! Description
|-
|0 || No packing || Uncompressed
|-
|1 || LZD || [[LZW]]. A fairly generic LZW implementation, with a dynamic code size from 9 to 13 bits, and two special codes: 256=Clear, 257=Stop.
|-
|2 || LZH || [[LZ77 with Huffman coding|LZ77+Huffman]]. Nearly identical to [[LHA]]'s "lh5" method. A small difference is that Zoo defines a way to mark the end of data. The format is pretty well documented in the comments in unzoo.c by Martin Schoenert (see below).
|}

== Identification ==
Files have bytes {{magic|DC A7 C4 FD}} at offset 20.

Zoo files typically start with "{{magic|ZOO ?.?? Archive.}}", followed by the byte <code>0x1a</code>, where each "<code>?</code>" is a digit. Regarding this field, the source code says "The contents of the text message are however not used by Zoo and they may be anything."

== Related formats ==
* [[Zoo filter format]]
* [[Zoo Z format‎]]
* [[Brik]] - same author

== Specifications ==
''[Ed. note: The Zoo format information in these documents is not of the highest quality. We haven't found a specification that is reasonably complete and correct.]''

* [https://www.fileformat.info/format/zoo/corion.htm The ZOO Archive File Format] from Corion.net, via FileFormat.info
* [http://apple2.org.za/gswv/a2zine/GS.WorldView/Resources/The.MacShrinkIt.Project/ARCHIVES.TXT Archive format info, including ZOO]
* [http://www.textfiles.com/programming/FORMATS/arc_fmts.txt ZOO file header format (among other archive types)]

== Software ==
* [[Konvertor]]
* Zoo - DOS binaries
** {{CdTextfiles|megarom/megarom1/ARC_LBR/ZOO120.ZIP|v1.20}} (1986-09-06)
** {{CdTextfiles|carousel344/013/ZOO121.ZIP|v1.21}} (1986-09-21)
** v1.22 (1986-10-01): [https://archive.org/details/ibm-wgam-wbiz-collection] → ibm-wgam-wbiz-collection/xtra0320-0329.zip → xtra0321.tar → FNW315-1.ZIP → ZOO122.EXE (bare executable)
** v1.40 (1987-01-24): [https://archive.org/details/shareware-grab-bag] → Shareware_Grab_Bag_1988.zip → 014/zoo140.arc
** v1.41: [https://archive.org/details/os2sharewarebbs OS/2 Shareware BBS Full Collection] → 08-Other.zip → S12215.ZIP → ZOO.EXE (bare executable)
** {{CdTextfiles|rbbsv3n1/zzap/zoo200.exe|v2.00}} (~1988-02-06)
** {{CdTextfiles|hof91/ARC/ZOO201.EXE|v2.01}} (1988-08-25) - Also includes Fiz 2.0 and Stuff 1.0
** {{CdTextfiles|simtel/simtel20/MSDOS/ARCHIVRS/ZOO210.EXE|v2.10}} (1991-07-09)
* Other official Zoo utilities:
** {{CdTextfiles|simtelctc/disk1/ZOO/BOOZ20.ZIP;1|Booz 2.0}} - Extractor; includes source code and DOS binary
** {{CdTextfiles|carousel344/013/LOOZ210.ZIP|Looz 2.1}}
** {{CdTextfiles|carousel/013B/OOZ130.ZIP|Ooz 1.30}}
** Sez {{CdTextfiles|carousel344/013/SEZ200.ZIP|2.0}}, {{CdTextfiles|microhaus/mhblackbox3/ARCHIVER/SEZ230.ZIP|2.3}} - Makes self-extracting archives
** {{CdTextfiles|simtelctc/disk1/ZOO/ATOZ112.ZIP;1|Atoz 1.12}} - Conversion helper
** {{CdTextfiles|simtel/simtel9211/msdos/zoo/stuff2.zip|Stuff 2.0}}
* [https://www.amiga-stuff.com/archivers-download.html amiga-stuff.com: Archivers] - Lists several versions of Zoo for Amiga
* [http://archives.math.utk.edu/software/multi-platform/gap/util/unzoo.c unzoo.c] by Martin Schoenert (Does not support LZD.)
* [https://github.com/PascalVault/Lazarus_Unpacker Open-source library in Free Pascal]
* {{Deark}}

Original(?) source code:
* v1.41 (1987-02-07): {{CdTextfiles|hof91/PROGLC/ZOO141_C.LZH|ZOO141_C.LZH}}
* v1.51 (1987-07-12)
** [https://archive.org/details/ASMEsMechanicalEngineeringToolkit1997December ASME's Mechanical Engineering Toolkit Dec. 1997] → C_LANG/Z150_SRC.LZH
** As posted to Usenet comp.sources.unix 1987-08: [{{CdTextfilesURL|sourcecode/usenet/compsrcs/unix/volume11/zoo/}}]
* v2.01 (1988-08-25)
** [http://www.nic.funet.fi/index/unix/tools/zoo/zoo-2.01.tar.gz zoo-2.01.tar.gz]
** [https://archive.org/details/ASMEsMechanicalEngineeringToolkit1997December ASME's Mechanical Engineering Toolkit Dec. 1997] → C_LANG/{Z201SRC1.Z00 + Z201SRC2.Z00}
** [https://archive.org/details/pegasus20 Pegasus 2.0] → pegasus/c_2/zoo_src.zip
* v2.10, a.k.a. v2.1 (1991-07-07)
** [http://www.nic.funet.fi/index/unix/tools/zoo/zoo-2.1.tar.gz zoo-2.1.tar.gz]
** {{CdTextfiles|aminet0294/AMINET/MISC/UNIX/ZOO_2_10.TAR|ZOO_2_10.TAR}}

Other source code and packages:
* [http://distcache.freebsd.org/ports-distfiles/zoo-2.10pl1.tar.gz zoo-2.10pl1.tar.gz]
* [http://www.ibiblio.org/pub/Linux/utils/compress/zoo-2.10-3.src.rpm zoo-2.10-3.src.rpm]
* [https://build.opensuse.org/package/show/openSUSE:Factory/zoo openSUSE maintained variant of zoo-2.10]
* [https://github.com/troglobit/zoo zoo - GitHub] - "A service to all computer archaeologists and acts as a focal point for patches and maintenance for the long haul." Provides updated licensing terms that it is now Unlicense/Public Domain.

== Sample files ==
* {{CdTextfilesURL|aminet0294/AMINET/UTIL/ARC/}} → *.ZOO
* {{CdTextfilesURL|garbo/PC/}} → .../*.ZOO
* {{CdTextfilesURL|geminiatari/ZIP/PROGRAM/}} → *.ZOO
* {{DexvertSamples|archive/zoo}}

== Links ==
* [[Wikipedia: Zoo (file format)]]
* {{CdTextfiles|rbbsv3n1/zzap/zooplan1.zip|zooplan1.zip}} → ZOOPLAN1.TXT - A Zoo Manifesto, by Rahul Dhesi (1986-11-01)
* {{CdTextfiles|simtelctc/disk1/ZOO/UGUIDE.ZIP;1|The zoo Archiver User's Guide}}, by Bill Davidsen (for Zoo 2.01)
* [https://manpages.ubuntu.com/manpages/bionic/man1/zoo.1.html Zoo man page] - Includes a change log going back to v1.71
* {{CdTextfiles|carousel344/013/ZOO121.ZIP|Zoo 1.21}} → ZOOHSTRY.TXT - Has some historical information about older versions

Zoo filter format

2025-09-04T14:08:43Z

Anonymoususer852: /* Format Info template */ Added compression details and magic bytes.

{{FormatInfo
|formattype=electronic
|subcat=Compression
|compression=lossless?; mandatory
|magic=32 5a
|released=1988
}}
'''Zoo filter format''' is a compressed file format related to the [[Zoo]] archive format. It is a streamable format, intended to be used e.g. in Unix-style pipes.

Support for it was introduced in Zoo 2.00, as a compile-time option. It does not appear to be enabled in the DOS releases of Zoo.

The filter format is distinct from [[Zoo Z format]].

Refer to the [[Zoo]] article for information related to the Zoo format and software.

== Discussion ==
The format is a simple container format consisting of a two-byte signature, then the compressed data, then a [[CRC]].

The compression method is either LZD or LZH. It is not recorded in the datastream. You must use the "<code>h</code>" option to compress or decompress LZH.

To compress, use "<code>zoo fc</code>" or "<code>zoo fch</code>". To decompress, use "<code>zoo fu</code>" or "<code>zoo fuh</code>".

== Identification ==
Files start with bytes {{magic|32 5a}}.

== Software ==
Refer to [[Zoo#Software]].

Zoo

2025-09-04T14:04:09Z

Anonymoususer852: Added magic to FormatInfo. Added a link to GitHub repository which, "acts as a focal point for patches and maintenance for the long haul", under Other source code and packages section.

{{FormatInfo
|subcat=Archiving
|extensions={{ext|zoo}}
|mimetypes={{mimetype|application/x-zoo}}
|pronom={{PRONOM|x-fmt/269}}
|wikidata={{wikidata|Q219983}}
|magic=DC A7 C4 FD (at offset 20)
|released=1986
}}
'''Zoo''' is a compressed archive format developed by Rahul Dhesi. It was one of the many competing archive formats in the 1980s, and was based (originally) on [[LZW]] compression. It had some popularity with Open/VMS and Amiga systems, but didn't ultimately win out over [[ZIP]].

== Discussion ==
The Zoo software encompassed several different utilities, which were sometimes distributed individually, and which had their own version numbers.
* '''Zoo''' - The main program
* '''Ooz''' - Extractor
* '''Booz''' (Barebones Ooz) - Extractor; some versions can also list files
* '''Looz''' (Ooz with List) - Extract/list/execute
* '''Atoz''' - Helper utility for converting other formats ([[ARC (compression format)|ARC]], [[LBR]]) to Zoo
* '''Fiz''' - Analyze damaged Zoo files
* '''Stuff''' - File finder
* '''Sez''' - Self-Extracting Zoo utility

== Compression methods ==

{| class="wikitable"
! ID !! Name !! Description
|-
|0 || No packing || Uncompressed
|-
|1 || LZD || [[LZW]]. A fairly generic LZW implementation, with a dynamic code size from 9 to 13 bits, and two special codes: 256=Clear, 257=Stop.
|-
|2 || LZH || [[LZ77 with Huffman coding|LZ77+Huffman]]. Nearly identical to [[LHA]]'s "lh5" method. A small difference is that Zoo defines a way to mark the end of data. The format is pretty well documented in the comments in unzoo.c by Martin Schoenert (see below).
|}

== Identification ==
Files have bytes {{magic|DC A7 C4 FD}} at offset 20.

Zoo files typically start with "{{magic|ZOO ?.?? Archive.}}", followed by the byte <code>0x1a</code>, where each "<code>?</code>" is a digit. Regarding this field, the source code says "The contents of the text message are however not used by Zoo and they may be anything."

== Related formats ==
* [[Zoo filter format]]
* [[Zoo Z format‎]]
* [[Brik]] - same author

== Specifications ==
''[Ed. note: The Zoo format information in these documents is not of the highest quality. We haven't found a specification that is reasonably complete and correct.]''

* [https://www.fileformat.info/format/zoo/corion.htm The ZOO Archive File Format] from Corion.net, via FileFormat.info
* [http://apple2.org.za/gswv/a2zine/GS.WorldView/Resources/The.MacShrinkIt.Project/ARCHIVES.TXT Archive format info, including ZOO]
* [http://www.textfiles.com/programming/FORMATS/arc_fmts.txt ZOO file header format (among other archive types)]

== Software ==
* [[Konvertor]]
* Zoo - DOS binaries
** {{CdTextfiles|megarom/megarom1/ARC_LBR/ZOO120.ZIP|v1.20}} (1986-09-06)
** {{CdTextfiles|carousel344/013/ZOO121.ZIP|v1.21}} (1986-09-21)
** v1.22 (1986-10-01): [https://archive.org/details/ibm-wgam-wbiz-collection] → ibm-wgam-wbiz-collection/xtra0320-0329.zip → xtra0321.tar → FNW315-1.ZIP → ZOO122.EXE (bare executable)
** v1.40 (1987-01-24): [https://archive.org/details/shareware-grab-bag] → Shareware_Grab_Bag_1988.zip → 014/zoo140.arc
** v1.41: [https://archive.org/details/os2sharewarebbs OS/2 Shareware BBS Full Collection] → 08-Other.zip → S12215.ZIP → ZOO.EXE (bare executable)
** {{CdTextfiles|rbbsv3n1/zzap/zoo200.exe|v2.00}} (~1988-02-06)
** {{CdTextfiles|hof91/ARC/ZOO201.EXE|v2.01}} (1988-08-25) - Also includes Fiz 2.0 and Stuff 1.0
** {{CdTextfiles|simtel/simtel20/MSDOS/ARCHIVRS/ZOO210.EXE|v2.10}} (1991-07-09)
* Other official Zoo utilities:
** {{CdTextfiles|simtelctc/disk1/ZOO/BOOZ20.ZIP;1|Booz 2.0}} - Extractor; includes source code and DOS binary
** {{CdTextfiles|carousel344/013/LOOZ210.ZIP|Looz 2.1}}
** {{CdTextfiles|carousel/013B/OOZ130.ZIP|Ooz 1.30}}
** Sez {{CdTextfiles|carousel344/013/SEZ200.ZIP|2.0}}, {{CdTextfiles|microhaus/mhblackbox3/ARCHIVER/SEZ230.ZIP|2.3}} - Makes self-extracting archives
** {{CdTextfiles|simtelctc/disk1/ZOO/ATOZ112.ZIP;1|Atoz 1.12}} - Conversion helper
** {{CdTextfiles|simtel/simtel9211/msdos/zoo/stuff2.zip|Stuff 2.0}}
* [https://www.amiga-stuff.com/archivers-download.html amiga-stuff.com: Archivers] - Lists several versions of Zoo for Amiga
* [http://archives.math.utk.edu/software/multi-platform/gap/util/unzoo.c unzoo.c] by Martin Schoenert (Does not support LZD.)
* [https://github.com/PascalVault/Lazarus_Unpacker Open-source library in Free Pascal]
* {{Deark}}

Original(?) source code:
* v1.41 (1987-02-07): {{CdTextfiles|hof91/PROGLC/ZOO141_C.LZH|ZOO141_C.LZH}}
* v1.51 (1987-07-12)
** [https://archive.org/details/ASMEsMechanicalEngineeringToolkit1997December ASME's Mechanical Engineering Toolkit Dec. 1997] → C_LANG/Z150_SRC.LZH
** As posted to Usenet comp.sources.unix 1987-08: [{{CdTextfilesURL|sourcecode/usenet/compsrcs/unix/volume11/zoo/}}]
* v2.01 (1988-08-25)
** [http://www.nic.funet.fi/index/unix/tools/zoo/zoo-2.01.tar.gz zoo-2.01.tar.gz]
** [https://archive.org/details/ASMEsMechanicalEngineeringToolkit1997December ASME's Mechanical Engineering Toolkit Dec. 1997] → C_LANG/{Z201SRC1.Z00 + Z201SRC2.Z00}
** [https://archive.org/details/pegasus20 Pegasus 2.0] → pegasus/c_2/zoo_src.zip
* v2.10, a.k.a. v2.1 (1991-07-07)
** [http://www.nic.funet.fi/index/unix/tools/zoo/zoo-2.1.tar.gz zoo-2.1.tar.gz]
** {{CdTextfiles|aminet0294/AMINET/MISC/UNIX/ZOO_2_10.TAR|ZOO_2_10.TAR}}

Other source code and packages:
* [http://distcache.freebsd.org/ports-distfiles/zoo-2.10pl1.tar.gz zoo-2.10pl1.tar.gz]
* [http://www.ibiblio.org/pub/Linux/utils/compress/zoo-2.10-3.src.rpm zoo-2.10-3.src.rpm]
* [https://build.opensuse.org/package/show/openSUSE:Factory/zoo openSUSE maintained variant of zoo-2.10]
* [https://github.com/troglobit/zoo zoo - GitHub] - "A service to all computer archaeologists and acts as a focal point for patches and maintenance for the long haul." Provides updated licensing terms that it is now Unlicense/Public Domain.

== Sample files ==
* {{CdTextfilesURL|aminet0294/AMINET/UTIL/ARC/}} → *.ZOO
* {{CdTextfilesURL|garbo/PC/}} → .../*.ZOO
* {{CdTextfilesURL|geminiatari/ZIP/PROGRAM/}} → *.ZOO
* {{DexvertSamples|archive/zoo}}

== Links ==
* [[Wikipedia: Zoo (file format)]]
* {{CdTextfiles|rbbsv3n1/zzap/zooplan1.zip|zooplan1.zip}} → ZOOPLAN1.TXT - A Zoo Manifesto, by Rahul Dhesi (1986-11-01)
* {{CdTextfiles|simtelctc/disk1/ZOO/UGUIDE.ZIP;1|The zoo Archiver User's Guide}}, by Bill Davidsen (for Zoo 2.01)
* [https://manpages.ubuntu.com/manpages/bionic/man1/zoo.1.html Zoo man page] - Includes a change log going back to v1.71
* {{CdTextfiles|carousel344/013/ZOO121.ZIP|Zoo 1.21}} → ZOOHSTRY.TXT - Has some historical information about older versions

EPWING

2025-09-02T18:48:36Z

Anonymoususer852: /* Modern use and alternatives */ Typo fixes for FreeUWING reference.

Talk:EBZip

2025-09-02T18:46:28Z

Anonymoususer852: /* LZMA benchmarks against other compression/utilities */ Typos and wording.

== <code>ebzip</code> caveats ==
Intended to be a place to note down issues, as well as tips and tricks with <code>eb-4.4.3</code>.

=== Hardcoded to look specifically for a file that begins with <code>"CATALOGS"</code> (case-insensitive), or <code>"CATALOGS.ebz"</code> ===
Custom dictionaries doesn't always follow standard convention to store their content(s), like having "<code>START.ebz</code>" in the top level. Even if they do, they may also compress "<code>CATALOGS</code>" as "<code>CATALOGS.ebz</code>." <code>eb-4.4.3</code> specifically looks for either "<code>CATALOGS</code>" or "<code>CATALOGS.ebz</code>", and either decompress or compress the files specified within, per user's request, e.g. <code>ebzip -u</code> will tell it to decompress and rename any of the listed files that are noted in either "<code>CATALOGS</code>" or "<code>CATALOGS.ebz</code>", and has "<code>.ebz</code>" extension, ignoring any other files in the same path, and doesn't decompress "<code>CATALOGS.ebz</code>" even if instructed. There are ways to workaround that:

==== Decompress "<code>CATALOGS.ebz</code>" ====
In my case,
* Simply rename the decompressed "<code>HONMON</code>" to something like "<code>HONMON.old</code>",
* Copy the "<code>CATALOGS.ebz</code>" there, but have it named as "<code>HONMON.ebz</code>",
* Run decompression via the usual <code>ebzip -u</code> from the top level where it doesn't contain "<code>HONMON.ebz</code>" but only "<code>CATALOGS.ebz</code>",
* Once decompressed, move decompressed "<code>HONMON</code>" (not "<code>HONMON.old</code>") back to where "<code>CATALOGS.ebz</code>" was, have it renamed as "<code>CATALOGS</code>",
* Rename "<code>HONMON.old</code>" back to "<code>HONMON</code>".
Optionally, delete "<code>CATALOGS.ebz</code>" considering that is now redundant.

==== Decompress other <code>.ebz</code> that were not listed ====
Same as above, but instead of copying, you can just move it, as,
* <code>ebzip -u</code> will only look for "<code>CATALOGS</code>" or "<code>CATALOGS.ebz</code>", and
* It saves having to manually delete the redundant file(s) afterwards.

=== Zero-length (un)compressed files not overwritten upon manual intervention ===
When a path that has zero-length files, but are listed in either "<code>CATALOGS</code>" or "<code>CATALOGS.ebz</code>", <code>ebzip -u</code> for instance may not overwrite it. This may also apply if <code>ebzip -uf</code> is used. To fix this, just delete the zero-length files, ensuring that they have a duplicate file that does not have <code>.ebz</code> extension, then re-run the command. When compressing, the opposite is true, to look for duplicate zero-length files with <code>.ebz</code> extension, delete them and run the utility to compress instead.

On modern [[Linux]] environments, these could be achieved via the following:

==== Enumerating ====
find . -size 0 -type f

==== Deleting ====
find . -size 0 -type f -exec sudo rm -iv "{}" \;
Care must be taken to avoid any potential losses when running this command.

[[User:Anonymoususer852|Anonymoususer852]] ([[User talk:Anonymoususer852|talk]]) 10:10, 18 August 2025 (UTC)

== LZMA benchmarks against other compression/utilities ==
In response to maintainer of <code>ebu</code>'s blog entry,<ref>[http://ikazuhiro.s206.xrea.com/article.php/20190701203038572 EPWINGを圧縮する (libdeflateで) / Compressing EPWING via libdeflate - Kazuhiro's blog]</ref> I decided to run a couple of benchmarks with different dictionaries, books or other referencing materials on my end. The following packages and their versions are as follows; kernel-6.15.8, zlib-1.3.1, libdeflate-1.24, xz-5.4.2, dwarfs-0.12.4, squashfs-tools-4.7.1.

=== zlib versus libdeflate versus LZMA benchmark (single electronic book) ===
For the following, file sizes are reported from <code>ls -l</code>, therefore size would be reported in blocks<ref>[https://unix.stackexchange.com/questions/28780/file-block-size-difference-between-stat-and-ls file block size - difference between stat and ls - Unix & Linux - Stack Exchange]</ref>. The tested dictionaries using no compression, zlib, libdeflate, or [[XZ]] are [[tar|tarballed]] for consistency. For compressed file systems, XZ's LZMA2 is used. Both zlib and libdeflate has a maximum compression level of 5, anything beyond 5 would result in error, however XZ goes up to level 9, and has further options like <code>--extreme</code> as well as various other tweaks which I will keep it simple and use <code>xz -9evv</code> instead.

[[DWARFS]] example parameters:
mkdwarfs --input . --output=../JMdict_eng_2021-02-26_UTC.dwarfs --block-size-bits=26 --compression=lzma:level=9:extreme --schema-compression=lzma:level=9:extreme --metadata-compression=lzma:level=9:extreme --no-history \
--pack-metadata=all,force --file-hash=sha3-512 --no-history-timestamp --no-create-timestamp --no-history-command-line

[[SquashFS]] example parameters:
mksquashfs . ../JMdict_eng_2021-02-26_UTC.squashfs -b 1048576 -comp xz -Xdict-size 100% -progress -info -no-xattrs

Percentages are rounded to two decimal points, otherwise rounded up. Compression ratio against uncompressed in percentage is offered instead.

{| border="1" align="center" style="text-align:center;" cellpadding="6"
|
| [[EBZip#Sample_file|kanjidic]]
| NHKAccent
| Eijiro
| Kenkyusha eiwa daijiten v6
| Kenkyusha waei daijiten v5
|-
| ''No compression''
| 130,385,920
| 1,578,055,680
| 1,374,771,200
| 484,945,920
| 328,192,000
|-
| <code>ebuzip -z -l 5</code> (zlib)
| 42,035,200
| 1,393,920,000
| 379,975,680
| 117,596,160
| 109,721,600
|-
| ''Compression ratio against uncompressed in percentage''
| 32.24%
| 88.33%
| 27.63%
| 24.25%
| 33.43%
|-
| <code>ebuzip -z -l 5</code> (libdeflate)
| 40,171,520
| 1,383,505,920
| 353,269,760
| 107,765,760
| 102,113,280
|-
| ''Compression ratio against uncompressed in percentage''
| 30.81%
| 87.67%
| 25.70%
| 22.22%
| 31.11%
|-
| <code>xz -9evv -T1</code> (liblzma)
| 21,607,608
| 1,350,900,536
| 197,134,768
| 55,220,788
| 50,109,852
|-
| ''Compression ratio against uncompressed in percentage''
| 16.57%
| 85.61%
| 14.34%
| 11.39%
| 15.27%
|-
| <code>xz -9evv -T0</code> (liblzma)
| 21,607,616
| 1,351,560,168
| 202,934,760
| 56,670,664
| 50,747,640
|-
| ''Compression ratio against uncompressed in percentage''
| 16.57%
| 85.65%
| 14.76%
| 11.69%
| 15.46%
|-
| <code>mk[[DWARFS|dwarfs]]</code> (liblzma)
| 22,145,746
| 1,353,699,364
| 212,307,176
| 60,043,728
| 54,541,980
|-
| ''Compression ratio against uncompressed in percentage''
| 16.99%
| 85.78%
| 15.44%
| 12.38%
| 16.62%
|-
| <code>mk[[SquashFS|squashfs]]</code> (liblzma)
| 31,793,152
| 1,367,302,144
| 272,068,608
| 81,805,312
| 160,956,416
|-
| ''Compression ratio against uncompressed in percentage''
| 24.38%
| 86.64%
| 19.79%
| 16.87%
| 49.04%
|}

XZ compression results are provided for the sake of discussion only, more on this in conclusion. While it does show extremely good results against all other utilities, the seek times are abysmally poor, worse than DWARFS with LZMA2 on machines with weaker hardware. Tests confirms liblzma having better compression regardless, against both zlib and libdeflate.

NHKAccent is the exception here, with poor compression results. This is due to electronic book containing multimedia files, which won't compress well if it was already compressed.

It is worthy to note here that compression/archiving utilities embracing liblzma, could by default take advantage of multi-threading, at the expense of producing potentially larger file. This however, is a small trade off, for effectively making full use of available CPU in the given environment.

=== Modern compressed file system with multiple electronic books shootout ===
Author of <code>ebu</code> mentioned the case with smartphones, which I do personally find that these are also limited in performance, not just with storage capacities. [[SquashFS]] was or is used in Android devices,<ref>[https://android.stackexchange.com/questions/258236/squashfs-explanation-for-android-system SquashFS explanation for Android system - Android Enthusiasts - Stack Exchange]</ref> so it makes sense to try using that on non system files in addition.

For these tests, the sample data consists of ~189 dictionaries in a single directory. Both [[DWARFS]] and [[SquashFS]] are configured to use XZ's LZMA2 where possible, as it is known to be one of the best, freely available (especially under Unix-like platforms) compression. Both DWARFS and SquashFS parameters are the same as in prior section. Due to the inherent way of how <code>ebuzip</code> works, requiring "<code>CATALOGS</code>" to be in the top directory, both variations of <code>ebuzip</code> using either zlib or libdeflate are omitted from this test. Besides, single compression job per dictionary, not including other files in the that path, e.g. "<code>START</code>" would make the overall process time consuming, if the process isn't already tedious that it requires extensive scripting.

{| border="1" align="center" style="text-align:center;" cellpadding="6"
| Method
| Size (blocks)
|-
| Uncompressed [[tar]]
| 29,014,026,240
|-
| [[DWARFS]]
| 10,613,886,184
|-
| [[SquashFS]]
| 12,269,518,848
|}

Where DWARFS shines, is the ability to also save on segmenting data, in addition to file de-duplication. For a small penalty I've given to DWARFS, in that it is to use sha3-512 as hash, it still compresses the collection better and faster, than SquashFS. Then again, the focus here is on overall file size, not performance in file creation.

=== Conclusion ===
Now of course, there are plenty of other compression utilities available, such as, but are not limited to [[Zstandard]], [[BZip3]], [[LZO]], etc. From the various results I have read, indicates that these other compression benchmarks, while trading better performance, these fall behind in final size constraints. The benchmark showcase here outlines the need to have as small of a compressed file as possible, with modern compression utilities and modern PC hardware, therefore making (speed-wise) performance testing out of the scope, as is the idea of creating the likes of [[ZIP]] bombs, or nested compressed files out of the scope.

I have also tried to run <code>mkfs.[[EROFS|erofs]]</code> in order to include it against the modern compressed file system with multiple dictionaries shootout. With version 1.8.10, no matter what parameters I set with LZMA <code>level=9</code> or <code>level=109</code> (109 is the extreme variant), even with lowering <code>dictsize=</code>, the results would cause system to run out of memory (16GB of RAM) before even completing the file system creation.

These benchmarks are in no way discrediting or disavowing the use of libdeflate over zlib. If file size is a concern, more modern compression algorithms and/or compression techniques could be considered. Modern EPWING viewer programs generally don't care if the electronic book is compressed using zlib or not compressed at all, and there are many ways of handling decompression transparently in modern Unix-like environments. Even more so, are the ever growing hardware performance as well as affordability with storage capacities in modern computing, questions the use of the what is widely considered as legacy compression algorithms for data archival. Last but not least, while single-threaded compression (as witnessed in the likes of <code>xz -9evv -T1</code> above) or tasks might be extremely beneficial in reducing overheads, especially with LZMA2, it is also a slow process which may subsequently not translate well in modern, demanding contexts, where time honored traditions are casted away for practical, convenient, modernized, and easy to use solutions.

<references />
[[User:Anonymoususer852|Anonymoususer852]] ([[User talk:Anonymoususer852|talk]]) 18:46, 2 September 2025 (UTC)

EBZip

2025-09-02T18:42:51Z

Anonymoususer852: Restructured content due to separation of EPWING into its own page. Split EBUZip into its own section.

{{FormatInfo
|formattype=electronic
|subcat=Compression
|extensions={{ext|ebz}}
|compression=lossless
|magic=45 42 5a 69 70 15 00 00 00 00 00 00 10 00 8b e8
|released=1999<ref>[https://web.archive.org/web/20010619200710/http://www.sra.co.jp/people/m-kasahr/eb/ChangeLog-ja.html ChangeLog - SRA website mirrored on Internet Archive]</ref>
}}
{{notice|1=Majority of the links/citations are in Japanese.}}

'''Electronic Book Zip or EBZip''' is a compression format used in electronic dictionaries that were distributed primarily on compact discs (CD). It is also one of the many frontend utilities that uses the EB library for manipulating [[EPWING]] format.

The electronic dictionaries stored on computers eliminates the need to have physical media, and allows remote referencing if it was connected to a network, using the likes of <code>dserver</code> or <code>ebnetd</code> for instance. It is likely that compression was needed due to the fact that hard disks were expensive compared to compact discs, when having a number of electronic dictionaries stored on a computer, it made little sense to retain physical media just to reference a number of words every so often, and to juggle around ejecting/inserting discs into the computer. For the expense of sacrificing few CPU cycles and potentially insignificant amount of memory, these electronic dictionaries could be made to take less storage space<ref>[https://web.archive.org/web/20170705202139/http://sci.lang.japan.narkive.com/X84h0clT/kenkyusha-dictionary-cd-and-epwing#post2 Kenkyusha dictionary CD and EPWING - sci.lang.japan NARKIVE - Internet Archive copy]</ref>, and allows faster dictionary lookups due to hard disks allowing random access.

The compression is lossless, initially using a modified [https://www.zlib.net/ zlib]<ref>[https://github.com/mistydemeo/eb/blob/742248c23cf7769d0fed412cbce458222a437795/zlib/README.altered Last version released with internal zlib - README.altered - version 4.1.3 - Misty De Méo]</ref>, then likely unmodified zlib by the original <code>ebzip</code> author<ref>[https://github.com/mistydemeo/eb/commit/742ddab4f255b7c34d7d1f1b4e3fd245bfc3ea18 Internal zlib removed - Commit 742ddab (4.2) - Misty De Méo]</ref>. Do keep in mind that not everything is compressed, EBZ format is not a container like [[ZIP]] for instance, selected files (typically <code>CATALOGS</code> and <code>HONMON</code>) are compressed and have their filenames bearing these extensions, e.g. <code>HONMON.ebz</code>.

== History ==
EB-like software for Unix-like began circulating on Japanese Usenet as early as September 1991 in the form of [[shar]] by Shingo Nishioka of Osaka University.<ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=NISHIOKA.91Sep3191513@sp4.sanken.osaka-u.ac.jp dserver/Electronic Book Dictionary server & clients (1/2) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=NISHIOKA.91Sep3191550@sp4.sanken.osaka-u.ac.jp dserver/Electronic Book Dictionary server & clients (2/2) - AncientFJ - katsu.watanabe.name]</ref> Versions before 2.1beta of <code>dserver</code> lacked compression, which was introduced as early as 1996-03-25 by Yura Shunsuke.<ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=4j5h0e$2am@isnews.is.s.u-tokyo.ac.jp dserver 2.1-beta patch for compressed dictionary - AncientFJ - katsu.watanabe.name]</ref> Gradually over time, Motoyuki Kasahara, a member of Software Research Associates (SRA) and the author of EB Library, began developing <code>ndtpd</code> which handles electronic dictionary or dictionaries over the network, replacing <code>dserver</code>.<ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5rp4qp$3sk$1@sranhh.sra.co.jp BookView 1.0 - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sctc3$irv$2@sranhh.sra.co.jp BookView 1.0.2 (Re: BookView 1.0) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=64emm8$mcf$1@sranhh.sra.co.jp BookView 1.1 (1/3) - AncientFJ - katsu.watanabe.name]</ref> The first release of such software was around 1997-06-04 as <code>ndtpd-1.0beta0</code> (via FTP) with which it included <code>eb</code> which was a library. Earliest release on fj.sources was 1997-08-07 as <code>ndtpd-1.0</code>, which also describes the relationship between <code>dserver</code> and <code>ndtpd</code>, as [[shar]].<ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc4ns$31a$2@sranhh.sra.co.jp ndtpd 1.0 (0/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc4pr$3o1$1@sranhh.sra.co.jp ndtpd 1.0 (1/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc4rn$3o1$2@sranhh.sra.co.jp ndtpd 1.0 (2/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc4te$3o1$3@sranhh.sra.co.jp ndtpd 1.0 (3/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc4vf$3o1$4@sranhh.sra.co.jp ndtpd 1.0 (4/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc514$3o1$5@sranhh.sra.co.jp ndtpd 1.0 (5/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc52p$3o1$6@sranhh.sra.co.jp ndtpd 1.0 (6/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc54s$3o1$7@sranhh.sra.co.jp ndtpd 1.0 (7/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc55j$3o1$8@sranhh.sra.co.jp ndtpd 1.0 (8/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc56d$3o1$9@sranhh.sra.co.jp ndtpd 1.0 (9/9) - AncientFJ - katsu.watanabe.name]</ref> Until the release of <code>ndtpd-3.0alpha0</code> (2000-06-04), <code>eb</code> has always been included. EB Library began as a separate project with the first release in 1999-06-16 as <code>eb-2.2</code>. Both of these projects continued until the discontinuation of <code>ndtpd</code> (as a release) on 2003-05-24 with the version 3.1.5. <code>eb</code> remained to be maintained until 2010-03-08 with the version 4.4.3.

== EBUZip ==
EBUZip is a fork of EBZip that is maintained by Ito Kazuhiro. The 'U' in EBUZip denotes the ability to support JIS X 4081 UTF-8 extensions provided that it is compiled and configured correctly. It also brings in new features that differs from EBZip such as,
* The ability to use libdeflate algorithm at compile time, which, "is a library for fast, whole-buffer DEFLATE-based compression and decompression."<ref>[https://github.com/ebiggers/libdeflate?tab=readme-ov-file#overview README.md Overview section - libdeflate - GitHub]</ref>, which was added on 2019-06-30.<ref>[http://green.ribbon.to/~ikazuhiro/dic/ebu.html EB Library with UTF-8 extension]</ref>. Additionally, the new maintainer provided benchmark results against uncompressed, zlib and libdeflate.<ref>[http://ikazuhiro.s206.xrea.com/article.php/20190701203038572 Compressing EPWING (using libdeflate)]</ref>
* Fixes to work on a newer codebase, as opposed to maintaining a series of patches on top of libeb.<ref>[https://sources.debian.org/data/main/e/eb/4.4.3-14/debian/patches/010_debian.patch 010_debian.patch - eb-4.4.3-14 - sources.debian.org]</ref>

Mr. Kazuhiro have also previously contributed to Mr. Kasahara's EBZip project, on 2000-06-24.<ref>[https://github.com/mistydemeo/eb/blob/master/ChangeLog.1#L1371 ChangeLog.1 (line 1371) - EB Library - Misty De Méo]</ref>.

== Identification ==
A <code>EBZ</code> file begins with bytes {{magic|45 42 5a 69 70 15 00 00 00 00 00 00 10 00 8b e8}}.

== Examples ==
'''These must be executed in the top level of (book) directory that contains <code>CATALOGS</code> file, otherwise it will default to your current directory <code>$(pwd)</code>.''' See [[EPWING#Content structure]] for the directory structure of book layout.
If you are using EBUZip, the name of the compiled binary is <code>ebuzip</code>, as opposed to <code>ebzip</code>.

Show information about the file:
ebzip --information
Decompress files:
ebzip --uncompress wikipedia-fpw
Compress files:
ebzip wikipedia-fpw
Test files:
ebzip --test wikipedia-fpw
Show help:
ebzip --help

== Sample file ==
{| border="1" align="center" style="text-align:center;" cellpadding="6"
| Filename
| Main server download link
| Mirror server download link
| Notes
|-
| kanjidic_en.fpwebz.tar
| [http://ftp.edrdg.org/pub/Nihongo/edict_en.fpwebz.tar EDRDG.org]
| [http://ftp.usf.edu/pub/ftp.monash.edu.au/pub/nihongo/kanjidic_en.fpwebz.tar USF.edu]
| Use [[7-Zip]] to open on Windows machines for instance, otherwise [[Tar]] should suffice.
|}

== Software (mostly Japanese only) ==
See also: [[Wikipedia:ja:EPWING#検索ソフトウェア]]
* Windows
** [ftp://ftp.sra.co.jp/pub/misc/eb/WinEBZip-0.0.exe WinEBZip.exe on SRA's FTP server - no longer maintained]
** [ftp://ftp.sra.co.jp/pub/misc/eb/WinEBZip-0.0.zip WinEBZip.zip on SRA's FTP server - no longer maintained]
** [https://ebstudio.info/manual/EBWin4/EBWin4.html EBWin4]
* Linux
** [ftp://ftp.sra.co.jp/pub/misc/eb/ EB Library directory on SRA's FTP server - no longer maintained]
** [http://green.ribbon.to/~ikazuhiro/dic/ebu.html EBU - EB Library with JIS X 4081 UTF-8 extension]
** [https://github.com/mvf/qolibri qolibri EPWING dictionary viewer]

== Links (mostly Japanese only) ==
* [[Wikipedia:ja:電子ブック (規格)]]
* [https://web.archive.org/web/20010606001016/http://www.sra.co.jp/people/m-kasahr/eb/index-ja.html Internet Archive's mirror of EB Library that was hosted on SRA's website]
* [http://green.ribbon.to/~ikazuhiro/dic/ebu.html EBU]

== References (mostly Japanese only) ==
<references/>

Electronic Publishing formats

2025-09-02T18:19:29Z

Anonymoususer852: /* E-book formats */ Added EPWING page.

{{FormatInfo
|formattype=electronic
|thiscat=Electronic Publishing formats
|image=Book.jpg
}}
Formats specific to the storage and transmission of publications in electronic form.

For ''desktop publishing'', see [[Document#Desktop Publishing]].

== Dictionaries, encyclopedias, etc. ==
* [[AARD]] [http://aarddict.org/1/index.html]
* [[Apple dictionary]] [https://josephg.com/blog/reverse-engineering-apple-dictionaries/] [https://developer.apple.com/library/archive/documentation/UserExperience/Conceptual/DictionaryServicesProgGuide/Introduction/Introduction.html]
* [[BGL]] (Babylon)
* [[CC-CEDICT]]
* [[Dic]] (Babylon) [http://ktranslator.sourceforge.net/dictionaries.html]
* [[Dictd]]
* [[Lingvo DSL]] [https://beijer.uk/software/dsl_abbyy_linvgvo]
* [[Ling]] [http://linguae.stalikez.info/]
* [[Lingoes]] [http://lingoes.net/]
* [[Mdict MDX]] [https://www.mdict.cn/wp/?lang=en]
* [[Mova]] [http://jalingo.sourceforge.net/dictionaries/]
* [[ODict]] [https://odict.org/]
* [[Slob]] [https://github.com/itkach/slob]
* [[SDictionary]] [http://swaj.net/sdict/index.html] [http://ktranslator.sourceforge.net/dictionaries.html]
* [[StarDic]]
* [[StarDict]]
* [[SugarDict]]
* [[XDXF]]

== E-book formats ==

* [[AZW]]
* [[Daisy]]
* [[DocBook]]
* [[EPUB]]
* [[EPWING]]
* [[IMP (Electronic publishing)|IMP]]
* [[KFX (Kindle)|KFX]] (Kindle Format X)
* [[LRF]]
* [[MOBI]]
* [[PalmDOC]]
* [[PDG]] (Superstar)
* [[Rocket eBook]]
* [[Shanda Bambook]]
* [[SiliconPage]]
* [[TomeRaider]]

== Journals, magazines, newspapers, articles, etc. ==
* [[Digital Replica Plus]]
* [[Issuu]]
* [[Journal Article Tag Suite]] (JATS / NLM)
* [[News Industry Text Format]]
* [[ScienceServer XML format]]

== Scanned or archived book formats ==

* [[Comic Book Archive]] (.cbr, .cbz, .cbt, .cba, .cb7)
* [[HathiTrust submission format]]

== Other formats ==

* [[FlipAlbum]] (.flp)
* [[OFIP]]
* [[Onix]] - [https://www.loc.gov/preservation/digital/formats/fdd/fdd000488.shtml LoC Description]
* [[Open Board Format]] (.obf, .obz)
* [[Portable Web Publications]]
* [[RePub]]
* [[Wordcruncher]] (.etb(u), .erl(u) .etg(u))

See also [[Document]], [[Markup]], and [[Web]], since e-publications might also be stored in formats (such as [[PDF]] and [[HTML]]) in those categories, and e-publishing formats may have roots in those formats.

== Identifiers ==

Naming and numbering systems for identifying published material.

* [[ARK]]
* [[DOI]]
* [[ISBN]]
* [[ISSN]]

== Links ==
* [https://scholasticahq.com/innovations-in-scholarly-publishing/announcement/one-of-the-biggest-bottlenecks-in-open-access-publishing-is-typesetting-it-shouldn-t-be Innovations in Scholarly Publishing - One of the biggest bottlenecks in Open Access publishing is typesetting. It shouldn't be.]
* [http://apprenticealf.wordpress.com/2012/09/10/calibre-plugins-the-simplest-option-for-removing-most-ebook-drm/ DeDRM plugin for calibre: the simplest option for removing DRM from most ebooks]
* [http://www.nbcnews.com/tech/gadgets/you-dont-own-your-kindle-books-amazon-reminds-customer-f1C6626211 You don't own your Kindle books, Amazon reminds customer]
* [http://boingboing.net/2013/10/22/even-amazon-cant-keep-up-the.html Even Amazon can't keep up the "you only license ebooks" shuck]
* [http://allthingsd.com/20130906/how-a-man-in-austria-used-legos-to-hack-amazons-kindle-e-book-security/ How a Man in Austria Used Legos to Hack Amazon’s Kindle E-Book Security]
* [http://www.mobileread.com/forums/showpost.php?p=2754561&postcount=319 Adobe is apparently introducing new "hardened" e-book DRM that will break compatibility with older readers]
* [http://www.the-digital-reader.com/2014/02/03/adobe-require-new-epub-drm-july-expects-abandon-existing-users/ Another article on Adobe DRM move]
* [http://sourceforge.net/projects/libebook/ libebook: library to read various e-book formats]
* [http://www.theverge.com/2014/5/9/5688146/the-fight-to-save-endangered-ebooks The fight to save endangered ebooks]
* [http://www.loc.gov/preservation/resources/rfs/textmus.html Library of Congress Recommended Format Specifications: Textual Works]
* [http://www.theguardian.com/books/2014/aug/19/readers-absorb-less-kindles-paper-study-plot-ereader-digitisation Readers absorb less on Kindles than on paper, study finds]
* [http://publishersweekly.com/pw/by-topic/industry-news/libraries/article/64271-check-it-out-with-michael-kelley-how-libraries-preserve-e-books.html Michael Kelley: How Libraries Preserve E-books]
* [http://the-digital-reader.com/2014/10/06/adobe-spying-users-collecting-data-ebook-libraries/ Adobe is Spying on Users, Collecting Data on Their eBook Libraries]
* [http://go-to-hellman.blogspot.com/2014/10/correcting-misinformation-on-adobe.html Correcting Misinformation on the Adobe Privacy Gusher]
* [http://www.theguardian.com/books/booksblog/2012/jun/07/war-and-peace-ebook-nook War and Peace ebook readers find a surprise in its Nooks]
* [http://vgel.me/posts/cracking-online-textbook/ Getting a full PDF from a DRM-encumbered online textbook]
* [http://www.trustedreviews.com/how-tos/how-to-remove-drm-from-your-ebooks Removing DRM from e-books]
* [https://bancroftdigitalcollectionsunit.wordpress.com/2016/03/11/how-to-read-an-ebook-from-1991/ How to Read an e-book…From 1991]
* [http://blog.kbresearch.nl/2016/04/04/valid-but-not-accessible-epub-crazy-fixed-layouts/ Valid, but not accessible EPUB: crazy fixed layouts]
* [http://wiki.dpconline.org/images/0/04/EBook_Summary_Assessment_v1.0.pdf File format assessment: eBook Summary (British Library)]

EPWING

2025-09-02T18:18:30Z

Anonymoususer852: Created page as a split from EBZip as the two are mutually distinct.

PC Screen Font

2025-08-24T20:33:10Z

Anonymoususer852: Added magic bytes for FormatInfo with reference as links. Updated links to https. Added discmaster.textfiles.com link for Sample files section.

{{FormatInfo
|formattype=electronic
|subcat=Fonts
|extensions={{ext|psf}}, {{ext|psfu}}
|magic=[https://github.com/file/file/blob/master/magic/Magdir/linux#L60 36 04] (version 1) [https://github.com/file/file/blob/master/magic/Magdir/linux#L60 72 b5 4a 86 00 00] (version 2)
|released=~1989
}}
The '''PC Screen Font''' ('''PSF''') format is a fixed-width bitmap font format originally used on the PC under MS-DOS. It was subsequently adopted and extended for Linux console fonts.

PSF has been extended to support [[Unicode]]. The '''.psfu''' file extension is sometimes used to indicate that the file includes Unicode mappings, though '''.psf''' files often have them as well.

== Identification ==
Files begin with bytes {{magic|0x36 0x04}} (version 1) or {{magic|0x72 0xb5 0x4a 0x86}} (version 2). The former may be a reference to [[vfont]] format, the magic number of which is 0436 when converted to octal.

== Specifications ==
* https://aeb.win.tue.nl/linux/kbd/font-formats-1.html

== Software ==
* [https://www.seasip.info/Unix/PSF/ PSF Tools]
* {{Deark}}

== Sample files ==
* [https://packages.debian.org/stable/console-setup-linux console-setup-linux package] → console-setup-linux_*.deb → /usr/share/consolefonts/*.psf.gz
* {{DexvertSamples|font/pcScreenFont}}
* [https://discmaster.textfiles.com/search?mode=deep&extension=psf&format=pcScreenFont Search results of files with PSF extension and has format pcScreenFont - Discmaster.textfiles.com]

== Links ==
* [[Wikipedia: PC Screen Font]]

SoundFont 2.0

2025-08-24T20:02:46Z

Anonymoususer852: Provided magic in FormatInfo template and a reference link. Partly updated the identification to include the hex detail. Updated links, old http/ftp links are no longer accessible, and rough replacements has been provided in lieu.

{{FormatInfo
| subcat = Audio and Music
| magic = [https://github.com/file/file/blob/master/magic/Magdir/riff#L678 73 66 62 6b] (at offset 0x08)
| extensions = {{ext|sf2}}
}}
'''SoundFont 2.0''' is a format used to store instrument patches for [[MIDI]] synthesisers. The SoundFont 2.0 format is based on [[RIFF]]; the RIFF type identifier is {{magic|"sfbk"}}.

See also [[SoundFont 1.0]].

== Identification ==
Files begin with {{magic|52 49 46 46}} which in ASCII is "{{magic|RIFF}}", and have {{magic|73 66 62 6b}} or in ASCII "{{magic|sfbk}}" at offset 8.

In most cases, this is followed by "{{magic|LIST}}" at offset 12, "{{magic|INFOifil}}" at offset 20, and bytes {{magic|02 00}} at offset 32. The last <code>02</code> byte distinguishes major version 2 from [[SoundFont 1.0|major version 1]].

== Software ==
* [http://www.swamiproject.org/ Swami]
* [https://www.fluidsynth.org/ FluidSynth]
* [https://audacious-media-player.org/ Audacious]<ref>[https://redmine.audacious-media-player.org/boards/1/topics/1143?r=1146#message-1146 How-to: MIDI playback - Audacious - OLD, PLEASE USE GITHUB DISCUSSIONS/ISSUES - redmine.audacious-media-player.org]</ref>

== Sample files ==
* [https://web.archive.org/web/20160308112401/http://freepats.zenvoid.org/sf2/acoustic_grand_piano_ydp_20080910.sf2 acoustic_grand_piano_ydp_20080910.sf2 - freepats.zenvoid.org - Internet Archive copy]
* [https://web.archive.org/web/20241220060025/http://www.personalcopy.com/Downloads/PC51f.sf2.gz PC51f.sf2.gz - www.personalcopy.com - Internet Archive copy] - File might be compressed twice.
* {{DexvertSamples|audio/soundFont2}}
* [https://discmaster.textfiles.com/search?mode=deep&extension=.sf2&format=soundFont2 Search results of files with SF2 extension and has format soundFont2 - Discmaster.textfiles.com]

== Links ==
* [[wikipedia:SoundFont|SoundFont on Wikipedia]]
* [https://docs.fileformat.com/audio/sf2/ SF2 File Format - SoundFont 2 Sound Bank - docs.fileformat.com]

== References ==
<references />

[[Category:RIFF based file formats]]

EBZip

2025-08-20T12:21:59Z

Anonymoususer852: Updated magic.

{{FormatInfo
|formattype=electronic
|subcat=Compression
|extensions={{ext|ebz}}
|wikidata={{wikidata|Q11197749}} (EPWING format only)
|compression=lossless
|magic=45 42 5a 69 70 15 00 00 00 00 00 00 10 00 8b e8
|released=1999<ref>[https://web.archive.org/web/20010619200710/http://www.sra.co.jp/people/m-kasahr/eb/ChangeLog-ja.html ChangeLog - SRA website mirrored on Internet Archive]</ref>
}}
{{notice|1=Majority of the links/citations are in Japanese.}}

'''Electronic Book Zip or EBZip''' is a compression format used in electronic dictionaries that were distributed primarily on compact discs (CD). The electronic dictionaries stored on computers eliminates the need to have physical media, and allows remote referencing if it was connected to a network, using the likes of <code>dserver</code> or <code>ebnetd</code> for instance. It is likely that compression was needed due to the fact that hard disks were expensive compared to compact discs, when having a number of electronic dictionaries stored on a computer, it made little sense to retain physical media just to reference a number of words every so often, and to juggle around ejecting/inserting discs into the computer. For the expense of sacrificing few CPU cycles and potentially insignificant amount of memory, these electronic dictionaries could be made to take less storage space<ref>[https://web.archive.org/web/20170705202139/http://sci.lang.japan.narkive.com/X84h0clT/kenkyusha-dictionary-cd-and-epwing#post2 Kenkyusha dictionary CD and EPWING - sci.lang.japan NARKIVE - Internet Archive copy]</ref>, and allows faster dictionary lookups due to hard disks allowing random access.

The compression is lossless, initially using a modified [https://www.zlib.net/ zlib]<ref>[https://github.com/mistydemeo/eb/blob/742248c23cf7769d0fed412cbce458222a437795/zlib/README.altered Last version released with internal zlib - README.altered - version 4.1.3 - Misty De Méo]</ref>, then likely unmodified zlib by the original <code>ebzip</code> author<ref>[https://github.com/mistydemeo/eb/commit/742ddab4f255b7c34d7d1f1b4e3fd245bfc3ea18 Internal zlib removed - Commit 742ddab (4.2) - Misty De Méo]</ref>. Under a new maintainer, [https://github.com/ebiggers/libdeflate libdeflate] was added on 2019-06-30.<ref>[http://green.ribbon.to/~ikazuhiro/dic/ebu.html EB Library with UTF-8 extension]</ref> in addition to having [http://green.ribbon.to/~ikazuhiro/dic/jisx4081u.html JIS X 4081 UTF-8 extension] support. Additionally, the new maintainer provided benchmark results against uncompressed, zlib and libdeflate.<ref>[http://ikazuhiro.s206.xrea.com/article.php/20190701203038572 Compressing EPWING (using libdeflate)]</ref> Do keep in mind that not everything is compressed, EBZ format is not a container like [[ZIP]] for instance, selected files (typically <code>CATALOGS</code> and <code>HONMON</code>) are compressed and have their filenames bearing these extensions, e.g. <code>HONMON.ebz</code>.

Originally intended for electronic dictionaries that were distributed primarily on CD, it is possible to create your own dictionaries using other referencing materials, such as Wikipedia. Modern software transparently handles decompression, eliminating the need to manually decompress.

== History ==
EPWING or '''E'''lectronic '''P'''ublishing-WING was a consortium of five members; Iwanami Shoten, Dai Nippon Printing, Toppan Printing, Fujitsu and Sony, produced the Electronic Book (EB) extension based on WING standard in circa 1991, with the aim to promote electronic publications in a common format, and to standardize WING onto ISO 9660 (the logical format for CD-ROMs), establishing the first edition (V1) of the EPWING convention.<ref>[[Wikipedia:ja:EPWING#成立の経緯]]</ref>

EB-like software for Unix-like began circulating on Japanese Usenet as early as September 1991 in the form of [[shar]] by Shingo Nishioka of Osaka University.<ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=NISHIOKA.91Sep3191513@sp4.sanken.osaka-u.ac.jp dserver/Electronic Book Dictionary server & clients (1/2) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=NISHIOKA.91Sep3191550@sp4.sanken.osaka-u.ac.jp dserver/Electronic Book Dictionary server & clients (2/2) - AncientFJ - katsu.watanabe.name]</ref> Versions before 2.1beta of <code>dserver</code> lacked compression, which was introduced as early as 1996-03-25 by Yura Shunsuke.<ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=4j5h0e$2am@isnews.is.s.u-tokyo.ac.jp dserver 2.1-beta patch for compressed dictionary - AncientFJ - katsu.watanabe.name]</ref> Gradually over time, Motoyuki Kasahara, a member of Software Research Associates (SRA) and the author of EB Library, began developing <code>ndtpd</code> which handles electronic dictionary or dictionaries over the network, replacing <code>dserver</code>.<ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5rp4qp$3sk$1@sranhh.sra.co.jp BookView 1.0 - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sctc3$irv$2@sranhh.sra.co.jp BookView 1.0.2 (Re: BookView 1.0) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=64emm8$mcf$1@sranhh.sra.co.jp BookView 1.1 (1/3) - AncientFJ - katsu.watanabe.name]</ref> The first release of such software was around 1997-06-04 as <code>ndtpd-1.0beta0</code> (via FTP) with which it included <code>eb</code> which was a library. Earliest release on fj.sources was 1997-08-07 as <code>ndtpd-1.0</code>, which also describes the relationship between <code>dserver</code> and <code>ndtpd</code>, as [[shar]].<ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc4ns$31a$2@sranhh.sra.co.jp ndtpd 1.0 (0/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc4pr$3o1$1@sranhh.sra.co.jp ndtpd 1.0 (1/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc4rn$3o1$2@sranhh.sra.co.jp ndtpd 1.0 (2/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc4te$3o1$3@sranhh.sra.co.jp ndtpd 1.0 (3/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc4vf$3o1$4@sranhh.sra.co.jp ndtpd 1.0 (4/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc514$3o1$5@sranhh.sra.co.jp ndtpd 1.0 (5/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc52p$3o1$6@sranhh.sra.co.jp ndtpd 1.0 (6/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc54s$3o1$7@sranhh.sra.co.jp ndtpd 1.0 (7/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc55j$3o1$8@sranhh.sra.co.jp ndtpd 1.0 (8/9) - AncientFJ - katsu.watanabe.name]</ref><ref>[https://katsu.watanabe.name/ancientfj/article.php?mid=5sc56d$3o1$9@sranhh.sra.co.jp ndtpd 1.0 (9/9) - AncientFJ - katsu.watanabe.name]</ref> Until the release of <code>ndtpd-3.0alpha0</code> (2000-06-04), <code>eb</code> has always been included. EB Library began as a separate project with the first release in 1999-06-16 as <code>eb-2.2</code>. Both of these projects continued until the discontinuation of <code>ndtpd</code> (as a release) on 2003-05-24 with the version 3.1.5. <code>eb</code> remained to be maintained until 2010-03-08 with the version 4.4.3.

It is notable that Ito Kazuhiro (author of EBU - EB Library with JIS X 4081 UTF-8 extension) made a contribution to Motoyuki Kasahara's EB Library on 2000-06-24.<ref>[https://github.com/mistydemeo/eb/blob/master/ChangeLog.1#L1371 ChangeLog.1 (line 1371) - EB Library - Misty De Méo]</ref>.

== Identification ==
A <code>EBZ</code> file begins with bytes {{magic|45 42 5a 69 70 15 00 00 00 00 00 00 10 00 8b e8}}.

== Examples ==
'''These must be executed in the top level of (book) directory that contains <code>CATALOGS</code> file, otherwise it will default to your current directory <code>$(pwd)</code>.''' See [[Wikipedia:ja:EPWING#ファイル構造]] for the directory structure of book layout.

Show information about the file:
ebzip --information
Decompress files:
ebzip --uncompress wikipedia-fpu
Compress files:
ebzip wikipedia-fpu
Test files:
ebzip --test wikipedia-fpu
Show help:
ebzip --help

== Sample file ==
{| border="1" align="center" style="text-align:center;" cellpadding="6"
| Filename
| Main server download link
| Mirror server download link
| Notes
|-
| kanjidic_en.fpwebz.tar
| [http://ftp.edrdg.org/pub/Nihongo/edict_en.fpwebz.tar EDRDG.org]
| [http://ftp.usf.edu/pub/ftp.monash.edu.au/pub/nihongo/kanjidic_en.fpwebz.tar USF.edu]
| Use [[7-Zip]] to open on Windows machines for instance, otherwise [[Tar]] should suffice.
|}

== Software (mostly Japanese only) ==
See also: [[Wikipedia:ja:EPWING#検索ソフトウェア]]
* Windows
** [ftp://ftp.sra.co.jp/pub/misc/eb/WinEBZip-0.0.exe WinEBZip.exe on SRA's FTP server - no longer maintained]
** [ftp://ftp.sra.co.jp/pub/misc/eb/WinEBZip-0.0.zip WinEBZip.zip on SRA's FTP server - no longer maintained]
** [https://ebstudio.info/manual/EBWin4/EBWin4.html EBWin4]
* Linux
** [ftp://ftp.sra.co.jp/pub/misc/eb/ EB Library directory on SRA's FTP server - no longer maintained]
** [http://green.ribbon.to/~ikazuhiro/dic/ebu.html EBU - EB Library with JIS X 4081 UTF-8 extension]
** [https://github.com/mvf/qolibri qolibri EPWING dictionary viewer]

== Links (mostly Japanese only) ==
* [[Wikipedia:ja:電子ブック (規格)]]
* [https://web.archive.org/web/20010606001016/http://www.sra.co.jp/people/m-kasahr/eb/index-ja.html Internet Archive's mirror of EB Library that was hosted on SRA's website]
* [http://green.ribbon.to/~ikazuhiro/dic/ebu.html EBU]

== References (mostly Japanese only) ==
<references/>

Talk:EBZip

2025-08-20T11:42:19Z

Anonymoususer852: /* zlib versus libdeflate versus LZMA benchmark (single electronic book) */ Wording on multi-threaded usage.

== <code>ebzip</code> caveats ==
Intended to be a place to note down issues, as well as tips and tricks with <code>eb-4.4.3</code>.

=== Hardcoded to look specifically for a file that begins with <code>"CATALOGS"</code> (case-insensitive), or <code>"CATALOGS.ebz"</code> ===
Custom dictionaries doesn't always follow standard convention to store their content(s), like having "<code>START.ebz</code>" in the top level. Even if they do, they may also compress "<code>CATALOGS</code>" as "<code>CATALOGS.ebz</code>." <code>eb-4.4.3</code> specifically looks for either "<code>CATALOGS</code>" or "<code>CATALOGS.ebz</code>", and either decompress or compress the files specified within, per user's request, e.g. <code>ebzip -u</code> will tell it to decompress and rename any of the listed files that are noted in either "<code>CATALOGS</code>" or "<code>CATALOGS.ebz</code>", and has "<code>.ebz</code>" extension, ignoring any other files in the same path, and doesn't decompress "<code>CATALOGS.ebz</code>" even if instructed. There are ways to workaround that:

==== Decompress "<code>CATALOGS.ebz</code>" ====
In my case,
* Simply rename the decompressed "<code>HONMON</code>" to something like "<code>HONMON.old</code>",
* Copy the "<code>CATALOGS.ebz</code>" there, but have it named as "<code>HONMON.ebz</code>",
* Run decompression via the usual <code>ebzip -u</code> from the top level where it doesn't contain "<code>HONMON.ebz</code>" but only "<code>CATALOGS.ebz</code>",
* Once decompressed, move decompressed "<code>HONMON</code>" (not "<code>HONMON.old</code>") back to where "<code>CATALOGS.ebz</code>" was, have it renamed as "<code>CATALOGS</code>",
* Rename "<code>HONMON.old</code>" back to "<code>HONMON</code>".
Optionally, delete "<code>CATALOGS.ebz</code>" considering that is now redundant.

==== Decompress other <code>.ebz</code> that were not listed ====
Same as above, but instead of copying, you can just move it, as,
* <code>ebzip -u</code> will only look for "<code>CATALOGS</code>" or "<code>CATALOGS.ebz</code>", and
* It saves having to manually delete the redundant file(s) afterwards.

=== Zero-length (un)compressed files not overwritten upon manual intervention ===
When a path that has zero-length files, but are listed in either "<code>CATALOGS</code>" or "<code>CATALOGS.ebz</code>", <code>ebzip -u</code> for instance may not overwrite it. This may also apply if <code>ebzip -uf</code> is used. To fix this, just delete the zero-length files, ensuring that they have a duplicate file that does not have <code>.ebz</code> extension, then re-run the command. When compressing, the opposite is true, to look for duplicate zero-length files with <code>.ebz</code> extension, delete them and run the utility to compress instead.

On modern [[Linux]] environments, these could be achieved via the following:

==== Enumerating ====
find . -size 0 -type f

==== Deleting ====
find . -size 0 -type f -exec sudo rm -iv "{}" \;
Care must be taken to avoid any potential losses when running this command.

[[User:Anonymoususer852|Anonymoususer852]] ([[User talk:Anonymoususer852|talk]]) 10:10, 18 August 2025 (UTC)

== LZMA benchmarks against other compression/utilities ==
In response to maintainer of <code>ebu</code>'s blog entry,<ref>[http://ikazuhiro.s206.xrea.com/article.php/20190701203038572 EPWINGを圧縮する (libdeflateで) / Compressing EPWING via libdeflate - Kazuhiro's blog]</ref> I decided to run a couple of benchmarks with different dictionaries, books or other referencing materials on my end. The following packages and their versions are as follows; kernel-6.15.8, zlib-1.3.1, libdeflate-1.24, xz-5.4.2, dwarfs-0.12.4, squashfs-tools-4.7.1.

=== zlib versus libdeflate versus LZMA benchmark (single electronic book) ===
For the following, file sizes are reported from <code>ls -l</code>, therefore size would be reported in blocks<ref>[https://unix.stackexchange.com/questions/28780/file-block-size-difference-between-stat-and-ls file block size - difference between stat and ls - Unix & Linux - Stack Exchange]</ref>. The tested dictionaries using no compression, zlib, libdeflate, or [[XZ]] are [[tar|tarballed]] for consistency. For compressed file systems, XZ's LZMA2 is used. Both zlib and libdeflate has a maximum compression level of 5, anything beyond 5 would result in error, however XZ goes up to level 9, and has further options like <code>--extreme</code> as well as various other tweaks which I will keep it simple and use <code>xz -9evv</code> instead.

[[DWARFS]] example parameters:
mkdwarfs --input . --output=../JMdict_eng_2021-02-26_UTC.dwarfs --block-size-bits=26 --compression=lzma:level=9:extreme --schema-compression=lzma:level=9:extreme --metadata-compression=lzma:level=9:extreme --no-history \
--pack-metadata=all,force --file-hash=sha3-512 --no-history-timestamp --no-create-timestamp --no-history-command-line

[[SquashFS]] example parameters:
mksquashfs . ../JMdict_eng_2021-02-26_UTC.squashfs -b 1048576 -comp xz -Xdict-size 100% -progress -info -no-xattrs

Percentages are rounded to two decimal points, otherwise rounded up. Compression ratio against uncompressed in percentage is offered instead.

{| border="1" align="center" style="text-align:center;" cellpadding="6"
|
| [[EBZip#Sample_file|kanjidic]]
| NHKAccent
| Eijiro
| Kenkyusha eiwa daijiten v6
| Kenkyusha waei daijiten v5
|-
| ''No compression''
| 130,385,920
| 1,578,055,680
| 1,374,771,200
| 484,945,920
| 328,192,000
|-
| <code>ebzip -z -l 5</code> (zlib)
| 42,035,200
| 1,393,920,000
| 379,975,680
| 117,596,160
| 109,721,600
|-
| ''Compression ratio against uncompressed in percentage''
| 32.24%
| 88.33%
| 27.63%
| 24.25%
| 33.43%
|-
| <code>ebzip -z -l 5</code> (libdeflate)
| 40,171,520
| 1,383,505,920
| 353,269,760
| 107,765,760
| 102,113,280
|-
| ''Compression ratio against uncompressed in percentage''
| 30.81%
| 87.67%
| 25.70%
| 22.22%
| 31.11%
|-
| <code>xz -9evv -T1</code> (liblzma)
| 21,607,608
| 1,350,900,536
| 197,134,768
| 55,220,788
| 50,109,852
|-
| ''Compression ratio against uncompressed in percentage''
| 16.57%
| 85.61%
| 14.34%
| 11.39%
| 15.27%
|-
| <code>xz -9evv -T0</code> (liblzma)
| 21,607,616
| 1,351,560,168
| 202,934,760
| 56,670,664
| 50,747,640
|-
| ''Compression ratio against uncompressed in percentage''
| 16.57%
| 85.65%
| 14.76%
| 11.69%
| 15.46%
|-
| <code>mk[[DWARFS|dwarfs]]</code> (liblzma)
| 22,145,746
| 1,353,699,364
| 212,307,176
| 60,043,728
| 54,541,980
|-
| ''Compression ratio against uncompressed in percentage''
| 16.99%
| 85.78%
| 15.44%
| 12.38%
| 16.62%
|-
| <code>mk[[SquashFS|squashfs]]</code> (liblzma)
| 31,793,152
| 1,367,302,144
| 272,068,608
| 81,805,312
| 160,956,416
|-
| ''Compression ratio against uncompressed in percentage''
| 24.38%
| 86.64%
| 19.79%
| 16.87%
| 49.04%
|}

XZ compression results are provided for the sake of discussion only, more on this in conclusion. While it does show extremely good results against all other utilities, the seek times are abysmally poor, worse than DWARFS with LZMA2 on machines with weaker hardware. Tests confirms liblzma having better compression regardless, against both zlib and libdeflate.

NHKAccent is the exception here, with compression results indicating poor results. This is due to electronic book containing multimedia files, which won't compress well if it was already compressed.

It is worthy to note here that compression/archiving utilities embracing liblzma, could by default take advantage of multi-threading, at the expense of producing potentially larger file. This however, is a small trade off, for effectively making full use of available CPU in the given environment.

=== Modern compressed file system with multiple electronic books shootout ===
Author of <code>ebu</code> mentioned the case with smartphones, which I do personally find that these are also limited in performance, not just with storage capacities. [[SquashFS]] was or is used in Android devices,<ref>[https://android.stackexchange.com/questions/258236/squashfs-explanation-for-android-system SquashFS explanation for Android system - Android Enthusiasts - Stack Exchange]</ref> so it makes sense to try using that on non system files in addition.

For these tests, the sample data consists of ~189 dictionaries in a single directory. Both [[DWARFS]] and [[SquashFS]] are configured to use XZ's LZMA2 where possible, as it is known to be one of the best, freely available (especially under Unix-like platforms) compression. Both DWARFS and SquashFS parameters are the same as in prior section. Due to the inherent way of how <code>ebuzip</code> works, requiring "<code>CATALOGS</code>" to be in the top directory, both variations of <code>ebuzip</code> using either zlib or libdeflate are omitted from this test. Besides, single compression job per dictionary, not including other files in the that path, e.g. "<code>START</code>" would make the overall process time consuming, if the process isn't already tedious that it requires extensive scripting.

{| border="1" align="center" style="text-align:center;" cellpadding="6"
| Method
| Size (blocks)
|-
| Uncompressed [[tar]]
| 29,014,026,240
|-
| [[DWARFS]]
| 10,613,886,184
|-
| [[SquashFS]]
| 12,269,518,848
|}

Where DWARFS shines, is the ability to also save on segmenting data, in addition to file de-duplication. For a small penalty I've given to DWARFS, in that it is to use sha3-512 as hash, it still compresses the collection better and faster, than SquashFS. Then again, the focus here is on overall file size, not performance in file creation.

=== Conclusion ===
Now of course, there are plenty of other compression utilities available, such as, but are not limited to [[Zstandard]], [[BZip3]], [[LZO]], etc. From the various results I have read, indicates that these other compression benchmarks, while trading better performance, these fall behind in final size constraints. The benchmark showcase here outlines the need to have as small of a compressed file as possible, with modern compression utilities and modern PC hardware, therefore making (speed-wise) performance testing out of the scope, as is the idea of creating the likes of [[ZIP]] bombs, or nested compressed files out of the scope.

I have also tried to run <code>mkfs.[[EROFS|erofs]]</code> in order to include it against the modern compressed file system with multiple dictionaries shootout. With version 1.8.10, no matter what parameters I set with LZMA <code>level=9</code> or <code>level=109</code> (109 is the extreme variant), even with lowering <code>dictsize=</code>, the results would cause system to run out of memory (16GB of RAM) before even completing the file system creation.

These benchmarks are in no way discrediting or disavowing the use of libdeflate over zlib. If file size is a concern, more modern compression algorithms and/or compression techniques could be considered. Modern EPWING viewer programs generally don't care if the electronic book is compressed using zlib or not compressed at all, and there are many ways of handling decompression transparently in modern Unix-like environments. Even more so, are the ever growing hardware performance as well as affordability with storage capacities in modern computing, questions the use of the what is widely considered as legacy compression algorithms for data archival. Last but not least, while single-threaded compression (as witnessed in the likes of <code>xz -9evv -T1</code> above) or tasks might be extremely beneficial in reducing overheads, especially with LZMA2, it is also a slow process which may subsequently not translate well in modern, demanding contexts, where time honored traditions are casted away for practical, convenient, modernized, and easy to use solutions.

<references />
[[User:Anonymoususer852|Anonymoususer852]] ([[User talk:Anonymoususer852|talk]]) 06:34, 20 August 2025 (UTC)

Talk:EBZip

2025-08-20T07:38:08Z

Anonymoususer852: /* zlib versus libdeflate versus LZMA benchmark (single electronic book) */ Split mkdwarfs' long command line syntax for legibility sake.

== <code>ebzip</code> caveats ==
Intended to be a place to note down issues, as well as tips and tricks with <code>eb-4.4.3</code>.

=== Hardcoded to look specifically for a file that begins with <code>"CATALOGS"</code> (case-insensitive), or <code>"CATALOGS.ebz"</code> ===
Custom dictionaries doesn't always follow standard convention to store their content(s), like having "<code>START.ebz</code>" in the top level. Even if they do, they may also compress "<code>CATALOGS</code>" as "<code>CATALOGS.ebz</code>." <code>eb-4.4.3</code> specifically looks for either "<code>CATALOGS</code>" or "<code>CATALOGS.ebz</code>", and either decompress or compress the files specified within, per user's request, e.g. <code>ebzip -u</code> will tell it to decompress and rename any of the listed files that are noted in either "<code>CATALOGS</code>" or "<code>CATALOGS.ebz</code>", and has "<code>.ebz</code>" extension, ignoring any other files in the same path, and doesn't decompress "<code>CATALOGS.ebz</code>" even if instructed. There are ways to workaround that:

==== Decompress "<code>CATALOGS.ebz</code>" ====
In my case,
* Simply rename the decompressed "<code>HONMON</code>" to something like "<code>HONMON.old</code>",
* Copy the "<code>CATALOGS.ebz</code>" there, but have it named as "<code>HONMON.ebz</code>",
* Run decompression via the usual <code>ebzip -u</code> from the top level where it doesn't contain "<code>HONMON.ebz</code>" but only "<code>CATALOGS.ebz</code>",
* Once decompressed, move decompressed "<code>HONMON</code>" (not "<code>HONMON.old</code>") back to where "<code>CATALOGS.ebz</code>" was, have it renamed as "<code>CATALOGS</code>",
* Rename "<code>HONMON.old</code>" back to "<code>HONMON</code>".
Optionally, delete "<code>CATALOGS.ebz</code>" considering that is now redundant.

==== Decompress other <code>.ebz</code> that were not listed ====
Same as above, but instead of copying, you can just move it, as,
* <code>ebzip -u</code> will only look for "<code>CATALOGS</code>" or "<code>CATALOGS.ebz</code>", and
* It saves having to manually delete the redundant file(s) afterwards.

=== Zero-length (un)compressed files not overwritten upon manual intervention ===
When a path that has zero-length files, but are listed in either "<code>CATALOGS</code>" or "<code>CATALOGS.ebz</code>", <code>ebzip -u</code> for instance may not overwrite it. This may also apply if <code>ebzip -uf</code> is used. To fix this, just delete the zero-length files, ensuring that they have a duplicate file that does not have <code>.ebz</code> extension, then re-run the command. When compressing, the opposite is true, to look for duplicate zero-length files with <code>.ebz</code> extension, delete them and run the utility to compress instead.

On modern [[Linux]] environments, these could be achieved via the following:

==== Enumerating ====
find . -size 0 -type f

==== Deleting ====
find . -size 0 -type f -exec sudo rm -iv "{}" \;
Care must be taken to avoid any potential losses when running this command.

[[User:Anonymoususer852|Anonymoususer852]] ([[User talk:Anonymoususer852|talk]]) 10:10, 18 August 2025 (UTC)

== LZMA benchmarks against other compression/utilities ==
In response to maintainer of <code>ebu</code>'s blog entry,<ref>[http://ikazuhiro.s206.xrea.com/article.php/20190701203038572 EPWINGを圧縮する (libdeflateで) / Compressing EPWING via libdeflate - Kazuhiro's blog]</ref> I decided to run a couple of benchmarks with different dictionaries, books or other referencing materials on my end. The following packages and their versions are as follows; kernel-6.15.8, zlib-1.3.1, libdeflate-1.24, xz-5.4.2, dwarfs-0.12.4, squashfs-tools-4.7.1.

=== zlib versus libdeflate versus LZMA benchmark (single electronic book) ===
For the following, file sizes are reported from <code>ls -l</code>, therefore size would be reported in blocks<ref>[https://unix.stackexchange.com/questions/28780/file-block-size-difference-between-stat-and-ls file block size - difference between stat and ls - Unix & Linux - Stack Exchange]</ref>. The tested dictionaries using no compression, zlib, libdeflate, or [[XZ]] are [[tar|tarballed]] for consistency. For compressed file systems, XZ's LZMA2 is used. Both zlib and libdeflate has a maximum compression level of 5, anything beyond 5 would result in error, however XZ goes up to level 9, and has further options like <code>--extreme</code> as well as various other tweaks which I will keep it simple and use <code>xz -9evv</code> instead.

[[DWARFS]] example parameters:
mkdwarfs --input . --output=../JMdict_eng_2021-02-26_UTC.dwarfs --block-size-bits=26 --compression=lzma:level=9:extreme --schema-compression=lzma:level=9:extreme --metadata-compression=lzma:level=9:extreme --no-history \
--pack-metadata=all,force --file-hash=sha3-512 --no-history-timestamp --no-create-timestamp --no-history-command-line

[[SquashFS]] example parameters:
mksquashfs . ../JMdict_eng_2021-02-26_UTC.squashfs -b 1048576 -comp xz -Xdict-size 100% -progress -info -no-xattrs

Percentages are rounded to two decimal points, otherwise rounded up. Compression ratio against uncompressed in percentage is offered instead.

{| border="1" align="center" style="text-align:center;" cellpadding="6"
|
| [[EBZip#Sample_file|kanjidic]]
| NHKAccent
| Eijiro
| Kenkyusha eiwa daijiten v6
| Kenkyusha waei daijiten v5
|-
| ''No compression''
| 130,385,920
| 1,578,055,680
| 1,374,771,200
| 484,945,920
| 328,192,000
|-
| <code>ebzip -z -l 5</code> (zlib)
| 42,035,200
| 1,393,920,000
| 379,975,680
| 117,596,160
| 109,721,600
|-
| ''Compression ratio against uncompressed in percentage''
| 32.24%
| 88.33%
| 27.63%
| 24.25%
| 33.43%
|-
| <code>ebzip -z -l 5</code> (libdeflate)
| 40,171,520
| 1,383,505,920
| 353,269,760
| 107,765,760
| 102,113,280
|-
| ''Compression ratio against uncompressed in percentage''
| 30.81%
| 87.67%
| 25.70%
| 22.22%
| 31.11%
|-
| <code>xz -9evv -T1</code> (liblzma)
| 21,607,608
| 1,350,900,536
| 197,134,768
| 55,220,788
| 50,109,852
|-
| ''Compression ratio against uncompressed in percentage''
| 16.57%
| 85.61%
| 14.34%
| 11.39%
| 15.27%
|-
| <code>xz -9evv -T0</code> (liblzma)
| 21,607,616
| 1,351,560,168
| 202,934,760
| 56,670,664
| 50,747,640
|-
| ''Compression ratio against uncompressed in percentage''
| 16.57%
| 85.65%
| 14.76%
| 11.69%
| 15.46%
|-
| <code>mk[[DWARFS|dwarfs]]</code> (liblzma)
| 22,145,746
| 1,353,699,364
| 212,307,176
| 60,043,728
| 54,541,980
|-
| ''Compression ratio against uncompressed in percentage''
| 16.99%
| 85.78%
| 15.44%
| 12.38%
| 16.62%
|-
| <code>mk[[SquashFS|squashfs]]</code> (liblzma)
| 31,793,152
| 1,367,302,144
| 272,068,608
| 81,805,312
| 160,956,416
|-
| ''Compression ratio against uncompressed in percentage''
| 24.38%
| 86.64%
| 19.79%
| 16.87%
| 49.04%
|}

XZ compression results are provided for the sake of discussion only, more on this in conclusion. While it does show extremely good results against all other utilities, the seek times are abysmally poor, worse than DWARFS with LZMA2 on machines with weaker hardware. Tests confirms liblzma having better compression regardless, against both zlib and libdeflate.

NHKAccent is the exception here, with compression results indicating poor results. This is due to electronic book containing multimedia files, which won't compress well if it was already compressed.

It is worthy to note here that compression/archiving utilities embracing liblzma, could be configured to take advantage of multi-threading by default, at the expense of producing potentially larger file. This however, is a small trade off, for effectively making full use of available CPU in the given environment.

=== Modern compressed file system with multiple electronic books shootout ===
Author of <code>ebu</code> mentioned the case with smartphones, which I do personally find that these are also limited in performance, not just with storage capacities. [[SquashFS]] was or is used in Android devices,<ref>[https://android.stackexchange.com/questions/258236/squashfs-explanation-for-android-system SquashFS explanation for Android system - Android Enthusiasts - Stack Exchange]</ref> so it makes sense to try using that on non system files in addition.

For these tests, the sample data consists of ~189 dictionaries in a single directory. Both [[DWARFS]] and [[SquashFS]] are configured to use XZ's LZMA2 where possible, as it is known to be one of the best, freely available (especially under Unix-like platforms) compression. Both DWARFS and SquashFS parameters are the same as in prior section. Due to the inherent way of how <code>ebuzip</code> works, requiring "<code>CATALOGS</code>" to be in the top directory, both variations of <code>ebuzip</code> using either zlib or libdeflate are omitted from this test. Besides, single compression job per dictionary, not including other files in the that path, e.g. "<code>START</code>" would make the overall process time consuming, if the process isn't already tedious that it requires extensive scripting.

{| border="1" align="center" style="text-align:center;" cellpadding="6"
| Method
| Size (blocks)
|-
| Uncompressed [[tar]]
| 29,014,026,240
|-
| [[DWARFS]]
| 10,613,886,184
|-
| [[SquashFS]]
| 12,269,518,848
|}

Where DWARFS shines, is the ability to also save on segmenting data, in addition to file de-duplication. For a small penalty I've given to DWARFS, in that it is to use sha3-512 as hash, it still compresses the collection better and faster, than SquashFS. Then again, the focus here is on overall file size, not performance in file creation.

=== Conclusion ===
Now of course, there are plenty of other compression utilities available, such as, but are not limited to [[Zstandard]], [[BZip3]], [[LZO]], etc. From the various results I have read, indicates that these other compression benchmarks, while trading better performance, these fall behind in final size constraints. The benchmark showcase here outlines the need to have as small of a compressed file as possible, with modern compression utilities and modern PC hardware, therefore making (speed-wise) performance testing out of the scope, as is the idea of creating the likes of [[ZIP]] bombs, or nested compressed files out of the scope.

I have also tried to run <code>mkfs.[[EROFS|erofs]]</code> in order to include it against the modern compressed file system with multiple dictionaries shootout. With version 1.8.10, no matter what parameters I set with LZMA <code>level=9</code> or <code>level=109</code> (109 is the extreme variant), even with lowering <code>dictsize=</code>, the results would cause system to run out of memory (16GB of RAM) before even completing the file system creation.

These benchmarks are in no way discrediting or disavowing the use of libdeflate over zlib. If file size is a concern, more modern compression algorithms and/or compression techniques could be considered. Modern EPWING viewer programs generally don't care if the electronic book is compressed using zlib or not compressed at all, and there are many ways of handling decompression transparently in modern Unix-like environments. Even more so, are the ever growing hardware performance as well as affordability with storage capacities in modern computing, questions the use of the what is widely considered as legacy compression algorithms for data archival. Last but not least, while single-threaded compression (as witnessed in the likes of <code>xz -9evv -T1</code> above) or tasks might be extremely beneficial in reducing overheads, especially with LZMA2, it is also a slow process which may subsequently not translate well in modern, demanding contexts, where time honored traditions are casted away for practical, convenient, modernized, and easy to use solutions.

<references />
[[User:Anonymoususer852|Anonymoususer852]] ([[User talk:Anonymoususer852|talk]]) 06:34, 20 August 2025 (UTC)

Linux

2025-08-20T07:29:09Z

Anonymoususer852: Linked David A. Wheeler's article to Linux presence in server space. The same article doesn't focus on server environments, but does make good references in where Linux is widely used in that same space. Updated Linux as utility section, with ref.

{|
|[[Software]]
| >
|[[Operating Systems]]
| >
|[[Linux]]
|}

[https://www.linux.com/ '''Linux'''] is an Open Source operating system (OS) closely related to [http://www.unix.org/ '''Unix'''], and is often referred to as "Unix-like". Technically, Linux is the kernel at the core of an operating system; the kernel alone isn't of much use by itself without the various utility programs that are typically "bundled" with the kernel, many of which came from the GNU project. The bundling of Linux kernel with typically GNU utility programs became what is better known as '''distributions''' or [http://distrowatch.com/ distros] for short.

Certain users calls it as '''GNU/Linux''' as the proper term for the operating system as a whole, major "religious wars" are fought over the correct naming schema.<ref>[[wikipedia:GNU/Linux_naming_controversy|GNU/Linux naming controversy on Wikipedia]]</ref><ref>[[wikipedia:GNU_Hurd#Development_history|Development history section - GNU Hurd - Wikipedia]]</ref> As are exceptions to that rule by using the likes of [[wikipedia:BusyBox|BusyBox]] instead GNU software components.

Owing to license being GPLv2, users are not held hostage over the Linux source code. As such, Linux is also noted for being supported by many free software projects as a solution, that replaces their dependency on proprietary counterparts which are often used in many other operating systems. There are free and open source word processors for example.

== History ==

Linux is created by [[wikipedia:Linus Torvalds|Linus Torvalds]] initially as a hobby<ref>[[wikipedia:https://en.wikipedia.org/wiki/Linus_Torvalds#cite_note-22|Cite note-22 - Linus Torvalds on Wikipedia]]</ref>, that used software utilities from [https://www.gnu.org/ '''GNU (GNU not Unix) Project''']. GNU created sets of Unix-like software utilities, but lacked a fully functional kernel, the "heart of the OS", which was needed to make a fully functional OS. Linus Torvalds combined his kernel with the GNU software and Linux was born.<ref>[https://www.cs.cmu.edu/~awb/linux.history.html LINUX's History by Linus Torvalds - Carnegie Mellon University of Computer Science]</ref><ref>[https://web.archive.org/web/20030818084123/http://www-2.cs.cmu.edu/~awb/linux.history.html LINUX's History by Linus Torvalds - Carnegie Mellon University of Computer Science - Internet Archive mirror]</ref>

== Who uses Linux? ==

Linux is a very small fraction (about 2%) of desktop computers worldwide, however,
* It is a leader in cellphones and tablets, it being the core of the Android, which is the most used OS in the world overall. The [[Android]] operating system (which uses non-GNU utility programs) and the related [[iOS|Apple iOS]], a BSD derivative, are the two major players in these markets.
* In the server market, websites specifically are dominated by Linux among Unix(-like) installations with about 60% of that market. Many server operators transitioned away from proprietary, but also generally costly solutions to Linux.<ref>[https://dwheeler.com/oss_fs_why.html Why Open Source Software / Free Software (OSS/FS, FLOSS, or FOSS)? Look at the Numbers! - David A. Wheeler]</ref>
* It sees users in cloud computing environments, for example Azure Linux,<ref>[[wikipedia:Azure_Linux|Azure Linux on Wikipedia]]</ref>
* It exists in networking devices (e.g. [https://openwrt.org/ OpenWRT]) along with other embedded devices like Single Board Computers (SBC) for example, [https://www.raspberrypi.com/software/operating-systems/ Rasbian for Raspberry Pi].
* It is also among the top 500 supercomputers worldwide.<ref>[https://www.top500.org/lists/top500/2025/06/ June 2025 - Top500]</ref>

=== Linux as utility ===

Many technicians supporting '''[[Microsoft Windows]]''' operating systems use a bootable live environment, for instance [http://livecdlist.com/ '''Linux LiveCD/DVD'''] to diagnose storage mediums, perform repairs, and retrieve data that may be lost, should their repair efforts require them to format and re-install Windows. In addition, there are tailored distros used by digital [[Forensics and Law Enforcement|forensics]] to analyze data. These are used by experienced computer security people, for instance to examine a suspect's computer for alleged computer crimes they have been accused of in court scenarios.<ref>[https://www.sans.org/cybersecurity-focus-areas/digital-forensics-incident-response Digital Forensics and Incident Response Training - SANS.org]</ref>

Linux is also used to diagnose variants of its own<ref>[https://android.stackexchange.com/questions/258236/squashfs-explanation-for-android-system SquashFS explanation for Android system - Android Enthusiasts - Stack Exchange]</ref>, where there is a somewhat growing community of hardware hackers interested in the likes of reverse engineering a device, and that the tools available within Linux makes it an easier choice overall.<ref>[https://embeddedbits.org/reverse-engineering-my-routers-firmware-with-binwalk-embeddedbits/ Reverse engineering my router's firmware with binwalk - embedditbits.org]</ref>

=== Linux for the technically inclined ===

"Power users" or tinkerers are generally experienced users who knows their way around OS may find Windows restrictive.<ref>[https://www.linkedin.com/pulse/why-windows-11-losing-trust-power-users-what-means-techs-dening-2ofrc Why Windows 11 Is Losing the Trust of Power Users - And What That Means for Tech’s Future - LinkedIn]</ref> Linux as a whole does not restrict users ability to perform certain dangerous tasks that could otherwise damage their installation. While it is possible to lock down users in Linux, and have it available as a Linux distro for convenience. This unfettered access across the OS could be seen as a benefit for those that do not appreciate the one-suit-fits-all approach Microsoft employs. If certain features are unavailable in Windows, it might also not be a feature that could be unlocked for free, contrary to power tools under Linux which are mostly available for free. [[wikipedia:WindowBlinds|WindowBlinds]] is an old example of this, in Linux, many modern popular graphical environments already offers (customizable) desktop/window transparency for free.

=== Linux for the privacy conscious individuals ===

There is a growing trend among Linux converts, typically from Windows, who are concerned with the likes of [[wikipedia:Telemetry#Software|software telemetry]] being added to Windows 11<ref>[https://www.extremetech.com/computing/342941-windows-11-collects-an-awful-lot-of-telemetry-about-your-pc Windows 11 Collects an Awful Lot of Telemetry About Your PC - Extreme Tech]</ref>, which constitutes as one of the many possible motives as to why Linux might be gaining some popularity among the heavily dominant Windows in desktop computer space. At the same time, telemetry may also exist in Linux, often creating heated discussions with those who value their privacy and software choices, over features in software that could otherwise potentially help software developers.<ref>[https://github.com/audacity/audacity/discussions/889 Actions we propose to take on PR #835 #889 - Audacity discussions - GitHub]</ref><ref>[https://discussion.fedoraproject.org/t/f40-change-request-privacy-preserving-telemetry-for-fedora-workstation-system-wide/85320 F40 Change Request: Privacy-preserving Telemetry for Fedora Workstation (System-Wide) - Fedora Discussions]</ref>

== Pros and Cons of running Linux ==

'''PROS:'''
* Linux distros, due to its nature, offers a wide range of choices to better suit individuals and their needs.
* Being a very small part of the overall market share for desktop computers, there are few Linux malware to contend with, the malware writers concentrate on the more dominant Windows platform.
* Linux installs are generally much smaller than Windows installs and can run on much older hardware. Many old Windows XP machines that cannot handle Windows 7 and beyond do well with a Linux install.
* Linux is generally a free OS, there is no direct cost to the user. Paid Linux distributions exists for those who desire specifics like priority support, especially if using in commercial contexts.
* It is a very stable OS.

'''CONS:'''
* While many modern distros are user-friendly out of the box, there is a significant learning curve with Linux. Many basic operations work differently and perhaps are a bit more complex on Linux. Broadly speaking, interested individuals are encouraged to have an open mind.
* [http://www.winehq.org/ '''[[Wine|WINE]]'''] for instance allows certain Windows software to run in Wine, but it doesn't always perform as well as it would in a native Windows environment, conversely sometimes they work better than a native Windows environment. Virtualization by means of using [[Virtualization Software]] may also be an option for running certain Windows software inside Linux, at the cost of increasing complexity as well as potentially having performance penalty, especially on older hardware. Gaming on Linux is somewhat possible via the likes of Steam.<ref>[https://fedoramagazine.org/gaming-on-fedora/ Gaming on Fedora - Fedora Magazine]</ref>
* Vendors (software or hardware) may simply choose to not support Linux on whatever befitting reason they have on the matter. Users are better off exercising their own discretion. When these are offered for "free", these may often not include free technical support, contrary to how Linux is regarded a "free OS".

== Links ==
* [https://www.kernel.org/ The Linux Kernel Archives]
* [http://archive.org/details/linux_distributions Archive of historical Linux distributions]
* [http://arstechnica.com/information-technology/2013/10/the-navys-newest-warship-is-powered-by-linux/ The Navy’s newest warship is powered by Linux]
* [[wikipedia:PlayStation_3#Use_in_supercomputing|Sony PlayStation 3 used in supercomputing - Wikipedia]]
* [http://www.extremetech.com/extreme/124287-the-worlds-slowest-linux-pc The world’s slowest Linux PC]
* [https://dmitry.gr/index.php?r=05.Projects&proj=07.%20Linux%20on%208bit Linux on an 8-bit micro]
* [https://linux-hardware.org/ linux-hardware.org - a crowd-sourced hardware compatibility database]

== References ==
<references/>

[[Category:Operating Systems]]
[[Category:Software]]

Squashfs

2025-08-20T06:55:30Z

Anonymoususer852: Adjusted layout in the top area for better legibility.

{{FormatInfo
|formattype=electronic
|subcat=Filesystem
|extensions={{ext|sfs}}<ref>[https://github.com/plougher/squashfs-tools/issues/259 What are the commonly accepted file name extension for squashfs? #259 - GitHub]</ref>
|wikidata={{wikidata|Q389314}}
|compression=lossless, optional
|magic=[https://github.com/file/file/blob/master/magic/Magdir/filesystems#L2194 68 73 71 73] (little-endian) [https://github.com/file/file/blob/master/magic/Magdir/filesystems#L2191 73 71 73 68] (big-endian)
|developed by=Phillip Lougher<ref>[https://sourceforge.net/p/squashfs/mailman/message/67563/ (Squashfs-announce) Squashfs1.2 released - 2003-03-13 05:30:45 - SourceForge mailman]</ref>
|maintained by=Phillip Lougher
|released=2002
}}
'''Squashfs''' is read-only filesystem that features optional compression<ref>[https://github.com/plougher/squashfs-tools/issues/206 <nowiki>[</nowiki>feature request<nowiki>]</nowiki>: mksquashfs: Add --no-compression flag to disable all compression #206 - squashfs-tools - GitHub]</ref> for Linux. Typically used as a "live image" for booting Linux off a removable storage (e.g. USB, CD/DVD), minus components like the bootloader, kernel and initramfs.<ref>[https://fedoraproject.org/wiki/Changes/EROFSforLiveMedia EROFS For Live Media - Fedora Project]</ref> It is also used in:
* Embedded devices like network routers<ref>[https://openwrt.org/docs/techref/flash.layout The OpenWrt Flash Layout - Technical Reference - openwrt.org]</ref><ref>[https://github.com/ReFirmLabs/binwalk/issues/301 problems extracting squashfs-image #301 - binwalk - GitHub]</ref><ref>[https://embeddedbits.org/reverse-engineering-my-routers-firmware-with-binwalk-embeddedbits/ Reverse engineering my router's firmware with binwalk - embeddedbits.org]</ref>, or in Android powered devices.<ref>[https://android.stackexchange.com/questions/258236/squashfs-explanation-for-android-system SquashFS explanation for Android system - Android Enthusiasts - Stack Exchange]</ref>
* Modern AppImage,<ref>[[wikipedia:AppImage#History|History of AppImage - Wikipedia]]</ref>, or [[Snap]]<ref>[[wikipedia:Snap_(software)#Support|Snap - support section - Wikipedia]]</ref> as a way portable software distribution on Linux.
* Storage of other data, such as disk images.<ref>[https://www.foo.be/2017/01/Squashfs_As_A_Forensic_Container Squashfs As a Forensic Container - foo.be]</ref>

Under [[Microsoft Windows|Windows]], SquashFS contents can be opened/extracted via [[7-Zip]].<ref>[https://stackoverflow.com/a/36877207 python2.7 How to handle squashfs in Windows - StackOverflow]</ref>

See also [[cramfs]].

== Identification ==
SquashFS files created on little-endian machines begins with hexadecimal {{magic|68 73 71 73}}, which translates to "hsqs" in ASCII. SquashFS files created on big-endian machines with hexadecimal {{magic|73 71 73 68}}, which translates to "sqsh" in ASCII.

== Examples ==
Create a Squashfs (file system) image from current/present directory and output it into its parent directory:
$ mksquashfs . ../example.squashfs -b 1048576 -comp xz -Xdict-size 100%

Open a Squashfs image (as root) from present directory and have its contents visible in <code>/mnt</code>:
# mount example.squashfs /mnt

Open a Squashfs image (as user) from present directory and have its contents visible in <code>/mnt/temp</code>:
$ squashfuse example.squashfs /mnt/temp

== Software ==
* [[7-Zip]]
* [https://github.com/vasi/squashfuse Squashfuse] (Unix-like only)

== Sample files ==
* {{DexvertSamples|archive/squashFS}}
* [https://discmaster.textfiles.com/search?mode=deep&extension=.sfs&format=squashFS Search results for .sfs extension and is squashFS format - Discmaster.textfiles.com]

== Links ==
* [[Wikipedia: SquashFS]]
* [https://squashfs.sourceforge.net/ Squashfs website]
* [https://sourceforge.net/projects/squashfs/ SourceForge project page]
* [https://github.com/plougher/squashfs-tools GitHub project page]
* [https://elixir.bootlin.com/linux/latest/source/Documentation/filesystems/squashfs.txt squashfs.txt]

== References ==
<references/>

Squashfs

2025-08-20T06:52:34Z

Anonymoususer852: Changed data compression to optional, and provide reference. Provide examples and references of where else SquashFS sees its deployment.

{{FormatInfo
|formattype=electronic
|subcat=Filesystem
|extensions={{ext|sfs}}<ref>[https://github.com/plougher/squashfs-tools/issues/259 What are the commonly accepted file name extension for squashfs? #259 - GitHub]</ref>
|wikidata={{wikidata|Q389314}}
|compression=lossless, optional
|magic=[https://github.com/file/file/blob/master/magic/Magdir/filesystems#L2194 68 73 71 73] (little-endian) [https://github.com/file/file/blob/master/magic/Magdir/filesystems#L2191 73 71 73 68] (big-endian)
|developed by=Phillip Lougher<ref>[https://sourceforge.net/p/squashfs/mailman/message/67563/ (Squashfs-announce) Squashfs1.2 released - 2003-03-13 05:30:45 - SourceForge mailman]</ref>
|maintained by=Phillip Lougher
|released=2002
}}
'''Squashfs''' is read-only filesystem that features optional compression<ref>[https://github.com/plougher/squashfs-tools/issues/206 <nowiki>[</nowiki>feature request<nowiki>]</nowiki>: mksquashfs: Add --no-compression flag to disable all compression #206 - squashfs-tools - GitHub]</ref> for Linux. Typically used as a "live image" for booting Linux off a removable storage (e.g. USB, CD/DVD), minus components like the bootloader, kernel and initramfs.<ref>[https://fedoraproject.org/wiki/Changes/EROFSforLiveMedia EROFS For Live Media - Fedora Project]</ref> It is also used in:
* Embedded devices like network routers<ref>[https://openwrt.org/docs/techref/flash.layout The OpenWrt Flash Layout - Technical Reference - openwrt.org]</ref><ref>[https://github.com/ReFirmLabs/binwalk/issues/301 problems extracting squashfs-image #301 - binwalk - GitHub]</ref><ref>[https://embeddedbits.org/reverse-engineering-my-routers-firmware-with-binwalk-embeddedbits/ Reverse engineering my router's firmware with binwalk - embeddedbits.org]</ref>, or in Android powered devices.<ref>[https://android.stackexchange.com/questions/258236/squashfs-explanation-for-android-system SquashFS explanation for Android system - Android Enthusiasts - Stack Exchange]</ref>
* Modern AppImage,<ref>[[wikipedia:AppImage#History|History of AppImage - Wikipedia]]</ref>, or [[Snap]]<ref>[[wikipedia:Snap_(software)#Support|Snap - support section - Wikipedia]]</ref> as a way portable software distribution on Linux.
* Storage of other data, such as disk images.<ref>[https://www.foo.be/2017/01/Squashfs_As_A_Forensic_Container Squashfs As a Forensic Container - foo.be]</ref> Under Windows, Squashfs contents can be opened/extracted via [[7-Zip]].<ref>[https://stackoverflow.com/a/36877207 python2.7 How to handle squashfs in Windows - StackOverflow]</ref>

See also [[cramfs]].

== Identification ==
SquashFS files created on little-endian machines begins with hexadecimal {{magic|68 73 71 73}}, which translates to "hsqs" in ASCII. SquashFS files created on big-endian machines with hexadecimal {{magic|73 71 73 68}}, which translates to "sqsh" in ASCII.

== Examples ==
Create a Squashfs (file system) image from current/present directory and output it into its parent directory:
$ mksquashfs . ../example.squashfs -b 1048576 -comp xz -Xdict-size 100%

Open a Squashfs image (as root) from present directory and have its contents visible in <code>/mnt</code>:
# mount example.squashfs /mnt

Open a Squashfs image (as user) from present directory and have its contents visible in <code>/mnt/temp</code>:
$ squashfuse example.squashfs /mnt/temp

== Software ==
* [[7-Zip]]
* [https://github.com/vasi/squashfuse Squashfuse] (Unix-like only)

== Sample files ==
* {{DexvertSamples|archive/squashFS}}
* [https://discmaster.textfiles.com/search?mode=deep&extension=.sfs&format=squashFS Search results for .sfs extension and is squashFS format - Discmaster.textfiles.com]

== Links ==
* [[Wikipedia: SquashFS]]
* [https://squashfs.sourceforge.net/ Squashfs website]
* [https://sourceforge.net/projects/squashfs/ SourceForge project page]
* [https://github.com/plougher/squashfs-tools GitHub project page]
* [https://elixir.bootlin.com/linux/latest/source/Documentation/filesystems/squashfs.txt squashfs.txt]

== References ==
<references/>

Talk:EBZip

2025-08-20T06:34:02Z

Anonymoususer852: /* LZMA benchmarks against other compression/utilities */ Added topic.

== <code>ebzip</code> caveats ==
Intended to be a place to note down issues, as well as tips and tricks with <code>eb-4.4.3</code>.

=== Hardcoded to look specifically for a file that begins with <code>"CATALOGS"</code> (case-insensitive), or <code>"CATALOGS.ebz"</code> ===
Custom dictionaries doesn't always follow standard convention to store their content(s), like having "<code>START.ebz</code>" in the top level. Even if they do, they may also compress "<code>CATALOGS</code>" as "<code>CATALOGS.ebz</code>." <code>eb-4.4.3</code> specifically looks for either "<code>CATALOGS</code>" or "<code>CATALOGS.ebz</code>", and either decompress or compress the files specified within, per user's request, e.g. <code>ebzip -u</code> will tell it to decompress and rename any of the listed files that are noted in either "<code>CATALOGS</code>" or "<code>CATALOGS.ebz</code>", and has "<code>.ebz</code>" extension, ignoring any other files in the same path, and doesn't decompress "<code>CATALOGS.ebz</code>" even if instructed. There are ways to workaround that:

==== Decompress "<code>CATALOGS.ebz</code>" ====
In my case,
* Simply rename the decompressed "<code>HONMON</code>" to something like "<code>HONMON.old</code>",
* Copy the "<code>CATALOGS.ebz</code>" there, but have it named as "<code>HONMON.ebz</code>",
* Run decompression via the usual <code>ebzip -u</code> from the top level where it doesn't contain "<code>HONMON.ebz</code>" but only "<code>CATALOGS.ebz</code>",
* Once decompressed, move decompressed "<code>HONMON</code>" (not "<code>HONMON.old</code>") back to where "<code>CATALOGS.ebz</code>" was, have it renamed as "<code>CATALOGS</code>",
* Rename "<code>HONMON.old</code>" back to "<code>HONMON</code>".
Optionally, delete "<code>CATALOGS.ebz</code>" considering that is now redundant.

==== Decompress other <code>.ebz</code> that were not listed ====
Same as above, but instead of copying, you can just move it, as,
* <code>ebzip -u</code> will only look for "<code>CATALOGS</code>" or "<code>CATALOGS.ebz</code>", and
* It saves having to manually delete the redundant file(s) afterwards.

=== Zero-length (un)compressed files not overwritten upon manual intervention ===
When a path that has zero-length files, but are listed in either "<code>CATALOGS</code>" or "<code>CATALOGS.ebz</code>", <code>ebzip -u</code> for instance may not overwrite it. This may also apply if <code>ebzip -uf</code> is used. To fix this, just delete the zero-length files, ensuring that they have a duplicate file that does not have <code>.ebz</code> extension, then re-run the command. When compressing, the opposite is true, to look for duplicate zero-length files with <code>.ebz</code> extension, delete them and run the utility to compress instead.

On modern [[Linux]] environments, these could be achieved via the following:

==== Enumerating ====
find . -size 0 -type f

==== Deleting ====
find . -size 0 -type f -exec sudo rm -iv "{}" \;
Care must be taken to avoid any potential losses when running this command.

[[User:Anonymoususer852|Anonymoususer852]] ([[User talk:Anonymoususer852|talk]]) 10:10, 18 August 2025 (UTC)

== LZMA benchmarks against other compression/utilities ==
In response to maintainer of <code>ebu</code>'s blog entry,<ref>[http://ikazuhiro.s206.xrea.com/article.php/20190701203038572 EPWINGを圧縮する (libdeflateで) / Compressing EPWING via libdeflate - Kazuhiro's blog]</ref> I decided to run a couple of benchmarks with different dictionaries, books or other referencing materials on my end. The following packages and their versions are as follows; kernel-6.15.8, zlib-1.3.1, libdeflate-1.24, xz-5.4.2, dwarfs-0.12.4, squashfs-tools-4.7.1.

=== zlib versus libdeflate versus LZMA benchmark (single electronic book) ===
For the following, file sizes are reported from <code>ls -l</code>, therefore size would be reported in blocks<ref>[https://unix.stackexchange.com/questions/28780/file-block-size-difference-between-stat-and-ls file block size - difference between stat and ls - Unix & Linux - Stack Exchange]</ref>. The tested dictionaries using no compression, zlib, libdeflate, or [[XZ]] are [[tar|tarballed]] for consistency. For compressed file systems, XZ's LZMA2 is used. Both zlib and libdeflate has a maximum compression level of 5, anything beyond 5 would result in error, however XZ goes up to level 9, and has further options like <code>--extreme</code> as well as various other tweaks which I will keep it simple and use <code>xz -9evv</code> instead.

[[DWARFS]] example parameters:
mkdwarfs --input . --output=../JMdict_eng_2021-02-26_UTC.dwarfs --block-size-bits=26 --compression=lzma:level=9:extreme --schema-compression=lzma:level=9:extreme --metadata-compression=lzma:level=9:extreme --no-history --pack-metadata=all,force --file-hash=sha3-512 --no-history-timestamp --no-create-timestamp --no-history-command-line

[[SquashFS]] example parameters:
mksquashfs . ../JMdict_eng_2021-02-26_UTC.squashfs -b 1048576 -comp xz -Xdict-size 100% -progress -info -no-xattrs

Percentages are rounded to two decimal points, otherwise rounded up. Compression ratio against uncompressed in percentage is offered instead.

{| border="1" align="center" style="text-align:center;" cellpadding="6"
|
| [[EBZip#Sample_file|kanjidic]]
| NHKAccent
| Eijiro
| Kenkyusha eiwa daijiten v6
| Kenkyusha waei daijiten v5
|-
| ''No compression''
| 130,385,920
| 1,578,055,680
| 1,374,771,200
| 484,945,920
| 328,192,000
|-
| <code>ebzip -z -l 5</code> (zlib)
| 42,035,200
| 1,393,920,000
| 379,975,680
| 117,596,160
| 109,721,600
|-
| ''Compression ratio against uncompressed in percentage''
| 32.24%
| 88.33%
| 27.63%
| 24.25%
| 33.43%
|-
| <code>ebzip -z -l 5</code> (libdeflate)
| 40,171,520
| 1,383,505,920
| 353,269,760
| 107,765,760
| 102,113,280
|-
| ''Compression ratio against uncompressed in percentage''
| 30.81%
| 87.67%
| 25.70%
| 22.22%
| 31.11%
|-
| <code>xz -9evv -T1</code> (liblzma)
| 21,607,608
| 1,350,900,536
| 197,134,768
| 55,220,788
| 50,109,852
|-
| ''Compression ratio against uncompressed in percentage''
| 16.57%
| 85.61%
| 14.34%
| 11.39%
| 15.27%
|-
| <code>xz -9evv -T0</code> (liblzma)
| 21,607,616
| 1,351,560,168
| 202,934,760
| 56,670,664
| 50,747,640
|-
| ''Compression ratio against uncompressed in percentage''
| 16.57%
| 85.65%
| 14.76%
| 11.69%
| 15.46%
|-
| <code>mk[[DWARFS|dwarfs]]</code> (liblzma)
| 22,145,746
| 1,353,699,364
| 212,307,176
| 60,043,728
| 54,541,980
|-
| ''Compression ratio against uncompressed in percentage''
| 16.99%
| 85.78%
| 15.44%
| 12.38%
| 16.62%
|-
| <code>mk[[SquashFS|squashfs]]</code> (liblzma)
| 31,793,152
| 1,367,302,144
| 272,068,608
| 81,805,312
| 160,956,416
|-
| ''Compression ratio against uncompressed in percentage''
| 24.38%
| 86.64%
| 19.79%
| 16.87%
| 49.04%
|}

XZ compression results are provided for the sake of discussion only, more on this in conclusion. While it does show extremely good results against all other utilities, the seek times are abysmally poor, worse than DWARFS with LZMA2 on machines with weaker hardware. Tests confirms liblzma having better compression regardless, against both zlib and libdeflate.

NHKAccent is the exception here, with compression results indicating poor results. This is due to electronic book containing multimedia files, which won't compress well if it was already compressed.

It is worthy to note here that compression/archiving utilities embracing liblzma, could be configured to take advantage of multi-threading by default, at the expense of producing potentially larger file. This however, is a small trade off, for effectively making full use of available CPU in the given environment.

=== Modern compressed file system with multiple electronic books shootout ===
Author of <code>ebu</code> mentioned the case with smartphones, which I do personally find that these are also limited in performance, not just with storage capacities. [[SquashFS]] was or is used in Android devices,<ref>[https://android.stackexchange.com/questions/258236/squashfs-explanation-for-android-system SquashFS explanation for Android system - Android Enthusiasts - Stack Exchange]</ref> so it makes sense to try using that on non system files in addition.

For these tests, the sample data consists of ~189 dictionaries in a single directory. Both [[DWARFS]] and [[SquashFS]] are configured to use XZ's LZMA2 where possible, as it is known to be one of the best, freely available (especially under Unix-like platforms) compression. Both DWARFS and SquashFS parameters are the same as in prior section. Due to the inherent way of how <code>ebuzip</code> works, requiring "<code>CATALOGS</code>" to be in the top directory, both variations of <code>ebuzip</code> using either zlib or libdeflate are omitted from this test. Besides, single compression job per dictionary, not including other files in the that path, e.g. "<code>START</code>" would make the overall process time consuming, if the process isn't already tedious that it requires extensive scripting.

{| border="1" align="center" style="text-align:center;" cellpadding="6"
| Method
| Size (blocks)
|-
| Uncompressed [[tar]]
| 29,014,026,240
|-
| [[DWARFS]]
| 10,613,886,184
|-
| [[SquashFS]]
| 12,269,518,848
|}

Where DWARFS shines, is the ability to also save on segmenting data, in addition to file de-duplication. For a small penalty I've given to DWARFS, in that it is to use sha3-512 as hash, it still compresses the collection better and faster, than SquashFS. Then again, the focus here is on overall file size, not performance in file creation.

=== Conclusion ===
Now of course, there are plenty of other compression utilities available, such as, but are not limited to [[Zstandard]], [[BZip3]], [[LZO]], etc. From the various results I have read, indicates that these other compression benchmarks, while trading better performance, these fall behind in final size constraints. The benchmark showcase here outlines the need to have as small of a compressed file as possible, with modern compression utilities and modern PC hardware, therefore making (speed-wise) performance testing out of the scope, as is the idea of creating the likes of [[ZIP]] bombs, or nested compressed files out of the scope.

I have also tried to run <code>mkfs.[[EROFS|erofs]]</code> in order to include it against the modern compressed file system with multiple dictionaries shootout. With version 1.8.10, no matter what parameters I set with LZMA <code>level=9</code> or <code>level=109</code> (109 is the extreme variant), even with lowering <code>dictsize=</code>, the results would cause system to run out of memory (16GB of RAM) before even completing the file system creation.

These benchmarks are in no way discrediting or disavowing the use of libdeflate over zlib. If file size is a concern, more modern compression algorithms and/or compression techniques could be considered. Modern EPWING viewer programs generally don't care if the electronic book is compressed using zlib or not compressed at all, and there are many ways of handling decompression transparently in modern Unix-like environments. Even more so, are the ever growing hardware performance as well as affordability with storage capacities in modern computing, questions the use of the what is widely considered as legacy compression algorithms for data archival. Last but not least, while single-threaded compression (as witnessed in the likes of <code>xz -9evv -T1</code> above) or tasks might be extremely beneficial in reducing overheads, especially with LZMA2, it is also a slow process which may subsequently not translate well in modern, demanding contexts, where time honored traditions are casted away for practical, convenient, modernized, and easy to use solutions.

<references />
[[User:Anonymoususer852|Anonymoususer852]] ([[User talk:Anonymoususer852|talk]]) 06:34, 20 August 2025 (UTC)

Talk:EBZip

2025-08-18T10:37:53Z

Anonymoususer852: /* Deleting */ Improved syntax to make it less disasterous "-i" implied. Also worksaround root/other owned files.

Talk:EBZip

2025-08-18T10:10:50Z

Anonymoususer852: Added talk page for topics that doesn't belong to main page.

DWARFS

2025-08-17T23:44:13Z

Anonymoususer852: Typo fixes for SHA-1 referenced link, concerning the name.

{{FormatInfo
|name=DWARFS
|formattype=electronic
|subcat=Filesystem
|compression=lossless, optional
|magic=[https://github.com/file/file/blob/master/magic/Magdir/dwarfs#L27 44 57 41 52 46 53]
|developed by=Marcus Holland-Moritz
|released=2020 <ref>[https://github.com/mhx/dwarfs/releases/tag/release-0.1.0 Release 0.1.0 - GitHub]</ref>
}}
'''D'''eduplicating '''W'''arp-speed '''A'''dvanced '''R'''ead-only '''F'''ile '''S'''ystem (DWARFS) is a read-only filesystem that optionally supports no compression (via <code>--compress-level=0</code> or <code>--compression=null</code> for instance) and no deduplication (via <code>--file-hash none</code>).

It is developed by Marcus Holland-Moritz and when compared to Squashfs, DWARFS also offers a choice of hashing algorithms,<ref>[https://github.com/mhx/dwarfs/issues/92 Choice of hash for duplicate detection #92 - GitHub]</ref> as well as a tool for checking DWARFS image via <code>dwarfsck</code>.

At maximum compression levels using LZMA, DWARFS (using <code>--compression=lzma:level=9:extreme</code>) can produce smaller file size compared to SquashFS with the rough equivalent of using XZ as <code>-comp xz</code>.

The software is currently available for both Unix-like (generally [[Linux]]) and [[Microsoft Windows]], with the caveat that, "Support for the Windows operating system is currently experimental. Having worked pretty much exclusively in a Unix world for the past two decades, my experience with Windows development is rather limited and I'd expect there to definitely be bugs and rough edges in the Windows code."<ref>[https://github.com/mhx/dwarfs?tab=readme-ov-file#windows-support Windows Support section of README.md - DWARFS - GitHub]</ref> There is even [https://formulae.brew.sh/formula/dwarfs <code>homebrew</code> formulae] for [[macOS]].

See also [[Squashfs]].

== Discussion ==
The aim with this software project is to ultimately create a compressed, deduplicated, read-only file system. While it is possible to disable all except for read-only, disabling it defeats the purpose.

DWARFS software project also intends to compete (performance-wise) with [[SquashFS]] at file system creation, in that a given directory, folder, or path is scanned, hashed, (optionally categorized) before the contents are compressed, adding only the unique copies. [[SquashFS]] tends to add and hash files as it creates the file system, similar to how [[7-Zip]] or [[ZIP]] files for instance are created, with the apparent ability to detect and avoid adding duplicate files during creation process that is unique to the likes of [[SquashFS]] and DWARFS, which however is considered slower, generally double-handling compared to DWARFS which does these during the scanning phase.

The developer's original motivation was with, "several hundred different versions of Perl that were taking up something around 30 gigabytes of disk space" and that there, "was unwilling to spend more than 10% of my hard drive keeping them around for when I happened to need them."<ref>[https://github.com/mhx/dwarfs?tab=readme-ov-file#history History section - DWARFS - GitHub]</ref> This may put the project in line for competing with <code>git</code>, which is a Versioning Control System (VCS), that is used to record changes to a given project at a more atomic level, compressing (and using zlib) changes with each commit, however, unlike <code>git</code> which introduces rather steep learning curves, contrasting to just adding several versions of the same software project into an "archive" is arguably more trivial to do. Ultimately, DWARFS, like [[SquashFS]] has many other potential use-cases, rather than whatever they were intentionally designed for, making it a somewhat popular choice in the race against other forms of archivers and software/file distribution methods, without requiring the use of bespoke compression software that happens to be ranked among the top in [https://www.mattmahoney.net/dc/text.html Matt Mahoney's data compression benchmarks].

== Identification ==
DWARFS files begins with the hexadecimal {{magic|44 57 41 52 46 53}} which translates to "DWARFS" in ASCII.<ref>[https://bugs.astron.com/view.php?id=449 0000449: Add magic for the DWARFS compressed file system format - bugs.astron.com]</ref>

== Examples ==
Create an extremely compressed DWARFS image, without history, all root owned contents, sha3-512 hash algorithm, idle task (lowest priority) from present directory to <code>example.dwarfs</code>:
mkdwarfs --input . --output=example.dwarfs --block-size-bits=26 --compression=lzma:level=9:extreme --compress-niceness=10 --schema-compression=lzma:level=9:extreme \
--metadata-compression=lzma:level=9:extreme --no-history --pack-metadata=all,force --file-hash=sha3-512 --set-owner=0 --set-group=0 --no-history-timestamp \
--no-create-timestamp --no-history-command-line

== <code>mkdwarfs</code> only features ==
The following below illustrates features that are only available upon creating the DWARFS file. Once the DWARFS file has been created, the only way to change these is to use <code>--recompress=all</code>.

=== Choice of available compression===
As of version 0.12.4, the following compressors are available:
{| border="1" cellpadding="6"
| Switch
| Comments
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/null.cpp null]</code>
| No compression.<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/null.cpp#L110 null.cpp (line 110) - DWARFS - GitHub]</ref>
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/lzma.cpp lzma]</code>
| liblzma compression<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/lzma.cpp#L413 lzma.cpp (line 413) - DWARFS - GitHub]</ref>, under modern [[Linux]] environments, [[XZ]]'s liblzma is used.
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/zstd.cpp zstd]</code>
| [[Zstandard|libzstd]] compression<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/zstd.cpp#L179 zstd.cpp (line 179) - DWARFS - GitHub]</ref>
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/lz4.cpp lz4]</code>
| lib[[LZ4|lz4]] compression,<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/lz4.cpp#L186 lz4.cpp (line 186) - DWARFS - GitHub]</ref> choice of either LZ4HC or LZ4 compression are available.
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/brotli.cpp brotli]</code>
| [[Brotli]] compressor<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/brotli.cpp#L169 brotli.cpp (line 169) - DWARFS - GitHub]</ref>
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/flac.cpp flac]</code>
| [[FLAC]] compression<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/flac.cpp#L496 flac.cpp (line 496) - DWARFS - GitHub]</ref>
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/ricepp.cpp ricepp]</code>
| RICEPP compression, likely [[wikipedia:Golomb coding|Golomb coding]]<ref>[https://github.com/glampert/compression-algorithms/blob/master/rice.hpp#L7-L9 rice.hpp (lines 7-9) - compression-algorithms - GitHub]</ref><ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/ricepp.cpp#L260 ricepp.cpp (line 260) - DWARFS - GitHub]</ref>
|}

=== Choice of available hash for duplication detection===
Since 0.7.0-RC1 introduced alternatives,<ref>[https://github.com/mhx/dwarfs/issues/92#issuecomment-1295209723 Choice of hash for duplicate detection #92 post #5 - DWARFS - GitHub]</ref> as of version [https://github.com/mhx/dwarfs/commits/main/src/checksum.cpp 0.12.4], the following hash algorithms are available:
{| border="1" cellpadding="6"
| Switch
| Remarks
|-
| <code>--file-hash=none</code>
| Disable file deduplication checks.<ref>[https://github.com/mhx/dwarfs/issues/92#issuecomment-1295209723 Choice of hash for duplicate detection #92, post #5 - DWARFS - GitHub]</ref>
|-
| <code>--file-hash=blake2b512</code>
|
|-
| <code>--file-hash=blake2s256</code>
|
|-
| <code>--file-hash=md5</code>
| "As of 2019, MD5 continues to be widely used, despite its well-documented weaknesses and deprecation by security experts."<ref>[[wikipedia:MD5#Security|Security section of MD5 - Wikipedia]]</ref> Vulnerabilities include collision attacks and preimage vulnerability.
|-
| <code>--file-hash=md5-sha1</code>
|
|-
| <code>--file-hash=ripemd160</code>
| Appeared as a checksum test in <code>git</code> hash <code>de5ec99</code>.<ref>[https://github.com/mhx/dwarfs/commit/de5ec99eca452d0ea2561bfcc6e9679683fa7463#diff-c4d301699de69e8106245704b87f73c0afa750915c8d396494c558aa3aaf3ab2R55 Commit de5ec99 - test/checksum_test.cpp (line 55) - test(checksum): add checksum tests - DWARFS - GitHub]</ref>
|-
| <code>--file-hash=sha1</code>
| Formerly used hash algorithm.<ref>[https://github.com/mhx/dwarfs/issues/92#issue-1299801734 Choice of hash for duplicate detection #92, post #1 - DWARFS - GitHub]</ref> Vulnerable to collision attacks.<ref>[[wikipedia:SHA-1#Attacks|Attacks section of SHA-1 - Wikipedia]]</ref>
|-
| <code>--file-hash=sha224</code>
|
|-
| <code>--file-hash=sha256</code>
|
|-
| <code>--file-hash=sha384</code>
|
|-
| <code>--file-hash=sha512</code>
|
|-
| <code>--file-hash=sha3-224</code>
|
|-
| <code>--file-hash=sha3-256</code>
|
|-
| <code>--file-hash=sha3-384</code>
|
|-
| <code>--file-hash=sha3-512</code>
|
|-
| <code>--file-hash=sha512-224</code>
|
|-
| <code>--file-hash=sha512-256</code>
|
|-
| <code>--file-hash=shake128</code>
| Disabled by author in <code>git</code> hash <code>afbd85e</code>.<ref>[https://github.com/mhx/dwarfs/commit/afbd85e4b1aad52b651fd42dc182e8109d0fa8b1 Commit afbd85e - fix(checksum): disable extended output algorithms (e.g. shake(128|256)) - DWARFS - GitHub]</ref>
|-
| <code>--file-hash=shake256</code>
| Disabled by author in <code>git</code> hash <code>afbd85e</code>.<ref>[https://github.com/mhx/dwarfs/commit/afbd85e4b1aad52b651fd42dc182e8109d0fa8b1 Commit afbd85e - fix(checksum): disable extended output algorithms (e.g. shake(128|256)) - DWARFS - GitHub]</ref>
|-
| <code>--file-hash=sm3</code>
|
|-
| <code>--file-hash=xxh3-64</code>
| Added by author in <code>git</code> hash <code>7ded26d</code>.<ref>[https://github.com/mhx/dwarfs/commit/7ded26d6a398af10fb05957b2fb4891cc533600c Commit 7ded26d - feat(checksum): add hexdigest() method - DWARFS - GitHub]</ref>
|-
| <code>--file-hash=xxh3-128</code>
| Current default choice when not specified.<ref>[https://github.com/mhx/dwarfs/releases/tag/v0.7.0-RC1 v0.7.0-RC1 - DWARFS releases by tag - GitHub]</ref>
|}

== Links ==
* [https://github.com/mhx/dwarfs DWARFS project page (GitHub)]

== References ==
<references/>

Squashfs

2025-08-17T16:18:48Z

Anonymoususer852: /* FormatInfo template */ Added developed by and maintained by. Likely developed by Phillip Lougher, with references to the earliest possible citation.

{{FormatInfo
|formattype=electronic
|subcat=Filesystem
|extensions={{ext|sfs}}<ref>[https://github.com/plougher/squashfs-tools/issues/259 What are the commonly accepted file name extension for squashfs? #259 - GitHub]</ref>
|wikidata={{wikidata|Q389314}}
|compression=lossless, mandatory
|magic=[https://github.com/file/file/blob/master/magic/Magdir/filesystems#L2194 68 73 71 73] (little-endian) [https://github.com/file/file/blob/master/magic/Magdir/filesystems#L2191 73 71 73 68] (big-endian)
|developed by=Phillip Lougher<ref>[https://sourceforge.net/p/squashfs/mailman/message/67563/ (Squashfs-announce) Squashfs1.2 released - 2003-03-13 05:30:45 - SourceForge mailman]</ref>
|maintained by=Phillip Lougher
|released=2002
}}
'''Squashfs''' is a compressed read-only filesystem for Linux. Typically used as a "live image" for booting Linux off a removable storage (e.g. USB, CD/DVD), minus components like the bootloader, kernel and initramfs.<ref>[https://fedoraproject.org/wiki/Changes/EROFSforLiveMedia EROFS For Live Media - Fedora Project]</ref> It is also used in modern AppImage,<ref>[[wikipedia:AppImage#History|History of AppImage - Wikipedia]]</ref>, or [[Snap]]<ref>[[wikipedia:Snap_(software)#Support|Snap - support section - Wikipedia]]</ref> as a way portable software distribution on Linux. Squashfs can also be used to store other data, such as disk images.<ref>[https://www.foo.be/2017/01/Squashfs_As_A_Forensic_Container Squashfs As a Forensic Container - foo.be]</ref> Under Windows, Squashfs contents can be opened/extracted via [[7-Zip]].<ref>[https://stackoverflow.com/a/36877207 python2.7 How to handle squashfs in Windows - StackOverflow]</ref>

See also [[cramfs]].

== Identification ==
SquashFS files created on little-endian machines begins with hexadecimal {{magic|68 73 71 73}}, which translates to "hsqs" in ASCII. SquashFS files created on big-endian machines with hexadecimal {{magic|73 71 73 68}}, which translates to "sqsh" in ASCII.

== Examples ==
Create a Squashfs (file system) image from current/present directory and output it into its parent directory:
$ mksquashfs . ../example.squashfs -b 1048576 -comp xz -Xdict-size 100%

Open a Squashfs image (as root) from present directory and have its contents visible in <code>/mnt</code>:
# mount example.squashfs /mnt

Open a Squashfs image (as user) from present directory and have its contents visible in <code>/mnt/temp</code>:
$ squashfuse example.squashfs /mnt/temp

== Software ==
* [[7-Zip]]
* [https://github.com/vasi/squashfuse Squashfuse] (Unix-like only)

== Sample files ==
* {{DexvertSamples|archive/squashFS}}
* [https://discmaster.textfiles.com/search?mode=deep&extension=.sfs&format=squashFS Search results for .sfs extension and is squashFS format - Discmaster.textfiles.com]

== Links ==
* [[Wikipedia: SquashFS]]
* [https://squashfs.sourceforge.net/ Squashfs website]
* [https://sourceforge.net/projects/squashfs/ SourceForge project page]
* [https://github.com/plougher/squashfs-tools GitHub project page]
* [https://elixir.bootlin.com/linux/latest/source/Documentation/filesystems/squashfs.txt squashfs.txt]

== References ==
<references/>

DWARFS

2025-08-17T16:07:21Z

Anonymoususer852: Added developer in FormatInfo. More typo fixes, "availble" → "available".

{{FormatInfo
|name=DWARFS
|formattype=electronic
|subcat=Filesystem
|compression=lossless, optional
|magic=[https://github.com/file/file/blob/master/magic/Magdir/dwarfs#L27 44 57 41 52 46 53]
|developed by=Marcus Holland-Moritz
|released=2020 <ref>[https://github.com/mhx/dwarfs/releases/tag/release-0.1.0 Release 0.1.0 - GitHub]</ref>
}}
'''D'''eduplicating '''W'''arp-speed '''A'''dvanced '''R'''ead-only '''F'''ile '''S'''ystem (DWARFS) is a read-only filesystem that optionally supports no compression (via <code>--compress-level=0</code> or <code>--compression=null</code> for instance) and no deduplication (via <code>--file-hash none</code>).

It is developed by Marcus Holland-Moritz and when compared to Squashfs, DWARFS also offers a choice of hashing algorithms,<ref>[https://github.com/mhx/dwarfs/issues/92 Choice of hash for duplicate detection #92 - GitHub]</ref> as well as a tool for checking DWARFS image via <code>dwarfsck</code>.

At maximum compression levels using LZMA, DWARFS (using <code>--compression=lzma:level=9:extreme</code>) can produce smaller file size compared to SquashFS with the rough equivalent of using XZ as <code>-comp xz</code>.

The software is currently available for both Unix-like (generally [[Linux]]) and [[Microsoft Windows]], with the caveat that, "Support for the Windows operating system is currently experimental. Having worked pretty much exclusively in a Unix world for the past two decades, my experience with Windows development is rather limited and I'd expect there to definitely be bugs and rough edges in the Windows code."<ref>[https://github.com/mhx/dwarfs?tab=readme-ov-file#windows-support Windows Support section of README.md - DWARFS - GitHub]</ref> There is even [https://formulae.brew.sh/formula/dwarfs <code>homebrew</code> formulae] for [[macOS]].

See also [[Squashfs]].

== Discussion ==
The aim with this software project is to ultimately create a compressed, deduplicated, read-only file system. While it is possible to disable all except for read-only, disabling it defeats the purpose.

DWARFS software project also intends to compete (performance-wise) with [[SquashFS]] at file system creation, in that a given directory, folder, or path is scanned, hashed, (optionally categorized) before the contents are compressed, adding only the unique copies. [[SquashFS]] tends to add and hash files as it creates the file system, similar to how [[7-Zip]] or [[ZIP]] files for instance are created, with the apparent ability to detect and avoid adding duplicate files during creation process that is unique to the likes of [[SquashFS]] and DWARFS, which however is considered slower, generally double-handling compared to DWARFS which does these during the scanning phase.

The developer's original motivation was with, "several hundred different versions of Perl that were taking up something around 30 gigabytes of disk space" and that there, "was unwilling to spend more than 10% of my hard drive keeping them around for when I happened to need them."<ref>[https://github.com/mhx/dwarfs?tab=readme-ov-file#history History section - DWARFS - GitHub]</ref> This may put the project in line for competing with <code>git</code>, which is a Versioning Control System (VCS), that is used to record changes to a given project at a more atomic level, compressing (and using zlib) changes with each commit, however, unlike <code>git</code> which introduces rather steep learning curves, contrasting to just adding several versions of the same software project into an "archive" is arguably more trivial to do. Ultimately, DWARFS, like [[SquashFS]] has many other potential use-cases, rather than whatever they were intentionally designed for, making it a somewhat popular choice in the race against other forms of archivers and software/file distribution methods, without requiring the use of bespoke compression software that happens to be ranked among the top in [https://www.mattmahoney.net/dc/text.html Matt Mahoney's data compression benchmarks].

== Identification ==
DWARFS files begins with the hexadecimal {{magic|44 57 41 52 46 53}} which translates to "DWARFS" in ASCII.<ref>[https://bugs.astron.com/view.php?id=449 0000449: Add magic for the DWARFS compressed file system format - bugs.astron.com]</ref>

== Examples ==
Create an extremely compressed DWARFS image, without history, all root owned contents, sha3-512 hash algorithm, idle task (lowest priority) from present directory to <code>example.dwarfs</code>:
mkdwarfs --input . --output=example.dwarfs --block-size-bits=26 --compression=lzma:level=9:extreme --compress-niceness=10 --schema-compression=lzma:level=9:extreme \
--metadata-compression=lzma:level=9:extreme --no-history --pack-metadata=all,force --file-hash=sha3-512 --set-owner=0 --set-group=0 --no-history-timestamp \
--no-create-timestamp --no-history-command-line

== <code>mkdwarfs</code> only features ==
The following below illustrates features that are only available upon creating the DWARFS file. Once the DWARFS file has been created, the only way to change these is to use <code>--recompress=all</code>.

=== Choice of available compression===
As of version 0.12.4, the following compressors are available:
{| border="1" cellpadding="6"
| Switch
| Comments
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/null.cpp null]</code>
| No compression.<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/null.cpp#L110 null.cpp (line 110) - DWARFS - GitHub]</ref>
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/lzma.cpp lzma]</code>
| liblzma compression<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/lzma.cpp#L413 lzma.cpp (line 413) - DWARFS - GitHub]</ref>, under modern [[Linux]] environments, [[XZ]]'s liblzma is used.
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/zstd.cpp zstd]</code>
| [[Zstandard|libzstd]] compression<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/zstd.cpp#L179 zstd.cpp (line 179) - DWARFS - GitHub]</ref>
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/lz4.cpp lz4]</code>
| lib[[LZ4|lz4]] compression,<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/lz4.cpp#L186 lz4.cpp (line 186) - DWARFS - GitHub]</ref> choice of either LZ4HC or LZ4 compression are available.
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/brotli.cpp brotli]</code>
| [[Brotli]] compressor<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/brotli.cpp#L169 brotli.cpp (line 169) - DWARFS - GitHub]</ref>
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/flac.cpp flac]</code>
| [[FLAC]] compression<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/flac.cpp#L496 flac.cpp (line 496) - DWARFS - GitHub]</ref>
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/ricepp.cpp ricepp]</code>
| RICEPP compression, likely [[wikipedia:Golomb coding|Golomb coding]]<ref>[https://github.com/glampert/compression-algorithms/blob/master/rice.hpp#L7-L9 rice.hpp (lines 7-9) - compression-algorithms - GitHub]</ref><ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/ricepp.cpp#L260 ricepp.cpp (line 260) - DWARFS - GitHub]</ref>
|}

=== Choice of available hash for duplication detection===
Since 0.7.0-RC1 introduced alternatives,<ref>[https://github.com/mhx/dwarfs/issues/92#issuecomment-1295209723 Choice of hash for duplicate detection #92 post #5 - DWARFS - GitHub]</ref> as of version [https://github.com/mhx/dwarfs/commits/main/src/checksum.cpp 0.12.4], the following hash algorithms are available:
{| border="1" cellpadding="6"
| Switch
| Remarks
|-
| <code>--file-hash=none</code>
| Disable file deduplication checks.<ref>[https://github.com/mhx/dwarfs/issues/92#issuecomment-1295209723 Choice of hash for duplicate detection #92, post #5 - DWARFS - GitHub]</ref>
|-
| <code>--file-hash=blake2b512</code>
|
|-
| <code>--file-hash=blake2s256</code>
|
|-
| <code>--file-hash=md5</code>
| "As of 2019, MD5 continues to be widely used, despite its well-documented weaknesses and deprecation by security experts."<ref>[[wikipedia:MD5#Security|Security section of MD5 - Wikipedia]]</ref> Vulnerabilities include collision attacks and preimage vulnerability.
|-
| <code>--file-hash=md5-sha1</code>
|
|-
| <code>--file-hash=ripemd160</code>
| Appeared as a checksum test in <code>git</code> hash <code>de5ec99</code>.<ref>[https://github.com/mhx/dwarfs/commit/de5ec99eca452d0ea2561bfcc6e9679683fa7463#diff-c4d301699de69e8106245704b87f73c0afa750915c8d396494c558aa3aaf3ab2R55 Commit de5ec99 - test/checksum_test.cpp (line 55) - test(checksum): add checksum tests - DWARFS - GitHub]</ref>
|-
| <code>--file-hash=sha1</code>
| Formerly used hash algorithm.<ref>[https://github.com/mhx/dwarfs/issues/92#issue-1299801734 Choice of hash for duplicate detection #92, post #1 - DWARFS - GitHub]</ref> Vulnerable to collision attacks.<ref>[[wikipedia:SHA-1#Attacks Attacks section of SHA-1 - Wikipedia]]</ref>
|-
| <code>--file-hash=sha224</code>
|
|-
| <code>--file-hash=sha256</code>
|
|-
| <code>--file-hash=sha384</code>
|
|-
| <code>--file-hash=sha512</code>
|
|-
| <code>--file-hash=sha3-224</code>
|
|-
| <code>--file-hash=sha3-256</code>
|
|-
| <code>--file-hash=sha3-384</code>
|
|-
| <code>--file-hash=sha3-512</code>
|
|-
| <code>--file-hash=sha512-224</code>
|
|-
| <code>--file-hash=sha512-256</code>
|
|-
| <code>--file-hash=shake128</code>
| Disabled by author in <code>git</code> hash <code>afbd85e</code>.<ref>[https://github.com/mhx/dwarfs/commit/afbd85e4b1aad52b651fd42dc182e8109d0fa8b1 Commit afbd85e - fix(checksum): disable extended output algorithms (e.g. shake(128|256)) - DWARFS - GitHub]</ref>
|-
| <code>--file-hash=shake256</code>
| Disabled by author in <code>git</code> hash <code>afbd85e</code>.<ref>[https://github.com/mhx/dwarfs/commit/afbd85e4b1aad52b651fd42dc182e8109d0fa8b1 Commit afbd85e - fix(checksum): disable extended output algorithms (e.g. shake(128|256)) - DWARFS - GitHub]</ref>
|-
| <code>--file-hash=sm3</code>
|
|-
| <code>--file-hash=xxh3-64</code>
| Added by author in <code>git</code> hash <code>7ded26d</code>.<ref>[https://github.com/mhx/dwarfs/commit/7ded26d6a398af10fb05957b2fb4891cc533600c Commit 7ded26d - feat(checksum): add hexdigest() method - DWARFS - GitHub]</ref>
|-
| <code>--file-hash=xxh3-128</code>
| Current default choice when not specified.<ref>[https://github.com/mhx/dwarfs/releases/tag/v0.7.0-RC1 v0.7.0-RC1 - DWARFS releases by tag - GitHub]</ref>
|}

== Links ==
* [https://github.com/mhx/dwarfs DWARFS project page (GitHub)]

== References ==
<references/>

DWARFS

2025-08-17T12:27:38Z

Anonymoususer852: More typo fixes with mentioning of "DwarFS" instead of DWARFS. Named the project page in Links section as appropriate.

{{FormatInfo
|name=DWARFS
|formattype=electronic
|subcat=Filesystem
|compression=lossless, optional
|magic=[https://github.com/file/file/blob/master/magic/Magdir/dwarfs#L27 44 57 41 52 46 53]
|released=2020 <ref>[https://github.com/mhx/dwarfs/releases/tag/release-0.1.0 Release 0.1.0 - GitHub]</ref>
}}
'''D'''eduplicating '''W'''arp-speed '''A'''dvanced '''R'''ead-only '''F'''ile '''S'''ystem (DWARFS) is a read-only filesystem that optionally supports no compression (via <code>--compress-level=0</code> or <code>--compression=null</code> for instance) and no deduplication (via <code>--file-hash none</code>).

It is developed by Marcus Holland-Moritz and when compared to Squashfs, DWARFS also offers a choice of hashing algorithms,<ref>[https://github.com/mhx/dwarfs/issues/92 Choice of hash for duplicate detection #92 - GitHub]</ref> as well as a tool for checking DWARFS image via <code>dwarfsck</code>.

At maximum compression levels using LZMA, DWARFS (using <code>--compression=lzma:level=9:extreme</code>) can produce smaller file size compared to SquashFS with the rough equivalent of using XZ as <code>-comp xz</code>.

The software is currently available for both Unix-like (generally [[Linux]]) and [[Microsoft Windows]], with the caveat that, "Support for the Windows operating system is currently experimental. Having worked pretty much exclusively in a Unix world for the past two decades, my experience with Windows development is rather limited and I'd expect there to definitely be bugs and rough edges in the Windows code."<ref>[https://github.com/mhx/dwarfs?tab=readme-ov-file#windows-support Windows Support section of README.md - DWARFS - GitHub]</ref> There is even [https://formulae.brew.sh/formula/dwarfs <code>homebrew</code> formulae] for [[macOS]].

See also [[Squashfs]].

== Discussion ==
The aim with this software project is to ultimately create a compressed, deduplicated, read-only file system. While it is possible to disable all except for read-only, disabling it defeats the purpose.

DWARFS software project also intends to compete (performance-wise) with [[SquashFS]] at file system creation, in that a given directory, folder, or path is scanned, hashed, (optionally categorized) before the contents are compressed, adding only the unique copies. [[SquashFS]] tends to add and hash files as it creates the file system, similar to how [[7-Zip]] or [[ZIP]] files for instance are created, with the apparent ability to detect and avoid adding duplicate files during creation process that is unique to the likes of [[SquashFS]] and DWARFS, which however is considered slower, generally double-handling compared to DWARFS which does these during the scanning phase.

The developer's original motivation was with, "several hundred different versions of Perl that were taking up something around 30 gigabytes of disk space" and that there, "was unwilling to spend more than 10% of my hard drive keeping them around for when I happened to need them."<ref>[https://github.com/mhx/dwarfs?tab=readme-ov-file#history History section - DWARFS - GitHub]</ref> This may put the project in line for competing with <code>git</code>, which is a Versioning Control System (VCS), that is used to record changes to a given project at a more atomic level, compressing (and using zlib) changes with each commit, however, unlike <code>git</code> which introduces rather steep learning curves, contrasting to just adding several versions of the same software project into an "archive" is arguably more trivial to do. Ultimately, DWARFS, like [[SquashFS]] has many other potential use-cases, rather than whatever they were intentionally designed for, making it a somewhat popular choice in the race against other forms of archivers and software/file distribution methods, without requiring the use of bespoke compression software that happens to be ranked among the top in [https://www.mattmahoney.net/dc/text.html Matt Mahoney's data compression benchmarks].

== Identification ==
DWARFS files begins with the hexadecimal {{magic|44 57 41 52 46 53}} which translates to "DWARFS" in ASCII.<ref>[https://bugs.astron.com/view.php?id=449 0000449: Add magic for the DWARFS compressed file system format - bugs.astron.com]</ref>

== Examples ==
Create an extremely compressed DWARFS image, without history, all root owned contents, sha3-512 hash algorithm, idle task (lowest priority) from present directory to <code>example.dwarfs</code>:
mkdwarfs --input . --output=example.dwarfs --block-size-bits=26 --compression=lzma:level=9:extreme --compress-niceness=10 --schema-compression=lzma:level=9:extreme \
--metadata-compression=lzma:level=9:extreme --no-history --pack-metadata=all,force --file-hash=sha3-512 --set-owner=0 --set-group=0 --no-history-timestamp \
--no-create-timestamp --no-history-command-line

== <code>mkdwarfs</code> only features ==
The following below illustrates features that are only availble upon creating the DWARFS file. Once the DWARFS file has been created, the only way to change these is to use <code>--recompress=all</code>.

=== Choice of available compression===
As of version 0.12.4, the following compressors are available:
{| border="1" cellpadding="6"
| Switch
| Comments
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/null.cpp null]</code>
| No compression.<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/null.cpp#L110 null.cpp (line 110) - DWARFS - GitHub]</ref>
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/lzma.cpp lzma]</code>
| liblzma compression<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/lzma.cpp#L413 lzma.cpp (line 413) - DWARFS - GitHub]</ref>, under modern [[Linux]] environments, [[XZ]]'s liblzma is used.
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/zstd.cpp zstd]</code>
| [[Zstandard|libzstd]] compression<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/zstd.cpp#L179 zstd.cpp (line 179) - DWARFS - GitHub]</ref>
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/lz4.cpp lz4]</code>
| lib[[LZ4|lz4]] compression,<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/lz4.cpp#L186 lz4.cpp (line 186) - DWARFS - GitHub]</ref> choice of either LZ4HC or LZ4 compression are available.
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/brotli.cpp brotli]</code>
| [[Brotli]] compressor<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/brotli.cpp#L169 brotli.cpp (line 169) - DWARFS - GitHub]</ref>
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/flac.cpp flac]</code>
| [[FLAC]] compression<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/flac.cpp#L496 flac.cpp (line 496) - DWARFS - GitHub]</ref>
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/ricepp.cpp ricepp]</code>
| RICEPP compression, likely [[wikipedia:Golomb coding|Golomb coding]]<ref>[https://github.com/glampert/compression-algorithms/blob/master/rice.hpp#L7-L9 rice.hpp (lines 7-9) - compression-algorithms - GitHub]</ref><ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/ricepp.cpp#L260 ricepp.cpp (line 260) - DWARFS - GitHub]</ref>
|}

=== Choice of available hash for duplication detection===
Since 0.7.0-RC1 introduced alternatives,<ref>[https://github.com/mhx/dwarfs/issues/92#issuecomment-1295209723 Choice of hash for duplicate detection #92 post #5 - DWARFS - GitHub]</ref> as of version [https://github.com/mhx/dwarfs/commits/main/src/checksum.cpp 0.12.4], the following hash algorithms are available:
{| border="1" cellpadding="6"
| Switch
| Remarks
|-
| <code>--file-hash=none</code>
| Disable file deduplication checks.<ref>[https://github.com/mhx/dwarfs/issues/92#issuecomment-1295209723 Choice of hash for duplicate detection #92, post #5 - DWARFS - GitHub]</ref>
|-
| <code>--file-hash=blake2b512</code>
|
|-
| <code>--file-hash=blake2s256</code>
|
|-
| <code>--file-hash=md5</code>
| "As of 2019, MD5 continues to be widely used, despite its well-documented weaknesses and deprecation by security experts."<ref>[[wikipedia:MD5#Security|Security section of MD5 - Wikipedia]]</ref> Vulnerabilities include collision attacks and preimage vulnerability.
|-
| <code>--file-hash=md5-sha1</code>
|
|-
| <code>--file-hash=ripemd160</code>
| Appeared as a checksum test in <code>git</code> hash <code>de5ec99</code>.<ref>[https://github.com/mhx/dwarfs/commit/de5ec99eca452d0ea2561bfcc6e9679683fa7463#diff-c4d301699de69e8106245704b87f73c0afa750915c8d396494c558aa3aaf3ab2R55 Commit de5ec99 - test/checksum_test.cpp (line 55) - test(checksum): add checksum tests - DWARFS - GitHub]</ref>
|-
| <code>--file-hash=sha1</code>
| Formerly used hash algorithm.<ref>[https://github.com/mhx/dwarfs/issues/92#issue-1299801734 Choice of hash for duplicate detection #92, post #1 - DWARFS - GitHub]</ref> Vulnerable to collision attacks.<ref>[[wikipedia:SHA-1#Attacks Attacks section of SHA-1 - Wikipedia]]</ref>
|-
| <code>--file-hash=sha224</code>
|
|-
| <code>--file-hash=sha256</code>
|
|-
| <code>--file-hash=sha384</code>
|
|-
| <code>--file-hash=sha512</code>
|
|-
| <code>--file-hash=sha3-224</code>
|
|-
| <code>--file-hash=sha3-256</code>
|
|-
| <code>--file-hash=sha3-384</code>
|
|-
| <code>--file-hash=sha3-512</code>
|
|-
| <code>--file-hash=sha512-224</code>
|
|-
| <code>--file-hash=sha512-256</code>
|
|-
| <code>--file-hash=shake128</code>
| Disabled by author in <code>git</code> hash <code>afbd85e</code>.<ref>[https://github.com/mhx/dwarfs/commit/afbd85e4b1aad52b651fd42dc182e8109d0fa8b1 Commit afbd85e - fix(checksum): disable extended output algorithms (e.g. shake(128|256)) - DWARFS - GitHub]</ref>
|-
| <code>--file-hash=shake256</code>
| Disabled by author in <code>git</code> hash <code>afbd85e</code>.<ref>[https://github.com/mhx/dwarfs/commit/afbd85e4b1aad52b651fd42dc182e8109d0fa8b1 Commit afbd85e - fix(checksum): disable extended output algorithms (e.g. shake(128|256)) - DWARFS - GitHub]</ref>
|-
| <code>--file-hash=sm3</code>
|
|-
| <code>--file-hash=xxh3-64</code>
| Added by author in <code>git</code> hash <code>7ded26d</code>.<ref>[https://github.com/mhx/dwarfs/commit/7ded26d6a398af10fb05957b2fb4891cc533600c Commit 7ded26d - feat(checksum): add hexdigest() method - DWARFS - GitHub]</ref>
|-
| <code>--file-hash=xxh3-128</code>
| Current default choice when not specified.<ref>[https://github.com/mhx/dwarfs/releases/tag/v0.7.0-RC1 v0.7.0-RC1 - DWARFS releases by tag - GitHub]</ref>
|}

== Links ==
* [https://github.com/mhx/dwarfs DWARFS project page (GitHub)]

== References ==
<references/>

DWARFS

2025-08-17T12:21:27Z

Anonymoususer852: /* Choice of available compression */ Fixed inconsistent period mark placement.

{{FormatInfo
|name=DWARFS
|formattype=electronic
|subcat=Filesystem
|compression=lossless, optional
|magic=[https://github.com/file/file/blob/master/magic/Magdir/dwarfs#L27 44 57 41 52 46 53]
|released=2020 <ref>[https://github.com/mhx/dwarfs/releases/tag/release-0.1.0 Release 0.1.0 - GitHub]</ref>
}}
'''D'''eduplicating '''W'''arp-speed '''A'''dvanced '''R'''ead-only '''F'''ile '''S'''ystem (DWARFS) is a read-only filesystem that optionally supports no compression (via <code>--compress-level=0</code> or <code>--compression=null</code> for instance) and no deduplication (via <code>--file-hash none</code>).

It is developed by Marcus Holland-Moritz and when compared to Squashfs, DWARFS also offers a choice of hashing algorithms,<ref>[https://github.com/mhx/dwarfs/issues/92 Choice of hash for duplicate detection #92 - GitHub]</ref> as well as a tool for checking DWARFS image via <code>dwarfsck</code>.

At maximum compression levels using LZMA, DWARFS (using <code>--compression=lzma:level=9:extreme</code>) can produce smaller file size compared to SquashFS with the rough equivalent of using XZ as <code>-comp xz</code>.

The software is currently available for both Unix-like (generally [[Linux]]) and [[Microsoft Windows]], with the caveat that, "Support for the Windows operating system is currently experimental. Having worked pretty much exclusively in a Unix world for the past two decades, my experience with Windows development is rather limited and I'd expect there to definitely be bugs and rough edges in the Windows code."<ref>[https://github.com/mhx/dwarfs?tab=readme-ov-file#windows-support Windows Support section of README.md - DWARFS - GitHub]</ref> There is even [https://formulae.brew.sh/formula/dwarfs <code>homebrew</code> formulae] for [[macOS]].

See also [[Squashfs]].

== Discussion ==
The aim with this software project is to ultimately create a compressed, deduplicated, read-only file system. While it is possible to disable all except for read-only, disabling it defeats the purpose.

DWARFS software project also intends to compete (performance-wise) with [[SquashFS]] at file system creation, in that a given directory, folder, or path is scanned, hashed, (optionally categorized) before the contents are compressed, adding only the unique copies. [[SquashFS]] tends to add and hash files as it creates the file system, similar to how [[7-Zip]] or [[ZIP]] files for instance are created, with the apparent ability to detect and avoid adding duplicate files during creation process that is unique to the likes of [[SquashFS]] and DWARFS, which however is considered slower, generally double-handling compared to DWARFS which does these during the scanning phase.

The developer's original motivation was with, "several hundred different versions of Perl that were taking up something around 30 gigabytes of disk space" and that there, "was unwilling to spend more than 10% of my hard drive keeping them around for when I happened to need them."<ref>[https://github.com/mhx/dwarfs?tab=readme-ov-file#history History section - DWARFS - GitHub]</ref> This may put the project in line for competing with <code>git</code>, which is a Versioning Control System (VCS), that is used to record changes to a given project at a more atomic level, compressing (and using zlib) changes with each commit, however, unlike <code>git</code> which introduces rather steep learning curves, contrasting to just adding several versions of the same software project into an "archive" is arguably more trivial to do. Ultimately, DWARFS, like [[SquashFS]] has many other potential use-cases, rather than whatever they were intentionally designed for, making it a somewhat popular choice in the race against other forms of archivers and software/file distribution methods, without requiring the use of bespoke compression software that happens to be ranked among the top in [https://www.mattmahoney.net/dc/text.html Matt Mahoney's data compression benchmarks].

== Identification ==
DWARFS files begins with the hexadecimal {{magic|44 57 41 52 46 53}} which translates to "DWARFS" in ASCII.<ref>[https://bugs.astron.com/view.php?id=449 0000449: Add magic for the DWARFS compressed file system format - bugs.astron.com]</ref>

== Examples ==
Create an extremely compressed DwarFS image, without history, all root owned contents, sha3-512 hash algorithm, idle task (lowest priority) from present directory to <code>example.dwarfs</code>:
mkdwarfs --input . --output=example.dwarfs --block-size-bits=26 --compression=lzma:level=9:extreme --compress-niceness=10 --schema-compression=lzma:level=9:extreme \
--metadata-compression=lzma:level=9:extreme --no-history --pack-metadata=all,force --file-hash=sha3-512 --set-owner=0 --set-group=0 --no-history-timestamp \
--no-create-timestamp --no-history-command-line

== <code>mkdwarfs</code> only features ==
The following below illustrates features that are only availble upon creating the DWARFS file. Once the DWARFS file has been created, the only way to change these is to use <code>--recompress=all</code>.

=== Choice of available compression===
As of version 0.12.4, the following compressors are available:
{| border="1" cellpadding="6"
| Switch
| Comments
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/null.cpp null]</code>
| No compression.<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/null.cpp#L110 null.cpp (line 110) - DWARFS - GitHub]</ref>
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/lzma.cpp lzma]</code>
| liblzma compression<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/lzma.cpp#L413 lzma.cpp (line 413) - DWARFS - GitHub]</ref>, under modern [[Linux]] environments, [[XZ]]'s liblzma is used.
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/zstd.cpp zstd]</code>
| [[Zstandard|libzstd]] compression<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/zstd.cpp#L179 zstd.cpp (line 179) - DWARFS - GitHub]</ref>
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/lz4.cpp lz4]</code>
| lib[[LZ4|lz4]] compression,<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/lz4.cpp#L186 lz4.cpp (line 186) - DWARFS - GitHub]</ref> choice of either LZ4HC or LZ4 compression are available.
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/brotli.cpp brotli]</code>
| [[Brotli]] compressor<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/brotli.cpp#L169 brotli.cpp (line 169) - DWARFS - GitHub]</ref>
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/flac.cpp flac]</code>
| [[FLAC]] compression<ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/flac.cpp#L496 flac.cpp (line 496) - DWARFS - GitHub]</ref>
|-
| <code>--compression=[https://github.com/mhx/dwarfs/blob/main/src/compression/ricepp.cpp ricepp]</code>
| RICEPP compression, likely [[wikipedia:Golomb coding|Golomb coding]]<ref>[https://github.com/glampert/compression-algorithms/blob/master/rice.hpp#L7-L9 rice.hpp (lines 7-9) - compression-algorithms - GitHub]</ref><ref>[https://github.com/mhx/dwarfs/blob/main/src/compression/ricepp.cpp#L260 ricepp.cpp (line 260) - DWARFS - GitHub]</ref>
|}

=== Choice of available hash for duplication detection===
Since 0.7.0-RC1 introduced alternatives,<ref>[https://github.com/mhx/dwarfs/issues/92#issuecomment-1295209723 Choice of hash for duplicate detection #92 post #5 - DWARFS - GitHub]</ref> as of version [https://github.com/mhx/dwarfs/commits/main/src/checksum.cpp 0.12.4], the following hash algorithms are available:
{| border="1" cellpadding="6"
| Switch
| Remarks
|-
| <code>--file-hash=none</code>
| Disable file deduplication checks.<ref>[https://github.com/mhx/dwarfs/issues/92#issuecomment-1295209723 Choice of hash for duplicate detection #92, post #5 - DWARFS - GitHub]</ref>
|-
| <code>--file-hash=blake2b512</code>
|
|-
| <code>--file-hash=blake2s256</code>
|
|-
| <code>--file-hash=md5</code>
| "As of 2019, MD5 continues to be widely used, despite its well-documented weaknesses and deprecation by security experts."<ref>[[wikipedia:MD5#Security|Security section of MD5 - Wikipedia]]</ref> Vulnerabilities include collision attacks and preimage vulnerability.
|-
| <code>--file-hash=md5-sha1</code>
|
|-
| <code>--file-hash=ripemd160</code>
| Appeared as a checksum test in <code>git</code> hash <code>de5ec99</code>.<ref>[https://github.com/mhx/dwarfs/commit/de5ec99eca452d0ea2561bfcc6e9679683fa7463#diff-c4d301699de69e8106245704b87f73c0afa750915c8d396494c558aa3aaf3ab2R55 Commit de5ec99 - test/checksum_test.cpp (line 55) - test(checksum): add checksum tests - DWARFS - GitHub]</ref>
|-
| <code>--file-hash=sha1</code>
| Formerly used hash algorithm.<ref>[https://github.com/mhx/dwarfs/issues/92#issue-1299801734 Choice of hash for duplicate detection #92, post #1 - DWARFS - GitHub]</ref> Vulnerable to collision attacks.<ref>[[wikipedia:SHA-1#Attacks Attacks section of SHA-1 - Wikipedia]]</ref>
|-
| <code>--file-hash=sha224</code>
|
|-
| <code>--file-hash=sha256</code>
|
|-
| <code>--file-hash=sha384</code>
|
|-
| <code>--file-hash=sha512</code>
|
|-
| <code>--file-hash=sha3-224</code>
|
|-
| <code>--file-hash=sha3-256</code>
|
|-
| <code>--file-hash=sha3-384</code>
|
|-
| <code>--file-hash=sha3-512</code>
|
|-
| <code>--file-hash=sha512-224</code>
|
|-
| <code>--file-hash=sha512-256</code>
|
|-
| <code>--file-hash=shake128</code>
| Disabled by author in <code>git</code> hash <code>afbd85e</code>.<ref>[https://github.com/mhx/dwarfs/commit/afbd85e4b1aad52b651fd42dc182e8109d0fa8b1 Commit afbd85e - fix(checksum): disable extended output algorithms (e.g. shake(128|256)) - DWARFS - GitHub]</ref>
|-
| <code>--file-hash=shake256</code>
| Disabled by author in <code>git</code> hash <code>afbd85e</code>.<ref>[https://github.com/mhx/dwarfs/commit/afbd85e4b1aad52b651fd42dc182e8109d0fa8b1 Commit afbd85e - fix(checksum): disable extended output algorithms (e.g. shake(128|256)) - DWARFS - GitHub]</ref>
|-
| <code>--file-hash=sm3</code>
|
|-
| <code>--file-hash=xxh3-64</code>
| Added by author in <code>git</code> hash <code>7ded26d</code>.<ref>[https://github.com/mhx/dwarfs/commit/7ded26d6a398af10fb05957b2fb4891cc533600c Commit 7ded26d - feat(checksum): add hexdigest() method - DWARFS - GitHub]</ref>
|-
| <code>--file-hash=xxh3-128</code>
| Current default choice when not specified.<ref>[https://github.com/mhx/dwarfs/releases/tag/v0.7.0-RC1 v0.7.0-RC1 - DWARFS releases by tag - GitHub]</ref>
|}

== Links ==
* https://github.com/mhx/dwarfs

== References ==
<references/>

FLAC

2025-08-17T12:17:59Z

Anonymoususer852: Updated http links; songbird is blank page, dustbury returns 404. Updated FormatInfo.

{{FormatInfo
|formattype=electronic
|subcat=Audio and Music
|extensions={{ext|flac}}
|mimetypes={{mimetype|audio/x-flac}},{{mimetype|audio/flac}}
|compression=lossless, mandatory
|pronom={{PRONOM|fmt/279}}
|wikidata={{wikidata|Q27881556}}
|magic=[https://github.com/file/file/blob/master/magic/Magdir/audio#L493 66 4C 61 43]
}}
'''FLAC''' is a Free Lossless Audio Codec. It can encode audio with a PCM bit resolution up to 32 bits per sample and sampling rates up to 640 kHz. FLAC-encoded audio is usually found either in a native container (which has the extension <code>.flac</code>), or in an [[Ogg]] container (when it's known as OggFLAC).

The format is open and royalty-free. The reference implementation is cross-platform and dual-licensed, command-line utilities (e.g. encoder, decoder and metadata editor) use GNU GPL and code libraries use BSD.

FLAC is suitable for archiving for many reasons:
* open format
* support for metadata tagging
* lossless (no generation loss if you need to convert to another format)
* disk size effective (audio is typically reduced to 50-60% of original size)
* data integrity
* error resistant (bit faults are contained within a frame, typically a fraction of a second)

== Identification ==
When FLAC is used as a file format, it begins with the ASCII signature "{{magic|fLaC}}".

In rare cases, this signature may appear following an ID3v2 segment; see [[ID3#Identification]].

== Playback ==
=== Hardware ===

Many home stereo and portable hardware music players support the FLAC format. See the [http://flac.sourceforge.net/links.html FLAC links] page for an up-to-date list.

=== Software ===

A number of popular audio players support the FLAC format, including:

* [https://amarok.kde.org/ Amarok] (cross-platform, open source)
* [https://www.foobar2000.org/ foobar2000] (Windows, non-commercial)
* [https://www.mediamonkey.com/ MediaMonkey] (Windows, commercial)
* [https://www.videolan.org/vlc/index.html VLC] (cross-platform, open source)
* [https://winamp.com/ Winamp] (Windows, commercial)

FLAC is also natively supported by Mozilla's Firefox browser, starting from Firefox 51. For more software products which support FLAC, see the [http://flac.sourceforge.net/links.html FLAC links page]

== Specifications ==
* [https://xiph.org/flac/format.html FLAC Format Specification]

== Sample files ==
* {{DexvertSamples|audio/flac}}
* https://samples.mplayerhq.hu/flac/ (some are in .ogg format (OggFLAC format))

== Links ==
* [https://xiph.org/flac/ Home of FLAC project]
* [https://xiph.org/flac/format.html FLAC format description]
* [https://xiph.org/flac/faq.html FLAC FAQ] (hey, that rhymes!)
* [https://web.archive.org/web/20191230043206/https://www.dustbury.com/archives/17227 Commentary about Audi car not playing high-bitrate FLACs - Internet Archive copy]
* [https://dericed.com/2013/flac-in-the-archives/ FLAC in the archives]
* [https://developer.mozilla.org/en-US/docs/Web/HTML/Supported_media_formats Mozilla Firefox supported media formats list]
* [https://www.ghacks.net/2016/08/30/firefox-51-flac-audio-codec-support/ Firefox 51: FLAC Audio Codec Support]