Deja Vu #05: CODING: Exploring MS-PACK and DEPACKER Functionality

SoundTrack: !-0 ;)-: BY SECTOR AFTER INSTY
__________________________________________

(C) Max/CYBERAX Software
Addition: dAn!!L
__________________________________________

This text will discuss the MS-PACK packer, specifically the DEPACKER that it attaches to packed files. In its original form, the UNPACKER of MS-PACK must operate with interrupts disabled. It is also not designed to be separated from the block (for example, if your program has many compressed files, it makes no sense to have a decompressor in all of them - one instance is sufficient). Working with interrupts enabled is useful if you need to unpack something without interrupting the music or animation. In this case, the music hangs on IM 2, while the DEPACKER operates in normal mode.

The following two programs are designed to implement all of the above.

The first program is written in BASIC. It is intended to separate the DEPACKER from the compressed block. When launched, it prompts for a file name (no more than 7 characters). It then writes a block without the decompressor to disk. An apostrophe is added to the original file name. The length of the resulting block will be 247 bytes smaller than the block with the DEPACKER.

1 INPUT "Name: ";A$ : RANDOMIZE USR VAL "15619" : REM : LOAD A$ CODE 2 LET O=VAL"PEEK 23782 + 256*PEEK 23783": LET A=VAL "PEEK (O+233) + 256*PEEK (O+234)" : LET L=VAL"3 + PEEK (O+239) + 256*PEEK(O+240)" 3 FOR J=NOT PI TO VAL "4" : POKE VAL"O + 238 + J", PEEK VAL "A - 4 + J" : NEXT J 4 RANDOMIZE USR VAL "15619" : REM : SAVE A$ + "'" CODE VAL "O + 238", L

The second program is written in assembly language. This is, in fact, the DEPACKER. (To be honest, I never figured out the algorithm of the decompressor. I only replaced all stack operations with IX register operations and tried to fit the new version of the DEPACKER into 256 bytes.)

ORG 40000
;+---------------------------------------+
;|*** MS-PACK UNPACKER WITH INTERRUPT ***|
;|---------------------------------------|
;| Originally written by 'MICROSPACE', |
;| remixed by Max 'CYBERAX Software' |
;+---------------------------------------+
UNPACK LD C,16
LD B,C
LD D,3
EXX
LD HL,SOURCE
;Address of the packed block without UNPACKER.
;(Where to unpack from)
;The block should be obtained by the previous BASIC program.
LD DE,BUF
LD BC,5
LDIR
PUSH HL
POP IX
LD DE,DESTIN
;Address in memory where unpacking will occur.
LD H,B
EXX
CALL POP_HL
JR ENTRY
FIRST CALL POP_AF
EXX
LD (DE),A
INC DE
EXX
ENTRY CALL NEWBIT
JR NC,FIRST
LD E,2
L1 XOR A
CALL NEWBIT
RLA
CALL NEWBIT
RLA
CP D
JR C,MET1
ADD A,E
LD E,A
CP 17
JR NZ,L1
XOR A
MET1 ADD A,E
CP 5
JR NC,MET2
CP 2
JR NZ,MET3
EXX
LD C,A
L3 CALL POP_AF
LD L,A
LD A,E
SCF
SBC A,L
LD L,A
LD A,D
SBC A,H
LD H,A
LDIR
LD H,B
EXX
JR ENTRY
L2 EXX
CALL POP_AF
INC A
JR Z,END
INC A
JR NZ,MET4
CALL POP_BC
JR MET5
MET4 ADD A,15
JR NC,MET6
INC B
JR MET6
MET2 JR Z,L2
DEC A
MET3 EXX
MET6 LD C,A
MET5 EXX
CALL NEWBIT
EXX
JR C,L3
EXX
XOR A
LD E,D
L4 CALL NEWBIT
RLA
DEC E
JR NZ,L4
CP 2
JR NC,MET7
INC A
LOOP EXX
LD H,A
JR L3
MET7 CALL NEWBIT
RLA
CP 8
JR NC,MET8
DEC A
JR LOOP
MET8 CALL NEWBIT
RLA
CP 23
JR NC,MET9
SUB 9
JR LOOP
MET9 CALL NEWBIT
RLA
AND 31
CP 31
JR C,LOOP
CALL POP_AF
JR LOOP
END LD HL,BUF
LD C,5
LDIR
;In case you will use this decompressor from BASIC,
;it is advisable to insert commands to restore the register
;HL here (otherwise BASIC will glitch...).
RET
NEWBIT ADD HL,HL
DJNZ MET
LD B,C
POP_HL LD L,(IX)
LD H,(IX+1)
FIN INC IX
INC IX
MET RET
POP_DE LD E,(IX)
LD D,(IX+1)
JR FIN
POP_BC LD C,(IX)
LD B,(IX+1)
JR FIN
POP_AF LD A,(IX)
JR FIN+2
BUF DEFS 5
SIZE EQU $-DEPACK

The addresses SOURCE and DESTIN are chosen not randomly. Rather, they are chosen according to certain rules. Let the length of the original block (unpacked) be LEN1, and the length of the packed one (without UNPACKER) be LEN2. There are two possible cases:

1) Suppose the DESTIN address is specified, then the SOURCE address may lie in the range from 16384 to DESTIN-LEN2 or in the range from DESTIN+LEN1-LEN2 to 65536-LEN2.

2) You can fix the SOURCE address, then DESTIN can lie in the range from 16384 to SOURCE-LEN1+LEN2 or from SOURCE+LEN2 to 65536-LEN1.

This is true on the condition that LEN1 is greater than LEN2 (i.e., if the block has compressed at least a little), otherwise these formulas do not apply. Moreover, if the decompressor is not separated from such a block, it will not be able to unpack this block correctly (for example, it will reset).

In conclusion, a few more words.

The modified UNPACKER works somewhat slower than the original (this is without considering the time taken by the IM 2 handler). If maximum speed is needed, you can do the following: launch the original UNPACKER with interrupts enabled...

To do this, it is necessary to separate the unpacking procedure from the block and modify its beginning similarly to my version. When calling, it is necessary to correct the SOURCE and DESTIN parameters so that the distance between these blocks increases by as many bytes as the IM 2 handler pushes onto the stack (or more). Let me explain a bit. The original DEPACKER sets the stack on the packed block and begins to unpack it to the DESTIN address, while the unpacked block grows upwards, and the packed one 'escapes' from it, decreasing in length. The bytes of the packed block are removed by POP commands. The most interesting thing is that the end of the unpacked block will never 'catch up' with the beginning of the unpacking, as a gap of 5 bytes is left between them. If we artificially increase this gap (by changing SOURCE or DESTIN), we will be able to save registers and return addresses during IM 2 processing. I would also note that I have not tested the second method, but the first one works successfully in one of my programs.

* * *

In addition from the editorial or as done in DEJA VU.

To avoid a long struggle and not to manually enter addresses, and if you do not need to work with interrupts enabled, perform the following manipulations:

1) Pack the code file with MS-PACK and be amazed by the compression ratio (if there are several files, of course, pack them all).

2) Run the following BASIC program:

10 CLEAR 25000
20 INPUT "MS:";N$
22 IF N$="" THEN GO SUB 100:GO TO 20
25 RANDOMIZE USR 15619:REM:LOAD N$ CODE
26 IF PEEK 23823 0 THEN GO TO 20
28 LET LEN=PEEK 23784+256*PEEK 23785
30 LET AD=PEEK 23782+256*PEEK 23783:
LET AD2=AD+36:LET AD3=AD+231
40 POKE AD3,PEEK AD2: POKE AD3+1, PEEK (AD2+1)
50 INPUT "NEW:";M$
60 IF M$="" THEN GO SUB 100:GO TO 50
70 RANDOMIZE USR 15619:REM:SAVE M$ CODE AD3,LEN-231
80 GO TO 20
100 RANDOMIZE USR 15619:REM:CAT
110 RETURN

3) Enter the name of the packed MS-PACK file (line 20), if such a file does not exist, nothing will happen; otherwise, it will prompt for the new file name; after entering the name, a new file will be written from the OLD ADDRESS+231, with a length of OLD LENGTH-231, i.e., 231 bytes shorter!

4) Create a universal decompressor by assembling the following listing in your favorite assembler:

ORG #5B00

LL5B00 LDDR ;or LDIR, see below
LD H,B
POP DE
EXX
POP HL
JR LL5B0E
LL5B08 DEC SP
POP AF
EXX
LD (DE),A
INC DE
EXX
LL5B0E ADD HL,HL
DJNZ LL5B13
POP HL
LD B,C
LL5B13 JR NC,LL5B08
LD E,#02
LL5B17 XOR A
ADD HL,HL
DJNZ LL5B1D
POP HL
LD B,C
LL5B1D RLA
ADD HL,HL
DJNZ LL5B23
POP HL
LD B,C
LL5B23 RLA
CP D
JR C,LL5B2E
ADD A,E
LD E,A
CP #11
JR NZ,LL5B17
XOR A
LL5B2E ADD A,E
CP #05
JR NC,LL5B5C
CP #02
JR NZ,LL5B5F
EXX
LD C,A
LL5B39 DEC SP
POP AF
LD L,A
LD A,E
SCF
SBC A,L
LD L,A
LD A,D
SBC A,H
LD H,A
LDIR
LD H,B
EXX
JR LL5B0E
LL5B49 EXX
DEC SP
POP AF
INC A
JR Z,LL5BAA
INC A
JR NZ,LL5B55
POP BC
JR LL5B61
LL5B55 ADD A,#0F
JR NC,LL5B60
INC B
JR LL5B60
LL5B5C JR Z,LL5B49
DEC A
LL5B5F EXX
LL5B60 LD C,A
LL5B61 EXX
ADD HL,HL
DJNZ LL5B67
POP HL
LD B,C
LL5B67 EXX
JR C,LL5B39
EXX
XOR A
LD E,D
LL5B6D ADD HL,HL
DJNZ LL5B72
POP HL
LD B,C
LL5B72 RLA
DEC E
JR NZ,LL5B6D
CP #02
JR NC,LL5B7F
INC A
LL5B7B EXX
LD H,A
JR LL5B39
LL5B7F ADD HL,HL
DJNZ LL5B84
POP HL
LD B,C
LL5B84 RLA
CP #08
JR NC,LL5B8C
DEC A
JR LL5B7B
LL5B8C ADD HL,HL
DJNZ LL5B91
POP HL
LD B,C
LL5B91 RLA
CP #17
JR NC,LL5B9A
SUB #09
JR LL5B7B
LL5B9A ADD HL,HL
DJNZ LL5B9F
POP HL
LD B,C
LL5B9F RLA
AND #1F
CP #1F
JR C,LL5B7B
DEC SP
POP AF
JR LL5B7B
LL5BAA LD HL,LL5BBC
LD C,#05
LDIR
LD HL,#0000
LL5BB2 EQU $-#02
EXX
LD SP,#0000
LL5BB6 EQU $-#02
EI ; or DI
RET ;
NOP ; or JP #nnnn NOP ;/
LL5BBC NOP
NOP
NOP
NOP
LL5BC0 NOP

;launching the decompressor with HL=address of the file without decompressor
DEPACK DI
LD (LL5BB6),SP
EXX
LD (LL5BB2),HL
LD C,#10
LD B,C
LD D,#03
EXX
LD SP,HL
POP HL
LD (LL5BE2),HL
POP HL
LD DE,LL5BC0
LD BC,#0005
LDDR
POP HL
POP DE
POP BC
LD SP,#0000
LL5BE2 EQU $-#02
JP LL5B00

Comments: the universal decompressor takes less than 256 bytes! Traditionally located in the printer buffer starting from address #5B00. (in this case, the launch address will be #5BC1=label DEPACK). At the entrance, pair HL must be set to the address of the loaded code block without the decompressor (what was obtained in point 3). To avoid problems with unpacked data overlapping with still unpacked ones, I recommend loading the file at the address specified in the directory! The unpacking will occur at the address that was specified in MS-PACK.

For example, we have a packed MS-PACK file:

"FILE" CODE 49152,10000;

Processing it in BASIC (points 2 and 3) will yield a new file without the decompressor:

"FILE+" CODE 49383,9769;

To unpack this file, place the decompressor at address #5B00 and unpack it as follows:

LD HL,49383
JP #5BC1 ;label DEPACK

The file will be unpacked at the address specified during packing in MS-PACK (for example #C000).

And finally, a small nuance - if the packed file turned out to be larger than the original during packing (and this can happen if we pack a small file or an already packed file - this is evident from the compression ratio given in percentages), then it is necessary to change the decompressor, placing not LDDR but LDIR at address #5B00! Thus, it makes sense to pack small files (2 sectors).

As a result, if we focus on the number of sectors, in 90% of cases it is possible to save one sector in the file!

If you use the method proposed by MAX, then in 96% of cases you save one sector! And most importantly - there is a choice! MS-PACK packs small text files well. If you are packing large code blocks, then use HRUST 1.0, it will surely show better results, and the possibility of saving the file without the decompressor is provided by the author.

Next, I pass the word to MAX.

-----------------------------------------******************************************
-----------------------------------------
Loading and launching BASIC files.

I think everyone knows how programs are usually launched in boots. The file name is thrown into the BASIC string (instead of spaces in the RUN command " "), then this string is executed. In this article, I will explain how to load and launch a BASIC program without using such tricks. The program in the listing is written with active use of TR-DOS ROM fragments, so do not be surprised by its awkward appearance.

Let there be a specific file that needs to be launched. Also, let’s assume that we know the header of this file (16 bytes). For loading, we will use some driver that can read 'B' sectors at address 'HL' from track and sector 'DE' (label LOAD). After loading, the driver should write the last track and sector to variable #5CF4. It is advisable to have CLEAR set to 65367. First, we will transfer the header to the address HEADER, equal to 23773. The file number should preferably be recorded in #5D1E. Then we run the following program:

BAS_RUN LD DE,(23635);start of BASIC program LD HL,(23641);address of the string DEC HL PUSH HL PUSH DE AND A SBC HL,DE ;in HL we got the length of the old BASIC program, ;i.e., the one that is (or is not) ;in memory before loading. LD DE,(HEADER+9) ;in DE we took from the header the length of the new BASIC, i.e., the one that needs to be loaded. PUSH DE PUSH HL LD HL,5 ADD HL,DE ;increased the length of the new program by 5 bytes. LD (23771),HL ;put it in the variable, as in TR-DOS. POP HL POP BC POP DE POP HL PUSH BC CALL #19E5 ;delete the old BASIC by shifting memory. POP BC CALL #1655 ;reserve space for the new one. INC HL LD BC,(HEADER+11) ;took from the header the length of the new BASIC ;without program variables. ADD HL,BC ;added the start address. LD (23627),HL ;set the system variable VARS. LD HL,(23635) ;HL - where we will load. LD A,(23772) ;high byte of the length. LD B,A LD DE,(HEADER+14) ;DE - track and sector. LD (#5CF4),DE DEC B INC B CALL NZ,LOAD ;if there is something to load - load it. LD A,(23771) AND A JR Z,NOLOW ;is the low byte of the length equal to zero? PUSH HL LD DE,(#5CF4) LD HL,BUF LD B,1 CALL LOAD ;loaded the last sector into the buffer. POP DE LD BC,(23771) LD B,0 LD HL,BUF LDIR ;the rest was transferred where needed. NOLOW LD HL,(23641) DEC HL LD (HL),128 ;for safety, wrote the end marker. INC HL INC HL LD E,(HL) INC HL LD D,(HL) ;took the auto-start string LD (23618),DE ;and wrote it to the corresponding variable. XOR A LD (23620),A ;zeroed the operator number in the string. LD (23824),A CALL #16B0 ;I don't exactly remember what this is, it seems to clean the ;calculator stack. LD HL,(23635) DEC HL LD (23639),HL ;made RESTORE. LD HL,(23613) LD (23827),HL EXX LD HL,10072 EXX LD IY,#5C3A LD (IY),255 SET 7,(IY+1) RES 5,(IY+1) ;other settings. JP 7037;launch... BUF DEFS 256;loading buffer.

It is better to place this program in the screen area. It would be nice to clear the screen before transferring control to the ROM, as TR-DOS does. For example, instead of JP 7037:

LD HL,7037
PUSH HL
JP 3435

In conclusion, I want to express my gratitude to Fedin P.Yu. for the book "Complete Description and Full Disassembler TR-SOS 5.04T (5.03)".

Contents of the publication: Deja Vu #05

  • Аперативчик - Max
    Detailed instructions on managing the DEJA VU interface, highlighting different input methods and navigation commands. Explanation of the new and old interfaces for enhanced user experience. Discussion on additional features like frame scrolling and music management.
  • Аперативчик - Max
    Discussion on supporting machines with more than 128k memory, leading to separate shells for 128k and 256k systems. Testing was mainly done on Scorpion and Profi, with functionality on other models anticipated. Article includes guidance on unpacking source files and insights on using improved algorithms.
  • Тема - M.M.A
    This article explores the theory behind digitizing sound on ZX Spectrum, focusing on sampling and quantization processes. It provides practical insights into converting sound files using specific hardware and software. Additionally, it offers methods to enhance sound quality while working within the hardware limitations.
  • Theme
    The article discusses the Save Our Scene initiative aimed at uniting Spectrum users and developers to promote software distribution and enhance the scene's development.
  • Charter of the Amazing Soft Making Association
    Discussion of the founding charter of the Amazing Soft Making association, detailing its goals, membership criteria, and operational principles.
  • Theory of Magazine Creation
    The article provides a detailed guide for aspiring magazine creators, focusing on technical aspects such as interface design, memory management, text formatting, and music integration for ZX Spectrum publications.
  • Solder Drop
    The article provides a personal account of purchasing and using the General Sound device for ZX Spectrum, detailing installation and sound performance. It discusses the initial issues encountered and praises the enhanced audio experience in compatible games. The author encourages further software adaptation for the device and reflects on multimedia capabilities with simultaneous hardware use.
  • Solder Drop
    The article discusses the capabilities of Sound Forge 4.0c for professional audio processing on PCs, highlighting its extensive features such as sound editing, effects, and restoration tools.
  • SOFTWARE
    The article reviews the latest software developments for the ZX Spectrum from Samara, including updates to MAXSOFT SCREEN PACKER, File Commander, and new applications like S-Terminal.
  • SOFTWARE - Card!nal
    Review and walkthrough of the logical graphic adventure game 'Operation R.R.' with detailed level instructions. Discussion on game elements like music choice and graphic design. Mentions new coder MAX/CYBERAX/BINARY DIMENSION's involvement.
  • SOFTWARE
    Discussion on the current state and evolution of the demoscene, highlighting the rise of 4K intros and upcoming competitions like FUNTOP'98.
  • CODING
    Article discusses assembly language coding techniques for optimizing screen scrolling on ZX Spectrum, featuring example code and performance analysis.
  • CODING - RLA
    The article explores stack manipulation techniques during second type interrupts for graphical effects on ZX Spectrum. It discusses solutions for preserving data integrity when interrupts disrupt graphical operations. Practical examples are provided to handle stack issues efficiently.
  • CODING
    The article describes the MS-PACK packer and its DEPACKER, detailing usage scenarios and providing BASIC and assembly code examples for handling packed files. It emphasizes optimizing performance by allowing unpacking with interrupts enabled and separating the DEPACKER from packed files. Additionally, it includes insights on programming techniques for loading and executing BASIC files on ZX Spectrum.
  • CODING
    The article discusses various coding techniques for ZX Spectrum, focusing on sprite rendering, rotation algorithms, and optimization methods to enhance performance.
  • ANOTHER WORLD
    Discussion on the evolution of multimedia technologies and their impact on various fields, including education and entertainment. It covers advances in computer hardware and software that have facilitated the integration of audio, video, and text. The article reflects on past developments and speculates on the future of multimedia systems.
  • ANOTHER WORLD
    Comparison of PC and Amiga systems highlighting performance, software costs, and user experience with multimedia capabilities.
  • Honor Roll
    Interview with PROGRESS discusses their creative journey on ZX Spectrum and AMIGA, addressing challenges in demomaking and the current state of the scene.
  • Honor Roll
    The article details the activities and future projects of the Eternity Industry team, based in Kovrov, including successful releases and collaborations with other groups.
  • Honor Roll
    Discussion of the Artcomp'98 festival, focusing on its mail-in format and guidelines for various competitions, including demo, graphics, and music categories.
  • Honor Roll
    The article provides a glossary of terms used in the demo scene, explaining roles such as musician, coder, and graphician, as well as different types of demos and effects. It serves as a useful resource for understanding the terminology and dynamics of the community. This is a descriptive piece aimed at educating readers about the jargon of the demo scene.
  • Honor Roll
    The article discusses the issues with mouse support in various ZX Spectrum magazines and the frustrations of users when encountering compatibility problems. It critiques developers for not adhering to standards, leading to poor user experiences. The author expresses the importance of consistent improvements in software for the ZX Spectrum community.
  • Honor Board
    The article discusses the process of creating tricolor images for ZX Spectrum using Photoshop and a simplified approach. It outlines how to divide an image into RGB channels and convert them for use on the Spectrum. Additionally, it provides tips on how to manage the files for optimal results.
  • Honor Roll
    The article discusses the comparison and perspectives on various computer systems, particularly emphasizing the strengths of AMIGA over PC and advocating for appreciation of all machines.
  • Seven and a Half
    Article discusses the humorous absurdities and peculiarities of military training and academia, blending satire with real anecdotes and witty observations.
  • Seven and a Half
    The article provides a satirical manual on programming methodologies, mocking the rigidity of formal programming practices and advocating for a more creative approach to coding.
  • Seven and a Half
    Instructions on safe sex practices, including guidelines on eligibility, preparation, and actions during and after the sexual session, along with handling emergency situations.
  • Seven and a Half
    The article discusses a call for a talented artist in Krasnodar for a ZX Spectrum group, raises concerns about the unethical practices of Scorpion regarding software rights, and critiques a video review of E'97.
  • Seven and a Half
    The article 'Семь и 1/2' narrates a humorous picnic adventure involving the editorial team of Deja Vu, highlighting their camaraderie and mishaps while preparing a barbecue.
  • Trial of the Pen
    The article is a humorous take on the fictional adventures of Winnie the Pooh as he interacts with computers and friends, discussing the absurdities of technology and daily life.
  • First Pen
    The article discusses the new section in Deja Vu dedicated to fantasy and science fiction literature, featuring book reviews and reader participation in content creation.
  • Advertisement
    The article is an advertisement section from Deja Vu #05, promoting collaborations with designers and musicians for future issues, and offering various software and hardware for ZX Spectrum.
  • News
    The article announces the launch of a new magazine, AMIGA RULES, focused on the AMIGA computer, addressing the lack of quality Russian-language publications. It aims to provide information on programming, hardware, software, and gaming, while fostering a community among AMIGA enthusiasts. The magazine will include contributions from readers and regular updates on the AMIGA scene.