Sometimes, though, it’s necessary to tell the CPU to stop executing a program, just for a moment, and direct its attention elsewhere. Perhaps some piece of hardware needs servicing: a user pressed a button on the keyboard or moved the mouse; a network request has completed; a disk read operation failed. It’s also possible that we want the CPU to execute more than one program at a time, switching between them every few microseconds. To do that, we set an alarm clock that rings, interrupting the CPU from its work.

In this section, we’ll look at what interrupts are.

This article is part of a series on toy operating system development.

View the series index

The dedicated CPU

Once you power on the CPU, it is a machine of sheer single mindedness. It operates in a simple loop: read instruction from memory, decode instruction, execute instruction, move pointer to next bit of memory, repeat. It never stops doing this; you’ll have to pull the plug to get it to stop - or feed it an instruction it doesn’t know what to do with. Short of these, nothing will distract it from its work.

Sometimes, though, the world around the CPU needs its attention.

• The CPU is surrounded by hardware that it has no knowledge about. The CPU doesn’t know what a keyboard is, or a mouse, or a disk drive. When you press a key on the keyboard, then this has no meaning for the CPU; it continues executing instructions. What’s needed is for the CPU to notice that something in its environment happened, and needs attention.
• Operating systems execute multiple processes at the same time. They do this by having the CPU execute some instructions from process A, then stop it, and set it work on process B. The CPU is moved from process to process every few microseconds - a concept known as scheduling. What’s needed is a way to set an alarm clock; when it rings, the CPU has to move to the next process. The process switching happens so frequently that to the user it appears that the CPU is executing all processes simultaneously.
• It’s possible for the CPU to encounter an error. It may, for example, try to divide by zero. That’s not the CPU’s fault; it is the program that it’s executing that made it attempt the division. Dividing by zero is impossible, and sends the CPU screaming. But who does it scream to? Itself, actually: it needs a way to stop attempting to divide and execute some other code that allows it to recover from the error.

Interrupting the CPU

The way to get the CPU’s attention is by interrupting it. When something of note happens to a piece of hardware, that hardware talks to a special chip known as the Programmable Interrupt Controller (PIC), sending it an Interrupt Request (IRQ). The PIC then communicates this request to the CPU. Receiving this interrupt, the CPU then stops doing what is was doing, acknowledges receipt to the PIC which in turns sends a vector (a number: one value for the keyboard, another value for the mouse, and so on). The CPU then executes a special bit of code to service the interrupt, appropriately known as an interrupt service routine. When it’s done, the CPU goes back to what it was doing before it was interrupted.

Note: in modern computers the 8259 PIC has given way to the APIC (“Advanced”): a local APIC on each CPU core, plus one or more I/O APICs for the devices. PCIe devices go further still, posting their interrupts as writes to memory (while PICs use a controller pin). For the theory in this post, none of this makes any difference.

The second use case - an alarm clock that interrupts the CPU so that it can tend to the next process that’s scheduled - works in a similar way. The kernel programs a special chip to fire off an interrupt every few microseconds, the CPU gets interrupted, and the interrupt service routine is written to call the kernel again, which will move some registers around so that the CPU will start executing the next process.

There are a number of errors that can happen during program execution: divide-by-zero is the typical example, but other things include an invalid opcode that the CPU doesn’t know what to do with, a stack overflow, or reading or writing to a region of memory where a process isn’t allowed. These will need dealing with by an operating system kernel: retry, or halt the offending process. On this occasion it is the CPU that fires off an interrupt, interrupting itself, so that an interrupt service routine (installed by the kernel) can run in order to put things right.

There’s a fourth use case still: a user program needing to talk to the kernel, in order to request a service (a software interrupt). The kernel knows how to speak to hardware on the program’s behalf, for example, and it can execute code at a higher privilege level than the user program. In previous chapters, we’ve already used the INT assembly instruction to call the BIOS. We’ve asked it to print a character to the screen, wait for a keypress, and produce a memory map. The difference is that we’ve moved to protected mode, and talking to the BIOS is out the window - we’ll have to provide all of these services ourselves, but more on that later.

The interrupt service routine

Whenever the CPU gets interrupted, then, it will execute an interrupt service routine. As we’ve seen, this can be the result of:

• Hardware interruption of the CPU (this includes the scheduling alarm clock)
• A program execution error
• A program executing the INT instruction

Each interrupt has a number. That number is an index into a table sitting somewhere in memory. The table is known as an interrupt vector table: for each interrupt, it contains the memory address of its interrupt service routine.

Note: the shape of the interrupt vector table has changed over the years, but the idea behind it hasn’t. A real-mode CPU has 256 entries of four bytes each, sitting at address 0x0, and each one is just a segment + offset. A protected-mode CPU has 256 gate descriptors instead, each containing more than just an address. Additionally, the modern table can live anywhere in memory, and we use LIDT to tell the CPU where (which we’ve done before).

The interrupt service routine is a piece of code which can do whatever it wants, subject to two constraints.

• The routine must leave the machine exactly as it found it: every register it touches has to be saved and put back, because the interrupted program doesn’t kow ut was interrupted and will carry on using those registers as though nothing had happened.

• An interrupt arrives between two arbitrary instructions, so an ISR may well have interrupted code that was halfway through updating a data structure. It cannot assume that anything it looks at is in a consistent state, and it must never wait for a lock that the interrupted code might already be holding - that code is not running, and will not release the lock until the ISR returns. A routine which can safely be entered while an earlier invocation of it is still in progress is called reentrant.

An ISR is special in one other way: while an ordinary subroutine returns with RET, an ISR uses IRET. This is because the CPU, when it gets interrupted, leaves a return address on the stack automatically and pushes the flags; IRET pops them off again. This is important because the interrupted program’s carry and zero flags have to survive, as does the interrupt-enable state.

The return address is a little bit special, and is different for three kinds of interrupts:

• For a fault, the saved address points at the offending instruction, so that it can be retried once the condition has been fixed. A page fault is the standard example: map the missing page, IRET, and the instruction now succeeds.
• For a trap, the saved address points after the instruction. Breakpoints, overflow, and the software INT instruction all behave this way.
• For an abort, there is no reliable address at all. A double fault is an abort: there is nothing sensible left to return to.

The barest (and not useful) implementation for interrupts is therefore a table of pointers, all the same, and at that pointer lives the IRET instruction. Or HLT, in case of an error.

Note: some of the error interrupts push an extra word onto the stack describing what went wrong, so this word must be removed before IRET can return to the correct place.

Interrupt masking

There’s a detail worth mentioning at this point: the CPU has special instructions allowing it to ignore interrupts. This is known as interrupt masking, and it’s the equivalent of the CPU inserting a finger in each ear and going “LA-LA-LA” until the masking is turned off. The CLI instruction turns off interrupts, and STI turns them back on again; instructions we have already used in the boot loader.

Interrupts are also masked automatically while an interrupt service routine is running, otherwise an interrupt could interrupt an interrupt and most of the time, we won’t have that. In protected mode this turns out to be a property of the table entry rather than of interrupts in general: an entry can be marked either to mask interrupts on the way in or to leave them alone.

Not everything can be masked, though. Some events are too serious to be ignored at the CPU’s convenience (imminent power loss, or a memory parity error) and these arrive as a non-maskable interrupt, which CLI cannot switch off. When one of those turns up, something is wrong with the machine rather than with the program it’s running.

Summary

In this article, we’ve explored what interrupts are: a way for either software, hardware, or a program error to make the CPU stop executing program code and jump to an interrupt service routine, returning to its original job when it’s done. When we first enter protected mode, there aren’t any interrupt service routines yet, so the CPU has no idea what to do when an interrupt is fired. In the next article, we’ll see how to set up an interrupt vector table.

In this section, we’ll be looking at the system’s memory map. As we’ll soon need to write memory management (to be done by our brand new kernel), we’ll need to know which areas of memory we can touch, and which are off-limits. Overwriting the kernel itself, for example, would be bad, and overwriting the global interrupt table would bring the system down, as well.

This article is part of a series on toy operating system development.

View the series index

Memory map

After we hit protected mode, all memory of the system becomes available to our kernel: vast fields of bytes, ready to be written, read, shifted, or fondled in other ways. But not all memory is equal! Some memory can’t be written to (ROM), and it’d be bad if we tried. Other memory is directly mapped to devices: the VGA card, for example, is tied to a specific area of memory. But there’s more: our kernel doesn’t even know how much memory our system has. It literally can’t see the end of those vast fields of bytes.

Which areas of memory are actually not available to us is something that varies from machine to machine. A different graphics card would use a different area of memory (CGA cards where at 0xB800, VGA at 0xA000). Our kernel doesn’t know these things - but the BIOS knows. The BIOS has a special interrupt that gets us a nice memory map: a list of sections, each marked as usable or reserved.

There’s but one catch: this is a BIOS interrupt, so it has to be called from real mode. This means we must call it before we ever go into protected mode, and so the code that we write in this article must be inserted into our second-stage bootloader. While that’s not a problem, we must also find a way to pass the memory map we obtain to the kernel, which will make use of it. This means we must place the map somewhere in memory - temporarily - and let the kernel find it there.

The interrupt

The interrupt that’ll get us our memory map is int 0x15, subfunction 0xe820. It’s documented here and here

The input is:

And the call will return:

The Address Range Descriptior that the BIOS will place at es:di looks like this:

… where “type” can be one of:

A “reserved” area of memory could be ROM, RAM in use by ROM, or memory mapped to a system device.

ACPI 3.0

In 2004, the Advanced Configuration and Power Interface specification (ACPI) added (among other things) a change to the memory map interrupt: rather than returning 20 bytes, it would now return 24 bytes. The last 32 bits have the following meaning:

A proper kernel will have to take these ACPI improvements into account, but also deal with older BIOSes that don’t carry them. More ifs and thens, therefore.

Getting the map - a recipe

Right, so there’s one interrupt, which we need to call repeatedly, until the return value indicates that there are no more ~lands to conquer~ memory runs to be found. We also need to take a potential ACPI extension into account. Note that the OSDev wiki’s page for memory detection discusses still other interrupts that might be used to detect memory, but interrupt 0x15/E820 is 99% of the process - since we’re not currently trying to build the new Linux, we’ll stick with it.

Our steps therefore:

• Select an area of memory where the memory map must be stored as it comes in from the interrupt. We must be careful not to overwrite other things. However, this is temporary: once the kernel is booted, it’ll move the memory map to a different, more suitable location.
• Call the interrupt
• Check the interrupt’s return value:
  • If the buffer > 20 bytes, then there is ACPI data. If not, assume the ACPI value for a usable memory run.
  • If the buffer is 0 bytes, the call returned no run and we continue.
  • If the Address Range Descriptior has a length of 0, then the BIOS reported a zero-byte run of memory. We’ll ignore it and continue.
• Advance the pointer where the interrupt stores descriptors.
• If a continuation value is present, do it all again.

Memory structure

We’re collecting our list of physical memory runs in the second-stage boot loader. This information has yet to make it to the kernel, which at this point hasn’t booted yet. We must therefore create a contract between the bootloader and the kernel: a precise definition of what the bootloader will place in memory, and where. In addition to the address range descriptors themselves, we’ll also want to tell the kernel how many descriptors were found. Let’s also stick a magic value on top, just to the kernel can verify that the memory map arrived to it unmolested.

In C terms, this’ll look like this:

struct bootinfo
{
  uint32_t magic;           // BOOTINFO_MAGIC
  uint32_t e820_count;      // zero means the BIOS returned nothing
  uint32_t e820_addr;       // -> struct e820_entry[e820_count]
  uint32_t boot_drive;      // what the BIOS put in dl at 0x7C00
} __attribute__((packed));

struct e820_entry
{
  uint32_t base_low,   base_high;
  uint32_t length_low, length_high;
  uint32_t type;
  uint32_t attrs;
} __attribute__((packed));

As for where to store this information… here is a list of what we’ve got going in memory up to the point that we start the kernel:

A diverse collection of things, but it’s important to note that virtually all of it is important only to the second-stage bootloader. It loaded an IDT and GDT just to be able to switch to protected mode. It has its own stack, we have the boot sector floating around as loaded by the BIOS, the FAT table, and the second stage’s own code. All of this can disappear as soon as the kernel runs, and the memory it occupies can be released.

I’ve picked 0x8000 to place the bootinfo structure, for the simple reason that it is free. The kernel will know where to pick it up, and then move it to somewhere more sensible - plenty of options, since just about everything else will disappear. The kernel will create its own IDT and GDT, and store them where it wants.

Calling the BIOS

Let’s actually call the BIOS then, and get down to assembly code. We’ll define a few things:

BOOTINFO_ADDR  = 0x8000        # 32 bytes (only 16 used)
E820_ADDR      = 0x8020        # memory map

# 32 entries of 24 bytes = 768 bytes, ending at 0x8320. It's not likely that
# real machines will have more entries, but the kernel will warn if that
# happens.
E820_MAX       = 32 # entry count
E820_ENTRY_SZ  = 24 # entry size
SMAP           = 0x534D4150    # 'SMAP'; magic signature that the BIOS returns
                               # to report the call went well.
# A little safety check. The pointer we pass to the kernel is used much later,
# so this helps us check that information wasn't clobbered anywhere.
BOOTINFO_MAGIC = 0xB007B007

These are all values we’ve gone over above, defined as constants for readability further on. Neat!

The call to the BIOS, then is done in a loop, since we must retrieve multiple runs of memory until the BIOS says that there are no more. We’ll do a quick setup:

xor   ax, ax                       # ax=0
mov   es, ax                       # es:di is then a linear address
mov   di, E820_ADDR                # es:di points to the memory map we'll build
xor   ebx, ebx                     # ebx=0: first call, not a continuation
xor   si, si                       # si=0: 0 entries stored so far
                                   #   (not part of interrupt call)

Now for the loop:

detect_memory__loop:
  mov   eax, 0xE820                  # Interrupt subfunction
  mov   edx, SMAP                    # magic code 
  mov   ecx, E820_ENTRY_SZ           # ask for 24 bytes: 20 of range, plus the
                                     # ACPI 3.0 attribute word
  int   0x15

Carry flag means end-of-list, or that E820 is not supported. We stop:

  jc    detect_memory__finished

Let’s check that the magic code (“SMAP”) is present on the response. Otherwise we give up:

  cmp   eax, SMAP                    # the BIOS must echo 'SMAP' back
  jne   detect_memory__finished      # If no 'SMAP', then done.

If the buffer size is 0, it’s not a valid run of memory so we skip it:

  test  ecx, ecx                     # cx = 0?
  jz    detect_memory__next          # skip

A little special treatment for ACPI 3.0. We’ve asked for 24 bytes to be returned; if the BIOS returns only 20, then there’s no ACPI extension. In that case we just pad the entry to 24 bytes and consider it “in use”.

  cmp   ecx, 20
  jne   detect_memory__have_attrs
  mov   es:[di+20], dword ptr 1      # A word value of 0x0001: memory in use.
detect_memory__have_attrs:

The BIOS may return memory runs of length 0. While these are valid, they are not useful to us and we skip them:

  # Zero-length regions may be returned but we don't want them. Skip.
  mov   eax, es:[di+8]               # length, low half
  or    eax, es:[di+12]              # ... or high half
  jz    detect_memory__next          # skip if eax=0

We come to the end of the loop. We keep track of how many runs we’ve found so far, and if there’s a continuation value, we jump to the loop again. Otherwise, we are done:

  inc   si                           # +1 areas found
  add   di, E820_ENTRY_SZ            # Move to next position in memory map

detect_memory__next:
  test  ebx, ebx                     # Continuation value of 0: last entry
  jz    detect_memory__finished      # so we are done.
  cmp   si, E820_MAX                 # out of room; the kernel will warn
  jb    detect_memory__loop          # LOOP AGAIN if there's room

At this point, we end up with the memory map in memory at our desired address, i.e. 0x8020. We’ll now prepend the boot information at 0x8000.

  movzx ebx, byte ptr iBootDrive

  xor   cx, cx                                 
  mov   es, cx                                 
  mov   di, BOOTINFO_ADDR                      # es:di=0:BOOTINFO_ADDR

  mov   es:[di],    dword ptr BOOTINFO_MAGIC
  movzx ecx, ax
  mov   es:[di+4],  ecx                        # e820_count
  mov   es:[di+8],  dword ptr E820_ADDR        # e820_addr
  mov   es:[di+12], ebx                        # boot_drive

… and that is that: the full structure is now sitting at 0x8000.

Handing over to the kernel

The structure’s ready, but we’re not quite done yet: the kernel has no idea that the boot info structure exists, much less where it is. This is easy to remedy: the kernel code simply reads the memory at 0x8000, and consults the memory map there. That’s not a very robust approach though: if we ever change the code to place the memory map somewhere else, the both the bootloader and the kernel must be updated. It’s prettier to pass a pointer to the kernel, so that the structure placement is entirely the bootloader’s decision.

.macro mJumpToKernel bootinfo
  mov ebx, \bootinfo
  jmp 0x08:0x20000
.endm

This give the kernel an additional responsibility though: it must not change the ebx register until the pointer is used. When the time comes to jump to C code, we simply push ebx onto the stack and call:

  push   ebx
  call   kernel_main

and the kernel’s C entry point looks like this:

void kernel_main(struct bootinfo *bi)

Extra safeguards

When making changes to the bootloader and the kernel’s assembly entry point, it’s easy to mess up this fragile way of handing the kernel a pointer. We may inadvertently change the pointer value, or overwrite the memory it points to, and it’d be difficult to debug when that happens. For this reason, and extra bonus points, we place a magic value at the boot info structure’s address, just before the actual boot info - then check it later:

if (handed_over == 0 || handed_over->magic != BOOTINFO_MAGIC) {
  kprintf("bootinfo: BAD MAGIC at %p - no memory map\n", (uint32_t)handed_over);
  return 0;
}

Getting the information into the kernel

At this point, our boot info structure lives at address 0x8000, and continues to be there as we load the kernel into memory, set up the IDT, GDT, and jump to protected mode, finally long-jumping to the kernel’s C code.

Before doing anything else in the kernel, we need to “adopt” the boot info structure, moving it into memory the kernel owns. We do this by copying the data into the kernel’s .bss section, which is the section for uninitialized variables. With that done, the data is safe. We can then mark the area at 0x8000 as “available”!

Summary

Somewhere down the line, we’ll want our kernel to manage the system’s memory, handing it out to processes as they start, and checking that those processes aren’t naughty and try to write beyond their allocated memory. Before we’re in a position to do so, however, the first thing to do is to create a map of the existing physical memory: how much of it is there, and which parts are reserved? Some of the memory is ROM, other memory is mapped to physical devices, yet other memory might be faulty. The BIOS can provide us with this information, so it is important we call it while we’re still in real mode - in protected mode, we can’t talk to the BIOS anymore.

In this section, we show how to call BIOS interrupt 0x15/E820 repeatedly to populate a physical memory map (in real mode). After we jump to protected mode, we then have the kernel move it to an area of its choosing, finally freeing up all the memory that was used by the second-stage boot loader, boot sector, and FAT tables.

Continue on to the next part of this guide!

The problem

There are a number of (paid) APIs on the internet that you can call to get data for a single lat/lon point on earth. You can obtain, say, ocean salinity at that point, or wave height, or wind direction and speed. But in order to show data on a map, you’ll want a grid of points, probably as many as 100x100, to render a smooth surface on your map. That would mean calling a point API 10,000 times, which is (a) very slow an (b) likely costly.

What’s needed here is a grid of points that you can download in a single API call, and then you do the interpolation between these points to present a smooth surface. This shouldn’t be hard to get - data offered by NOAA’s GFS, ocean state from HYCOM, air quality from Copernicus CAMS, elevation from GEBCO - it’s all already in grid form, just not in a suitable format for browser consumption. It’s heavy, world-spanning NetCDF files, or GRIBs. The missing piece is a way to get this data into the browser efficiently.

Values baked into pixels

The way to render data on a map is through map tiles. Using tiles, a massive raster can be delivered to the browser in a lazy way: when the users zooms, or pans (sufficiently), the map requests map tiles from the server and renders these. Only the data for the region that the user is looking at is downloaded, and only at the required zoom level. At any given time, 5-10 map tiles are downloaded (depending on the user’s screen resolution), and the total download is usually not more than ~100KB.

With map tiles established as the way to get data onto a map, the trick then is to pour all the NetCDF and GRIB data from different providers into map tiles, regularly (with data updated every 6 hours). Having such tiles at hand already solves part of the problem. But there’s a twist: we do better by not producing raster tiles that show an immutable image, but by encoding values into the image’s pixels. If each pixel doesn’t represent a color but a value, then the browser can decide how to map that value to a color on the user’s map.

This means that a dataset such as air temperature could be rendered in shades of blue, or using a red gradient, or a rainbow gradient, and the user could switch between these colors on the fly. All that’s necessary is to put a bit WebGL in the rendering pipeline that translates values to colors. Better yet, wind or ocean current speed tiles could encode not just scalar values, but vector u and v values. With wind direction known, a bit of WebGL code can render animated particles on a map, fast!

This approach isn’t new: Mapbox’s terrain RGB uses this, too. What’s new is applying it to a variety of datasets (ocean current speed, wind speed, water temperature etc.) and updating the information every 6 hours. Elevation can be offered as well (Mapbox-style), and doesn’t need refreshing every 6 hours, of course.

About Mercator

The product that offers all this is mercator.blue. It has two halves: the first is the tile service: an HTTP endpoint serving the value-encoded tiles plus a metadata.json that fully documents the encoding, so any developer can decode the pixels without my code. The second is an open-source SDK that ships the shader decoders and ready-made visualisations (colormapped rasters, animated wind and current streamlines, arrows, contours, value labels) for MapLibre, Mapbox, Leaflet, OpenLayers, deck.gl and React. If you decide to use one of these SDKs, then a few lines of code are enough to get an animated wind field or ocean current particles on a map.

Mercator offers a catalogue of datasets, all updated every 6 hours (with the exception of elevation, which is static). There is atmospheric weather, ocean currents and waves, air quality and elevation, all including forecasts (12h ahead, 24h ahead, 48h ahead). The whole thing follows the same shape as Mapbox GL JS or Deck.gl: the SDK is free and open (Apache-2.0), and the paid part is the data API, with a free tier of 10,000 tiles a month. Because the tile format is documented, you are never locked to my renderer.

Try it out

You’re welcome to test drive both the datasets and the SDK. The SDK is open source; the tiles have a free tier that allows you to experiment all the datasets (it’s 10K tiles free).

There is a live interactive globe on the homepage with wind streaming over it, a live playground for each framework, and the full tile-encoding and API docs if you want to decode the tiles yourself.

Using a 3rd-party service?

If you don’t want to code the email sending yourself, you can reach for a third-party solution. Mailgun, Sendgrid and Mailchimp come to mind, but they all cost money. Mailgun has a free tier, but it’ll only work for sending out email to up to 5 known email addresses, and that’s not what you want - you want to send email out into the world (but only one email at a time). Even if you did pay for a third-party solution, it may have a free tier of up to, say, 1,000 emails per month, after which you start paying. God forbid that your API key falls into the wrong hands and someone sends out a million emails with it!

Using your own server

In my case, I have a shared-hosting server that runs PHP. Through CPanel, I am able to create email addresses, so I can go ahead and create info@myserver.org and use that to send email from.

Problem 1: Email sent manually from my server does not arrive

With so many spammers around, email has become something like the Wild West. Anyone can send out emails, and users are bombarded with spam. To fight some of this, email providers try to block emails where it can’t be determined where the email came from - in which case it could be a spoofed address.

Sending out a simple email manually from my CPanel email client to a GMail address will result in this (I’ll get an email back from CPanel or from Gmail):

550-5.7.26 This message does not pass authentication checks (SPF and DKIM both 
550-5.7.26 do not pass).
550-5.7.26 This mail is unauthenticated, which poses a security risk to the 
550-5.7.26 sender and Gmail users, and has been blocked. The sender must 
550-5.7.26 authenticate with at least one of SPF or DKIM. For this message, 
550-5.7.26 DKIM checks did not pass and SPF check for [example.com] 
550-5.7.26 did not pass with ip: [x.y.z.n]. The sender should visit 
550-5.7.26 https://support.google.com/mail/answer/81126#authentication for 
550 5.7.26 instructions on setting up authentication.

In other words, if your domain doesn’t have SPF and DKIM records, GMail (and likely others) will reject your mails straight out of the gate.

A solution is brought by CPanel itself: the Email deliverability option.

If you have no DKIM or SPF records setup, then the Email Deliverabilty screen will report that there is a problem and offer to repair it:

If your DNS is actually under CPanel’s control, you can CPanel create SPF and DKIM records automatically. If not, and like me, you have your domain at GoDaddy while you’re hosting somewhere else, it’s not going to be automatic. But never fear, CPanel can still help. If you choose the Manage option, CPanel will tell you that no DKIM record exists, and provide you with a suggested record:

This system does not control DNS for the “myserver.org” domain and the system did not find any authoritative nameservers for this domain. You can install the suggested “DKIM” record locally. However, this server is not the authoritative nameserver. If you install this record, this change will not be effective. Contact your domain registrar to verify this domain’s registration.

Suggested DKIM (TXT) record:
Name = default._domainkey.www.myserver.org.
Value = v=DKIM1; k=rsa; p=a-long-encrypted-value;

Suggested SPF (TXT) record:
Name = www.myserver.org.
Value = v=spf1 +mx +a +ip4:(ip number) ~all

You can actually take these values to GoDaddy and create the DNS records yourself. Note that both need to be TXT records. Create these and save them, wait a few minutes, and then manually send an email again. It should now arrive without issue into a GMail inbox.

Getting Laravel’s mailer to behave

With the TXT records configured, I found that while I could send emails manually through the email web client and receive them, emails from PHP would not be received.

It turns out that for Lumen, after configuring all that’s necessary for Illuminate/mail, no email would be sent. I also found that the email sending route returns rather quickly; ordinarily it would be processing a little while before returning.

I there checked my STMP settings using the GMass SMTP Test Tool which came in handy. I found that I was able to send email email through this and receive it in a GMail inbox, so my SMTP settings were correct. I also noted that sending the email took a few seconds, so it was suspicious that the Laravel code returned immediately.

Finding no way to get the Laravel code to tell me why it failed, I installed the venerable PHPMailer instead. Testing it out by pasting its sample code into my back-end implementation, it immediately worked.

Stepping away from Laravel’s SwiftMailer was made easier by using Laravel’s ENV variables in PHPMailer instead of using hardcoded values (these are the settings from the .env file):

$mail->Host       = env('MAIL_HOST');            // Set the SMTP server to send through
$mail->SMTPAuth   = true;                        // Enable SMTP authentication
$mail->Username   = env('MAIL_USERNAME');        // SMTP username
$mail->Password   = env('MAIL_PASSWORD');        // SMTP password
$mail->SMTPSecure = PHPMailer::ENCRYPTION_SMTPS; // Enable implicit TLS encryption
$mail->Port       = env('MAIL_PORT', 2525);      // TCP port

Further, it’s still possible to use Laravel’s Blade templates by rendering them to string:

$mail->isHTML(true);
$mail->Subject = 'Database password reset';
$mail->Body    = view('reset', compact('user'))->render();
$mail->send();

I know I’ve had Laravel’s SwiftMailer work for me in the past, but PHPMailer does me the courtesy of letting me put a try…catch around the mail sending process so that I’ll know when it’s failed. Also, a full SMTP debug can be echoed to the output to see precisely what went wrong.

DENTHOR, coder for ...
_____   _____   ____   __   __  ___  ___ ___  ___  __   _____
/  _  \ /  ___> |  _ \ |  |_|  | \  \/  / \  \/  / |  | /  _  \
|  _  | \___  \ |  __/ |   _   |  \    /   >    <  |  | |  _  |
\_/ \_/ <_____/ |__|   |__| |__|   |__|   /__/\__\ |__| \_/ \_/
smith9@batis.bis.und.ac.za
The great South African Demo Team! Contact us for info/code exchange!  

Free Direction Texture Mapping

There are two things you should know before we begin.

Firstly, I am cheating. The texture mapping I am going to show you is not perspective-correct, with clever divides for z-placement etc. This method looks almost as good and is quite a bit faster too.

Secondly, you will find it all rather easy. The reason for this is that it’s all rather simple. I first made the routine by sitting down with some paper and a pencil and had it on the machine in a few hours. A while later when people on the net started discussing their methods, they were remarkably similar.

Let me show you what I mean.

Let us assume you have a texture of 128x128 (a straight array of bytes [0..127, 0..127]) which you want to map onto the side of a polygon. The problem of course being that the polygon can be all over the place, with one side longer then the other etc.

Our first step is to make sure we know which end is up. Let me demonstrate:

                      1
                    +
                 /    \
              /         \
          4 +            +  2
              \        /
                \   /
                  +
                  3

Let us say that the above is the chosen polygon. We have decided that point 1 is the top left, point 3 is bottom right. This means that

• 1 - 2 is the top of the texture
• 2 - 3 is the right of the texture
• 3 - 4 is the bottom of the texture
• 4 - 1 is the left of the texture

The same polygon, but rotated:

                      3
                    +
                 /    \
              /         \
          2 +            +  4
              \        /
                \   /
                  +
                  1

Although the positions of the points are different, point 1 is still the top-left of our texture.

How to put it to screen

Okay, so now you have four points and know which one of them is also the top-left of our texture. What next?

If you think back to our tutorial on polygons, you will remember we draw it scanline by scanline. We do texture mapping the same way.

Let’s look at that picture again:

                      1
                    +
               a /    \  b
              /         \
          4 +            +  2
              \        /
                \   /
                  +
                  3

We know that point 1 is at [0,0] in our texture. Point 2 is at [127,0], Point 3 is at [127,127], and point 4 is at [0,127].

The clever bit, and the entire key to texture mapping, is making the logical leap that precisely half way between Point 1 and Point 2 (b), we are at [64,0] in our texture. (a) is in the same manner at [0,64].

That’s it. All we need to know per y scanline is:

• The starting position on the x axis of the polygon line
• The position on the x in the texture map referenced by that point
• The position on the y in the texture map referenced by that point
• The ending position on the x axis of the polygon line
• The position on the x in the texture map referenced by that point
• The position on the y in the texture map referenced by that point

Let me give you an example. Let’s say that (a) and (b) from the above picture are on the same y scanline. We know that the x of that scanline is (say) 100 pixels at the start and 200 pixels at the end, making it’s width 100 pixels.

We know that on the left hand side, the texture is at [0,64], and at the right hand side, the texture is at [64,0]. In 100 pixels we have to traverse our texture from [0,64] to [64,0].

Assume at the start we have figured out the starting and ending points in the texture:

  textureX = 0;
  textureY = 64;
  textureEndX = 64;
  textureEndY = 0;

  dx := (TextureEndX-TextureX)/(maxx-minx);
  dy := (TextureEndY-TextureY)/(maxx-minx);
  for loop1 := minx to maxx do BEGIN
    PutPixel (loop1, ypos, texture [textureX, textureY], VGA);
    textureX = textureX + dx;
    textureY = textureY + dy;
  END;

Do the above for all the scanlines, and you have a texture mapped polygon! It’s that simple.

We find our beginning and ending positions in the usual fashion. We know that Point 1 is [0,0]. We know that Point 2 is [127,0]. We know the number of scanlines on the y axis between Point 1 and Point 2.

  textureDX = 127/abs (point2.y - point1.y)

We run though all the y scanlines, starting from [0,0] and adding the above formula to the X every time. When we hit the last scanline, we will be at point [127,0] in the texture.

Repeat for all four sides, and you have the six needed variables per scanline.

In closing

As you can see, texture mapping (this type at least) is quite easy, and produces quite a good result. You will however notice a bit of distortion if you bring the polygon too close. This can be fixed by a) Subdividing the polygon, so the one is made up of four or more smaller polygons. Much bigger, but works; b) Using more accurate fixed point; or c) Figuring out perspective correct texture mapping, mapping along constant-z lines etc.

DENTHOR, coder for ...
_____   _____   ____   __   __  ___  ___ ___  ___  __   _____
/  _  \ /  ___> |  _ \ |  |_|  | \  \/  / \  \/  / |  | /  _  \
|  _  | \___  \ |  __/ |   _   |  \    /   >    <  |  | |  _  |
\_/ \_/ <_____/ |__|   |__| |__|   |__|   /__/\__\ |__| \_/ \_/
smith9@batis.bis.und.ac.za
The great South African Demo Team! Contact us for info/code exchange!  

Face Sorting

There are many ways to sort faces in a 3D object. For now, I will show you just about the easiest one of the lot.

Say you have two polygons….

                ------P1

           ------------------P2

                   Eye

As you can see, P1 has to be drawn before P2. The easiest way to do this is as follows:

On startup, find the midpoint of each of the polys, through the easy equations:

x = (P2.1.x + P2.2.x + P2.3.x + p2.4.x)/4
y = (P2.1.y + P2.2.y + P2.3.y + p2.4.y)/4
z = (P2.1.z + P2.2.z + P2.3.z + p2.4.z)/4

NOTE: For a triangle you would obviously only use three points and divide by three.

Anyway, now you have the X,Y,Z of the midpoint of the polygon. You can then rotate this point with the others. When it is time to draw, you can compare the resulting Z of the midpoint, sort all of the Z items, and then draw them from back to front.

In the sample program I use a simple bubble sort… basically, I check the first two values against each other, and swap them if the first is bigger than the second. I continue doing this to all the numbers until I run through the entire list without swapping once. Bubble sorts are standard computer science topics… perhaps borrow a text book to find out more about them and other (better) sorting methods.

The above isn’t perfect, but it should work 90% of the time. But it still means that when you are drawing a cube, you have to draw all 6 sides every frame, even though only three or so are visible. That is where hidden face removal comes in…

Hidden Face Removal

Pick up something square. A stiffy disk will do fine. Face it towards you, and number all the corners from one to four in a clockwise direction.

                1 +-------------+ 2
                  |             |
                  |             |
                  |             |
                  |             |
                4 +-------------+ 3

Now rotate the stiffy disk on all three axes, making sure that you can still see the front of the disk. You will notice that whenever you can see the front of the disk, the four points are still in alphabetical order. Now rotate it so that you can see the back of the stiffy. Your points will now be:

                2 +-------------+ 1
                  |             |
                  |             |
                  |             |
                  |             |
                3 +-------------+ 4

The points are now anti-clockwise! This means, in its simplest form, that if you define all your polygon points in a clockwise order, when drawing you ignore the polys that are anticlockwise. (Obviously when you define the 3D object, you define the polygons facing away from you in an anticlockwise order.)

To find out whether a poly’s points are clockwise or not, we need to find its normal. Here is where things start getting fun.

In school, you are told that a normal is perpendicular to the plane. In ASCII:

                      | Normal
                      |
                      |
        --------------------------- Polygon

As you can see, the normal is at 90 degrees to the surface of the poly. We must extend this to three dimensions for our polygons. You’ll have to trust me on that, I can’t draw it in ASCII :)

To find a normal, you only need three points from your poly (ABC):

A(x0,y0,z0), B(X1,Y1,Z1), C(X2,Y2,Z2)

then the vector normal = AB^AC = (Xn,Yn,Zn) with

Xn=(y1-y0)(z0-z2)-(z1-z0)(y0-y2)
Yn=(z1-z0)(x0-x2)-(x1-x0)(z0-z2)
Zn=(x1-x0)(y0-y2)-(y1-y0)(x0-x2)

We are interested in the Z normal, so we will use the function:

normal:=(x1-x0)(y0-y2)-(y1-y0)(x0-x2);

The result is something of a sine wave when you rotate the poly in three dimensions. A negative value means that the poly is facing you, a positive value means that it is pointing away.

The above means that with a mere two MULs you can discount an entire poly and not draw it. This method is perfect for “closed” objects such as cubes etc.

I am anything but a maths teacher, so go borrow someone’s math book to find out more about surface normals. Trust me, there is a lot more written about them than you think.

An extension of calculating your normal is finding out about light-sourcing your polygons. Watch for more information in one of the next few trainers.

A combination of the above two routines should work quite nicely in creating 3d objects with little or no overlapping.

DENTHOR, coder for ...
_____   _____   ____   __   __  ___  ___ ___  ___  __   _____
/  _  \ /  ___> |  _ \ |  |_|  | \  \/  / \  \/  / |  | /  _  \
|  _  | \___  \ |  __/ |   _   |  \    /   >    <  |  | |  _  |
\_/ \_/ <_____/ |__|   |__| |__|   |__|   /__/\__\ |__| \_/ \_/
smith9@batis.bis.und.ac.za
The great South African Demo Team! Contact us for info/code exchange!  

Assembler - the short version

Okay, there are many assembler trainers out there, many of which are probably better than this one. I will focus on the areas of assembler that I find important… if you want more, go buy a book (go for the Michael Abrash ones), or scour the ‘net for others.

First, let us start off with the basic set up of an assembler program.

DOSSEG

This tells your assembler program to order your segments in the same manner that high level languages do.

.MODEL <MODEL>

<MODEL> can be:

Tiny       Code + Data < 64k   (Can be made a COM file)
Small      Code < 64k          Data < 64k
Medium     Code > 64k          Data < 64k
Compact    Code < 64k          Data > 64k
Large      Code > 64k          Data > 64k
Huge       Arrays > 64k

Enable 286 instructions … can be .386 ; .386P etc.:

.286

Set the stack. <size> will be the size of your stack. I usually use 200h:

.STACK <size>

Tells the program that the data is about to follow. (Everything after this will be placed in the data segment):

.DATA

Tells the program that the code is about to follow. (Everything after this will be placed in the code segment)

.CODE

Tells the program that this is where the code begins:

START:

Tells the program that this is where the code ends:

END START

To compile and run an assembler file, we run:

tasm bob
tlink bob

I personally use tasm; you will have to find out how your assembler works.

Now, if we ran the above file as follows:

DOSSEG
.MODEL SMALL
.286
.STACK 200h
.DATA
.CODE

START
END START

You would think that is would just exit to DOS immediately, right? Wrong. You have to specifically give DOS back control, by doing the following:

START
        mov     ax,4c00h
        int     21h
END START

Now if you compiled it, it would run and do nothing.

Okay, let us kick off with registers.

Firstly: A bit is a value that is either 1 or 0.

This is obviously quite limited, but if we start counting in them, we can get larger numbers. Counting with ones and zeros is known as binary, and we call it base 2. Counting in normal decimal is known as base 10, and counting in hexadecimal is known as base 16.

    Base 2 (Binary)     Base 10 (Decimal)    Base 16 (Hexadecimal)
         0                      0                       0
         1                      1                       1
         10                     2                       2
         11                     3                       3
         100                    4                       4
         101                    5                       5
         110                    6                       6
         111                    7                       7
         1000                   8                       8
         1001                   9                       9
         1010                   10                      A
         1011                   11                      B
         1100                   12                      C
         1101                   13                      D
         1110                   14                      E
         1111                   15                      F

As you can see, you need four bits to count up to 15, and we call this a nibble. With eight bits, we can count up to 255, and we call this a byte. With sixteen bits, we can count up to 65535, and we call this a word. With thirty-two bits, we can count up to lots, and we call this a double word.

A quick note: Converting from binary to hexadecimal is actually quite easy. You break up the binary into groups of four bits, starting on the right, and convert these groups of four to hex.

      1010 0010 1111 0001
  =      A    2    F    1

Converting to decimal is a bit more difficult. What you do, is you multiply each number by its base to the power of its index…

  A2F1 hex
= (A*16^3) + (2*16^2) + (F*16^1) + (1*16^0)
= (10*4096) + (2*256) + (15*16) + (1)
= 40960 + 512 + 240 + 1
= 41713 decimal

The same system can be used for binary.

To convert from decimal to another base, you divide the decimal value by the desired base, keeping a note of the remainders, and then read the results backwards.

               16   |   41713
               16   |   2607    r   1       (41713 / 16 = 2607 r 1)
               16   |   162     r   F       (2607 / 16 = 162 r 15)
               16   |   10      r   2       (162 / 16 = 10 r 2)
                    |   0       r   A       (10 / 16 = 0 r 10)

Read the remainders backwards, our number is A2F1 hex. Again, the same method can be used for binary.

The reason why hex is popular is obvious: using bits, it is impossible to get a reasonable base 10 (decimal) system going, and binary gets unwieldly at high values. Don’t worry too much though: most assemblers (like tasm) will convert all your decimal values to hex for you.

You have four general purpose registers: AX, BX, CX and DX. Think of them as variables that you will always have. On a 286, these registers are 16 bytes long, or one word.

As you know, a word consists of two bytes, and in assembler you can access these bytes individually. They are separated into high bytes and low bytes per word.

   High byte | Low byte
   0000 0000 | 0000 0000  bits
   [--------Word-------]

The method of access is easy. The high byte of AX is AH, and the low byte is AL. You can also access BH, BL, CH, CL, DH and DL.

A 386 has extended registers: EAX, EBX, ECX, EDX… you can access the lower word normally (as AX, with bytes AH and AL), but you cannot access the high word directly … you must ror EAX,16 (rotate the binary value through 16 bits), after which the high word and low word swap … do it again to return them. Acessing EAX as a whole is no problem: mov eax, 10; add eax,ebx … these are all valid.

Next come segments. As you have probably heard, computer memory is divided into various 64k segments (note: 64k = 65,536 bytes, sound familiar?) A segment register points to which segment you are looking at. An offset register points to how far into that segment you are looking. One way of looking at it is like looking at a 2D array… the segments are your columns and your offsets are your rows. Segments and offsets are displayed as Segment:Offset … so $a000:50 would mean the fiftieth byte in segment $a000.

The segment registers are ES, DS, SS and CS. A 386 also has FS and GS. These values are words (0-65,535), and you cannot access the high or low bytes separately. CS points to your code segment, and usually if you touch this your program will explode. SS points to your stack segment, again, this baby is dangerous. DS points to your data segment, and can be altered, if you put it back after you use it, and don’t use any global variables while it is altered. ES is your extra segment, and you can do what you want with it.

The offset registers are DI, SI, IP, SP, BP. Offset registers are generally associated with specific segment registers, as follows: ES:DI DS:SI CS:IP SS:SP … On a 286, BX can be used instead of the above offset registers, and on a 386, any register may be used. DS:BX is therefore valid.

If you create a global variable (let’s say bob), when you access that variable, the compiler will actually look for it in the data segment. This means that the statement:

ax = bob

could be

ax = ds:[15]

A quick note: A value may be signed or unsigned. An unsigned word has a range from 0 to 65,535. A signed word is called an integer and has a range -32,768 to 32,767. With a signed value, if the leftmost bit is equal to 1, the value is in the negative.

Next, let us have a look at the stack. Let us say that you want to save the value in ax, use ax to do other things, then restore it to its origional value afterwards. This is done by utilizing the stack. Have a look at the following code:

mov   ax, 50      ; ax is equal to 50
push  ax          ; push ax onto the stack
mov   ax, 27      ; ax is equal to 27
pop   ax          ; pop ax off the stack

At this point, ax is equal to 50.

Remember we defined the stack to be 200h further up? This is part of the reason we have it. When you push a value onto the stack, that value is recorded on the stack heap (referenced by SS:SP, SP is incremented) When you pop a value off the stack, the value is placed into the variable you are popping it back in to, SP is decremented and so forth. Note that the computer does not care what you pop the value back into.

mov   ax, 50
push  ax
pop   bx

This would set the values of both ax and bx to 50. (There are faster ways of doing this, pushing and popping are fairly fast though).

push ax
push bx
pop  ax
pop  bx

This would swap the values of ax and bx. As you can see, to pop the values back in to the original variables, you must pop them back in the opposite direction to which you pushed them.

push ax
push bx
push cx

pop cx
pop bx
pop ax

would result in no change for any of the registers.

When a procedure is called, all the parameters for that procedure are pushed onto the stack. These can actually be read right off the stack, if you want to.

As you have already seen, the mov command moves a value…

mov  <dest>, <source>

Note that dest and source must be the same number of bits long.

mov  ax, dl

would not work, and neither would

mov  cl,bx

However:

mov  cx,dx
mov  ax,50
mov  es,ax

are all valid.

shl I have explained before, it is where all the bits in a register are shifted one to the left and a zero added on to the right. This is the equivalent of multiplying the value by two. shr works in the opposite direction.

rol does the same, except that the bit that is removed from the left is replaced on the right hand side. ror works in the opposite direction.

div <value> divides the value in ax by value and returns the result in al if value is a byte, placing the remainder in ah. If value is a word, the double word DX:AX is divided by value, the result being placed in ax and the remainder in dx. Note that this only works for unsigned values.

idiv <value> does the same as above, but for signed variables.

mul <value> If value is a byte, al is multiplied by value and the result is stored in ax. If value is a word, ax is multiplied by value and the result is stored in the double word DX:AX.

imul <value> does the same as above, but for signed variables.

The j* commands are fairly simple: if a condition is met, jump to a certain label.

jz  <label>   Jump if zero
ja  <label>   Jump above     (unsigned)
jg  <label>   Jump greater   (signed)

and so forth.

An example:

cmp  ax,50    ; Compare ax to 50
je   @Equal   ; If they are equal, jump to label @equal
call MyProc   ; Runs procedure MyProc and then returns to the next line of code.

Procedures are declared as follows:

MyProc   proc near
         ret    ; Must be here to return from where it was called
MyProc   endp

Variables are also easy:

bob  db 50

creates a variable bob, a byte, with an initial value of 50.

bob2 dw 50

creates a variable bob2, a word, with an initial value of 50.

bob3 db 1,2,3,4,5,65,23

creates bob3, an array of 7 bytes.

bob4 db 100 dup (?)

creates bob4, an array of 100 bytes, with no starting value.

Go back and look at Part 7 for a whole lot more assembler commands, and get some sort of reference guide to help you out with others. I personally use the Norton Guides help file to program assembler.

Fire Routines

To demonstrate how to write an assembler program, we will write a fire routine in 100% assembler. The theory is simple.

Set the palette to go from white to yellow to red to blue to black. Create a 2D array representing the screen on the computer. Place high values at the bottom of the array (screen) for each element, do the following:

• Take the average of the four elements under it:

                         * Current element
                        123
                         4  Other elements

• Get the average of the four elements, and place the result in the current element.
• Repeat

Easy, no? I first saw a fire routine in the Iguana demo, and I just had to do one ;) … it looks very effective.

With the sample file, I have created a batch file, make.bat. It basically says:

tasm fire
tlink fire

So to build and run the fire program, type:

make
fire

The source file is commented quite well, so there shouldn’t be any problems.

In closing

As you can see, the sample program is in 100% assembler. For the next tutorial I will return to Pascal, and hopefully your newfound assembler skills will help you there too.

DENTHOR, coder for ...
_____   _____   ____   __   __  ___  ___ ___  ___  __   _____
/  _  \ /  ___> |  _ \ |  |_|  | \  \/  / \  \/  / |  | /  _  \
|  _  | \___  \ |  __/ |   _   |  \    /   >    <  |  | |  _  |
\_/ \_/ <_____/ |__|   |__| |__|   |__|   /__/\__\ |__| \_/ \_/
smith9@batis.bis.und.ac.za
The great South African Demo Team! Contact us for info/code exchange!  

Loading a PCX file

This is actually quite easy. The PCX file is divided into three sections, namely a 128 byte header, a data section, and a 768 byte palette.

You can usually ignore the 128 byte header, but here it is:

0  Manufacturer     10 = ZSoft .PCX file
1  Version
2  Encoding
3  Bits Per Pixel
4  XMin, Ymin, XMax, YMax  (2 bytes each)
12 Horizontal Resolution (2 bytes)
14 Vertical Resolution (2 bytes)
16 Color palette setting (48 bytes)
64 Reserved
65 Number of color planes
66 Bytes per line (2 bytes)
68 1 = Color    2 = Grayscale  (2 bytes)
70 Blank (58 bytes)

That makes 128 bytes.

The palette file, which is 768 bytes, is situated at the very end of the PCX file. The 769’th byte back should be the decimal 12, which indicates that a VGA color palette is to follow.

There is one thing that we have to do to get the palette correct in VGA… the PCX palette values for R,G,B are 0 to 255 respectively. To convert this to our standard (VGA) palette, we must divide the R,G,B values by 4, to get them into a range of 0 to 63.

Actually decoding the image is very simple. Starting after the 128 byte header, we read a byte.

If the top two bits of this byte are not set, we dump the value to the screen.

We check bits as follows:

if ((temp and $c0) = $c0) then ...(bits are set)... else ...(bits are not set)

C0 in hex = 11000000 in binary = 192 in decimal

Let’s look at that more closely…

  temp  and  c0h
  temp  and  11000000b

That means, when represented in bit format, both corresponding bits have to be set to one for the result to be one.

0 and 0 = 0     1 and 0 = 0    0 and 1 = 0    1 and 1 = 1

In the above case then, both of the top two bits of temp have to be set for the result to equal 11000000b. If it does not equal that, the top two bits are not both set, and we can put the pixel.

So:

  if ((temp and $c0) = $c0) then BEGIN
  END else BEGIN
    putpixel (screenpos,0,temp,vga);
    inc (screenpos);
  END;

If the top two bits are set, things change. The bottom six bits become a loop counter, which the next byte is repeated.

  if ((temp and $c0) = $c0) then BEGIN
    { Read in next byte, temp2 here.}
    for loop1:=1 to (temp and $3f) do BEGIN
      putpixel (screenpos,0,temp2,vga);
      inc (screenpos);
    END;
  END else BEGIN
    putpixel (screenpos,0,temp,vga);
    inc (screenpos);
  END;

There is our PCX loader. You will note that by and’ing temp by $3f; I am and’ing it by 00111111b… in other words, clearing the top two bits.

The sample program has the above in assembler, but it is the same procedure… and you can read the next tutorial for more info on how to code in assembler.

In the sample I assume that the pic is 320x200, with a maximum size of 66,432 bytes.

File packing

The question is simple: how do I get all my files into one executable? Having many small data files can start to look unprofessional.

An easy way to have one .exe and one .dat file when dealing with many cels etc. is easy… you would alter your load procedure, which looks as follows:

VAR temp : Array [1..5,1..256] of byte;
Procedure Init;
BEGIN
  loadcel ('pic1.cel',temp[1]);
  loadcel ('pic2.cel',temp[2]);
  loadcel ('pic3.cel',temp[3]);
  loadcel ('pic4.cel',temp[4]);
  loadcel ('pic5.cel',temp[5]);
END;

For one compile you would do this:

VAR temp : Array [1..5,1..256] of byte;
Procedure Init;
VAR f:File;
BEGIN
  loadcel ('pic1.cel',temp[1]);
  loadcel ('pic2.cel',temp[2]);
  loadcel ('pic3.cel',temp[3]);
  loadcel ('pic4.cel',temp[4]);
  loadcel ('pic5.cel',temp[5]);
  assign (f,'pic.dat');
  rewrite (f,1);
  blockwrite (f,temp,sizeof(temp));
  close (f);
END;

From then on, you would do:

VAR temp : Array [1..5,1..256] of byte;
Procedure Init;
VAR f:File;
BEGIN
  assign (f,'pic.dat');
  reset (f,1);
  blockread (f,temp,sizeof(temp));
  close (f);
END;

This means that instead of five data files, you now have one! You have also stripped the 800 byte cel header too. Note that this will work for any form of data, not just cel files.

The next logical step is to include this data in the .exe file, but the question is how? In an earlier tutorial, I converted my data file to constants and placed it inside my main program. This is not good mainly because of space restrictions … you can only have so many constants, and what if your data file is two megs big?

Attached with this tutorial is a solution. I have written a program that combines your data files with your executable file, no matter how big the data is. The only thing you have to worry about is a small change in your data loading methods. Let’s find out what.

Using the file packer

Normally you would load your data as follows:

Procedure Init;
BEGIN
  loadcel ('bob.bob',temp);
  loadpcx ('den.pcx',VGA);        { Load a PCX file }
  loaddat ('data.dat',lookup);    { Load raw data into lookup }
END;

Easy, hey? Now, using the file packer, you would change this to:

USES fpack;
Procedure Init;
BEGIN
  total := 3;
  infodat[1] := 'bob.bob';
  infodat[2] := 'den.pcx';
  infodat[3] := 'data.dat';
  loadcel (1,temp);
  loadpcx (2,VGA);
  loaddat (3,lookup);
END;

Not too difficult? Now, this is still using the normal data files on your hard drive. You would then run PACK.EXE, select the program’s .exe as the base, then select “bob.bob”, “den.pcx” and “data.dat”, in order (1, 2, 3). Hit “c” to contine, and it will combine the files. Your programs .exe file will be able to run independently of the separate data files on disk, because the data files are imbedded with the .exe.

Let us take a closer look at the load procedures. Normally a load procedure would look as follows:

Procedure LoadData (name:string; p:pointer);
VAR f:file;
BEGIN
  assign (f,name);
  reset (f,1);
  blockread (f,p^,filesize(f);
  close (f);
END;

In FPack.pas, we do the following:

Function LoadData (num:integer; p:pointer):Boolean;
VAR f:file;
BEGIN
  If pack then BEGIN
    assign (f,paramstr(0));
    reset (f,1);
    seek (f,infopos[num]);
    blockread (f, p^, infopos[num+1]-infopos[num]);
    close (f);
  END else BEGIN
    assign (f,infodat[num]);
    reset (f,1);
    blockread (f, p^, filesize (f));
    close (f);
  END;
END;

As you can see, we just have two special cases depending on whether or not the .exe file has been packed yet.

NOTE: After you have packed the file, you CAN NOT pklite it. You can however pklite the .exe_before you run pack.exe … in other words, you cannot use pklite to try pack your data files.

PACK.EXE does have a limitation … you can only pack 24 data files together. If you would like it to do more, mail me … It should be easy to increase the number.

In the sample program, FINAL.EXE is the same as temp.pas, except it has its PCX embedded inside it. I ran pack2.exe, selected final.exe and eye.pcx, hit “C”, and there it was. You will notice that eye.pcx is not included in the directory … it is now part of the exe!

In closing

Well, that’s about it for this trainer… next one (as I have mentioned twice already ;) will be on assembler, with a flame routine thrown in.

This tut has been a bit of a departure from normal tuts … aside from the PCX loading routines, the rest has been “non programming” oriented … don’t worry, next week’s one will be back to normal.

DENTHOR, coder for ...
_____   _____   ____   __   __  ___  ___ ___  ___  __   _____
/  _  \ /  ___> |  _ \ |  |_|  | \  \/  / \  \/  / |  | /  _  \
|  _  | \___  \ |  __/ |   _   |  \    /   >    <  |  | |  _  |
\_/ \_/ <_____/ |__|   |__| |__|   |__|   /__/\__\ |__| \_/ \_/
smith9@batis.bis.und.ac.za
The great South African Demo Team! Contact us for info/code exchange!  

Pixel Morphing

Have you ever lain down on your back in the grass and looked up at the cloudy sky? If you have, you have probably seen the clouds move together and create wonderful shapes… that cloud plus that cloud together make a whale… a ship… a face etc.

We can’t quite outdo Mother Nature, but we can sure give it a shot. The effect I am going to show you is where various pixels at different starting points move together and create an overall picture.

The theory behind it is simple: each pixel has bits of data associated with it, most important of which is as follows:

This is my color
This is where I am
This is where I want to be.

The pixel, keeping its color, goes from where it is to where it wants to be. Our main problem is how it moves from where it is to where it wants to be. A obvious approach would be to say “If its destination is above it, decrement its y value, if the destination is to the left, decrement its x value and so on.”

This would be bad. The pixel would only ever move at set angles, as you can see below:

                Dest   O-----------------\
                                           \  <--- Path
                                             \
                                               \
                                                O Source

Doesn’t look very nice, does it? The pixels would also take different times to get to their destination, whereas we want them to reach their points at the same time, i.e.:

     Dest 1   O-------------------------------O Source 1
     Dest 2   O-----------------O Source 2

Pixels 1 and 2 must get to their destinations at the same time for the best effect. The way this is done by defining the number of frames or “hops” needed to get from source to destination. For example, we could tell pixel one it is allowed 64 hops to get to its destination, and the same for point 2, and they would both arrive at the same time, even though pixel 2 is closer.

The next question, it how do we move the pixels in a straight line? This is easier than you think…

Let us assume that for each pixel, x1,y1 is where it is, and x2,y2 is where it wants to be.

   (x2-x1) = The distance on the X axis between the two points
   (y2-y1) = The distance on the Y axis between the two points

If we do the following:

  dx := (x2-x1)/64;

we come out with a value in dx which is very useful. If we added dx to x1 64 times, the result would be x2! Let us check…

  dx = (x2-x1)/64
  dx*64 = x2-x1         { Multiply both sides by 64 }
  dx*64+x1 = x2         { Add x1 to both sides }

This is high school math stuff, and is pretty self explanatory. So what we have is the x movement for every frame that the pixel has to undergo. We find the y movement in the same manner.

  dy := (y2-y1)/64;

So our program is as follows:

  { Set x1,y1 and x2,y2 values }
  dx:= (x2-x1)/64;
  dy:= (y2-y1)/64;

  for loop1:=1 to 64 do BEGIN
    putpixel (x1,y1)
    wait;
    clear pixel (x1,y1);
    x1:=x1+dx;
    y1:=y1+dy;
  END;

If there was a compiler that could use the above pseudocode, it would move the pixel from x1,y1 to x2,y2 in 64 steps.

So, what we do is set up an array of many pixels with this information, and move them all at once… voilá, we have pixel morphing! It is usually best to use a bitmap which defines the color and destination of the pixels, then randomly scatter them around the screen.

Why not use pixel morphing on a base object in 3d? It would be the work of a moment to add in a Z axis to the above.

The sample program uses fixed point math in order to achieve high speeds, but it is basically the above algorithm.

Static

A static screen was one of the first effects Asphyxia ever did. We never actually released it because we couldn’t find anywhere it would fit. Maybe you can.

The easiest way to get a screen of static is to tune your TV into an unused station … you even get the cool noise effect too. Those people who build TVs really know how to code ;-)

For us on a PC however, it is not as easy to generate a screen full of static (unless you desperately need a new monitor)

What we do is this:

• Set colors 1-16 to various shades of grey.
• Fill the screen up with random pixels between colors 1 and 16
• Rotate the palette of colors 1 to 16.

That’s it! You have a screenful of static! To get two images in one static screen, all you need to do is fade up/down the specific colors you are using for static in one of the images.

A nice thing about a static screen is that it is just palette rotations … you can do lots of things in the foreground at the same time (such as a scroller).

In closing

Well, that is about it … as I say, I will be doing more theory stuff in future, as individual demo effects can be thought up if you know the base stuff.

Note the putpixel in this GFX3.PAS unit … it is very fast .. but remember, just calling a procedure eats clock ticks… so embed putpixels in your code if you need them. Most of the time a putpixel is not needed though.

DENTHOR, coder for ...
_____   _____   ____   __   __  ___  ___ ___  ___  __   _____
/  _  \ /  ___> |  _ \ |  |_|  | \  \/  / \  \/  / |  | /  _  \
|  _  | \___  \ |  __/ |   _   |  \    /   >    <  |  | |  _  |
\_/ \_/ <_____/ |__|   |__| |__|   |__|   /__/\__\ |__| \_/ \_/
smith9@batis.bis.und.ac.za
The great South African Demo Team! Contact us for info/code exchange!  

How do plasmas work?

I will only cover one type of plasma here: a realtime plasma of course. Other types of plasmas include a static picture with a pallette rotation.

When you get right down to it, this method of realtime plasmas is merely an intersection of four COS waves. We get our color at a particular point by saying:

      col := costbl[one]+costbl[two]+costbl[three]+costbl[four];

The trick is getting the four indexes of that cos table array to create something that looks nice. This is how we organize it: have two of them being indexes for vertical movement and two of them being indexes for horizontal movement.

This means that by changing these values we can move along the plasma. To draw an individual screen, we pass the values of the four to another four so that we do not disturb the original values. For every pixel across, we add values to the first two indexes, then display the next pixel. For every row down, we add values to the second two indexes. Sound complex enough? Good, because that what we want, a complex shape on the screen.

By altering the original four values, we can get all sorts of cool movement and cycling of the plasma. The reason we use a cos table is as follows: a cos table has a nice curve in the value of the numbers… when you put two or more together, it is possible to get circular pictures… circles are hard to do on a computer, so this makes it a lot easier…

Okay, now you can have a look at the source file, all I do is put the above into practice. I did add one or two things though…

Background: This is just a large array, with the values in the array being added to the plasma at that pixel.

Psychedelic: This cycles through about 7000 colors instead of just rotating through the base 256.

Clever fading

You will notice when the sample program fades in and out that the colors all reach their destination at the same time … in other words, they don’t all increment by one until they hit the right color then stop. When done in that way the fading does not look as professional.

Here is how we do a step-crossfade:

Each r,g,b value can be between 0 and 64. Have the palette we want to get to in bob and the temporary pallette in bob2. For each step, from 0 to 63 do the following:

     bob2[loop1].r:=bob[loop1].r*step/64;

That means if we are halfway through the crossfade (step=32) and the red value is meant to get to 16, our equation looks like this:

    r:=16*32/64
     r=8

Which is half of the way to where it wants to be. This means all colors will fade in/out with the same ratios… and look nicer.

Rotating the pallette

I have done this one before, I think. Here it is:

move color 0 into temp

move color 1 into color 0
move color 2 into color 1
move color 3 into color 2
and so on till color 255

move temp into color 255

And your palette is rotating. Easy huh? Recheck Part 2 for more info on palette rotation.

In closing

The tutorial was a bit short this time, but that is mostly because the sample file is self-explanatory. The file can however be speeded up, and of course you can add certain things which will totally change the look of the plasma.