About my driver, HIMEM/EMM386 and interrupts

View: New views
6 Messages — Rating Filter:   Alert me  

About my driver, HIMEM/EMM386 and interrupts

by Lucas Kiwi :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Guys,

   I'm building a driver (for DOS) that's supposed to have several functions hooked at an interrupt. When applications call these functions, the driver should load and unload some tables or activate/deactivate them somehow. Because these tables are big, I don't want to keep them in conventional memory with the driver code altogether, but instead prefer them to by dynamic.
   The problem is as follows. If I want to have the tables in files, then loading them in and unloading them from conventional memory will cause several problems, which include slowdowns, risk of instability because of hard-drive access and a big amount of conventional memory still occupied while the tables are in use. I want, therefore, have the tables permanently loaded in high, extended or expanded memory while the driver is running. But... this means that the driver will have to do different things to get to the tables depending on whether HIMEM, or EMM386, or both, or none of them are currently there.
   I am really not very experienced with high memory, but I understand that, even without HIMEM or EMM386, I can access high memory by setting up real-flat mode. Also, if HIMEM only is there, this mode is already available and I can easily access the tables without having to move memory (that is, using 32bit pointers in real mode instead of calling HIMEM functions to copy data). If EMM386 is loaded, I know I can use the expanded memory functions, but then virtual-86 mode is set and I can no loger use 32bit real-mode pointers!  Yet, there has to be a way, because HIMEM works even though EMM386 is loaded. This clearly shows me that there is a big deal of things I don't know about these two drivers!
   I need help understanding how these two drivers work, as well as what happens in real-flat and virtual-86 modes. I also need to know how I can make my interrupt functions flexible enough so that they can work no matter which HIMEM/EMM386 is currently working and, if possible, I would like my functions to also be compatible with protected mode (DPMI) applications calling them, because the functions have to access memory and return pointers and therefore, might not work by using real-mode-interrupt calling functions of the DPMI.
   Can anybody help me with all this, please?  :)

                Lucas

Need mail bonding? Bring all your contacts to Yahoo!Xtra with TrueSwitch
------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Freedos-devel mailing list
Freedos-devel@...
https://lists.sourceforge.net/lists/listinfo/freedos-devel

Re: About my driver, HIMEM/EMM386 and interrupts

by Christian Masloch :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

>    I'm building a driver (for DOS) that's supposed to have several  
> functions hooked at an interrupt. When applications call these  
> functions, the driver should load and unload some tables or  
> activate/deactivate them somehow. Because these tables are big, I don't  
> want to keep them in conventional memory with the driver code  
> altogether, but instead prefer them to by dynamic.
>    The problem is as follows. If I want to have the tables in files,  
> then loading them in and unloading them from conventional memory will  
> cause several problems, which include slowdowns, risk of instability  
> because of hard-drive access and a big amount of conventional memory  
> still occupied while the tables are in use. I want, therefore, have the  
> tables permanently loaded in high, extended or expanded memory while the  
> driver is running.

I'd disregard EMS (expanded memory), or at least depending on EMS. It's  
supported when installing an EMM386 (+HIMEM) only, and if the user  
switched it on. If you still want to use EMS, write your program so that  
it doesn't need a page frame if running under the correct EMM.

> But.... this means that the driver will have to do different things to  
> get to the tables depending on whether HIMEM, or EMM386, or both, or  
> none of them are currently there.

Your program can't check for EMM386 or HIMEM, it can only check whether  
XMS or EMS is available.

>    I am really not very experienced with high memory, but I understand  
> that, even without HIMEM or EMM386, I can access high memory by setting  
> up real-flat mode.

You're talking about extended memory. High memory is a 64 KiB (- 16 byte)  
area behind the real mode memory (1 MiB). High memory is available in  
normal real mode if the A20 line is switched on. HIMEM provides an  
interface for switching the A20 line.

> Also, if HIMEM only is there, this mode is already available and I can  
> easily access the tables without having to move memory (that is, using  
> 32bit pointers in real mode instead of calling HIMEM functions to copy  
> data).

Not necessarily. HIMEM isn't restricted to this method, and even if your  
HIMEM uses it, you'd still have to set up your own "flat real mode".

>    I need help understanding how these two drivers work, as well as what  
> happens in real-flat and virtual-86 modes.

I'd disregard using "flat real mode" at all.

> I also need to know how I can make my interrupt functions flexible  
> enough so that they can work no matter which HIMEM/EMM386 is currently  
> working and, if possible, I would like my functions to also be  
> compatible with protected mode (DPMI) applications calling them, because  
> the functions have to access memory and return pointers

What does your interface look like? Does it return a pointer to the table?  
Then the table has to stay in the memory addressable in real mode, i.e.  
1088 KiB (including the high memory area). Otherwise, not only your driver  
but any application which calls it would have to switch into the "flat  
real mode" on itself.

With XMS (extended memory), your applications either would have to switch  
into "flat real mode" or some sort of protected mode (set up by itself,  
VCPI or DPMI) to access the table directly as well. XMS move requests  
would allow parts of the table (or the full table) to be read into real  
mode memory allocated by the application.

> and therefore, might not work by using real-mode-interrupt calling  
> functions of the DPMI.

Any memory accessible from real mode is accessible by DPMI applications as  
well. Just make sure to retrieve the correct segment and pointer registers  
 from the real mode call structure, and to convert the passed real mode  
segment to a selector correctly.

Regards,
Christian

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Freedos-devel mailing list
Freedos-devel@...
https://lists.sourceforge.net/lists/listinfo/freedos-devel

Re: About my driver, HIMEM/EMM386 and interrupts

by Lucas Kiwi :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Thank you, Christian

   You've cleared out some doubts and misunderstandings I had. Still, now this leads me to new questions. Also, I would like to clarify some of the things I wrote in my last e-mail.
   When you say a program would still need to set up flat real mode, I suppose you mean that they still need a jump to protected mode and back to initially setup the segment registers. However, I understood that HIMEM did something like establishing a routine that was run whenever you tried to use an offset greater than 0FFFFh together with a "real-mode-made" segment. This routine would "trap" the event and handle the switch to protected mode and back, keeping the segment base, but extending the limit to 4G, then returning to the instruction that generated the exception. Am I right?  Does this really happen?  If so, programs should not mind about their segments being initially setup in real mode.
   I agree that using the EMS functions would make the interface lack of good compatibility and would like to avoid that. What I don't want is to do something that will not work if EMM386 happens to be running. I believe that, while EMM386 is running and the system is in virtual-86 mode, it is no longer possible to extend the segment limits to 4G and they will always generate the exception. This makes me wonder how come everything still seems to run properly (programs that use XMS) when EMM is there..
   My driver is a set of functions that use 32-bit registers and runs hooked on a single interrupt. Sometimes the caller will provide a pointer to a buffer and the functions will fill these buffers, but also, some very big tables have to be loaded, that can occupy as much as one or two megabytes. I could do moving parts of them to conventional memory as needed, but it would be faster if I could access them directly somehow. The driver leads with a sound interface sometimes and needs to set up big buffers that, depending on the sound card, may have a fix hardware address beyond the conventional and high memory limit. Basically, the driver may return pointers to these areas, but could also copy the data to a buffer provided by the caller.
   Do you think I should write the functions like pure real-mode code that accesses HIMEM XMS functions as needed to move the data?  Would that be entirely compatible with real-mode- as well as DPMI-applications calling the functions?  If so, how should I handle the events when I need to read or write in a fixed extended memory area?
   Is it better to always ask the caller to provide buffers or is it more efficient (and safe) to return pointers to the buffers?  In some cases, the problem is that one buffer is to be used by many applications (i.e.: a running program and a couple of TSRs in memory) and having the caller create a buffer would mean the driver would have to update several identical buffers instead of getting all tasks refer to the same one. That would not be practical and would use up more memory.
   Well, I think I've got into big trouble building this driver!  I hope at least I learn a lot in the process. Thanks in advance for any more hints you can give me,

             Lucas


--- On Mon, 26/10/09, Christian Masloch <cm@...> wrote:

From: Christian Masloch <cm@...>
Subject: Re: [Freedos-devel] About my driver, HIMEM/EMM386 and interrupts
To: freedos-devel@...
Received: Monday, 26 October, 2009, 3:28 PM

>    I'm building a driver (for DOS) that's supposed to have several 

> functions hooked at an interrupt. When applications call these 
> functions, the driver should load and unload some tables or 
> activate/deactivate them somehow. Because these tables are big, I don't 
> want to keep them in conventional memory with the driver code 
> altogether, but instead prefer them to by dynamic.
>    The problem is as follows. If I want to have the tables in files, 
> then loading them in and unloading them from conventional memory will 
> cause several problems, which include slowdowns, risk of instability 
> because of hard-drive access and a big amount of conventional memory 
> still occupied while the tables are in use. I want, therefore, have the 
> tables permanently loaded in high, extended or expanded memory while the 
> driver is running.

I'd disregard EMS (expanded memory), or at least depending on EMS. It's 
supported when installing an EMM386 (+HIMEM) only, and if the user 
switched it on. If you still want to use EMS, write your program so that 
it doesn't need a page frame if running under the correct EMM.

> But.... this means that the driver will have to do different things to 
> get to the tables depending on whether HIMEM, or EMM386, or both, or 
> none of them are currently there.

Your program can't check for EMM386 or HIMEM, it can only check whether 
XMS or EMS is available.

>    I am really not very experienced with high memory, but I understand 
> that, even without HIMEM or EMM386, I can access high memory by setting 
> up real-flat mode.

You're talking about extended memory. High memory is a 64 KiB (- 16 byte) 
area behind the real mode memory (1 MiB). High memory is available in 
normal real mode if the A20 line is switched on.. HIMEM provides an 
interface for switching the A20 line.

> Also, if HIMEM only is there, this mode is already available and I can 
> easily access the tables without having to move memory (that is, using 
> 32bit pointers in real mode instead of calling HIMEM functions to copy 
> data).

Not necessarily. HIMEM isn't restricted to this method, and even if your 
HIMEM uses it, you'd still have to set up your own "flat real mode".

>    I need help understanding how these two drivers work, as well as what 
> happens in real-flat and virtual-86 modes.

I'd disregard using "flat real mode" at all.

> I also need to know how I can make my interrupt functions flexible 
> enough so that they can work no matter which HIMEM/EMM386 is currently 
> working and, if possible, I would like my functions to also be 
> compatible with protected mode (DPMI) applications calling them, because 
> the functions have to access memory and return pointers

What does your interface look like? Does it return a pointer to the table? 
Then the table has to stay in the memory addressable in real mode, i.e. 
1088 KiB (including the high memory area). Otherwise, not only your driver 
but any application which calls it would have to switch into the "flat 
real mode" on itself.

With XMS (extended memory), your applications either would have to switch 
into "flat real mode" or some sort of protected mode (set up by itself, 
VCPI or DPMI) to access the table directly as well. XMS move requests 
would allow parts of the table (or the full table) to be read into real 
mode memory allocated by the application.

> and therefore, might not work by using real-mode-interrupt calling 
> functions of the DPMI.

Any memory accessible from real mode is accessible by DPMI applications as 
well. Just make sure to retrieve the correct segment and pointer registers 
from the real mode call structure, and to convert the passed real mode 
segment to a selector correctly.

Regards,
Christian

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Freedos-devel mailing list
Freedos-devel@...
https://lists.sourceforge.net/lists/listinfo/freedos-devel

Reading this email at work? Make a change with Yahoo!Xtra Jobs
------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Freedos-devel mailing list
Freedos-devel@...
https://lists.sourceforge.net/lists/listinfo/freedos-devel

Re: About my driver, HIMEM/EMM386 and interrupts

by Christian Masloch :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

>    When you say a program would still need to set up flat real mode, I  
> suppose you mean that they still need a jump to protected mode and back  
> to initially setup the segment registers.

Yes.

> However, I understood that HIMEM did something like establishing a  
> routine that was run whenever you tried to use an offset greater than  
> 0FFFFh together with a "real-mode-made" segment. This routine would  
> "trap" the event and handle the switch to protected mode and back,  
> keeping the segment base, but extending the limit to 4G, then returning  
> to the instruction that generated the exception. Am I right?  Does this  
> really happen?  If so, programs should not mind about their segments  
> being initially setup in real mode.

No, I'm not aware of any Extended Memory Manager (XMM) that does such a  
thing.

>    I agree that using the EMS functions would make the interface lack of  
> good compatibility and would like to avoid that. What I don't want is to  
> do something that will not work if EMM386 happens to be running. I  
> believe that, while EMM386 is running and the system is in virtual-86  
> mode, it is no longer possible to extend the segment limits to 4G and  
> they will always generate the exception. This makes me wonder how come  
> everything still seems to run properly (programs that use XMS) when EMM  
> is there.

You're correct that with EMM386 (which sets up DOS to run in v86 mode),  
others can no longer set up flat real mode. I don't know whether such a  
thing as flat v86 mode is possible, but if it is you'd certainly have to  
work together with EMM386 to get there.

I don't know whether Microsoft's HIMEM handles it the same way, but HIMEMX  
determines whether the CPU is currently in real or protected mode first.  
If it's in real mode, HIMEMX sets up flat real mode (it can be build to  
use protected mode instead) and does the XMS move itself. If it's in  
protected mode, HIMEMX simply calls Int15.87 (BIOS Move extended memory).  
Int15.87 is hooked by EMM386.

EMM386 internally just catches this request and handles it with its  
protected mode code. Note however that EMM386 sets things up so anything  
runs inside protected mode all the time. v86 mode is just a fast emulation  
of real mode that runs in protected mode.

>    My driver is a set of functions that use 32-bit registers and runs  
> hooked on a single interrupt. Sometimes the caller will provide a  
> pointer to a buffer and the functions will fill these buffers, but also,  
> some very big tables have to be loaded, that can occupy as much as one  
> or two megabytes.. I could do moving parts of them to conventional  
> memory as needed, but it would be faster if I could access them directly  
> somehow. The driver leads with a sound interface sometimes and needs to  
> set up big buffers that, depending on the sound card, may have a fix  
> hardware address beyond the conventional and high memory limit.

Is this some sort of mapped I/O space for the hardware, or do you actually  
have to allocate a buffer at this address?

Also, doesn't the hardware work with physical 32-bit addresses? If that's  
the case, notice that in any protected mode environment (and flat real  
mode too?) memory might be remapped. For example, EMM386 maps extended  
memory into the UMA to provide UMBs for the v86 mode DOS. Since the  
physical address used by the hardware still accesses the ROM or RAM that  
was in the UMA before EMM386 remapped UMBs, access to EMM386 UMBs with  
physical addresses doesn't work. Potentially extended memory could be  
remapped too.

> Basically, the driver may return pointers to these areas, but could also  
> copy the data to a buffer provided by the caller.
>    Do you think I should write the functions like pure real-mode code  
> that accesses HIMEM XMS functions as needed to move the data?  Would  
> that be entirely compatible with real-mode- as well as DPMI-applications  
> calling the functions?

Yes, using HIMEM exclusively is compatible with both real mode and DPMI  
applications. That would of course require the user to load some HIMEM  
program, but I think that isn't too restricting. There aren't many reasons  
you wouldn't want to load any HIMEM.

Note that if you lock your XMS memory block (XMS function 0Ch), HIMEM  
returns the linear 32-bit address of that block. DPMI applications can  
benefit from locking the block because you can provide them the linear  
address (as well as the XMS handle). This linear address can then be used  
in the DPMI environment to access the memory block directly. (For this,  
the DPMI application allocates a LDT descriptor (Int31.0000) and sets its  
base (Int31.0007) to the provided linear address as well as the limit  
(Int31.0008) to one large enough to access the memory block.)

> If so, how should I handle the events when I need to read or write in a  
> fixed extended memory area?

You're right, this can't be handled with XMS functions. You'd either have  
to set up your own protected mode/flat real mode/VCPI/DPMI environment  
(depending on whether an EMM or DPMI host is loaded) to access this  
memory, or use Int15.87. As mentioned, proper Int15.87 handling is  
provided by an installed EMM and your BIOS should provide it otherwise.  
(Consider that EMM's Int15.87, DPMI and VCPI only allow access to  
"remapped" memory using the linear address. VCPI might allow you to see  
how memory is remapped, though.)

In case you actually have to allocate a buffer at that fixed address, your  
driver has to request, lock, resize and free XMS memory blocks to try  
allocating a block which returns the correct (linear) address when locked.

>    Is it better to always ask the caller to provide buffers or is it  
> more efficient (and safe) to return pointers to the buffers?  In some  
> cases, the problem is that one buffer is to be used by many applications  
> (i.e.: a running program and a couple of TSRs in memory) and having the  
> caller create a buffer would mean the driver would have to update  
> several identical buffers instead of getting all tasks refer to the same  
> one. That would not be practical and would use up more memory.

If the buffer is updated by your driver only (i.e. the applications aren't  
allowed to write to the buffer, not even by sending requests to the  
driver), then it's better to use a single buffer which is provided to the  
applications by the driver. Otherwise, every application should use its  
own buffer because with a single buffer you might run into reentrancy  
problems. (I.e. after one application requests something from the driver  
but isn't finished using the data, another application requests something  
else and that second request overwrites the data the first application is  
still using.)

Regards,
Christian

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Freedos-devel mailing list
Freedos-devel@...
https://lists.sourceforge.net/lists/listinfo/freedos-devel

Re: About my driver, HIMEM/EMM386 and interrupts

by japhethx gmail :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

>> However, I understood that HIMEM did something like establishing a  
>> routine that was run whenever you tried to use an offset greater than  
>> 0FFFFh together with a "real-mode-made" segment. This routine would  
>> "trap" the event and handle the switch to protected mode and back,  
>> keeping the segment base, but extending the limit to 4G, then returning  
>> to the instruction that generated the exception. Am I right?  Does this  
>> really happen?  If so, programs should not mind about their segments  
>> being initially setup in real mode.
>
> No, I'm not aware of any Extended Memory Manager (XMM) that does such a  
> thing.

MS Himem does indeed "install" such a routine, but it is installed only during
XMS block moves inside Himem.

See the MS Himem source for details. It's Public Domain, IIRC.


------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Freedos-devel mailing list
Freedos-devel@...
https://lists.sourceforge.net/lists/listinfo/freedos-devel

Re: About my driver, HIMEM/EMM386 and interrupts

by Lucas Kiwi :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Guys,

   I was pretty sure I had got it working before, so I didn't want to just leave it as it is. What I just did is restart my computer with plain FreeDOS and wrote, compiled and executed a programme like this:

   mov ax,0b800h



--- On Tue, 27/10/09, japhethx gmail <japhethx@...> wrote:

From: japhethx gmail <japhethx@...>
Subject: Re: [Freedos-devel] About my driver, HIMEM/EMM386 and interrupts
To: freedos-devel@...
Received: Tuesday, 27 October, 2009, 4:56 PM

>> However, I understood that HIMEM did something like establishing a 

>> routine that was run whenever you tried to use an offset greater than 
>> 0FFFFh together with a "real-mode-made" segment. This routine would 
>> "trap" the event and handle the switch to protected mode and back, 
>> keeping the segment base, but extending the limit to 4G, then returning 
>> to the instruction that generated the exception. Am I right?  Does this 
>> really happen?  If so, programs should not mind about their segments 
>> being initially setup in real mode.
>
> No, I'm not aware of any Extended Memory Manager (XMM) that does such a 
> thing.

MS Himem does indeed "install" such a routine, but it is installed only during
XMS block moves inside Himem.

See the MS Himem source for details. It's Public Domain, IIRC.


------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Freedos-devel mailing list
Freedos-devel@...
https://lists.sourceforge.net/lists/listinfo/freedos-devel

Reading this email at work? Make a change with Yahoo!Xtra Jobs
------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Freedos-devel mailing list
Freedos-devel@...
https://lists.sourceforge.net/lists/listinfo/freedos-devel