Forth Lesson 16
Finding PCI Physical Addresses
This isn't really a lesson per se, but rather a long-winded answer to an email question. I made it into a sort of lesson because it has a lot of obscure but useful information.
Jordan Crouse wrote: > On 06/09/07 14:29 +0100, David Woodhouse wrote: > >> On Thu, 2007-09-06 at 20:57 +0800, Luna.Huang@quantatw.com wrote: >> >>> Could you teach me how find the DCOM register in CPU? >>> >>> For example, I want to read a DC register, DC_GENERAL_CFG register. >>> >>> I know its memory offset is 004h but I don’t know how find the base >>> address. >>> >>> Study the CPU spec. but not really understand. >>> >> I believe the easiest way to find those is through a PCI BAR, although >> actually it's set by magic GeodeLink stuff. Jordan is the best person to >> ask. >> > > The DC lives at 0xfe004000. All the video base addresses are in PCI > header device 00:01.1, PCI id 1022:2081. > > Bar 0 is the framebuffer, > bar 1 is the graphics processor > bar 2 is the display controller > bar 3 is the video processor > > Jordan
Here are some techniques you can use from Open Firmware:
ok screen-ih iselect ok 4 dc@ . 6501
"screen-ih iselect" goes into the context of the display driver device so you can access its methods. "4 dc@ ." accesses the DC register at offset 4 and displays its value. Here are some access words for various registers in the display device.
dc@ ( offset -- value ) \ Read the value from a display controller register onto the stack dc! ( value offset -- ) \ Set the value of a display controller register gp@ ( offset -- value ) \ Read the value from a graphics processor register onto the stack gp! ( value offset -- ) \ Set the value of a graphics processor register vp@ ( offset -- value ) \ Read the value from a video processor register onto the stack vp! ( value offset -- ) \ Set the value of a video processor register
Let's look at how dc@ works:
ok see dc@ : dc@ dc-base + rl@ ;
It adds the base address to the offset on the stack ( dc-base + ), then uses rl@ to read a 32-bit (l) register at the resulting address. dc-base is the virtual base address of the display controller registers.
ok dc-base . ffae8000
ffae8000 is a virtual address, not a physical address. OFW runs with paging enabled, using virtual addresses. The reason it does so is because OFW wants to live at very high virtual addresses, so it can be co-resident with Linux if desired. If OFW used physical addressing (i.e. x86 paging turned off), the OFW code would have to either live at the bottom of the physical memory address space, where it would be in the way of Linux, or else it would have to live at the very top of physical memory, but then its execution address would depend on the memory size.
How can we find the physical address of the display controller registers?
ok dc-base map? VA: ffae8000 PA: fe004000 Mode: 73 Dirty Accessed Don'tCache Supervisor R/W
map? displays the page mapping information for a virtual address taken from the stack.
Here is another way to get just the physical address.
ok dc-base >physical . fe004000
">physical ( vadr -- padr )" converts a virtual address to a physical address (by looking in the page tables).
We could also use PCI config space to find the physical address (but the physical address by itself wouldn't necessarily let us access the register since paging is enabled).
ok 918 config-l@ . fe004000
The "918 config-l@ ." displays the value of BAR 1. How did I get the 918 number? The general pattern for the PCI config space address of a BAR is:
(bus# * 0x10000) + (device# * 0x800) + (function# * 0x100) + 0x10 + (BAR# * 4)
(The 0x10 is the starting offset of the BARs in the device's block of configuration registers.)
In this case, bus# =0 (because this is the root PCI bus - there are no PCI to PCI bridges in the system - this is the 00 in 00:01.1), device# is 1 (the 01 in 00:01.1), function# is 1 (the .1 in 00:01.1), BAR# is 2:
(0 * 0x10000) + (1 * 0x800) + (1 * 0x100) + 0x10 + (2*4) = 0x918
How did Jordan know which BAR is which? Well, one way is to read the source code for either the Linux driver or the OFW driver. That information is not in the LX hardware specification. The reason it is not there is because the graphics hardware is not really on the PCI bus! It is really on an internal bus inside the LX processor. The hardware manual cannot document which BAR is which because there really are no BARs in the LX hardware!
Software pretends it is on the PCI bus because OSs have well-developed frameworks for handling PCI devices, and do not have frameworks for handling the real GeodeLink internal hardware architecture. How does the software do this pretending?
In the case of a conventional BIOS, there is special "VSA" code that runs in system management mode. The VSA code intercepts accesses to the PC I/O port addresses that are used for PCI configuration space access, and runs complicated system-management-mode code that emulates PCI configuration registers for the devices that are inside the LX CPU and the CS5536 I/O chip.
In the Open Firmware case, I didn't want to use VSA because it is hard to maintain (for example we don't have a compiler that will compile the code) and slow (it would have made startup and resume take longer). So instead of emulating the PCI configuration access by intercepting the I/O port accesses, I added a small amount of code to the PCI configuration access subroutines in the Linux kernel and OFW. That code does effectively the same thing as VSA, but at a slightly higher level - it notices that the configuration space address is for one of the internal devices, and returns the right value from a table in memory.
So back to the question of where the BAR numbers are documented: [1] is an AMD document that describes the emulated PCI config registers that the VSA provides.
How does the hardware actually establish the physical address of the DC? Well, the LX CPU has some "P2D" MSRs that control address assignment. Any P2D MSR can map an address to one of several devices, so there is not a fixed assignment of P2Ds to specific devices. OFW chooses to use the P2D MSR at MSR number 0x1000002a to map the DC. The value in that MSR is 0x801ffcfe.007fe004 . You can look up the bits in that MSR if you want to understand how it really works. OFW maps the GP with MSR 0x10000022, and the VP+VIP with MSR 0x10000023.
One more topic: You can find PCI device and function numbers with:
ok show-devs /pci /pci/usb@f,5 /pci/usb@f,4 /pci/audio@f,3 /pci/camera@c,2 /pci/sd@c,1 /pci/nandflash@c /pci/pci1022,2082@1,2 /pci/host@1 /pci/display@1,1 /pci/isa@f /pci/usb@f,5/wlan@0,0 /pci/sd@c,1/disk /pci/isa@f/rtc@i70 /pci/isa@f/8042@i60 /pci/isa@f/serial@i3f8 /pci/isa@f/timer@i40 /pci/isa@f/interrupt-controller@i20 /pci/isa@f/dma-controller@i00 /pci/isa@f/8042@i60/mouse@aux /pci/isa@f/8042@i60/keyboard@kbd
Here we see "/pci/display@1,1" - the display device is device# 1, function# 1. Similarly, the audio device (audio@f,3) is device# f, function# 3. There are two USB devices - the one at f,5 is the EHCI, i.e. the USB2.0 controller, and the one at f,4 is OHCI, i.e. the USB 1.1 controller.
Thus endeth the lesson.