Forth Lesson 16

Finding PCI Physical Addresses

This isn't really a lesson per se, but rather an answer to an email question. I made it into a sort of lesson because it has a lot of obscure but useful information.

Jordan Crouse wrote:
> On 06/09/07 14:29 +0100, David Woodhouse wrote:
>  
>> On Thu, 2007-09-06 at 20:57 +0800, Luna.Huang@quantatw.com wrote:
>>    
>>> Could you teach me how find the DCOM register in CPU?
>>>
>>> For example, I want to read a DC register, DC_GENERAL_CFG register.
>>>
>>> I know its memory offset is 004h but I don’t know how find the base
>>> address.
>>>
>>> Study the CPU spec. but not really understand.
>>>       
>> I believe the easiest way to find those is through a PCI BAR, although
>> actually it's set by magic GeodeLink stuff. Jordan is the best person to
>> ask.
>>     
>
> The DC lives at 0xfe004000. All the video base addresses are in PCI
> header device 00:01.1, PCI id 1022:2081.
>
> Bar 0 is the framebuffer,
> bar 1 is the graphics processor
> bar 2 is the display controller
> bar 3 is the video processor
>
> Jordan

Here are some techniques you can use from Open Firmware:

ok screen-ih iselect

ok see dc@
: dc@
  dc-base + rl@
;

ok dc-base map?
VA: ffae8000 PA: fe004000
Mode: 73 Dirty Accessed Don'tCache Supervisor R/W

ok dc-base >physical .
fe004000

ok 918 config-l@ .
fe004000

The "918 config-l@ ." displays the value of BAR 1. How did I get the 918 number? The general pattern for the PCI config space address of a BAR is:

(bus# * 0x10000)  +  (device# * 0x800)  + (function# * 0x100)  + 0x10  + (BAR# * 4)

(The 0x10 is the starting offset of the BARs in the device's block of configuration registers.)

In this case, bus# =0 (because this is the root PCI bus - there are no PCI to PCI bridges in the system - this is the 00 in 00:01.1), device# is 1 (the 01 in 00:01.1), function# is 1 (the .1 in 00:01.1), BAR# is 2:

(0 * 0x10000) + (1 * 0x800) + (1 * 0x100) + 0x10 + (2*4) = 0x918

How did Jordan know which BAR is which? Well, one way is to read the source code for either the Linux driver or the OFW driver. That information is not in the LX hardware specification. The reason it is not there is because the graphics hardware is not really on the PCI bus! It is really on an internal bus inside the LX processor. The hardware manual cannot document which BAR is which because there really are no BARs in the LX hardware!

Software pretends it is on the PCI bus because OSs have well-developed frameworks for handling PCI devices, and do not have frameworks for handling the real GeodeLink internal hardware architecture. How does the software do this pretending?

In the case of a conventional BIOS, there is special "VSA" code that runs in system management mode. The VSA code intercepts accesses to the PC I/O port addresses that are used for PCI configuration space access, and runs complicated system-management-mode code that emulates PCI configuration registers for the devices that are inside the LX CPU and the CS5536 I/O chip.

In the Open Firmware case, I didn't want to use VSA because it is hard to maintain (for example we don't have a compiler that will compile the code) and slow (it would have made startup and resume take longer). So instead of emulating the PCI configuration access by intercepting the I/O port accesses, I added a small amount of code to the PCI configuration access subroutines in the Linux kernel and OFW. That code does effectively the same thing as VSA, but at a slightly higher level - it notices that the configuration space address is for one of the internal devices, and returns the right value from a table in memory.

So back to the question of where the BAR numbers are documented: AMD has a document that describes the emulated PCI config registers that the VSA provides. That document is hard to get; it is on AMD's restricted web site. It is easier to get the information either by reading the driver code or by asking Jordan.

How does the hardware actually establish the physical address of the DC? Well, the LX CPU has some "P2D" MSRs that control address assignment. Any P2D MSR can map an address to one of several devices, so there is not a fixed assignment of P2Ds to specific devices. OFW chooses to use the P2D MSR at MSR number 0x1000002a to map the DC. The value in that MSR is 0x801ffcfe.007fe004 . You can look up the bits in that MSR if you want to understand how it really works. OFW maps the GP with MSR 0x10000022, and the VP+VIP with MSR 0x10000023.

One more topic: You can find PCI device and function numbers with:

ok show-devs /pci
/pci/usb@f,5
/pci/usb@f,4
/pci/audio@f,3
/pci/camera@c,2
/pci/sd@c,1
/pci/nandflash@c
/pci/pci1022,2082@1,2
/pci/host@1
/pci/display@1,1
/pci/isa@f
/pci/usb@f,5/wlan@0,0
/pci/sd@c,1/disk
/pci/isa@f/rtc@i70
/pci/isa@f/8042@i60
/pci/isa@f/serial@i3f8
/pci/isa@f/timer@i40
/pci/isa@f/interrupt-controller@i20
/pci/isa@f/dma-controller@i00
/pci/isa@f/8042@i60/mouse@aux
/pci/isa@f/8042@i60/keyboard@kbd

Here we see "/pci/display@1,1" - the display device is device# 1, function# 1. Similarly, the audio device (audio@f,3) is device# f, function# 3. There are two USB devices - the one at f,5 is the EHCI, i.e. the USB2.0 controller, and the one at f,4 is OHCI, i.e. the USB 1.1 controller.

Thus endeth the lesson.